GenAIT: Development and Validation of an Objective Generative AI Literacy Test for High School Students

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: We're looking at the paper: "GenAIT: Development and Validation of an Objective Generative AI Literacy Test for High School Students".

Marcus: Here's the gist of the paper. The gist The study reports the iterative development and validation of GenAIT, an 18-item objective test measuring high school students’ conceptual knowledge about GenAI, which supports its use for group-level research.

Ines: GenAI Literacy Definition and Scope Literacy was originally understood in relation to the skills and knowledge required to read, understand, and use written language As societies became more digitalized, researchers proposed the construct of digital literacy to describe skills and knowledge related to digital tools Long and Magerko (2020) defined AI literacy as a set of competencies that enables individuals to critically evaluate AI technologies, collaborate with AI, and use AI as a tool in different areas of life To distinguish competencies specifically related to generative technologies from the broader AI literacy, the term “GenAI literacy” has been used GenAI literacy can be understood as a focused extension of AI literacy, rather than a completely distinct construct The operationalization of the scope of conceptual knowledge required for GenAI literacy in high school students relied on the tri-partite framework draft provided by the Council of Europe (2026) This framework includes technological (How Artificial Intelligence works and how it might be developed), practical (How Artificial Intelligence can be used effectively), and human (The impact of AI on humans, human rights, democracy and the rule of law) dimensions The technical domain contains the basics about how GenAI works and solves problems The practical domain is related to students’ understanding of their active role in getting GenAI to do what they want, including via prompting and critical evaluation of its outputs The human domain focuses on the potential implications of GenAI on human cognition, society, and the environment Instrument Development and Content Validity The development process involved multiple stages, including expert review and a pilot study Prior to instrument development, the construct should be clearly defined, and the content domain specified The content blueprint was operationalized as consisting of three content domains: technical, practical, and human Items were generated through an iterative cycle of drafting, review, and refinement conducted by five members of the Artificial and Natural Intelligence Lab at the University of Tartu Content validity was assessed by a panel of five experts who rated the relevance, clarity, and comprehensiveness of the item pool The primary metric for content validity was the item-level content validity index (I-CVI), with values above.90 indicating excellent content validity For the initial 21-item version of GenAIT, an S-CVI-Ave value of.96 was obtained The final version of GenAIT included 18 items following item selection described in the following sections Psychometric Validation and Reliability The psychometric research questions were RQ1: To what extent do expert judgments support the relevance, clarity, and comprehensiveness of the initial GenAIT item pool for the defined content domain?

Marcus: RQ2: What are the internal structure, item functioning, and score reliability and precision of the final GenAIT form?

Ines: The validation occurred in multiple stages, including a large-scale survey of 7,432 Estonian high school students Internal structure and score reliability were assessed using confirmatory factor analysis (CFA), classical test theory (CTT), and item response theory (IRT) The final instrument was evaluated using the 3PL model, which demonstrated acceptable fit with RMSEA =.013, TLI =.987, CFI =.990, and SRMSR =.021 The marginal reliability value was.72 and the KR-20 value was.

Paper summary: Ines: So we're diving into "GenAIT: Development and Validation of an Objective Generative AI Literacy Test for High School Students" to see what this whole piece is about. Essentially, the authors are arguing that there’s a real need to objectively measure whether high school students actually understand generative AI.

Marcus: They claim that current tests are often too general, focusing on broad AI literacy instead of the specific skills needed for generative technologies.

Yuki: And they point out that most existing GenAI literacy tests rely on subjective self-reports, which can lead to inaccurate results when we're trying to gauge real knowledge and understanding.

Ines: So the thesis is that they developed this eighteen-item test, GenAIT, specifically designed to measure conceptual knowledge across technical, practical, and human impact areas <ref:2608.25815#pg1>.

Marcus: They emphasize that the content of this test covers three domains: how the AI works technically, how students can actually use it through prompting and evaluation, and the broader implications for society.

Yuki: The framework they used is based on a tri-partite structure that goes beyond just technical skills; it includes human rights, democracy, and the rule of law as areas to consider.

Ines: They also made sure this test was rigorously developed through an iterative process involving expert review to check for content validity before they even started testing students.

Marcus: The validation involved a large survey of seven thousand four hundred thirty-two Estonian high school students, and they used statistical methods like confirmatory factor analysis and item response theory to check the test's internal structure <ref:2608.25815#pg1>.

Yuki: The results showed that the test had approximate unidimensionality, meaning it measures one core concept consistently across all items tested.

Ines: And for reliability, they found broadly adequate metrics for group-level research, specifically a marginal reliability of point seven two and a KR-twenty value of point six nine <ref:2608.25815#pg1>.

Marcus: The paper also reported good fit statistics for a three-parameter logistic model, showing that the statistical structure of the test data is sound.

Yuki: This work matters because it provides one step toward having an objective assessment tool for GenAI literacy in this student population, rather than relying on less reliable subjective measures.

Ines: It moves the conversation from 'do students think they know enough' to 'what do we actually measure when we ask them about generative AI'.

Conclusion: Marcus: So looking at the "GenAIT: Development and Validation of an Objective Generative AI Literacy Test for High School Students" paper, the core contribution is providing a validated, objective instrument for assessing GenAI literacy among high school students.

Ines: They focused on moving beyond general AI literacy to target generative technologies specifically, using a structured framework that covers technical application and human impact.

Yuki: What this means in practice is that educators and researchers have a tool that can measure conceptual knowledge systematically, rather than just relying on informal classroom discussions.

Marcus: It’s about establishing a measurable starting point for education—a way to see if the curriculum is actually building the necessary literacy skills before we scale up.

Ines: The authors themselves flagged that while the test works well for group research, it's not designed for high-stakes individual classification, so we have to be careful about how we use those scores.

Yuki: And they also highlighted that because it's related to LLM use frequency and perceived ease of use in a negative way, using those things as direct measures of conceptual understanding is not supported by this data.

Marcus: So the implication is that we should treat these test scores as indicators for group-level research, not definitive judgments on an individual student's capability.

Ines: The main takeaway is that developing a reliable, objective tool like GenAIT is a necessary first step in tackling the growing international interest in AI education assessment.