Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI

This paper, authored by Modhurita Mitra and colleagues from Utrecht University, presents a schema-based framework for extracting complex, structured information from unstructured text documents using generative AI, followed by automated semantic evaluation against a gold standard.

The abstract states: "We present a schema-based framework for extracting complex, structured information from unstructured text documents using generative AI, followed by automated semantic evaluation of the extracted information against a gold standard. The schema, serving as an information model encoding domain knowledge, provides a unified, systematic, and consistent framework for extraction of hierarchical, nested information, with attributes of variable cardinality, and subsequent evaluation of the results. Information extraction from a document is performed in a single call to the model, in zero-shot mode."

The paper introduces a path-based semantic matching algorithm for evaluation, uses generative AI for semantic comparison of extracted and gold standard values, and introduces a rubric to classify comparison results as exact, semantic, useful, or non-match, according to domain-specific considerations.

Key results reported in the abstract: "We were able to extract 12 out of 14 attributes with an F1 score of >90% from documents published by the health technology assessment organisation NICE, using the generative AI model Claude Opus 3. The time needed to extract the attributes from a document was ∼30 times lower than the time taken by a human domain expert. We further demonstrate generalisability of this framework across different generative AI models and transferability across different HTA organisations and languages."

The method emerged from efforts to extract attributes from health technology assessment (HTA) organisation documents, which are described as highly specialised, requiring deep domain knowledge to interpret, with content and structure differing across documents, organisations, and time. The authors note that the structure of the desired attributes is complex and involves hierarchical relationships between attributes of variable cardinality, making this a difficult information extraction problem.

HTA is defined as "the process of systematically reviewing a health technology (e.g., a drug, a medical device, or a medical therapy) based on clinical evidence summarised in reports that include conclusions regarding various attributes (e.g. clinical effectiveness and cost-effectiveness of drugs) that affect the adoption of that health technology in patient care. In the European Union, HTA is carried out independently by each country, resulting in heterogeneous data in different languages and formats. The scientific goal of the project is to create an Open Science database of a number of attributes of interest extracted from these documents in order to facilitate their downstream use by different stakeholders."

The authors identified 14 research-relevant attributes to extract: HTA ID, treatment type, assessment type, assessment date, internal identifier, indication, INN (international nonproprietary name), brand name, comparators, relative effectiveness assessment outcome, cost-effectiveness assessment outcome, final recommendation, managed entry agreement, and clinical restrictions. These attributes have a nested structure—for example, each drug in a document has its own corresponding brand name, relative effectiveness assessment, and final recommendation. The schema structure includes indications as a list of dictionaries containing indication name, and technologies as a list of dictionaries containing inn, brand name, comparators, outcome rea, outcome cea, final recommendation, managed entry agreement, and clinical restrictions.

The paper positions its work relative to existing approaches. Information extraction has traditionally fallen into rule-based or machine learning-based methods. With LLMs, fine-tuning BERT, BART, or open-weights generative models has become popular, but these fine-tuning methods require labelled training data, the technical know-how for fine-tuning LLMs, and the infrastructure to fine-tune and deploy the models, making them inaccessible to domain experts with limited resources.

The authors discuss prompt engineering techniques such as chain-of-thought prompting, LLM-as-a-judge, and zero-, few-, and many-shot prompting. They also discuss constrained decoding, where a response schema is passed to a generative AI API along with but separate from the prompt, and the AI produces a response consistent with the schema.

Prior related studies include: LLMs fine-tuned on materials science data to extract hierarchical attributes of variable cardinality (though only one level deep and with exact string matching only); a study populating knowledge bases with nested hierarchical information by recursively traversing a schema, where the authors note this kind of grounding is not an evaluation of the inherent correctness of the extracted information – it is only validation of the existence of the extracted information in the ontology; and a histopathology study using a flat JSON schema, which was flat – hierarchical, nested data with attributes of variable cardinality were not extracted.

The data consists of HTA documents from the National Institute for Health and Care Excellence (NICE) in the United Kingdom. The development set comprised 8 representative documents covering a diverse range of attribute values—some initial assessments, some reassessments; some positive and negative outcomes; some with multiple drugs for one indication, others with one drug for multiple indications. The test set consisted of 50 randomly chosen HTA documents. Gold standards were manually created for both sets. The gold standard for the test set contained a total of 57 indications, 60 technologies, and 837 instances of leaf-level attributes, where a leaf-level attribute is one without further sub-attributes.

The extraction was performed using Claude Opus 3 from Anthropic via its public API, in a single call per document, in zero-shot mode, on entire documents of 20–40 pages. The temperature parameter was set to zero for maximal determinism and reproducibility.

The methodology initially used a Q&A method with flat JSON structure, which proved insufficient: "Some documents assessed one drug for treating multiple indications, while others assessed multiple drugs for treating one indication. It became evident that we needed a nested JSON structure to represent this kind of hierarchical data structure with attributes of variable cardinality." The schema was provided in the prompt (not as a hard API constraint): "A schema in the prompt provides a soft guideline for the output structure, whereas a response schema enforces the structure as a hard constraint. Nevertheless, every JSON output we obtained was schema-compliant, though we provided the JSON schema only in the prompt."

The prompt consisted of three parts: the document text (obtained via PyPDF), the instructions (including setting the AI's role, task description, and output format requirements), and the schema specifying attributes and output format.

The schema, expressed in JSON Schema format, plays a central role in three ways:

1. Encoding domain information: Leaf-level keys specify attribute names; the description field provides attribute descriptions and domain knowledge; type, pattern, and enum fields constrain output values (e.g., pattern YYYY-MM-DD for dates; enum restricting values to positive, negative, or null). The type field specifies nested hierarchical structure—array for variable-cardinality lists, object for dictionaries with sub-attributes.

2. Specifying output format: The schema specifies both individual key-value pair formats and the entire JSON object structure.

3. Providing context during evaluation: During evaluation, if values are not an exact string match, generative AI checks semantic match using the attribute name and the associated description field from the schema to provide context about the values being compared.

Domain experts specified the attributes, their hierarchical structure, and provided the domain knowledge in the schema; Research Engineers designed the schema in JSON Schema format.

The evaluation process compares extracted results against a human-generated gold standard, with both in the schema-prescribed format. Because of the nested structure, direct comparison is non-trivial—lists may have different lengths and element orderings. The authors developed an automated approach: "We approach the problem of identifying the corresponding elements in the extracted and gold standard JSON objects by finding the path to each leaf-level element in one JSON object, and then finding the matching path in the other JSON object."

For non-leaf attributes that are lists of dictionaries (indications and technologies), matching dictionary list elements requires choosing an identifier (anchor) attribute: indication name at the indications level; at the technologies level, the primary identifier is inn, with brand name as secondary if no INN is present. Best one-to-one matches are formed by ranking match types (exact > semantic > useful), with greedy search used to break ties.

At each comparison step, the match is classified as True Positive (TP), True Negative (TN), False Positive (FP), or False Negative (FN) according to a table: both absent = TN; gold standard absent but extracted present = FP; gold standard present but extracted absent = FN; both present with matching values = TP; both present with non-matching values = FP.

When both values are present, an exact string match is first checked via simple string comparison, yielding TP if exact. If not exact, generative AI classifies the comparison into four categories:

1. Exact match: Near-identical values with minor differences (e.g., em dash vs. en dash, alpha vs. α) — classified as TP.

2. Semantic match: Lexically different but semantically identical paraphrase (e.g., pyrexia in children vs. fever in kids) — classified as TP.

3. Useful match: Neither lexically nor semantically identical, but still useful for comparative HTA research (e.g., fever in children under 12 years of age vs. fever) — classified as TP. This classification is justified because HTA deals with reimbursement decisions and not clinical decisions.

4. Non-match: Neither lexically identical nor semantically similar — classified as FP.

The evaluation model was Claude Opus 4.6, the most advanced model from Anthropic that was available at the time we performed the evaluation part of this study (March 2026). All evaluations used Claude Opus 4.6 for uniformity.

The authors performed six analyses: (1) performance metrics (precision, recall, accuracy, F1 score); (2) consistency analysis over two runs with identical model, parameters, prompt, and data; (3) time, costs, and scalability computations; (4) comparison with rule-based and traditional machine learning methods; (5) generalisability across different models; and (6) transferability across HTA organisations and languages.

For 12 of 14 attributes, all metrics exceeded 90%. Two attributes performed more poorly: outcome rea with F1 of 0.88 and clinical restrictions with F1 of 0.83. The authors explain that extracting outcome rea is difficult even for humans—The relative effectiveness discussion in HTA documents is often long, nuanced, and ambiguous—giving the example that when a document says This drug seems to be clinically effective, but the evidence presented is insufficient, the AI typically classified outcome rea as positive while the human expert classified it as negative. For clinical restrictions, the issue is that it is reported alongside indication name in the same Recommendations section, making them difficult to decouple; the paper provides a toy example where gold standard is indication name: fever in children under 12 years of age, clinical restrictions: null but AI extracts indication name: fever, clinical restrictions: only for children under 12 years of age, causing a false positive for clinical restrictions.

During evaluation, 634 of 796 comparisons yielded exact string matches. The remaining comparisons were classified by Claude Opus 4.6 as 20 exact, 19 semantic, 84 useful, and 39 non-matches. Human verification confirmed agreement on all non-matches; for other classifications, humans sometimes classified differently but always within the three TP-category types: Since a non-match is classified as FP and exact, semantic, and useful matches are all classified as TP, this discrepancy is immaterial for the purposes of calculating the metrics.

Using results from a first run as gold standard for a second run with the same model and settings, "the values of all the metrics for all the attributes were > 96%."

"On average, it took Claude Opus 3 ∼0.5 minute to extract the attributes from one HTA document, and the API cost was ∼0.5 euro per document. For comparison, it takes a domain expert ∼15 minutes to extract the attributes from one document. Thus there is a ∼30-fold increase in time-efficiency when generative AI is employed instead of a human for this task. The time and API costs scale linearly with the number of documents."

No existing traditional method can extract the complete complex hierarchical structure in a single call. Attribute-by-attribute comparison showed that a rule-based method extracted four attributes (hta id, assessment date, internal identifier, indication name) with 90% accuracy and three more (assessment type, inn, brand name) with 70% accuracy, but could not extract other attributes with significant accuracy. For machine learning classification of the three roughly binary attributes (outcome rea, outcome cea, final recommendation), the best results were: Gradient Boosting and Random Forest with F1 0.76 for outcome rea; Gradient Boosting with F1 0.98 for outcome cea and 0.93 for final recommendation. The generative AI approach outperformed both, and additionally extracts all attributes in a single call, making implementation simpler than rule-based methods (which need a separate rule per attribute) or ML methods (which need labelled training data per attribute).

The framework was tested with three additional models—Claude Opus 4.6, Gemini 3.1 Pro, and the open-weights GPT OSS 120B (accessed through the AI-Hub provided by SURF)—with Claude Opus 4.6 used for evaluation in all cases. It can be seen that we get comparable results from all four models, demonstrating the generalisability of this framework across different generative AI models.

The pipeline designed for NICE was applied as-is, without modification, to 25 documents each from Zorginstituut Nederland (ZIN, Dutch) and Haute Autorité de Santé (HAS, French), with Claude Opus 4.6 for both extraction and evaluation. The approach performs reasonably well for several attributes for ZIN and HAS too, thus illustrating the broad transferability of our schema-based framework across different HTA organisations and languages. The paper notes that HAS does not report internal identifier and outcome cea, and that for managed entry agreement for HAS, either both AI and gold standard were null or the AI did not classify future assessment requirements as outcome-based managed entry agreements as the human expert did.

The authors conclude: **"Using a schema-based framework, we have successfully extracted data with a hierarchical structure, containing attributes of variable cardinality, from HTA documents. We have also eva…

Based on this paper, I can improve AI systems in the following concrete ways:

Improvement: Enable the AI to accept a JSON Schema as part of the prompt and produce structured, nested output in one call without fine-tuning or constrained decoding.

What the improved system can do:

• Extract hierarchical, nested information with attributes of variable cardinality (e.g., lists of dictionaries containing sub-attributes) directly from unstructured 20–40 page documents.

• Encode domain knowledge through schema descriptions, enums, patterns, and type constraints so the AI knows exactly what each attribute means and what values are allowed.

• Handle one-to-many and many-to-one relationships (e.g., one drug for multiple indications, multiple drugs for one indication) using nested JSON structures.

• Produce schema-compliant outputs reliably even when the schema is only a soft guideline in the prompt.

In summary, the improved AI system can extract deeply nested, multi-cardinality structured data from lengthy unstructured documents in a single zero-shot call, automatically evaluate that extraction against a gold standard using path-based semantic matching, classify matches into domain-meaningful categories, transfer to new domains and languages by changing only the schema, run on open-weights models for privacy and reproducibility, and preserve human oversight where validity is critical.

We present a schema-based framework for extracting complex, structured information from unstructured text documents using generative AI, followed by automated semantic evaluation of the extracted information against a gold standard. The schema, serving as an information model encoding domain knowledge, provides a unified, systematic, and consistent framework for extraction of hierarchical, nested information, with attributes of variable cardinality, and subsequent evaluation of the results. Information extraction from a document is performed in a single call to the model, in zero-shot mode. In the evaluation step, we introduce a path-based semantic matching algorithm to align the nested, variable-cardinality attributes in the extracted results with those in the gold standard. We use generative AI for semantic comparison of the extracted and gold standard values of an attribute, and introduce a rubric to classify the result of the comparison, according to domain-specific considerations, as an exact, semantic, useful, or non-match. We were able to extract 12 out of 14 attributes with an F1 score of > 90% from documents published by the health technology assessment organisation NICE, using the generative AI model Claude Opus 3. The time needed to extract the attributes from a document was about 30 times lower than the time taken by a human domain expert. We further demonstrate generalisability of this framework across different generative AI models and transferability across different HTA organisations and languages.