TKGB: A Comprehensive Benchmark for Temporal Knowledge Graph Forecasting

The gist The authors introduce a new TKG evolution benchmark to address shortcuts in existing temporal knowledge graph forecasting datasets by incorporating bias-corrected datasets and novel tasks that simulate knowledge evolution stages.

Problem Identification

Existing benchmarks inadvertently introduce a shortcut where near state-of-the-art performance can be achieved by counting co-occurrences without using any temporal information The authors find that this strong performance stems from co-occurrence biases inherent in these datasets and the oversimplified formulation of the evaluation task that can be exploited. Specifically, they observe that each entity interacts with only a very limited set of entities and relations, and most of these interactions have already been seen during training. This leads to the conclusion that a strong performance of TKG forecasting can be achieved merely by memorizing such preferred sets of entities for each entity and (e, r) pair.

Benchmark Construction

The authors introduce a comprehensive temporal knowledge graph evolution benchmark comprising four newly constructed datasets and two novel evaluation tasks designed to simulate different stages of TKG evolution. The four datasets are FinWiki, ICEWSWiki, YAGO130K, and WIKI500K, which are drawn from three distinct domains and vary in scale. A key feature of these new datasets is that each dataset incorporates aligned timestamp and time-interval knowledge, revealing mutual influences within knowledge evolution. Furthermore, they are enriched with extensive textual annotations that describe knowledge semantics. To mitigate co-occurrence shortcuts, the datasets were constructed using strategies such as a popularity-based strategy during dataset construction and a k-core-based filtering strategy.

Novel Evaluation Tasks

The benchmark includes two novel tasks closely aligned with the evolution process to promote a more accurate understanding of the challenges in TKG evolution modeling. 1) Generative knowledge forecasting is proposed, which directly generates all knowledge involving s in the next timestamp rather than completing a missing element in future facts, thereby assessing a model’s ability to learn how knowledge forms. 2) Knowledge obsolescence prediction aims to determine whether currently valid knowledge will become invalid in the future, triggered by other new knowledge, which evaluates a model’s ability to understand how knowledge becomes obsolete.

Experimental Results and Findings

The experiments compare various models on the new benchmark, demonstrating that Co-occurrence-based scoring performs poorly on our proposed benchmark datasets, often 10 times worse than baselines. Existing models show that they tend to fit this co-occurrence shortcut, i.e., they learn to predict by counting co-occurrences rather than by capturing underlying evolutionary mechanisms. The authors find that TKG-ICL outperforms other models on this task, revealing the ability of LLMs in understanding the knowledge evolution mechanism with sufficient semantic information. Specifically, TKG-ICL's performance improves significantly on both generative knowledge forecasting and knowledge obsolescence prediction tasks using textual annotations. The LLM-based model maintains consistent performance in the inductive setting, demonstrating the potential of leveraging the inherent knowledge in LLMs to build more generalizable TKG forecasting models.

Conclusion

The paper reveals the co-occurrence shortcut problem in current TKG benchmarks, tracing it to dataset bias and oversimplified task formulations. The authors address this by proposing the first TKG evolution benchmark, which eliminates shortcuts and provides a more comprehensive testbed for evaluating knowledge evolution learning.

--- Page 1 ---

Towards Better Evolution Modeling for Temporal Knowledge Graphs Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Trovato et al.

The gist The authors introduce a new TKG evolution benchmark to address shortcuts in existing temporal knowledge graph forecasting datasets by incorporating bias-corrected datasets and novel tasks that simulate knowledge evolution stages.

How it works

The authors examine the root cause of the shortcut problem by identifying inherent biases in current datasets and oversimplified form of evaluation task. They observe that in current TKG datasets, each entity interacts with only a very limited set of entities and relations, and most of these interactions have already been seen during training. This observation suggests that the missing object entity can be predicted merely by memorizing these preferred entities.

Dataset Details

The proposed benchmark includes four newly constructed datasets designed to mitigate co-occurrence biases. These datasets are enriched with aligned timestamp and time-interval knowledge, revealing mutual influences within knowledge evolution. For instance, YAGO130K and WIKI500K are derived from YAGO and Wikidata, both of which inherently contain timestamp and time-interval knowledge. The authors also enrich these datasets with textual annotations by retrieving the ‘label’ property associated with each code to enhance semantic understanding.

Novel Tasks

The benchmark proposes two novel tasks to simulate different stages of knowledge evolution. 1) Generative knowledge forecasting directly generates future knowledge from history, assessing a model’s ability to learn how knowledge forms. 2) Knowledge obsolescence prediction aims to determine whether currently valid knowledge will become invalid in the future, triggered by other new knowledge, which evaluates a model’s ability to understand how knowledge becomes obsolete.

Conclusion

The core issue stems from the neglect of knowledge semantics, which is crucial for understanding the causality behind knowledge evolution. The authors introduce a new TKG evolution benchmark, which consists of four newly constructed datasets with corrected biases and external knowledge semantic information. This provides a more accurate understanding of the challenges in TKG evolution modeling.

--- Page 11 ---

Towards Better Evolution Modeling for Temporal Knowledge Graphs Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Trovato et al.

The gist The authors introduce a new TKG evolution benchmark to address shortcuts in existing temporal knowledge graph forecasting datasets by incorporating bias-corrected datasets and novel tasks that simulate knowledge evolution stages.

How it works

The authors employ prompts from Appendix F to incorporate time-interval knowledge as background context for TKG-ICL, while converting such knowledge into repeating timestamp knowledge for other models. They also use textual annotations to enhance the semantic understanding of the evolution process, showing that TKG-ICL’s performance improves significantly on both generative knowledge forecasting and knowledge obsolescence prediction tasks using textual annotations.

Effectiveness of Integrating Knowledge

Figure 4 shows that existing methods generally achieve better performance after integrating time-interval knowledge, demonstrating its effectiveness in helping understanding the knowledge formation process. Similarly, textual annotations improve performance on the prediction tasks, indicating that precise semantics for understanding the causality behind knowledge evolution are crucial.

Inductive Setting Performance

In Table 6, the results show that while performance of learning-based methods drops significantly, the LLM-based model TKG-ICL maintains consistent performance. This demonstrates the potential of leveraging the inherent knowledge in LLMs to build more generalizable TKG forecasting models.

Conclusion

The paper reveals the co-occurrence shortcut problem in current TKG benchmarks, tracing it to dataset bias and oversimplified task formulations. They address this by proposing the first TKG evolution benchmark, which eliminates shortcuts and provides a more comprehensive testbed for evaluating knowledge evolution learning.

1. Bold header: Mitigation of Co-occurrence Shortcut via Data Construction

The improved system will incorporate strategies like popularity-based strategy during dataset construction and k-core-based filtering strategy to ensure datasets exhibit lower train–test overlap or a larger average size of entity preference sets, thereby directly addressing the paper's finding that competitive TKG forecasting performance can be achieved merely by leveraging co-occurrence frequency.

2. Bold header: Integration of Time-Interval Knowledge for Causal Understanding

The system will utilize both timestamp and time-interval knowledge, as detailed in the benchmark construction, to allow for the understanding of mutual influences within knowledge evolution, which is crucial because the occurrence of a timestamp knowledge is conditioned on existing time-interval knowledge.

3. Bold header: Enrichment with Semantic Textual Annotations

The improved system will enrich entity and relation annotations by replacing ambiguous names or abbreviations with conceptual descriptions, as seen in the datasets, to provide richer semantics for analyzing causal relationships among knowledge and enable more effective anonymization in LLM-based approaches.

4. Bold header: Novel Task for Knowledge Obsolescence Prediction

The system will be equipped with a Knowledge obsolescence prediction task, which aims to determine whether currently valid knowledge will become invalid in the future, triggered by other new knowledge, simulating how knowledge becomes obsolete, which is described as largely overlooked in prior work.

5. Bold header: Enhanced Generative Forecasting via Direct Knowledge Generation

The improved system for generative forecasting will be designed to move beyond predicting a missing object entity by directly generating future knowledge from history, as proposed in the benchmark task: directly generate all knowledge involving s in the next timestamp (i.e., (s, r', o', t + 1) (s, r', o', t + 1) ∈ Gt+1), to alleviate shortcut problem by reducing redundant information of future knowledge.

6. Bold header: LLM-based Reasoning with Semantic Context

The TKG-ICL model will be specifically trained to leverage textual annotations, as evidenced by its superior performance on the obsolescence task, demonstrating the ability of LLMs in understanding the knowledge evolution mechanism with sufficient semantic information rather than relying solely on numerical IDs.

7. Bold header: Tailored Negative Sampling for Training

The system's training process will incorporate a negative sampling strategy that perturbs both r and o when training models, as this approach is found to yield significantly better results, as this approach aligns more closely with the task formulation—providing less redundant information about the query knowledge.

Temporal knowledge graph (TKG) forecasting has advanced remarkably in recent years. Yet we found existing benchmarks inadvertently introduce a shortcut: For a subset of test queries, models can achieve optimal evaluation metrics without using any time information. We trace this to two pieces of evidence: inherent dataset biases and an overly simplistic task formulation that allows models to exploit biases, revealing fundamental flaws in current evaluation protocols for differentiating true forecasting ability. Moreover, existing datasets rely on fixed history that poorly reflects real-world dynamics, lack semantic context, and fail to explicitly mitigate shortcut, preventing thorough assessment of forecasting abilities. To address these, we propose TKGB, a comprehensive benchmark for evaluating the capability of TKG forecasting models. By integrating four complementary tasks with an automatic pipeline for constructing up-to-date and context-enriched TKG datasets with controllable shortcuts, TKGB reveals capability differences and failure modes hidden under conventional evaluation.