SECite: Analyzing and Summarizing Citations in Software Engineering Literature

The gist The SECite framework introduces a novel approach for evaluating scholarly impact through sentiment analysis of citation contexts by integrating citation sentiment analysis with LLM-based summarization to capture both strengths and limitations of research papers.

SECite Overview

SECite is a novel approach for evaluating scholarly impact through sentiment analysis of citation contexts by integrating citation sentiment analysis with LLM-based summarization to capture both strengths and limitations of research papers The goal is to provide a deeper, more practical understanding of its contributions and shortcomings. This framework addresses the limitation of traditional summaries that reflect only the authors’ selfpresented perspective by analyzing how other researchers discuss and cite the paper. The study reveals meaningful patterns in how the academic community perceives these works, highlighting areas of alignment and divergence between external citation feedback and the authors’ own presentation.

SECite Methodology

The SECite framework delivers citation sentiment analysis with intelligent summary generation through three major steps. These steps are:

1. semiautomated extraction of citation statements from research papers.

2. NLP-based sentiment classification of citations into positive, or negative using an unsupervised K-means clustering algorithm.

3. Generative AI–powered summarization that highlights both the strengths and weaknesses of the referenced works.

Data Collection and Cleaning

The data collection phase involves a semi-automated pipeline designed to interface with Google Scholar to efficiently extract citation information. The procedure consists of several precise steps to ensure comprehensive data acquisition, including:

(i) Automated Navigation and Search:

The pipeline opens the Google Scholar website using Selenium for browser automation and enters the target research paper title to locate the exact paper.

(ii) Accessing Citation References:

After identifying the correct paper, the system clicks on the citations link to access all citing works.

(iii) Link Collection and Paper Retrieval:

The pipeline gathers the links to all citing papers, sequentially opens each link, downloads the associated paper, and saves it to a designated location for processing.

(iv) PDF Parsing and Extraction:

Each downloaded paper is loaded and processed through a custom PDF parser function implemented in Python to extract citations.

Challenges faced during data collection include:

i) Reading PDFs and Extracting References: Parsing PDFs to extract reference numbers and cited sentences presents a significant challenge due to variations in textual formatting and encoding across different documents.

ii) Converting Table References: In some cases, references appear in tabular form, requiring careful interpretation for conversion into clear sentences or association rules.

iii) Handling CAPTCHA Challenges: Some sources require CAPTCHA verification, which demands human intervention to continue the data collection process.

Data Transformation and Modeling

After data collection, the dataset is cleaned by removing stopwords, numbers, and non-alphabetic characters using NLTK and regular expressions to enhance quality. This cleaning process focuses on removing words like “the,” “and,” and “is” which carry little semantic value in analytical contexts.

The data transformation stage utilizes unsupervised machine learning techniques, specifically K-means clustering [16], to group the data into three sentiment categories: Positive, Neutral, and Negative. For feature extraction, BERT sentence embeddings via Hugging Face are employed to capture nuanced contextual and semantic information from text. The performance of the clustering is assessed using the Silhouette Score [29], which measures cluster cohesion and separation. Finally, t-SNE is applied for dimensionality reduction to visualize the high-dimensional embeddings in a two-dimensional space.

Text Summarization and Evaluation

The analysis involves applying summarization techniques to categorize data points as positive or negative sentiment. The comparison between data-point summaries and full-paper summaries, also categorized by sentiment, is performed using Gemini 1.5 via LangChain and prompt engineering. This multi-tiered approach allows for cross-validation to evaluate consistency and accuracy across different aggregation levels.

The evaluation metrics include:

(i) Unsupervised Sentiment Analysis:

The Silhouette Score is used to evaluate the K-Means model, where higher values indicate that a data point fits well within its own cluster and lower values suggest poor clustering quality.

(ii) Summary Comparison:

The Semantic Similarity Score is calculated using BERT or Sentence-BERT embeddings to quantify the semantic overlap between the original text and the generated summary.

Results and Analysis

The effectiveness of the automated data-collection pipeline was assessed, revealing a success rate of 44% due to obstacles like captcha verification, text parsing errors, and timeout issues on five out of nine research papers. Regarding K-means performance, the Silhouette Score for two clusters was 0.1384 and for three clusters was 0.1086, indicating weak separation in both cases. The study selected the two-cluster K-Means model due to its superior Silhouette Score. For summary quality, empirical study comparisons showed an average similarity of 0.64 across summaries and manual analysis indicated that positive summaries achieved high conceptual and factual scores, while negative summaries showed reduced conceptual correctness and informativeness.

Discussion and Limitations

The SECite framework faces limitations in the semi-automated citation extraction process, specifically regarding captcha handling, text parsing errors, and occasional timeouts. Furthermore, K-Means clustering can be sensitive to the initial placement of centroids and less effective on imbalanced datasets. While transformer-based LLMs are powerful, their internal decision-making can appear opaque, suggesting an opportunity to incorporate explainability techniques. The study's focus on software engineering literature may limit its generalizability across other academic disciplines.

Conclusion and Future Work

The study successfully investigates the positive and negative aspects of research papers from both external citations and internal content by developing a semi-automated approach that categorizes citations into positive and negative sentiments using K-Means clustering. By comparing citation-based and full-text perspectives, SECite offers a holistic view of each paper’s impact. Future work will focus on enhancing automation and accuracy by improving the reliability of citation discovery, parsing, and extraction to address technical challenges like captcha handling. For sentiment classification, exploring advanced alternatives such as zero-shot prompt engineering with LLMs could improve grouping without requiring labeled data. Additionally, evaluation metrics for summaries can be developed to better capture coherence and completeness.

References

[1] D. Malcom, “It’s time we fix the peer review system,” American Journal of Pharmaceutical Education, vol. 82, no. 5, 2018.

[2] H. Staines and M. E. Martone, “Community feedback on scholarly content: why it is important and why it should be preserved,” Insights, vol. 31, 2018.

[3] B. Yu, “Automated citation sentiment analysis: What can we learn from biomedical researchers,” Proceedings of the American Society for Information Science and Technology, vol. 50, no. 1, pp. 1–9, 2013.

[4] S. Ghosh, D. Das, and T. Chakraborty, “Determining sentiment in citation text and analyzing its impact on the proposed ranking index,” 2017.

[5] T. Yu, T. S. Zaman, and C.

1. Bold Header: Adaptive Clustering Algorithm Implementation

This improvement involves replacing standard K-means with alternative clustering techniques like DBSCAN or HDBSCAN to enhance grouping accuracy and better capture the nuances of citation sentiment. This allows the system to identify non-spherical or noise-affected clusters more effectively, addressing the finding that K-means remains appropriate for this task because it provides a systematic method to partition citation statements into meaningful sentiment groups (positive/negative).

2. Bold Header: Explainable AI Integration for LLM Outputs

To address the opaque nature of transformer-based LLMs, incorporate explainability techniques to better understand which features and linguistic patterns influence the model’s outputs. This would allow researchers to validate why a specific citation was classified positively or negatively, moving beyond just the final summary.

3. Bold Header: Robust Data Acquisition Pipeline

Improve the semi-automated extraction success rate from 44% to a higher level by developing specialized modules to handle identified failure points, specifically captcha verification, text parsing errors, and occasional timeouts. This involves creating more resilient parsers for varying document formats and implementing intelligent retry mechanisms for access-restricted sources.

4. Bold Header: Multi-Tiered Summary Validation

Enhance the summary quality check by refining the comparison between data points and full papers to better evaluate consistency and accuracy across different aggregation levels. This directly tackles the finding that the relationship between summarization granularity and sentiment accuracy needs deeper investigation, ensuring that summaries generated from citation feedback accurately reflect the source material.

5. Bold Header: Domain Generalization Strategy

Mitigate the risk of domain bias by implementing a strategy to test SECite's adaptability across diverse fields beyond software engineering literature. This involves systematically applying the framework to different datasets to determine whether citation patterns, terminology, and sentiment expressions may vary significantly across academic disciplines.

Identifying the strengths and limitations of a research paper is a core component of any literature review. However, traditional summaries reflect only the authors' self-presented perspective. Analyzing how other researchers discuss and cite the paper can offer a deeper, more practical understanding of its contributions and shortcomings. In this research, we introduce SECite, a novel approach for evaluating scholarly impact through sentiment analysis of citation contexts. We develop a semi-automated pipeline to extract citations referencing nine research papers and apply advanced natural language processing (NLP) techniques with unsupervised machine learning to classify these citation statements as positive or negative. Beyond sentiment classification, we use generative AI to produce sentiment-specific summaries that capture the strengths and limitations of each target paper, derived both from clustered citation groups and from the full text. Our findings reveal meaningful patterns in how the academic community perceives these works, highlighting areas of alignment and divergence between external citation feedback and the authors' own presentation. By integrating citation sentiment analysis with LLM-based summarization, this study provides a comprehensive framework for assessing scholarly contributions.