RAG4Tickets: AI-Powered Ticket Resolution via Retrieval-Augmented Generation on JIRA and GitHub Data

The gist: The proposed Retrieval-Augmented Generation (RAG) framework integrates Sentence-Transformers, FAISS, and an LLM to deliver context-aware ticket resolution recommendations by embedding historical JIRA tickets and GitHub metadata to synthesize grounded suggestions.

How it works

1. The system utilizes a unified architecture for ingesting and linking JIRA and GitHub dataA unified architecture for ingesting and linking JIRA and GitHub data.

2. A decision pipeline is established where retrieved evidence guides LLM-generated resolution suggestionsA decision pipeline where retrieved evidence guides LLM-generated resolution suggestions.

3. The embedding layer converts heterogeneous artifacts into dense vector representations using Sentence-Transformers and CodeBERT/GraphCodeBERTThe embedding layer converts heterogeneous artifacts into dense vector representations using Sentence-Transformers and CodeBERT/GraphCodeBERT.

4. Embeddings are indexed using FAISS to enable scalable Approximate Nearest Neighbor (ANN) retrieval across millions of vectorsEmbeddings are indexed using FAISS to enable scalable Approximate Nearest Neighbor (ANN) retrieval across millions of vectors.

5. The generation layer synthesizes the final response by concatenating top-ranked retrievals into a structured prompt for an LLMThe generation layer synthesizes the final response by concatenating top-ranked retrievals into a structured prompt for an LLM.

Data and Embedding Strategy

The data sources include JIRA tickets with titles, descriptions, priority, status, and resolutionJIRA tickets with titles, descriptions, priority, status, and resolution, user comments capturing discussions and error logs for implicit knowledgeuser comments capturing discussions and error logs for implicit knowledge, and GitHub PRs providing commit messages, code diffs, summaries, and review commentsGitHub PRs providing commit messages, code diffs, summaries, and review comments. The embedding layer employs Sentence-Transformers like all-MiniLM-L6-v2 and multi-qa-MPNet-base-dot-v1 for textual dataThe embedding layer employs Sentence-Transformers like all-MiniLM-L6--v2 and multi--qa--MPNet--base--dot--v1 for textual data, while CodeBERT and GraphCodeBERT embeddings are integrated to capture programming-language semanticswhile CodeBERT and GraphCodeBERT embeddings are integrated to capture programming-language semantics. Embeddings are partitioned by artifact type for targeted search spacesEmbeddings are partitioned by artifact type for targeted search spaces, and embeddings are normalized and periodically refreshed to capture evolving terminologyembeddings are normalized and periodically refreshed to capture evolving terminology.

Retrieval Mechanism

The retrieval layer uses FAISS (Facebook AI Similarity Search) to index all embeddings, supporting Approximate Nearest Neighbor (ANN) search for scalable retrievalFAISS (Facebook AI Similarity Search) to index all embeddings, supporting Approximate Nearest Neighbor (ANN) search for scalable retrieval. The workflow involves encoding a new ticket, querying the FAISS index to retrieve top-k similar tickets and PRs, and then re-ranking them using cosine similarity before passing them to the generation layerThe workflow involves encoding a new ticket, querying the FAISS index to retrieve top-k similar tickets and PRs, and then re-ranking them using cosine similarity before passing them to the generation layer. The methodology empirically selected HNSW for its balance of speed and recall in enterprise-scale datasetsThe methodology empirically selected HNSW for its balance of speed and recall in enterprise-scale datasets, finding it reduced average query latency by 73% compared to Flat index, with only a 2% drop in recallfinding it reduced average query latency by 73% compared to Flat index, with only a 2% drop in recall.

Generation and Evaluation

The RAG workflow synthesizes the final response by concatenating top-ranked retrievals into a structured prompt for an LLMThe RAG workflow synthesizes the final response by concatenating top-ranked retrievals into a structured prompt for an LLM. The LLM produces candidate solutions including step-by-step resolution plans and hyperlinks to related PRs for immediate traceabilityThe LLM produces candidate solutions including step-by-step resolution plans and hyperlinks to related PRs for immediate traceability. Evaluation metrics include Recall@k, Mean Reciprocal Rank (MRR), BLEU, ROUGE-L, and a factual consistency score against ground-truth resolutionsEvaluation metrics include Recall@k, Mean Reciprocal Rank (MRR), BLEU, ROUGE-L, and a factual consistency score against ground-truth resolutions. The case study demonstrated a 45% improvement in average resolution time and a 32% self-reported productivity uplift for developersThe case study demonstrated a 45% improvement in average resolution time and a 32% self-reported productivity uplift for developers.

Limitations and Future Work

Key challenges include factual reliability, where occasional hallucinations were observed when the LLM extrapolated beyond retrieved evidenceKey challenges include factual reliability, where occasional hallucinations were observed when the LLM extrapolated beyond retrieved evidence, and dataset drift due to framework evolution which degrades retrieval precisionand dataset drift due to framework evolution which degrades retrieval precision. Future work plans include incorporating a retrieval verification layer and reinforcement learning with human feedback (RLHF) to penalize hallucinationsFuture work plans include incorporating a retrieval verification layer and reinforcement learning with human feedback (RLHF) to penalize hallucinations, and investigating hybrid retrieval architectures combining dense FAISS indexing with sparse retrievers like BM25 for precision-recall tradeoff optimizationand investigating hybrid retrieval architectures combining dense FAISS indexing with sparse retrievers like BM25 for precision-recall tradeoff optimization.

The results collectively validate the promise of AI-augmented DevOps workflows to mitigate cognitive overload, accelerate triage, and enhance organizational knowledge reuse across software delivery lifecycles. The study demonstrates how retrieval and generation can be effectively fused to emulate human-like diagnostic and decision-making behaviorThe study demonstrates how retrieval and generation can be effectively fused to emulate human-like diagnostic and decision-making behavior.

This paper is 2510.08667, a Retrieval-Augmented Generation (RAG) framework that unifies JIRA tickets, developer discussions, and GitHub pull requests into an integrated pipeline for AI-assisted ticket triage and resolution generationa Retrieval-Augmented Generation (RAG) framework that unifies JIRA tickets, developer discussions, and GitHub pull requests into an integrated pipeline for AI-assisted ticket triage and resolution generation.

1. Improve factual reliability by incorporating retrieval verification layers or cross-encoder re-ranking to ensure that generated recommendations remain strictly supported by retrieved artifacts. This directly addresses the limitation that LLM occasionally generates responses not grounded in retrieved evidence, leading to unreliable or misleading suggestions.

2. Mitigate historical bias and data quality issues by implementing temporal relevance weighting and human-in-the-loop curation of training and retrieval datasets. This strategy counters the risk that Legacy tickets and PRs may contain outdated or low-quality fixes, propagating technical debt or suboptimal patterns.

3. Enhance system stability against framework evolution by establishing a mechanism for periodic re-indexing and embedding refresh coupled with continuous fine-tuning using recent artifacts. This ensures the system maintains relevance despite Dataset drift (Framework Evolution) and prevents older embeddings from degrading retrieval precision.

4. Improve developer trust by ensuring Evidence-linked outputs (PRs, diffs, tickets) are displayed alongside suggestions. This directly supports the finding that developers placed greater trust in AI-generated resolutions when transparent evidence—such as retrieved PR links and rationale—was displayed.

5. Optimize for scalability and cost efficiency by investigating hybrid retrieval architectures that combine dense FAISS indexing with sparse retrievers such as BM25 or ColBERT for precision–recall tradeoff optimization. This aims to manage the operational costs associated with indexing tens of thousands of artifacts in FAISS while maintaining high performance.

Modern software teams frequently encounter delays in resolving recurring or related issues due to fragmented knowledge scattered across JIRA tickets, developer discussions, and GitHub pull requests (PRs). To address this challenge, we propose a Retrieval-Augmented Generation (RAG) framework that integrates Sentence-Transformers for semantic embeddings with FAISS-based vector search to deliver context-aware ticket resolution recommendations. The approach embeds historical JIRA tickets, user comments, and linked PR metadata to retrieve semantically similar past cases, which are then synthesized by a Large Language Model (LLM) into grounded and explainable resolution suggestions. The framework contributes a unified pipeline linking JIRA and GitHub data, an embedding and FAISS indexing strategy for heterogeneous software artifacts, and a resolution generation module guided by retrieved evidence. Experimental evaluation using precision, recall, resolution time reduction, and developer acceptance metrics shows that the proposed system significantly improves resolution accuracy, fix quality, and knowledge reuse in modern DevOps environments.