Self-evolving Agentic Customer Support System at LinkedIn

LinkedIn’s self-evolving agentic support system integrates retrieval-augmented generation with evolutionary auto-prompting and a modular, production-aligned evaluation framework to enable safe, continuous improvement without retraining foundation models. The system treats prompts, retrieval, and evaluation as a closed-loop, versioned workflow with operational guardrails. Offline simulations and ablations show clear quality gains over vanilla RAG and baseline agents, including reduced hallucinations and improved response completeness. In a two-week user-randomized A/B test on LinkedIn’s production support traffic, the integrated self-evolved workflow increased QA self-serve by 9.0 percentage points, cancellation self-serve by 4.8 points, and routing accuracy by 30.6 points. These results demonstrate a practical path to scalable, self-evolving AI agents in real-world enterprise settings.

The system architecture exposes two coupled loops—an inner inference loop (agent ↔ RAG ↔ content lake) and an outer optimization loop (auto-prompt → agent → evaluator → auto-prompt)—enabling scalable self-improvement and rapid adaptation across products and languages without modifying the base model or retrieval corpus. At inference time the agent is conditioned by a system prompt from the Automatic Prompt Engine that encodes task instructions, tone, and implicit policies for reasoning depth and tool use. RAG is exposed as an explicit action rather than a preprocessing step, letting the agent decide when to retrieve and how to integrate evidence. Outputs are scored by a modular evaluation framework that decomposes quality into grounding, intent alignment, multilingual fidelity, and stylistic compliance using rule-based diagnostics and LLM-based judges; aggregate fitness signals feed back into the prompt engine, which refines the system prompt via evolutionary search.

The Automatic Prompt Engineering Engine (“Auto-Prompt”) formulates prompt optimization as evolutionary search inspired by genetic algorithms. Domainspecific business rules guide an LLM-driven initialization stage, after which a population of candidate prompts evolves through selection, crossover, and mutation, evaluated each generation against task-specific metrics; only the strongest survive into the next iteration. To prevent evolutionary drift from violating safety or policy requirements, business rules are treated as immutable hard constraints rather than optimization objectives: violating prompts are filtered or assigned zero fitness prior to selection. Crossover and mutation therefore operate within a constrained prompt space that preserves required behaviors by construction.

The unified knowledge base consolidates three content sources: help articles, learning content, and product documentation microsites. They are refreshed by an ETL pipeline that chunks, embeds, facets, and retires content before loading it into the versioned content lake. Retirement is enforced through a snapshot pointer, so each refresh scopes retrieval to the latest snapshot and supersedes prior content without deleting it. Each document is tagged with three facets—product (line of business), locale (7+ languages), and snapshot date—enabling scoped, reproducible retrieval. At query time, hybrid dense–sparse retrieval is followed by semantic re-ranking, returning ranked grounding documents with provenance (URL, locale, source) for citation and audit.

The evaluation framework uses a modular, agent-aware framework that decomposes quality into interpretable dimensions—grounding, intent understanding, tool-execution correctness, and multilingual fidelity—each scored by specialized evaluators combining rule-based checks with LLM judgments. Multilingual quality is assessed by dedicated modules that detect untranslated spans, verify terminology, and measure fluency via XGLM perplexity, while an LLM evaluator scores semantic fidelity under realistic dialog conditions. Unlike static benchmarks, evaluation is conditioned on the context retrieved at inference time, addressing knowledge drift directly. Signals are aggregated through a consensus layer calibrated against periodic human annotations, stored in a versioned replayable format, and consumed by upstream optimization (auto-prompt and RAG iteration), closing the loop between agent behavior, measurement, and controlled self-evolution.

The production architecture is structured around four principles: Reliability as a prerequisite for self-evolution (a lightweight orchestration and fallback layer mediates user traffic with deterministic routing, error handling, and graceful degradation); Runtime as a controlled mutation boundary (prompts are assembled dynamically, context-construction policies are swappable, and tool usage is declaratively configured, so behavioral improvements propagate without redeploying the runtime itself); Memory as versioned artifacts, not latent state (the runtime carries no cross-session latent state beyond short-horizon, task-scoped context, and longer-lived knowledge is promoted only through explicitly versioned and evaluated artifacts); and Telemetry as an evaluation substrate (structured event streams feed nearline quality checks and offline evaluators measuring groundedness, relevance, completeness, and policy compliance). The outer optimization loop typically runs weekly and can also be triggered by content refreshes or detected regressions; candidate prompts and retrieval configurations are promoted only after offline regression checks and staged rollout.

In production, three patterns proved load-bearing: centralized execution through configuration, modular prompts, and declarative tool selection—rather than per-LoB code forks—was critical to avoiding behavioral divergence; layered fail-safes such as targeted query redirection allowed traffic to route to a known-safe content pool when a polluted vector index required hours-long reindexing, cutting recovery from hours to minutes; and staged rollout of new execution patterns in greenfield workflows before extension to mature traffic let evolution mechanisms and evaluation signals mature before broader deployment.

In the RAG simulation comparing four response-generation configurations (Vanilla RAG, RAG-only agents with ReAct-style and OpenAI tool invocation, and the full Support AI Agent), vanilla and agentic RAG configurations hallucinate at 4.8–6.2%, while the full Support AI Agent reaches <0.1% with the highest overall score (2.78) and completeness (87.8%). The gain comes from explicit agentic constraints that prioritize synthesis from retrieved authoritative content and suppress reliance on parametric knowledge when grounded sources exist.

In the Auto-Prompt simulation on the routing intent detection dataset, prompt accuracy improves consistently only when both crossover and mutation are enabled—increasing average accuracy from 62.6% to 68.0% after two generations, with the best prompt reaching 73.3%. Removing either operator slows improvement and saturates early, indicating that prompt evolution, not random sampling, is what drives sustained gains.

In the evaluator-signal ablation, response groundedness is the dominant driver of evaluator reliability—removing it causes the largest drop in alignment with human labels (from 87% to 76%), while removing completeness barely changes it (86%), and collapsing all signals into a single overall score performs worst (67%), obscuring critical failure modes.

In the multilingual translation evaluation on the En–Zh slice, accuracy rises from 49.7% (COMET) to 76.5% (single-agent LLM judge) to 84.8% (modular multi-agent evaluator): decomposing translation quality into specialized dimensions captures the domain-specific, context-sensitive demands of support interactions that metric-based scoring misses.

The online experiment was a two-week A/B test on continuous live member and customer support traffic with fixed 50/50 assignment. The control was the original production agent—a handcrafted prompt, fixed retrieval pipeline, and periodic human QA. The treatment was the integrated self-evolved workflow, which bundles (i) evolved Auto-Prompt, (ii) agent-invoked RAG, and (iii) closed-loop evaluator-driven iteration with gated rollout. QA self-serve increased from 33.7% to 42.7% (+9.0 pp, 95% CI [8.4, 9.6], z = 27.6); cancellation self-serve increased from 61.9% to 66.6% (+4.8 pp, 95% CI [3.8, 5.7], z = 10.0); routing accuracy increased from 38.2% to 68.8% (+30.6 pp, 95% CI [23.6, 37.6], z = 8.2). All three outcomes have p ≪ 10−4 and remain significant after Holm correction. The experiment also monitored escalation rate, thumbs-up/down feedback, latency, customer-satisfaction score, and moderation incidents; none regressed under treatment.

Limitations include evaluator dependence on LLM-as-judge evaluators (GPT-4.1), cost and latency of evolution, single-tenant evaluation (two weeks on one deployment surface), online component attribution (the treatment bundles all components), retrieval ceiling (response quality is bounded by retrieval coverage), limited multilingual stress test (English↔Chinese only), and closed-source dependencies (GPT-4o-mini, GPT-4.1, and a proprietary search backend).

Improvements to AI Systems:

1. Implement a closed-loop self-evolution architecture where prompts, retrieval, and evaluation form a versioned, continuously improving workflow—enabling the system to autonomously refine its own behavior without retraining the base model, adapting to new content and user patterns weekly.

2. Add evolutionary auto-prompting with hard safety constraints—use genetic-algorithm-style selection, crossover, and mutation to optimize system prompts, while treating business rules as immutable filters that reject or zero-score any violating prompt, ensuring policy compliance by construction rather than by optimization trade-off.

3. Expose retrieval as an explicit agent action rather than a preprocessing step—allow the AI to decide when to retrieve, what to retrieve, and how to integrate evidence, reducing hallucinations from 4.8–6.2% to under 0.1% by prioritizing synthesized grounded content over parametric knowledge.

4. Decompose evaluation into modular, interpretable dimensions (grounding, intent alignment, multilingual fidelity, stylistic compliance) with specialized judges—instead of a single overall score, which drops human-label alignment from 87% to 67%—enabling targeted detection and correction of specific failure modes.

5. Use hybrid dense-sparse retrieval with semantic re-ranking and provenance tagging—return ranked documents with URL, locale, and snapshot date for citation and audit, while scoping retrieval to the latest content snapshot via a pointer that supersedes old content without deletion, ensuring reproducibility and freshness.

6. Implement a consensus layer for evaluator signals that calibrates LLM-based judgments against periodic human annotations—improving alignment with human labels from 76% to 87% when groundedness is included, and storing signals in a versioned, replayable format for regression testing and continuous calibration.

7. Add a lightweight orchestration and fallback layer with deterministic routing, error handling, and graceful degradation—including targeted query redirection to a known-safe content pool during index corruption, cutting recovery time from hours to minutes.

8. Make runtime configuration-driven—assemble prompts dynamically, swap context-construction policies, and declare tool usage declaratively, so behavioral improvements propagate without redeploying the runtime, preventing per-line-of-business code forks and behavioral divergence.

9. Promote memory as versioned artifacts, not latent state—carry no cross-session latent state beyond short-horizon task context, and promote longer-lived knowledge only through explicitly versioned and evaluated artifacts, enabling safe rollback and auditability.

10. Use multi-agent evaluators for multilingual quality—decompose translation quality into specialized dimensions (untranslated spans, terminology, fluency via perplexity, semantic fidelity), improving accuracy from 49.7% (COMET) to 84.8% on En–Zh support interactions.

What the Improved AI System Can Do:

• Self-improve weekly without retraining, evolving its own prompts and retrieval strategies based on real user traffic and evaluator feedback, while maintaining hard policy guardrails.

• Reduce hallucinations to near-zero in grounded response generation by treating retrieval as an explicit, auditable action with provenance.

• Increase self-serve resolution rates by 9.0 percentage points for QA and 4.8 points for cancellations, and improve routing accuracy by 30.6 points in production support environments.

• Detect and correct specific quality failures (e.g., ungrounded claims, untranslated spans, stylistic non-compliance) through modular evaluators, rather than relying on opaque aggregate scores.

• Recover quickly from data corruption by redirecting traffic to safe content pools, minimizing downtime from hours to minutes.

• Scale across products and languages without code forks, using configuration-driven execution and versioned content lakes with multi-facet tagging (product, locale, snapshot date).

• Maintain auditability and reproducibility through versioned prompts, retrieval snapshots, and replayable evaluation signals, enabling regression checks before any promotion to production.

Enterprise support agents operate in rapidly changing environments where policies, product capabilities, and knowledge bases evolve continuously, making static assistants brittle and costly to maintain. We present LinkedIn's self-evolving agentic support system, which integrates retrieval-augmented generation with evolutionary auto-prompting and a modular, production-aligned evaluation framework to enable safe, continuous improvement without retraining foundation models. The system treats prompts, retrieval, and evaluation as a closed-loop, versioned workflow with operational guardrails. Offline simulations and ablations show clear quality gains over vanilla RAG and baseline agents, including reduced hallucinations and improved response completeness. In a two-week user-randomized A/B test on LinkedIn's production support traffic, the integrated self-evolved workflow increased QA self-serve by 9.0 percentage points, cancellation self-serve by 4.8 points, and routing accuracy by 30.6 points. These results demonstrate a practical path to scalable, self-evolving AI agents in real-world enterprise settings.