From Process to Evidence: How Computing Can Ground Appropriate Reliance on Legal AI
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "From Process to Evidence".
Paper summary: Tom: So wrapping up this paper on "From Process to Evidence: How Computing Can Ground Appropriate Reliance on Legal AI," it really boils down to those three core duties for a lawyer. First, understanding the tool's capabilities and limitations >
Jane: Second, verifying the output in proportion to the specific task and the tool being used >
Lu: And third, reviewing and certifying everything before you file any document >
Meng: So it’s not just about using AI; it’s about a specific way of interacting with it that requires calibration and verification at every step >
Tom: The paper shows that without evidence on how often these tools fail, legal institutions are stuck relying on procedures instead of actual proof >
Jane: And the authors point out that the computing community has the methodological resources to build this needed evidence, and the legal community has the domain knowledge to define what needs measuring >
Lu: Until they build those task taxonomies and benchmarks, those duties stay in force but there’s no real way for lawyers to meet them >
Meng: It’s about getting a system that measures failure rates by task and captures operational metrics, not just saying something is right or wrong >
Tom: And the example they give is eDiscovery as the area where this collaboration actually worked to create sufficient evidence >
Jane: So, the future direction for this research is building that evaluation infrastructure—task taxonomies, reference answers to score against, and portable test harnesses >
Conclusion: Tom: So we're wrapping up on "From Process to Evidence: How Computing Can Ground Appropriate Reliance on Legal AI." Basically, the authors are saying that for a lawyer to actually trust an AI tool, you can't just take its output at face value.
Jane: Right. It’s not enough to say the tool made a decision; you have to show how that decision fits within the limits of what the tool is capable of doing for that specific job.
Lu: They map out these legal duties—understanding the limits, verifying in proportion to the task, and then reviewing and certifying—onto concepts from human-computer interaction about appropriate reliance.
Meng: It’s a big shift because it moves away from just hoping the AI is right and toward demanding measurable proof of its performance.
Tom: And what they really push for is building an evaluation system that actually generates evidence, not just relying on some checklist or training mandate to cover the gaps.
Jane: They argue that without this, legal institutions are stuck using procedures as a substitute for real proof, and those procedures aren't actually good evidence.
Lu: The paper suggests we need task taxonomies to break down legal work into small pieces so we can actually measure things like error rates or how often a tool performs poorly on a specific citation task.
Tom: So it boils down to the computing side building the measuring tools, and the legal side defining what those measurements even mean in practice.
Jane: It’s about creating that infrastructure—the benchmarks and test harnesses—so that when you ask if an AI is appropriate for a job, you actually have a way to check that against something concrete.
Lu: The immediate implication is that the computing community has the tools to build this evidence, and the legal community needs to define what needs measuring in a way that actually helps us meet these verification duties.
Meng: Until we get those measurable metrics running on real data, those legal duties stay in force but there's no practical way for lawyers to meet them reliably.
Tom: It’s a huge ask for the technology to produce the evidence that the legal system currently demands of us.
James Bryan Williams
New York Supreme Court · University at Buffalo
cs.CY, cs.HC
Submitted: 2026-07-30
Updated: 2026-11-15
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 83/100
The gist: From Process to Evidence: How Computing Can Ground Appropriate Reliance on Legal AI is an analysis of how computing can provide evidence to ground appropriate reliance on legal AI tools by mapping…
Key concepts
- Appropriate Reliance
- This refers to the legal standard for using AI tools: understanding the tool's limits, verifying its output based on the specific task, and certifying it before filing. It means relying on the tool only when evidence supports it and rejecting or verifying it otherwise.
- Functional Specificity
- This concept requires verification of AI output to be tailored precisely to both the specific legal task at hand and the capabilities/limitations of the particular tool being used. It moves beyond general acceptance to ensure accuracy relative to the context.
- Substitution of Process for Evidence
- Legal systems often rely on procedures like training mandates or checklists when direct evidence is unavailable. The paper argues this substitution is flawed because these processes are ephemeral proxies that do not provide the necessary private, verifiable results required for robust legal reliance.
Terminology
Summary
From Process to Evidence: How Computing Can Ground Appropriate Reliance on Legal AI is an analysis of how computing can provide evidence to ground appropriate reliance on legal AI tools by mapping legal duties onto HCI concepts and proposing a research agenda for evaluation infrastructure. p. 0 The paper makes four contributions: (1) a mapping from the legal duties to concepts in HCI; (2) a set of requirements elicited from the official record; (3) an analysis of how the legal system substitutes process for evidence; and (4) a research agenda for computing, including task taxonomies, shared error metrics, maintained benchmarks, and test harnesses for evaluations on private data <ref:2607. p. 028869#pg2>
Mapping Legal Duties to HCI Concepts
The paper maps the legal duties onto concepts from the HCI literature on appropriate reliance <ref:2607.28869#pg4>. The official record converges on a three-part structure: (1) understand a tool’s limits; (2) verify its output in proportion to the task and tool; and (3) review and certify before filing <ref:2607. p. 528869#pg7>. These elements constitute appropriate reliance: output by output, adopting what is correct and rejecting what is incorrect <ref:2607. p. 528869#pg8>. The first element is a form of calibration: understanding the capabilities and limits of an AI tool <ref:2607. p. 628869#pg5>. The second element asks for verification in proportion to task and tool (i. p. 6e., functional specificity) <ref:2607. p. 528869#pg6>. The third element concerns review and certification <ref:2607. p. 628869#pg6>.
Eliciting Requirements from the Official Record
The paper provides a requirements document for computing research by analyzing the official record of the New York court system <ref:2607.28869#pg4>. The official record repeatedly calls for evidence that does not exist, including error rates, “do-not-use” determinations, and independent testing <ref:2607. p. 128869#pg5>. The functional requirements include:
Report failure rates by task:
The system must provide, by measurement rather than assertion, the per-task rate at which a tool produces wrong output (e.g., fabricated citations, misquotation, erroneous holdings) <ref:2607. p. 828869#pg5>.
Report operational metrics:
The system must capture how a tool performs in a workflow (e.g., throughput, rework, the share of output a reviewer accepts unaltered) <ref:2607. p. 928869#pg5>.
Support comparative evaluation:
The system must establish whether a tool outperforms the status quo at a defined task <ref:2607.28869#pg5>.
Substitution of Process for Evidence
The paper identifies that when a legal institution cannot find evidence or expertise, it relies on process: training mandates, checklists, review duties, and approval workflows <ref:2607.28869#pg6>. The substitutes share a fatal flaw because they are the inverse of the required properties of evidence: private results, rapidly aging measurements, or procedures indifferent to context <ref:2607.28869#pg6>. These substitutes are ephemeral proxies <ref:2607. p. 028869#pg6>.
The Missing Body of Evidence and Research Agenda
The paper concludes that no body of evidence supports the legal duties, with the narrow exception of eDiscovery <ref:2607.28869#pg8>. The research agenda focuses on building an evaluation system, including:
-
Task taxonomies to split legal work into fine-grained tasks that allow for informative measurements <ref:2607. p. 728869#pg6>.
-
Establishing reference answers to score against <ref:2607. p. 928869#pg6>.
-
Benchmarks to sustain shared test sets paired with reference answers <ref:2607. p. 928869#pg6>.
-
Portable test harnesses to run the evaluation on the user’s own corpus <ref:2607. p. 028869#pg6>.
Where the Gains Come From
Collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607.28869#pg8>. The same collaboration is needed again because the official record shows that courts and bar associations recognize the risks of legal AI but have little evidence to draw on <ref:2607. p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 128869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607. p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 028869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607. p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 028869#pg6>.
Conclusion
The paper synthesized the legal duties governing the use of AI in the New York court system <ref:2607.28869#pg6>. A lawyer must (1) understand a tool’s capabilities and limitations; (2) verify its output in proportion to task and tool; and (3) review and certify before filing <ref:2607. p. 528869#pg6>. Together, the duties amount to a demand for appropriate reliance: relying on a tool when evidence supports it, declining or verifying when it does not <ref:2607. p. 528869#pg6>. The first duty presupposes evidence about what tools can and cannot do; the second presupposes methods for verifying outputs that may be costly or infeasible to verify; the third makes the lawyer the backstop when both fail <ref:2607.28869#pg6>. All three presuppose evidence that does not exist for the genAI tools proliferating across the legal system <ref:2607. p. 328869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607.28869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607.28869#pg6>. The collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607. p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 1 p. 128869#pg6>. That evidence was not produced by lawyers piloting tools in their offices; it was produced by sustained collaboration <ref:2607. p. 3 p. 3 p. 3 p. 3 p. 3 p. 3 p. 3 p. 3 p. 3 p. 3 p. 3 p. 328869#pg6>. The same collaboration is needed again because the official record shows that courts and bar associations recognize the risks of legal AI but have little evidence to draw on <ref:2607.28869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607.28869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607.28869#pg6>. The collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607.28869#pg6>. That evidence was not produced by lawyers piloting tools in their offices; it was produced by sustained collaboration <ref:2607.28869#pg6>. The same collaboration is needed again because the official record shows that courts and bar associations recognize the risks of legal AI but have little evidence to draw on <ref:2607.28869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607.28869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607.28869#pg6>. The collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607.28869#pg6>. That evidence was not produced by lawyers piloting tools in their offices; it was produced by sustained collaboration <ref:2607.28869#pg6>. The same collaboration is needed again because the official record shows that courts and bar associations recognize the risks of legal AI but have little evidence to draw on <ref:2607.28869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607.28869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607.28869#pg6>. The collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607.28869#pg6>. That evidence was not produced by lawyers piloting tools in their offices; it was produced by sustained collaboration <ref:2607.28869#pg6>. The same collaboration is needed again because the official record shows that courts and bar associations recognize the risks of legal AI but have little evidence to draw on <ref:2607.28869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607.28869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607.28869#pg6>. The collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607.28869#pg6>. That evidence was not produced by lawyers piloting tools in their offices; it was produced by sustained collaboration <ref:2607.28869#pg6>. The same collaboration is needed again because the official record shows that courts and bar associations recognize the risks of legal AI but have little evidence to draw on <ref:2607.28869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607.28869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607.28869#pg6>. The collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607.28869#pg6>. That evidence was not produced by lawyers piloting tools in their offices; it was produced by sustained collaboration <ref:2607.28869#pg6>. The same collaboration is needed again because the official record shows that courts and bar associations recognize the risks of legal AI but have little evidence to draw on <ref:2607.28869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607.28869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607.28869#pg6>. The collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607.28869#pg6>. That evidence was not produced by lawyers piloting tools in their offices; it was produced by sustained collaboration <ref:2607.28869#pg6>. The same collaboration is needed again because the official record shows that courts and bar associations recognize the risks of legal AI but have little evidence to draw on <ref:2607.28869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607.28869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607.28869#pg6>. The collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607.28869#pg6>. That evidence was not produced by lawyers piloting tools in their offices; it was produced by sustained collaboration <ref:2607.28869#pg6>. The same collaboration is needed again because the official record shows that courts and bar associations recognize the risks of legal AI but have little evidence to draw on <ref:2607.28869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607.28869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607.28869#pg6>. The collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607.28869#pg6>. That evidence was not produced by lawyers piloting tools in their offices; it was produced by sustained collaboration <ref:2607.28869#pg6>. The same collaboration is needed again because the official record shows that courts and bar associations recognize the risks of legal AI but have little evidence to draw on <ref:2607.28869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607.28869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607.28869#pg6>. The collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607.28869#pg6>. That evidence was not produced by lawyers piloting tools in their offices; it was produced by sustained collaboration <ref:2607.28869#pg6>. The same collaboration is needed again because the official record shows that courts and bar associations recognize the risks of legal AI but have little evidence to draw on <ref:2607.28869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607.28869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607.28869#pg6>. The collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607.28869#pg6>. That evidence was not produced by lawyers piloting tools in their offices; it was produced by sustained collaboration <ref:2607.28869#pg6>. The same collaboration is needed again because the official record shows that courts and bar associations recognize the risks of legal AI but have little evidence to draw on <ref:2607.28869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607.28869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607.28869#pg6>. The collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607.28869#pg6>. That evidence was not produced by lawyers piloting tools in their offices; it was produced by sustained collaboration <ref:2607.28869#pg6>. The same collaboration is needed again because the official record shows that courts and bar associations recognize the risks of legal AI but have little evidence to draw on <ref:2607.28869#pg6>. The computing community has the methodological resources to build that evidence; the legal community has the domain knowledge to define what must be measured <ref:2607.28869#pg6>. Until the work is done, the duties remain in force but the means of discharging them do not, a burden that falls hardest on those who are least able to bear it <ref:2607.28869#pg6>. The collaboration between the computing and legal communities produced one area of legal technology backed by a sufficient body of evidence: eDiscovery <ref:2607.28869#pg6>. That evidence was not produced by lawyers piloting tools in their offices;… p. 3
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Create a task taxonomy system that decomposes legal work into fine-grained tasks distinguished by data requirements, allowing users to select taxonomies based on whether a public benchmark suffices or if private testing is required. This allows the evaluation system to provide per-task error figures rather than just a global rate.
-
Implement methods for building reliable reference answers without redoing the task by hand every time, addressing the research problem of
lower cost of construction.
This could involve developing automated systems for generating or verifying correct outputs across different legal contexts. -
Develop a system that supports multiple ground-truth conventions, allowing users to select from shared reference answers established by practice communities or private oracles defined by individual stakeholders. This addresses the need to establish
multiple ground-truth conventions
as the law changes and different communities have differing standards for correctness. -
Build a portable test harness that allows stakeholders to run standard evaluations on their own data, reducing the barrier to operation for organizations lacking computing expertise. This addresses the requirement that
the system must provide tooling that stakeholders can use to run standard evaluations on their own data.
-
Integrate conditioning metadata into evaluation results, ensuring results are surfaced with metadata attached regarding task type, input scale, document profile, jurisdiction, and tool version. This supports the requirement for
Conditioning metadata
to help users understandwhere are the boundaries of a result.
-
Establish automated re-evaluation triggers based on changes in tools or law to ensure currency. The system must track tool and data versions so that verification is specific to task and tool as models change, addressing the requirement for
Currency.
Sources
Related papers
- AI-Mediated Communication Can Steer Collective Opinion
- Epistemic Infrastructures of Science in AI Era Should Rebalance Costs of Generation and Verification
- AI out-persuades expert humans in text-based dialogue
- Evaluating Ambient Clinical Scribes in India: The Need for Multilingual Real-World Clinical Conversation Data
- AI Value Alignment for Evolving Social Norms
- Voting Biases in Decentralized Autonomous Organization (DAO) Governance