It's standard practice to store a fixed reference answer for every eval case. When the…
This is a AI post classified by Jev as AI infra & evals (research), kept by the AI Radar because it carries real work, not commentary.
Nice work from Adobe. It's standard practice to store a fixed reference answer for every eval case. When the underlying data changes daily, that stored answer goes stale. Adobe researchers write each reference answer as a Python function instead. The function runs against the live system at evaluation time, so the expected answer follows the data, and an upstream API change makes the test fail visibly. An LLM judge then splits the agent's response and the computed answer into atomic facts and scores precision and recall, whatever the output format. Against expert labels, this raises agree
Posted by DAIR.AI (132.3k followers) 2 days ago · 84 likes · 11.3k views · view the original post on X. Kept by the AI Radar as AI infra & evals. Tools mentioned: academy.dair.ai.
More AI work like this
- Trains your own large language model from scratch using plain PyTorch. — @tom_doerr
- Here's a useful use-case for @typesafeai for a change. — @iam_zachi
- This is genius. — @daniel_mac8
- Which GPU should you use for embedding workloads? — @runpod
- Big update to the @Alibaba_Qwen Qwen3.8-Flash-Next single DGX Spark recipe! — @jvr0x
- 🔰 The Databricks Advanced Learning Festival runs September 16 through October 14, with… — @databricks
- Q: What is a trace? — @HamelHusain
- Inference scaling part 1. — @rasbt
Every post is read and classified by Jev (TypeSafe): what it is, which market it belongs to, and whether the link is a real tool. 27.2k posts from 4.8k X accounts over the last 14 days, 1.2k tools, 19 markets. Collected every 5 minutes, fully re-ranked every hour — last update 2026-09-19 18:08 UTC. Full method.