01 / The problem
What makes the decision difficult
A data agent can write fluent SQL and still query the wrong table, misunderstand business meaning or return a plausible wrong answer.
OpenAI-inspired
Artificial Intelligence · 2026 public case
OpenAI's in-house data agent supports thousands of users across a very large data platform. Its most transferable lessons are context retrieval, reusable workflows and systematic evaluation: an agent becomes trustworthy through the system around the model, not the model alone.
9 min read
System blueprint
From evidence to action
The case brief
The model is only one part of the story. A useful case study makes the problem, evidence, trade-offs and final action easy to explain.
01 / The problem
What makes the decision difficult
A data agent can write fluent SQL and still query the wrong table, misunderstand business meaning or return a plausible wrong answer.
02 / The approach
How the system creates a useful output
Retrieve schema context, plan and execute analysis with tools, then continuously evaluate SQL and result correctness against golden cases.
A closer look
Four notes to help you reason about the case instead of simply memorizing its stack.
Why this is genuinely hard
OpenAI described an internal data platform serving more than 3,500 users across 600 petabytes and 70,000 datasets. At that scale, translating a question into SQL is not mainly a syntax problem. The agent must identify the correct table among near-duplicates, understand grain and lineage, recover business definitions from code or documents, apply permissions and decide whether a surprising empty result is real or caused by a bad join. Two SQL queries can look different and still be equivalent; a fluent explanation can also hide a materially wrong result. The difficult product problem is preserving an inspectable chain from question to context, query, output and conclusion. Trust comes from that chain and repeatable evaluation, not from how confident the answer sounds.
Design decisions that matter
A layered context system can combine table usage, expert annotations, code-derived enrichment, institutional knowledge, memory and live inspection. Precomputing and embedding context keeps retrieval focused across thousands of tables, while live queries resolve stale or missing details; the trade-off is predictable latency versus freshness. Memory preserves corrections schemas cannot express, but saved guidance needs scope, ownership and an edit path or yesterday's exception becomes tomorrow's rule. Reusable workflows make recurring analysis consistent, while open planning remains necessary for novel questions. Evaluation cannot rely on SQL text because different queries may return the same correct result. A learner system should compare generated SQL and executed data with golden cases, then pair that with read-only execution, inherited permissions, visible assumptions and direct links to results.
Where a project like this can fail
Build a smaller version yourself
Create a miniature analytics agent over a synthetic commerce warehouse with 8–12 related tables and 50,000–100,000 rows: orders, refunds, customers, sessions, experiments and daily exchange rates, plus metadata describing grain, keys, freshness and exclusions. Write 60–80 natural-language questions with reviewed golden SQL, deliberately including ambiguous dates, joins, nulls and changing metric definitions. Deliver an evaluation harness that executes only read-only SQL, compares result tables rather than exact query strings, records retrieved context and reports accuracy by failure type. Add a small interface showing assumptions, generated query, source tables and raw results, then convert five observed mistakes into regression tests.
How the system works
Read this as a design walkthrough, not a recipe. In a real project, each step is tested, revised and connected to the next one.
Table names are not enough. Index descriptions, ownership, lineage, common joins and business definitions so the agent can retrieve the right context for a question.
The agent plans, chooses retrieval or query tools, inspects results and decides when the task is complete. Tools should be narrow, documented and permission-aware.
Package recurring reports and validations as workflows. This captures organizational knowledge once and gives routine analysis a stable, reviewable path.
Compare generated SQL and resulting data against curated golden cases. Grade acceptable variations, track regressions and add production failures back into the test set.
A practical stack
Real tools you can learn and recognize
What this case teaches
The goal is not to copy a company's private implementation. It is to understand the reasoning well enough to design, test and explain your own version.
What to measure
A model is only as useful as its evaluation
Whether executed results match the trusted expected answer.
Questions resolved without unsafe queries or unnecessary handoffs.
Tool calls, tokens and elapsed time per successful analysis.
What to remember
Ideas worth carrying into your own project
Read the original work
This is an independent learning adaptation. Use the sources to inspect the company's own explanation, evidence and technical detail.
Next case study
This independent learning case is inspired by public industry patterns and is not affiliated with or endorsed by the named company.