Data lineage explained: tracing an AI answer back to its source

Data lineage is the record of where data came from and what happened to it on the way: which sources, which jobs, which transformations, and when. To trace an AI answer back to its source, you need that record joined to a second one, written at answer time, that lists exactly which documents, queries and dataset versions the model used. With both, any number in an answer can be followed back to the rows it was computed from.

Most companies have some of the first record and almost none of the second.

A wrong number, followed backwards

Here is an illustration. A sales head asks the internal assistant for South region revenue in Q3. The assistant says ₹4.2 crore. Finance has ₹3.9 crore in its board pack, and the meeting is on Monday.

Without lineage, someone spends Friday afternoon rerunning queries and guessing: a made-up figure, a stale table, a different definition of revenue, a draft document.

With lineage, the work is mechanical. You open the assistant's run for that question and see it called a SQL tool against a table called revenue_kpi. You see the exact query. You see that revenue_kpi was last written by a pipeline run at 06:10 that morning, and that the run read the orders table but not the refunds table, because a filter on order status excluded refunded orders from the subtraction step. The ₹0.3 crore gap was refunds. The model did nothing wrong. It faithfully reported a number that was built wrong two steps upstream.

When an AI answer is wrong, the fault is often not in the model, and you cannot fix what you cannot locate. We covered lineage briefly as one item of a data readiness checklist; this article is about how to actually build the trail.

Two records that have to meet

The story above crossed a boundary that most tooling does not.

The first half (answer, agent run, tool call) is what I'll call answer provenance. It lives in the AI runtime and is written while the model works. The second half (table, pipeline run, source rows) is classic data lineage. It lives in the data platform and is written when pipelines run.

The question people actually ask ("why did it say that?") needs both, joined on something precise: the name and version of the dataset the tool call read. If the AI side only logs "queried the revenue table" and the data side only knows it refreshes daily, you can't tell which refresh the answer saw.

This join is the reason CloudOEngine keeps data connections, pipelines and the AI runtime on one platform with one permission model and one audit trail, so the records on both sides of this join sit in one audit trail rather than in separate tools.

The vocabulary, borrowed from two standards

Two open specifications already describe all of this, and they agree more than they differ.

The W3C published the PROV family of documents in April 2013. PROV defines provenance as information about the entities, activities and people involved in producing a piece of data or thing, and notes that it can be used to judge that data's quality, reliability or trustworthiness. Its data model, PROV-DM, has three core types. An entity is a thing with some fixed aspects, such as a file, a table version or an answer. An activity is something that occurs over a period of time and acts upon or with entities, such as a query or a pipeline run. An agent bears some form of responsibility for an activity taking place, which can be a person, a service or, fittingly, an AI agent. Activities use entities and generate new ones, and one entity can be derived from another.

OpenLineage, an open framework for data lineage collection and analysis and a graduate project of the LF AI & Data Foundation, uses a more operational version of the same idea: datasets, jobs and runs. A job is a process that consumes or produces datasets (a workflow task, a query, even a model). A run is one occurrence of a job in time, with its own unique ID. Runs report events as they progress, typically starting with START and ending with COMPLETE, and each event can carry facets: extra detail such as the schema of a dataset, the SQL a job ran, an error message, or the parent run that started it.

An agent answering a question is just another job. It has a run ID, it reads datasets (tables, document chunks), and it produces an output. Treat it that way and AI runs can live in the same lineage graph as the pipelines that feed them.

Table-level lineage is not enough for numbers

Table-level lineage says revenue_kpi was built from orders and refunds. That would not have found the bug in our illustration, because the refunds table was correctly listed as an input. The problem was inside the transformation.

Column-level lineage goes one layer down. The OpenLineage column lineage facet maps each output column to the input columns it depends on, and it splits dependencies into two kinds. A direct dependency means the output value is derived from the input: copied unchanged (identity), modified within the same row (transformation), or combined across rows (aggregation). An indirect dependency means the input affects the output without being a source of it: used in a join, a group-by, a filter, a sort, a window, or a conditional such as CASE WHEN.

That indirect category is where AI-facing numbers go wrong. A WHERE status != 'refunded' clause never shows up as a source of the revenue column, but it decides which rows count. If your lineage only tracks direct dependencies, filters and joins are invisible, and those are exactly the places where two teams' definitions of "revenue" drift apart.

So for any table an assistant quotes numbers from, I'd want column-level lineage including indirect dependencies. Elsewhere, table-level is often fine.

Declared lineage and observed lineage

There are two ways to learn the graph, and they answer different questions.

Declared lineage comes from code. In dbt, for example, the ref() function references another model, and dbt uses those references to build the dependency graph automatically and run models in the right order. The documentation notes a catch: if a reference is hidden from the parser, say inside a conditional block, dbt may not see the dependency until run time. Declared lineage tells you what is supposed to feed what, and it is available before anything runs.

Observed lineage comes from execution. An OpenLineage-style run event records what a particular run actually read and wrote, with the dataset version where the system knows it (the spec gives an Iceberg snapshot ID as an example of a version facet). This is the only kind that can tell you which version of revenue_kpi existed at 10:42 when the assistant answered.

For tracing AI answers, observed lineage matters most, because answers are tied to moments. The declared graph earns its keep in impact analysis: before you change the refunds logic, you want every table, dashboard and agent downstream of it.

What to record at answer time

The data side has standards and tools. The answer side is where most teams have nothing, so here is what I would record for every AI run that touches company data.

• The question, the user, their role, and the time.
• The prompt template version and the model used.
• Every retrieved chunk, with its document ID, section and document version.
• Every tool call, with the exact arguments (the full SQL text, not a summary), the row count or result returned, and the dataset name and version it read.
• Any intermediate reasoning steps the runtime exposes, and the final answer.
• A run ID that links all of the above, and that the data side can be joined to.

The tool call arguments are the part people skip, and they are the most valuable. "Queried revenue" is useless for debugging; the actual query, with its date range and filters, often shows the mistake at a glance. In CloudOEngine, every agent step and tool call is traced and inspectable in the AI runtime, so the run behind a disputed answer can be opened rather than reconstructed.

For RAG answers, a chunk from "Travel policy" is not enough if the policy changed in August. Store a version on every chunk at indexing time, as our RAG explainer suggests, and copy it into the run record on retrieval.

Citations are a pointer, not a trail

Many assistants now show sources under the answer. That helps a reader, but it is not lineage, for two reasons.

First, a citation is the model's claim about what it used, and models are not always right about that. The ALCE benchmark from Gao and colleagues (EMNLP 2023) measured whether generated citations actually support the statements they are attached to. On its ELI5 dataset, even the best models lacked complete citation support 50% of the time. A citation can point to a real document that does not say what the answer says.

Second, a citation stops at the document or table. It says "from revenue_kpi" but not which version, run or filter. That may be fine for a policy question, but not for a number headed into a board pack.

Show citations to users, and trust the run record when the two disagree.

Which record answers which question

Different people ask different questions about a disputed answer, and each is answered by a different piece of the trail.

If a row in that table has no answer in your setup, that is where to start.

Tracing a disputed answer, step by step

When the ₹4.2 versus ₹3.9 call comes in, this is the order I would work in.

1. Find the run. Search by user and time, or by the answer text, and get the run ID.
2. Read the tool calls and retrieved chunks. Check whether the model's answer matches what the tools returned. If it doesn't, the problem is in the model or the prompt, and lineage of the data won't help.
3. If it does match, take the dataset name and version from the tool call and find the pipeline run that produced that version.
4. Walk the column-level lineage for the disputed column, paying attention to indirect dependencies: filters, joins and conditionals.
5. Compare with the other number's lineage. Two figures that disagree usually differ in one filter, one join or one date boundary.
6. Fix it at the source, then use declared lineage to list every downstream table, dashboard and agent that needs to be rerun or rechecked.

Step 2 decides whether the model team or the data team owns the problem, so get to it fast. Many arguments about "the AI is hallucinating" end there.

There is a quieter fix too. The ₹4.2 and ₹3.9 numbers came from two definitions of revenue. If the assistant and the board pack read the same KPI definition, built from the real schema and agreed once, the dispute doesn't arise. CloudOEngine's AI reads your actual data schemas to propose KPIs, which gives teams one place to agree that definition before anyone quotes it.

Lineage and the rules on traceability

Regulators rarely say "data lineage", but traceability requirements point the same way. The EU AI Act's Article 12 requires high-risk AI systems to technically allow automatic recording of events over the lifetime of the system, so that situations presenting a risk can be identified and the system's operation monitored. For certain biometric identification systems, it sets minimum log contents, including the reference database the input was checked against and the people who verified the results.

Whether it applies to you depends on what your system does and where; this is not legal advice. Indian teams handling personal data should also read our note on AI governance under the DPDP Act. The practical point holds either way: the logs you build for debugging are much of what an auditor will ask for, so build them once and keep them.

Where to start

Don't start by mapping every table in the company. Start from the answers.

Pick the one AI assistant or agent that quotes numbers people act on. List the tables and documents it can read. For those, and only those, make sure every run records the full tool call arguments and the dataset version, and that the pipelines building those tables emit run-level lineage with column detail. Then take five real answers from last week and try to trace each one back to source rows using only what was recorded. Wherever you get stuck is your next piece of work.

How CloudOEngine helps

CloudOEngine is one platform and one console for connecting data (databases, files and camera streams), running pipelines, and running AI. The AI runtime includes an LLM gateway, agents with tool access, and RAG. Because the data, the pipelines and the assistant that answers from them sit in one place, they share one permission model and one audit trail. For lineage, that means the runtime record of an answer and the record of the data behind it are not scattered across separate tools' logs.

Agents run real tools, for example filing a ticket, and every step and tool call is traced and inspectable. When an answer is disputed, you open its run and see what was retrieved, what was queried and what came back, which is step 2 of the process above.

The AI reads your real data schemas to propose KPIs, so the definitions an assistant quotes can be agreed from the schema rather than reinvented in each query. For actions with consequences, Autopilot runs triggered agents with approval rules: a person approves the action in the app or through an email approve link before it runs.

Governance is included, not upsold: tenant isolation, role-based access, audit logs and per-tenant encrypted secrets. CloudOEngine is built in Bengaluru, India.

Book a demo or see the data layer.

Frequently asked questions

What is data lineage?

Data lineage is the record of where a piece of data came from and what happened to it on the way: which sources it was read from, which jobs transformed it, and which datasets it ended up in. Good lineage also records when each step ran and which version of each dataset it used.

What is the difference between data lineage and data provenance?

The terms overlap and are often used interchangeably. Lineage usually means the flow of data through pipelines, while provenance, as the W3C PROV standard defines it, is broader information about the entities, activities and people involved in producing a piece of data or thing.

What is column-level lineage?

Column-level lineage records which input columns each output column depends on, rather than stopping at tables. The OpenLineage column lineage facet also separates direct dependencies, where the value is derived from the input, from indirect ones such as joins and filters that affect the value without being its source.

Are citations in an AI answer the same as lineage?

No. A citation points to a document the model says it used, but it does not prove the document supports the claim, and it says nothing about how a number in a table was calculated. A 2023 benchmark found even the best models lacked complete citation support about half the time on one dataset.

What should be logged to trace an AI answer?

At minimum: the question and who asked it, the prompt and model version, every retrieved chunk with its document version, every tool call with its exact query and result, and the dataset version or pipeline run behind each table that was read. A shared run ID ties them together.

Is data lineage required by law?

Not under that name in most places, but rules on traceability point the same way. The EU AI Act's Article 12, for example, requires high-risk AI systems to allow automatic logging of events over their lifetime. This is not legal advice; check the rules that apply to you.

Sources

1. OpenLineage documentation (LF AI & Data)
2. OpenLineage object model: runs, jobs and datasets
3. OpenLineage column lineage dataset facet
4. PROV-Overview: An Overview of the PROV Family of Documents (W3C, 2013)
5. PROV-DM: The PROV Data Model (W3C Recommendation, 2013)
6. About ref function (dbt documentation)
7. Enabling Large Language Models to Generate Text with Citations (Gao et al., 2023)
8. EU AI Act, Article 12: Record-keeping

#data lineage #provenance #openlineage #ai governance #traceability