Can You Reconstruct an LLM Run Later? A Practical Provenance Checklist
An answer by itself rarely explains how an LLM reached it. Learn what to record for investigation or comparison—and how to make missing provenance and data-handling choices visible.
Why an answer alone is not enough
A saved response tells you what the system returned, but not necessarily the context that shaped it. The same response text can be interpreted differently depending on the system prompt, retrieved documents, model version and generation parameters. The supplied thesis on tool-mediated systems notes that preserving response text alone may not be enough for later interpretation or investigation ([Tool-mediated systems also complicate attribution and reconstruction, Politecnico di Torino](https://webthesis.biblio.polito.it/41347/1/tesi.pdf)).
For an important response, consider a record that distinguishes input material, system constraints, user instructions, retrieved material, tool-assisted transformations, model output, generated interpretation, uncertainty and post-processing. This list follows the proposed record elements in FIELD COMPUTE ATLAS; it is a design checklist, not a guarantee that every system can preserve every element ([FIELD COMPUTE ATLAS, version 2.1](https://zenodo.org/records/23014733/files/FIELD_COMPUTE_ATLAS_v2_1_FULL_INTEGRATED_20260926.pdf?download=1)).
- Can you identify the workflow that produced the response?
- Can you distinguish the model’s output from changes made by your application afterward?
- Which parts of the invocation context are present, and which are missing?
Choose the goal: replay, reconstruction or comparison
Decide what you want a record to support before choosing what to retain. Replay, diagnosis and comparison are different goals, and a record that helps explain an earlier run does not necessarily make it possible to reproduce the exact output.
The precise internal model state may be unavailable, as FIELD COMPUTE ATLAS notes. Be clear about that limitation when describing what a record can support ([FIELD COMPUTE ATLAS, version 2.1](https://zenodo.org/records/23014733/files/FIELD_COMPUTE_ATLAS_v2_1_FULL_INTEGRATED_20260926.pdf?download=1)).
- For replay, retain the inputs and configuration you need and are permitted to keep; state any dependencies you cannot restore.
- For diagnosis, prioritize the context and transformations needed to interpret what happened.
- For comparison, identify what changed between the runs and what could not be verified.
- Do not promise exact replay when relevant inputs or model state are unavailable.
Record prompt and application context
A template name alone may not show the instructions and inputs used for a particular invocation. Where your system permits, preserve the assembled prompt or the relevant parts of it, together with a prompt or application revision identifier. The supplied thesis recommends preserving invocation context and provenance alongside the response artifact ([Tool-mediated systems also complicate attribution and reconstruction, Politecnico di Torino](https://webthesis.biblio.polito.it/41347/1/tesi.pdf)).
Make the limits of the record explicit. If a field or dependency was not retained, mark it as unavailable rather than implying that the record is complete.
- Keep system constraints, user instructions and relevant variable values distinguishable where your system represents them separately.
- Record the prompt or application revision when available, and retain the corresponding revision only when your governance rules allow it.
- Note material pre-processing or post-processing that may affect how the response is interpreted.
Capture model, retrieval and tool provenance
Record the model and request details that your integration can observe, and distinguish observed values from assumptions. Provider interfaces differ, so consult the documentation for the specific service rather than relying on a generic checklist to define its available fields.
In retrieval-augmented or tool-using workflows, a reference to a source may not show the exact material or result used. Where permitted, preserve relevant retrieved passages, tool inputs and results, and transformations between them. FIELD COMPUTE ATLAS proposes distinguishing retrieved material, tool-assisted transformations and generated interpretation in a run record ([FIELD COMPUTE ATLAS, version 2.1](https://zenodo.org/records/23014733/files/FIELD_COMPUTE_ATLAS_v2_1_FULL_INTEGRATED_20260926.pdf?download=1)).
- Record model and request details that are available to your system; note when a value is unavailable.
- For retrieved material, retain useful identifiers or the relevant content when permitted, and state if the record does not preserve what was supplied to the model.
- For tool activity, distinguish the tool’s input and result from the model’s interpretation of them.
- Identify missing provenance rather than implying that source links or a tool name are a complete record.
Use structured records carefully
When recording prompts, retrieved documents, tool outputs or responses, structured serialization can make field boundaries explicit. The supplied thesis also discusses canonicalization, escaping untrusted content and hashing over a canonical representation as logging mechanisms ([Tool-mediated systems also complicate attribution and reconstruction, Politecnico di Torino](https://webthesis.biblio.polito.it/41347/1/tesi.pdf)).
A hash can help check whether a retained artifact matches a prior record; it cannot recover content that was discarded. If you use hashes, document the representation being checked and do not present the hash as a substitute for the underlying content.
- Define clear boundaries between recorded fields.
- If using canonicalization or hashing, document the representation and escaping choices.
- State whether the underlying content is retained, transformed or unavailable.
Set access and retention rules
Prompts, retrieved passages, tool results and responses may include information that should not be broadly accessible. Decide what to collect, who may access it and how long to keep it as part of your data-governance choices. The appropriate policy depends on your context; this checklist does not prescribe a particular retention period.
Consider whether sensitive fields should be excluded, redacted or stored separately. If content is omitted or transformed, make that visible so readers of the record understand what it can and cannot support.
- Classify trace fields according to your investigation needs and data rules.
- Limit access according to your organization’s governance choices.
- Document redaction, retention and deletion decisions.
- State what was retained, transformed, redacted or unavailable.
Frequently asked questions
Can an LLM response be reproduced exactly later?
Not necessarily. Exact reproduction depends on whether the relevant inputs and context are available and whether the model service exposes or preserves the state needed to repeat the request. FIELD COMPUTE ATLAS notes that precise internal model state may be unavailable. A record can still support investigation without guaranteeing identical output.
What is useful to keep for investigating an LLM run?
Consider retaining the invocation context, relevant model and application details, retrieved material, tool activity and output where permitted. Also state what is missing. The proposed record elements in FIELD COMPUTE ATLAS include input material, system constraints, user instructions, retrieved material, tool-assisted transformations, model output, uncertainty and post-processing.
Is a trace ID enough to investigate a response?
No. A run or trace reference can help organize related records, but it is not a substitute for the prompt, retrieved material, tool results or output. The supplied checklist calls for trace provenance as well as disclosure when provenance is absent.
Should teams keep full prompts and retrieved documents?
That depends on the investigation need and your data rules. Retaining content may provide more context, but you should decide what to collect, who can access it and how long to keep it. State clearly when content was omitted or transformed.
Do hashes make discarded prompt or context content reproducible?
No. A hash over a canonical representation can help check whether a retained artifact matches a prior record, but it cannot recover content that was not kept.
Sources and further reading
- FIELD COMPUTE ATLAS, version 2.1 — Zenodo
- Tool-mediated systems also complicate attribution and reconstruction — Politecnico di Torino thesis repository
- The design cannot distinguish these explanations — Zenodo