Choosing what to keep in an AI execution trace

What a debugging session actually uses

The questions on-call engineers ask first are structural: why did the agent call that tool, why did the run take forty seconds, why did it cost three times the usual amount, where did it fail. The span tree answers most of them, with operation names, requested and returned model, token usage, tool names, error types and durations, plus an identifier that links the run to the business case. OpenTelemetry's generative AI semantic conventions, still marked as in development, define attributes for most of these. Full content is needed for narrower jobs: reproducing a wrong answer, judging output quality and investigating a suspected prompt injection.

Three tiers of trace data

1. 1

   Tier 1: operational metadata

   Span structure, model names, token counts, tool names, status, timings and a pseudonymous user reference. No message content. Kept under your normal operational log retention.

2. 2

   Tier 2: content captured by decision

   Prompts, system instructions, outputs, tool arguments and results, retrieved passages. Captured only for a defined sample, flagged runs or a time-boxed investigation, stored apart from tier 1 with narrower access and a shorter period set by your policy owners.

3. 3

   Tier 3: never stored

   Access tokens, API keys, passwords, connection strings and raw session identifiers. The OWASP logging cheat sheet lists these among data that should usually be removed, masked, hashed or encrypted rather than logged directly.

Keep content out of traces by default

The OpenTelemetry generative AI conventions treat model instructions, user messages and outputs as sensitive. They say instrumentations should not capture them by default but should offer an opt-in, and for production they recommend storing content externally and recording only references on spans, so that content can have its own access controls. Libraries outside OpenTelemetry may behave differently, and OpenTelemetry's own guidance on sensitive data makes reviewing what your instrumentation emits your responsibility. Check each library's actual output in a test run instead of trusting its documentation alone.

Redaction as a second line

The OpenTelemetry Collector can delete, hash or rewrite attributes, drop whole spans, and with the redaction processor remove attributes that are not on an allowlist. Use an allowlist for tier 1 so that a new attribute does not slip through unnoticed. Do not rely on pattern-based redaction of free text: personal data in prompts appears in forms no pattern anticipates. OpenTelemetry also warns that hashing a small or predictable value, such as a numeric user ID, may not anonymise it, because the hash can be reversed by trying every possible input.

Who sets the retention period

How long each tier is kept is a decision for your data protection, legal and security owners, not an engineering default, and this article is not legal advice. The OWASP logging cheat sheet puts the principle plainly: log data should not be destroyed before the required retention period, nor kept beyond it. Engineering makes the decision enforceable:

• Separate storage per tier with expiry configured in the store itself
• Deletion that also reaches exports, copies in evaluation sets and backups
• An access log for tier 2, reviewed by someone outside the team that reads it
• A register of what each service captures, updated when instrumentation changes

If traces can show individual employees' work, bring in whoever handles employee data questions early.

A hypothetical debugging session under these rules

An AI support agent quotes a customer the wrong refund amount. On-call opens the trace by case ID. Tier 1 shows the agent called the order lookup, then the refund calculation tool, which succeeded quickly with normal token counts, so the fault is in the inputs or the logic. Tool arguments are tier 2 and were not captured. The engineer requests a time-boxed capture for that one tool, the data owner approves it, and a run with a synthetic order in staging shows the amount passed without its currency. After the fix, capture is switched off and the captured content expires on the agreed date.

AI engineering and architecture