The data and system access an AI project needs before it starts

What a process project actually needs

Teams often postpone AI work until a data lake or a data catalogue is finished. For an agent or an assistant built on an existing model, that is rarely the constraint. The model is already trained. The project needs the records one process touches, a sanctioned way to read them and cases that show what a correct result looks like. The AWS Generative AI Lens names fragmented data, inconsistent governance and unclear ownership as what slows generative AI experiments, and all three are questions of people and access more than of volume.

A start checklist with a name on every line

1. 1

   A data owner for each source

   A person who can approve that this data may be used for this purpose, in which environment and for how long. Their approval is what your data protection officer will ask about.

2. 2

   A service account with read access

   Access through an API with the narrowest scope the task needs. Write access comes later, once results have been reviewed.

3. 3

   Representative past cases

   Closed tickets, booked invoices or corrected extractions where the right result is recorded. Anthropic's evaluation guidance suggests drawing early test tasks from real failures in bug trackers and support queues.

4. 4

   An expert who can judge results

   Someone from the process with time set aside each week to resolve unclear labels and read outputs. Without that time, nobody can confirm that a result is right.

5. 5

   An approved place to work

   An environment where the test data may be stored and model calls may be made, with the data flows your security team has signed off.

6. 6

   A short description of each dataset

   Where the records came from, which period they cover and what is known to be missing. Gebru and colleagues proposed datasheets for datasets for this purpose, and half a page per source is enough for a first project.

What makes a case set representative

A random export of last month's cases is a start, but it will contain mostly routine work and almost none of the cases that cause trouble. The AWS lens recommends stratified sampling: divide the cases into relevant groups and sample from each. In a hypothetical set of 300 service requests, that could mean 180 routine requests, 75 exceptions such as missing documents or disputed amounts, and 45 rare types that are deliberately over-represented. The proportions are illustrative and should follow your own case mix. Report results per group. Because rare types are over-represented, the overall score of this set is not your production error rate, and estimates for production should weight each group by its real share.

Check the outcomes as well as the inputs. A recorded outcome can be wrong, for example when a ticket was closed to meet a target. Have the expert review a sample of labels before they become the standard the system is measured against.

When production data may not be used

Some organisations do not allow models to be tested on production records. The AWS lens covers this case: the ground truth set should then contain data that is functionally equivalent, with mock data and mock endpoints for agent workflows. In practice that means pseudonymised extracts in an approved environment, or synthetic cases that experts write to match real patterns. Whether pseudonymisation is sufficient is a decision for your data protection officer and counsel, and engineering can supply the facts they need: which fields, where they flow and how long they are kept.

Access that will still work in production

Pilots are tempted by shortcuts: a personal API token, a spreadsheet export sent by email, an administrator account borrowed for a week. They work in the first sprint and block the move to production, because nobody can approve them for permanent use. The AWS lens recommends treating a model that reads your data stores like a privileged user and granting it only the minimum access its task requires. Setting that up at the start spends administrator time early instead of on rebuilding the integration later.

Deciding whether to start now

Split the work by what it touches. Scoping, test design and a prototype running on synthetic or mock data can start once the data owners and the expert are named. Anything that reads real records or sends them to a model waits until the service account, the approved environment and the data flow have all been signed off. Dataset descriptions can be written in parallel. If there are no known outcomes and no expert time, do not start the build at all. Start recording outcomes in the current process instead. Our AI engineering work begins with that scoping, and a first conversation needs no access to your systems or data.

Choosing the first workflow to automateWhat an AI pilot must prove