AmouAI Hub/Courses/Software Engineering for AI Engineers/Chapter 11
Lifecycle, Governance and Data an Agent Can Use
Data is collected, kept, corrected, exported and deleted, and every one of those verbs is an obligation somebody has to implement. This chapter is about doing that deliberately — and about the specific way a badly governed corpus fails an AI system, which is not an error message but a fluent, well-cited, out-of-date answer.
By the end of this chapter you can
- Write a retention policy that names an age, a reason and a mechanism for every table
- Implement a deletion that reaches the derived copies as well as the row
- Spot the join that re-identifies data you called anonymous
- Run a schema migration with traffic on the table, in three reversible phases
- Explain why a stale retrieval index is more dangerous than an empty one
- Chunk a document so that retrieval returns something a model can use
1Data has a lifespan, and somebody has to choose it
Most systems have no retention policy. What they have is an accident: everything is kept forever because deleting things is scary and nobody was asked.
“Keep everything” feels like the safe default and it is not one. Every row you hold is a row you must secure, back up, restore, migrate, include in a subject access request, and disclose in a breach. Storage is the cheapest part of holding data and it is the only part anyone budgets for.
A retention policy is three columns wide, and the third is the one that makes it real:
| Data | Kept for, and why | The mechanism |
|---|---|---|
| Invoices and customers | 7 years — statutory accounting retention | Never deleted by the application; archived to cold storage after 2 years |
| Webhook deliveries | 30 days — long enough to debug an integration | Nightly job deletes older rows; index size bounded as a side effect |
| Session tokens | 14 days — the sign-in window | A TTL in the store; expiry is the store’s job, not a job’s |
| Application logs | 90 days — the realistic investigation window | Log platform lifecycle rule |
| Audit trail | Indefinite — it is the record of who did what | Append-only, never updated, archived not deleted |
| Invoice embeddings | As long as the invoice — derived data | Deleted by the same cascade that deletes the invoice |
Read the last row again, because it is the one that gets missed and it is the reason this chapter exists in a course about AI systems. Derived data inherits the lifecycle of its source. An embedding of a deleted invoice is not a harmless leftover: it is a copy of the content, held past its retention, reachable by search, and invisible to every process that thinks the invoice is gone.
The mechanism column also has a rule of its own: a policy with no mechanism is a document, not a policy. “We delete webhook logs after 30 days” is true only if something runs. The honest version of a retention policy names the job, and the job has monitoring, because a deletion job that silently stopped six months ago looks exactly like one that is working.
2Deletion that actually deletes
“Delete” in most codebases means setting a flag. That is a legitimate design and it is not deletion, and the gap between the two is where the incidents live.
A soft delete — deleted_at, or is_deleted — keeps the row and hides
it. It is genuinely useful: it makes undo possible, preserves referential integrity, and keeps historical
reports intact. It also means the data is still there, still in every backup, still in the search index if
the index was built from a query that forgot the flag, and still returned by the one endpoint whose author
did not know the convention.
The question to ask of any delete is not “did we delete it?” but “where are the copies?” A single invoice in a mature system is typically in six places:
| Copy | What removes it | How long it can survive |
|---|---|---|
| The row itself | The delete statement | Immediate |
| Rows referencing it | ON DELETE CASCADE, or explicit deletes in the same transaction | Immediate, if you wrote it |
| The search or vector index | An explicit index deletion, in the same transaction if the index is in the same store | Forever, if nobody wrote that step |
| Caches | Invalidation, or a TTL | The TTL — which is why an unbounded cache is a retention problem |
| Analytics extracts and warehouses | A propagated deletion, usually a separate pipeline | Often forever; this is the most commonly missed one |
| Backups | Backup expiry | The backup horizon, typically 30–90 days |
Backups are the honest exception and worth stating plainly rather than pretending otherwise. You cannot surgically remove one person’s rows from a point-in-time backup without destroying the backup’s purpose, and regulators broadly accept this. What is expected is that you can say how long a deleted row persists in backups, that the period is bounded and documented, and that a restored backup goes through the deletion queue before it serves traffic. That last clause is the part teams forget, and it is what turns a restore into a re-creation of data somebody asked you to erase.
3PII: minimise, isolate, and the join that re-identifies
The cheapest way to protect personal data is not to have it. The second cheapest is to keep it in one place. Everything after that is expensive.
Minimise. Every personal field should have a named use. A date of birth collected because the form template had one is a liability with no offsetting benefit — and “we might want it for analytics later” is not a named use, it is a wish. The question that settles most of these is: if this field leaked tomorrow, what would we say it was for?
Isolate. Personal data in one table, referenced by id from everywhere else, gives you one place to encrypt, one place to audit access, and one row to delete. Personal data denormalised across forty tables because a join felt slow gives you forty places to miss.
And then the part that catches careful teams. Anonymisation is not a property of a column; it is a property of a dataset. Remove the name and the email from a table and you have not anonymised it if what remains — a postcode, a date of birth, a job title, an employer — identifies one person by combination. This is not theoretical: it is the finding that a small number of quasi-identifiers uniquely picks out most individuals in a population, and it is why “we stripped the PII columns” is a claim that needs checking rather than a conclusion.
A practical consequence for AI systems: the same logic applies to a retrieval corpus, and it is easier to
get wrong there because the content is unstructured. A chunk of an invoice that mentions a contact by name
is personal data even though no column is called name, and it is subject to the same erasure
obligation as the row it came from. This is another reason to store the source id beside every vector: it
is the only thing that makes “delete everything derived from this record” a query rather than
a search.
4Expand, migrate, contract
A schema change on a live system is not one deploy. It is three, and the discipline is that every one of them is safe to stop at.
The failure this avoids is the one everyone has seen: a deploy that changes the schema and the code at the same time, so that for the ninety seconds it takes to roll out, some instances are writing the old shape and some are reading the new one. The fix is to make the schema and the code compatible in both directions until they no longer need to be.
Two properties of that sequence are worth naming, because they generalise well beyond renaming a column.
The first is that the backfill is idempotent. WHERE contact_email IS NULL means running
it twice does nothing the second time, which means it can be interrupted, resumed, and re-run by a nervous
engineer at 2 am without consequence. A backfill that is only correct if it runs exactly once is a
backfill you cannot safely retry, and every long-running job gets interrupted eventually.
The second is that only the last step destroys anything. Steps one through six can each be rolled back by deploying the previous release. That is what makes the sequence usable under pressure: at every point before the contract, the answer to “something looks wrong, what do we do?” is “roll back”, and it works.
5Freshness, and the confident wrong answer
A conventional system that has lost touch with its data returns an error. A retrieval system that has lost touch with its data returns a paragraph.
This asymmetry is the single most important thing in the chapter. If your cache is stale, a user sees an old number and often notices. If your retrieval index is stale, a model reads an old document, reasons correctly about it, and produces a fluent, specific, well-cited answer that is wrong — and the citation is what makes it convincing. There is no exception thrown, no 500, no alert. The system is working exactly as designed.
Which leads to a claim worth arguing about: a stale index is more dangerous than a missing one. With no index the system says it cannot find anything, and the user goes and looks. With a stale index the system answers, and the user does not.
Three mechanisms, in increasing order of effort, and most systems should have all three:
| Mechanism | What it does | What it costs |
|---|---|---|
| Re-embed on write | The chunk is regenerated in the same job that changed the source. The index is never more than a queue-depth behind. | An embedding call on the write path, usually asynchronous. The main cost is remembering to do it on every write path, including bulk imports and admin edits |
| A content hash per chunk | Turns “what is stale?” into a join, so a nightly sweep can catch whatever the write path missed | One column, and the discipline of computing it from the same text you embedded |
| A generated-at timestamp, used at query time | Lets retrieval refuse. A chunk older than a threshold can be excluded, or returned with an explicit warning the answer layer must handle | One column, and a product decision about what to do when there is nothing fresh enough — which is the hard part |
The third one is where the design work is, because it forces a question the other two let you avoid: what should the system say when the only relevant document is eight months old? “I found something but it may be out of date, here it is with its date” is almost always the right answer, and it is only available if the date was stored. A system that cannot distinguish fresh from stale has no way to be honest, whatever its prompt says.
6Build: making Ledger’s data retrieval-ready
Everything above converges on one question: what has to be true of a dataset before an agent can safely act on it?
Four things, and the first is the one that gets the least attention:
It has to be chunked along its own structure. A fixed-size split at 800 characters cuts sentences in half, separates a table from its heading, and puts the amount in one chunk and the invoice number in another — after which no retrieval quality work can recover the connection. Splitting on the document’s own boundaries (sections, line items, paragraphs), with a small overlap so a sentence that straddles a boundary appears in both, produces chunks that are individually meaningful. The rule of thumb: a chunk should make sense read alone, because that is exactly how the model will read it.
The other three follow from earlier sections and are worth stating as a checklist rather than prose:
- Every chunk carries its provenance — source id, model version, content hash, generated-at. Without the source id, deletion is a search rather than a query; without the rest, staleness is undetectable and migration is unestimatable.
- Deletion propagates — removing an invoice removes its chunks, in the same transaction if they live in the same store. This is the single strongest argument for keeping them there.
- Facts come from the source row — retrieval finds candidates, a query supplies numbers. Chapter 10 argued this from correctness; this chapter adds the governance argument, which is that a chunk is a copy and a copy has its own lifecycle.
Notice how much of “making data agent-ready” turns out to be ordinary data engineering done properly. That is the finding, not a disappointment. The AI-specific part — chunking, embedding, similarity — is a thin layer over a foundation of retention, provenance, deletion and migration discipline, and a team that has never had that foundation does not get to skip it because the new layer is more interesting.
✓Checkpoint
▶Playground
Invoices, their embeddings, and a deletion queue. Try deleting a customer and then checking every place their data could still be.
✓Exercise set
Twelve problems. The SQL ones are graded partly on being re-runnable — the tests execute several of your statements a second time and require that nothing changes, because a migration you cannot safely retry is a migration you cannot safely run. Your work is saved in this browser.
Chapter 12 — Requirements Into Numbers
Part 3 begins. “It should be fast” is not a requirement; 200 ms at the 99th percentile under 3,000 requests a second is. Next chapter you turn vague asks into figures you can design against.