AmouAI Hub/Courses/Software Engineering for AI Engineers/Chapter 19
Security, Shifted Left
You do not need to become a security specialist. You do need to be the person who notices, at the moment the code is written, that this endpoint takes an id from the user and never checks who owns it — because that particular defect is invisible to every scanner you can buy, and it is the one that keeps happening.
By the end of this chapter you can
- Threat model a flow in fifteen minutes with four questions
- Parameterise every query, including the ones that look like exceptions
- Write an ownership check that fails closed
- Validate a user-supplied URL against private ranges and redirects
- Explain why broken access control is the class scanners miss
- Keep secrets out of logs, build output and error messages
1You are partly a security engineer now
Not because anyone appointed you, but because the decisions that matter are made in the code and cannot be inspected back in afterwards.
A security team can run scanners, set policy, and review a design. What they cannot do is notice that the
getInvoice function takes an id and returns whatever it finds — because that code
looks exactly like correct code, and the missing thing is a line that was never written. The people who
can catch it are the person who wrote it and the person who reviewed it.
The good news is that the vast majority of real vulnerabilities are a small number of shapes, and knowing those shapes is a few hours of study rather than a career. You are not being asked to find novel classes of attack; you are being asked to recognise five or six patterns when they appear in a diff.
The framing that makes this manageable: every input from outside your system is hostile until it has been through something that makes it safe — and “outside” includes another team’s service, a webhook payload, a filename, a URL you were asked to fetch, and text a model produced from content you did not write.
Ask two questions of every handler: who is allowed to do this, and where does this value come from. Almost every vulnerability in this chapter is one of those two questions going unasked.
2Threat modelling in fifteen minutes
Threat modelling has a reputation for being a two-day workshop. The version that fits in a design review is four questions, and it catches most of what the two-day version catches.
- What are we building? Draw the flow. Boxes and arrows, five minutes, on a whiteboard.
- What can go wrong? Walk each arrow and ask the STRIDE questions — can someone pretend to be someone else, change data in flight, deny having done it, read what they should not, take the system down, or gain more privilege than they had?
- What are we going to do about it? Per finding: mitigate, accept with a reason, or transfer.
- Did we do it? The step that gets skipped, and the one that makes the exercise real.
Two practical notes. Do it on the flow rather than the system — “a user exports their invoices” is a tractable subject and “Ledger” is not. And write the findings down where the code is, because a threat model in a document nobody opens has the same fate as the diagram in Chapter 16.
3The surface that keeps appearing
Four classes account for most of what actually happens to ordinary applications. None is exotic and all four are cheap to prevent at the moment the code is written.
| Class | The shape | The fix, stated once |
|---|---|---|
| Injection | User data is concatenated into something that gets interpreted — SQL, a shell command, a template, HTML | Never build the instruction from the data. Parameterise, escape on output, and pass arguments as an array rather than a string |
| Broken access control | The system checks who you are and forgets to check what you may touch | Scope every query by the actor, in one enforced place. Section four is entirely about this |
| SSRF | Your server fetches a URL the user chose, from inside your network | Resolve, validate against private ranges, allow only expected schemes and ports, re-check on redirect |
| Secrets | A credential reaches somewhere it can be read — a log, a build artefact, an error page, a repository | Never in code; redact at the logging boundary; rotate. Section five |
Injection deserves one clarification because the standard advice gets misapplied. The fix is not
“escape the input” — it is never construct the instruction out of the data. A
parameterised query sends the SQL and the values over separate channels, so no value can become syntax;
an escaping function tries to neutralise syntax inside a single channel, and it is a rule with exceptions
you will eventually meet. The same distinction explains why execFile(cmd, [args]) is safe
where exec(cmd + ' ' + arg) is not.
And the awkward cases are where this breaks down in practice. You cannot parameterise a column name in an
ORDER BY, which is exactly why somebody concatenates it. The answer is an allowlist —
map the user’s sort=due_date to a known column through a lookup table, and reject
anything not in it — and the reason this matters is that the awkward case is where the vulnerability
always is.
4Why broken access control is the one scanners miss
A scanner can find a SQL string built by concatenation, because that is a syntactic pattern. It cannot find a missing ownership check, because the correct code and the vulnerable code look identical.
Consider two handlers. One returns an invoice by id. The other returns an invoice by id after checking that it belongs to the caller’s organisation. The difference is one predicate, and no tool that has not been told your authorisation model can tell which one is right — because for a public endpoint, the first one is right.
This class has a name — insecure direct object reference, or IDOR — and it is consistently near the top of every vulnerability survey, for a reason worth internalising: it is a missing thing, and static analysis is much better at finding present things than absent ones.
Three defences, in increasing order of durability:
| Defence | What it does | Why it is not enough on its own |
|---|---|---|
| Check in the handler | if (invoice.orgId !== session.orgId) throw | Correct where it appears, and it has to appear in every handler, forever, including the one written next Tuesday |
| Scope in the repository | Every query goes through a method that takes the actor and adds the predicate | Much better — the check cannot be forgotten because it is not a step. It can still be bypassed by writing a raw query |
| Enforce in the database | Row-level security, so the predicate is applied by the engine | The strongest, and it costs operational complexity and a per-request session variable |
For most teams the repository layer is the right answer: it removes the possibility of forgetting, it is testable against a database containing two organisations, and it puts the security-relevant code in one file somebody can review. What makes it work is the discipline that nothing queries the tables directly — which is a rule the build can enforce, exactly as in Chapter 13.
Two more properties are worth naming because they are cheap and frequently missed. Fail closed: if
the actor is missing, unknown, or the check throws, the answer is no — a check written as
if (user && user.orgId !== target) throw passes silently when user is
undefined, which is precisely the case an attacker will produce. And return the same answer for
“does not exist” and “not yours”: a 404 for both, because a 403 tells the
attacker the id was real, which is how enumeration works.
5Secrets: where they end up, and how to stop it
Secrets rarely leak from the place they are stored. They leak from the six places they are copied to on the way.
| Where it ends up | How | The control |
|---|---|---|
| Version control | A key committed “temporarily” | A pre-commit scanner, and treat any committed secret as burned — rotate, do not just remove the commit |
| Logs | Logging the whole request object, headers included | Redact at the logging boundary, by key name, so nobody has to remember at each call site |
| Error pages and stack traces | A connection string in an exception message | Generic errors to the client; details to the log, redacted |
| Build output | A CI step echoing its environment | Masked variables, and no set -x in a step that touches secrets |
| Client bundles | An environment variable inlined at build time | Know which prefix your framework exposes to the browser, and check the bundle |
| A model prompt | A key in context that gets echoed back | Redact before the call. Anything in a prompt may appear in an output |
The redaction one is worth doing properly because it is the highest-volume leak. Redacting at each call
site fails the first time somebody logs an object they did not construct; redacting in the logger, by key
name, against a list including password, token, secret,
authorization, api_key and cookie, catches the ones nobody thought
about. And it should preserve the shape — replace the value, keep the key — because a
log line that silently drops fields is a debugging problem you have traded for a security one.
One habit worth more than any tool: treat a leaked secret as burned. Removing the commit does not help, because the repository was cloned, the CI cached it, and the value was in a log. The only response that works is rotation, and a team that can rotate a credential in ten minutes handles this as an inconvenience rather than an incident.
6Build: threat model one Ledger flow
Fifteen minutes on the export flow, written out, so the shape is concrete.
The flow. A user clicks Export. The API queues a job. A worker reads the organisation’s invoices, writes a CSV to object storage, and POSTs a notification to the integrator’s webhook URL. The user downloads the file from a signed URL.
| Boundary | The threat that matters | The control | Verified by |
|---|---|---|---|
| Browser → API | A request supplying its own orgId | Tenant comes from the session, never the request | A test asserting a request with a foreign orgId returns the caller’s data, not the foreign one |
| API → queue | Unbounded exports as a denial of service | Rate limit plus one running export per organisation | A test that a second export returns the running one |
| Worker → database | The tenant predicate missing on the export query | Repository scoping, enforced by the build | An integration test with two organisations in the database |
| Worker → object storage | A predictable object key readable by anyone | Random key, private bucket, signed URL with a one-hour expiry | A test that the unsigned URL returns 403 |
| Worker → webhook | SSRF to an internal address | Resolve, reject private ranges, https on 443, re-check on redirect, no response body stored | Tests for a private address, a hostname resolving to one, and a redirect to one |
| Signed URL → user | The link shared or leaked after the fact | Short expiry, one-time use, audit entry on access | A test that a second fetch of the same URL fails |
The last column is what makes this a piece of engineering rather than a document. Every control has a test named for the threat it prevents — which is Chapter 17’s point applied to security, and Chapter 16’s point applied to a decision nobody should have to rediscover.
Notice also how much of this table is the same two questions from section one. Four of the six rows are “who is allowed to do this”; one is “where did this value come from”; and one is rate limiting. That distribution is typical, and it is why access control deserves more of your attention than the exotic categories.
✓Checkpoint
▶Playground
A second flow — the AI drafting path, where the untrusted input is text rather than a parameter. Walk the arrows and notice that the categories are the same ones.
✓Exercise set
Twelve problems, most of them building the specific controls this chapter argues for. The tests are adversarial: they send the inputs an attacker would send, not the ones a user would. Your work is saved in this browser.
Chapter 20 — The AI Supply Chain
You execute far more third-party code than your own, and an agent now chooses some of it. Next chapter closes Part 4: package hallucination, reviewing what was added rather than what was written, and prompt injection when the agent holds real tools.