"We don't train on your data" is a true answer to a question that does not protect you. Interrogate where the data travels and where it rests, not what the model learns.
A researcher recently ran a small, cruel experiment on an AI coding tool. As reported, they pointed Grok Build v0.2.93 at a repository with one instruction: reply OK, and open no files. At the model layer the agent obeyed — the traffic suggested it touched nothing. Meanwhile the product wrapped around it uploaded a Git bundle of the full tracked repository and its entire commit history, including a canary file the model never opened. Even if not every detail survives scrutiny, the mechanism is the alarming part.
I run security and platform teams in a regulated fintech, and I sit in a lot of AI vendor reviews. Almost every one opens with the same reassurance, delivered like a closing argument: we don't train on your data. It is usually true. It is also an answer to a question that, on its own, does not protect you — and in the Grok Build account it would have been perfectly true while the entire codebase, canary and all, walked out the door.
Here is the reframe I want to make stick. Data governance has three legs: what a system trains on, how the data travels, and where the data rests — and who can read it there. "We don't train on your data" answers the first leg only. It is also the leg least likely to hurt you first, which is exactly why vendors volunteer it. When someone leads with the answer they can most cleanly clear, that is a tell, not a comfort.
The pledge is true. It just answers one leg of three.
Give the assurance its due; the strong version is genuinely strong. Microsoft's Azure OpenAI commitments state that your prompts, your training files, your outputs, and any model you fine-tune are not used to improve the foundation model or shared with other customers without permission, and that a fine-tuned model is exclusive to you, encrypted at rest, and deletable on demand. That is precise, testable, and contractual — a clean answer to the training question.
It is also doing an enormous amount of load-bearing work right now, because it is the sentence that makes leaders comfortable pushing regulated data in. Enterprises are racing to fine-tune small models on sensitive material inside a "controlled" cloud boundary, and the cited results are real. Microsoft's own customer stories describe Bayer fine-tuning a Phi model on proprietary crop-protection label data so expert questions that took days now resolve in under thirty seconds, and Discovery Bank fine-tuning five variants across Azure OpenAI's 4o-mini and 4.1-mini to cut average response time from five or six seconds to under two. These are exactly the workloads where the most sensitive data goes in — and where "we don't train on it" is the phrase quieting the room.
The model obeyed. The product still shipped the repo.
The lesson underneath that account is that the model turn and the data path are two different systems, and you can win one while losing the other. An agent can honor an instruction perfectly at the model layer — open no files, touch nothing — while the product around it moves your bytes for reasons of its own: telemetry, context bundling, a background upload to "improve your experience." The model was obedient. The plumbing was never asked. A vendor can truthfully swear it never trained on that repository while its product copies the repository into a bundle or a log.
If that shape feels familiar, it should. This is the exfiltration class one layer up from injection. When Varonis disclosed SearchLeak — CVE-2026-42824, a one-click leak in Microsoft 365 Copilot, on June 15, 2026 — nothing broke: the data walked out through trusted, allowlisted infrastructure turned into a courier, because the egress was already permitted. The Grok Build story is the same silhouette with a different courier, the product's own bundling-and-upload path. Your perimeter can hold perfectly and still leak, because the thing carrying the data out is a service you approved.
And then there is the canary. A canary file is a plant that exists to never move; if it turns up somewhere it should not, you know a transmission path is live. In the account, it moved. That is precisely the primitive we use to test whether a boundary leaks, and almost nobody has thought to point one at their AI vendor's product rather than at their own network. The model is not the thing you most need to instrument. The product is.
Interrogate the transmission and storage paths, not the training claim
So retire "do you train on our data" as a standalone checkbox and put the weight on the two legs the pledge skips. A control-plane vendor review should force three answers:
- Where does the payload travel? Bundle, log, telemetry, context cache, a secondary service, a sub-processor. Enumerate every hop your bytes take before the model sees them and after it responds. The Grok Build bundle lived on one of those hops.
- Where does it rest, and for how long? Retention is where a transient prompt becomes a durable record with residency, deletion, and DLP obligations attached. "We don't keep it" is a claim you can hold a retention window and a deletion API against, or it is marketing.
- Who can read it there? Not hypothetical. Under standard Azure abuse monitoring, a sample of flagged prompts and completions can be stored and selected for restricted human review by authorized Microsoft employees — a storage-plus-human-read path that coexists perfectly happily with "we don't train on your data."
That last one is also where the control lever hides. Eligible managed customers can apply for modified abuse monitoring, which turns the human-read path off — a documented, requestable knob that a real control-plane review exists to find and turn. The no-train pledge is a contract term; the abuse-monitoring configuration is a data-flow decision. Only one of them is a diagram of where your data actually goes.
The enforcement mechanism underneath all of this is classification, not trust. Map a Red/Amber/Green router onto your existing fintech data classification: material non-public information, credentials, and privileged legal advice are Red, and Red never crosses a boundary whose transmission and storage paths you cannot audit. Green — public or synthetic material — can go almost anywhere. And self-hosting moves this boundary without removing the review: LM Studio will serve requests from other devices on your local network with authentication off by default the moment a second device connects, a non-human-identity hole about who can read what, not about training. Run the model yourself and you still owe the same three answers.
The correction record is the asset, not the weights
Now the part that actually reframes the vendor relationship. The lock-in was never the model weights. Weights are rented reasoning; you can point at a different set this afternoon. The thing that is genuinely hard to move is the correction record: the accumulated ledger of which exceptions your legal team has accepted, which numbers finance has ruled material, which contract clauses carry risk, plus the permissions, prompts, and evaluations you built while teaching the system your world. Nobody hands that to you. You rebuild it from scratch, over months, every time.
Here is what a CIO should see: that record is two things at once — a regulated data asset and an audit artifact, your decision-provenance trail wearing a different hat. When an examiner asks how an automated system reached a call, it is a large part of the answer. This is a different object from the one I have written about before, where a well-built pipeline generates its own audit trail as a byproduct of running. This is about a record whose whole value is that it is yours to export, not the vendor's to hold.
Which makes the requirement blunt: the correction record must be exportable and vendor-independent. If it lives only inside a vendor's product, you have concentration risk on your single most valuable AI asset, and you have handed a regulator's question — prove you control this decision system — to a company whose answer is a support ticket. Under the US Treasury's Financial Services AI Risk Management Framework — 230 control objectives across seven domains published February 19, 2026 — the accepted-exception, material-number, risky-clause record is precisely what you will be asked to produce, and to prove is yours to control. A record you cannot extract is a control you do not have.
The model-independence test: swap one workflow, inventory the rebuild
All of this stays theoretical until you make it concrete, so here is the test I would run this quarter. Pick one recurring, sensitive workflow. Demand a live model swap on it — not a slide, an actual run of that workflow on a second model behind your own gateway — and then inventory everything the team has to rebuild to make the swap work: the memory, the review rules, the integrations, the tuned behavior, and the correction record itself.
That inventory is the number. The length and cost of the rebuild list is your portability and concentration-risk exposure expressed as work, not vibes — a figure a CIO reports to the board like any single-vendor dependency. I have argued that model selection is capacity planning, routing work across models by risk tier. This is the other half of that discipline: not selection but extraction. Own the harness, rent the model, and prove the rent clause works by exercising it before you are forced to.
Because you will be forced to eventually. The test doubles as a continuity rehearsal: when a model is pulled on a regulator's timeline — as two US-lab flagships were in June, suspended three days after launch under an export-control directive — or a vendor simply changes its terms, the rebuild inventory you already produced is your recovery runbook. The portability metric and the disaster-recovery plan turn out to be the same document.
One discipline makes the whole thing auditable: pin what you can hold. Log the stable model id, claude-opus-4-8, not a floating alias like chat-latest that reassigns under you without a changelog. Pin the version and schedule the re-validation. Then "swap the model" is a config change instead of an archaeology project, and every entry in your correction record stays attributable to a specific version an examiner can name.
Rewrite the questionnaire, then price the rebuild
Rewrite the questionnaire. Demote the training checkbox and add the questions that carry the risk: where does the payload travel, where does it rest and for how long, who can read it, which sub-processors touch it, and can we get modified abuse monitoring in writing. Then require a canary test of the transmission path before any Red data touches a product — point the plant at the vendor, not just at your own network.
Classify and route: Red stays off any hosted path whose transmission and storage you cannot audit. Own the record — export it on a schedule and hold it as regulated audit evidence under your own retention and provenance, mapped once to the Treasury and NIST controls. Measure the lock-in: run one model-independence test this quarter and report the rebuild inventory to the board as concentration risk, with a named owner and a date. And close the auth-off-by-default hole on any self-hosted inference.
The pledge is true and nearly beside the point. Ask where the data travels, where it rests, and who can read it. Keep the correction record in your own hands. And price the rebuild yourself, before a vendor's roadmap or a regulator's directive prices it for you.
I would like to hear from anyone who has actually pointed a canary at their AI vendor's product path — not the model, the product — and watched what came back. Tell me in the comments.
