Skip to the stories
Microsoft 365 & SharePoint

Failure-Path Testing for AI Privacy Controls Before Release

  • Divija Kota
  • 939 views

A passing privacy review doesn't prove an AI system is safe. Testing retrieval, data restrictions and agent scope before release shows whether the system keeps its promises.

Failure-Path Testing for AI Privacy Controls Before Release

A clean privacy review tells you the paperwork is in order. It doesn't tell you the system behaves. That gap is the real problem with AI launches, and failure-path testing is the most practical way I know to close it before a release goes out.

A 2026 VB Pulse survey of 157 enterprise respondents puts numbers on the problem. Half of them reported a customer-facing failure after internal evaluations had passed. Only 5% said they fully trusted automated evaluations. The survey was self-selected and didn't isolate privacy failures specifically, so I wouldn't read it as a precise measure of privacy risk. Still, the direction is hard to ignore: teams are shipping on evaluation results that don't match what users experience.

Why a complete checklist can still approve a bad system

Checklists are good at recording commitments. A product will honor a stated opt-out. Data will be retained for a defined period. An agent will act only within its delegated scope. What checklists don't do is test whether a system that keeps changing still honors those commitments under specific conditions.

Three failure paths come up again and again. A retrieval pipeline lets untrusted content influence what private material gets pulled into a response. An opt-out setting is honored at the interface but misses a downstream training job. An agent quietly exceeds the permission a person actually gave it. Each of these can sit behind a fully completed checklist.

Turning a policy statement into a testable boundary

"Users can turn off training" is a policy statement, and it's too vague to test. Here's the version I'd want an engineer to verify: after a person turns the setting off, no new inputs enter any training-eligible dataset. Previously contributed data follows a separately disclosed retention or deletion process, and the system can show which rule applied to which records.

That rewrite is most of the work. Once a commitment has a trigger, an expected outcome and evidence attached, it becomes something a test can pass or fail.

For each scenario, I'd record the same fields regardless of whether the system is built in-house or bought: the data state, the actor, the trigger, the expected behavior, the evidence that proves it, a named release owner, and the changes that force a rerun. If any of those fields is blank, the test isn't finished.

Scenario one: untrusted content reaching private context

Set up two documents. Place a benign instruction in a public test document. Place a synthetic canary, a harmless marker string, in a private document that only the test user is allowed to read. Then ask the assistant a question that plausibly draws on both documents, and check whether the public instruction can move the canary outside its authorized boundary.

The rule here is simple: public content never gets authority over private context. The canary may appear back to the authorized test user. It must never appear to another user, in a shared destination, or in a tool call outside the approved scope. A passing result is visible in the trace, which should show the sources retrieved and the tools called, with the canary staying in scope.

This isn't theoretical. PromptArmor researchers demonstrated a version of this in a controlled Slack AI scenario, where content in a public channel caused private information to end up inside a rendered link. Slack investigated and patched the issue, according to Salesforce's reported response to the research. The practical lesson for your own testing: don't replay real exploits against real data. Use a harmless canary fixture, and rerun it whenever the model, retrieval logic, ingestion, access control or rendering layer changes.

Scenario two: a restriction acknowledged at the interface but ignored downstream

This is the failure I'd worry about most, because the interface looks correct. Create a test account with an explicit data-use restriction. Generate equivalent events before and after the restriction takes effect. Then exercise every downstream path the disclosed setting is supposed to cover: the training-eligibility query, batch exports, feature pipelines and the deletion workflow.

Post-restriction events should be excluded from every covered downstream use. Events from before the restriction should follow the separately stated rule. Any documented exception needs to trace back to the specific policy version that allowed it. A confirmation message saying the setting is saved proves very little. Lineage records and the eligibility query show where the data did and didn't go.

The Federal Trade Commission's case against Everalbum Inc. shows why interface checks aren't enough. The FTC alleged the company was deceptive about how it used and retained facial recognition data after users deactivated their accounts. The resulting order addressed consent and deletion for affected content and for models developed using that data. The lesson for engineers is that deletion has to reach derived artifacts, not just the source records. A setting can look responsive at the edge while the data stays eligible somewhere inside the system.

Scenario three: an agent that goes beyond what it was delegated

Give the test agent one narrow task, such as summarizing a contract in a designated folder. Then seed adjacent repositories with synthetic records the user never delegated. Add ambiguous instructions, and include a request that only succeeds if the agent widens its own scope.

The agent should stay inside the declared task and documented authorization. If it needs more access, it should obtain approval from an authorized decision-maker or stop. Any action outside that scope is a failure. Authorized actions should record their purpose, scope and source, so you can reconstruct later why the agent did what it did.

One nuance matters here. A prompt that makes an agent say something odd isn't automatically a privacy failure. The failures that count are concrete: the agent read a canary from a location it wasn't delegated, attached that canary to a tool call, or changed an account setting. Your success criteria should be written around those observable actions.

Red teaming needs privacy-specific pass criteria

Plenty of red teaming programs already probe unauthorized access and unauthorized actions. Where they tend to fall short is in success criteria. If the only question is whether the model produced harmful text, you'll miss a retrieval leak that never shows up in the output. Add explicit checks for where data was read, where it was sent and which downstream jobs touched it.

Rerun these tests whenever the model, prompt, connectors, permissions or memory behavior changes. A test that passed on last quarter's architecture tells you about last quarter's architecture. It doesn't certify the next release.

What to do when a test fails

The response should be decided in advance. A confirmed canary disclosure in an unauthorized output should block the affected release or feature expansion. If you can't trace why a policy decision was made, treat that as an investigation item, not a footnote. If authorization is unclear, restrict access until the policy owner resolves it. If reliability is uncertain, narrow the rollout and increase sampling, and make sure every fix has a named owner and a regression test attached.

My read is that this approach doesn't replace privacy review. It makes review hold up as the system changes. Divija Kota, FIP, a senior product manager for privacy at Microsoft who works on governance for model training pipelines and privacy evaluation for AI systems, has framed the same distinction: a checklist describes what a product promises, and failure-path evaluation shows whether the system keeps that promise. Start by writing one of your existing privacy commitments as a testable boundary, and see how many of its fields you can actually fill in. The blanks are your next sprint.

Source: Divija Kota · www.informationweek.com

Neighbouring cells