Notion AI chats leaked from Fireworks AI, an inference vendor that had given Notion a contractual commitment to zero data retention. Two things went wrong at once, and they are worth separating. Notion's zero-retention promise covered a request path, not the data. And the promise it got from Fireworks was a clause, not a control. Neither one can delete a byte. The only data a vendor can't keep is data you never sent.
What Happened
From Fireworks' update: on October 23 a third-party cloud provider hosting Fireworks R&D machines had an incident. On October 24 an unauthorized party used it to reach Fireworks dev machines and run a workload; Fireworks detected it the same day. Fireworks attributes it to a financially motivated group that sells LLM tokens to illicit routing gateways, LLMjacking. Depending on the customer, what was accessed included billing details, stored secrets for a limited set of accounts, and "a subset of incomplete model weights and datasets."
Notion's notice explains where its data came from. "For the Insights feature, we sent the stored requests and responses from Notion AI use to a third party model provider", for "classification jobs to help customers understand their own AI usage." The affected transcripts come from fewer than 1% of workspaces, and for most of them Notion hadn't added metadata tying a transcript to a workspace. As of November 3 Notion has disconnected from Fireworks and deleted its data there, and Insights is disabled.
Then the two sentences that carry the story:
"Fireworks provided us with a contractual commitment to zero data retention for our use of its platform."
"Unfortunately, during our investigation we learned that data was retained from the Fireworks endpoints that we used."
Neither company has said which endpoint retained it or why, so I won't guess.
The Mechanism
Look at what Notion's own promise says. Zero data retention "applies to the API requests we send to third party model vendors we use to generate responses to Notion AI requests", and "our zero data retention commitments are not intended to cover these use cases", meaning chat history and Insights.
That's an honest, precise scope, and it shows the shape of the problem:
- The inference path. A user asks, the request goes to a model, the answer comes back. Retention terms are written for this.
- The second path. Notion stores chat history, because that is the product. A feature built on top of it sends those stored chats out again, to be classified. Same vendor, same data, different path, and the customer-facing promise doesn't follow it.
So every new job over stored customer content is a new copy, and a copy only has the protection someone wrote down for that path. Then the second failure: even where a contract did cover the path, the data was still there for an attacker to find. A retention clause is a statement about what a vendor intends to do. You can't audit a deletion from the outside, and you find out it didn't happen from an incident notice.
This is the supply chain problem I keep coming back to: it isn't a missing tool, it's a data flow nobody owns.
The Control
Keep the vendor inventory by data flow, not by vendor. "Fireworks: model inference" describes the first path and hides the second. One line per flow:
| Flow | Data sent | Covered by your promise to customers? | Would a leak at the vendor matter? |
|---|---|---|---|
| AI answers | live prompt and response | yes | yes |
| Usage insights | stored chat history | no | yes |
Any row with "yes" in the last column is data sitting at a vendor, whatever the contract says. Treat it like a database you host there and ask whether it needs to exist:
- Send derived data, not raw content. A classification job usually needs a category, a length, a language. Strip or reduce the text before it leaves, or run the job where the content already lives.
- Assume retention unless you've turned it off yourself. VentureBeat reports that Fireworks' docs say a few features "do store customer content, such as stored responses, batch jobs, and training", and that its account-wide zero-retention policy is off unless set. Read the defaults for every path you use.
For users it's shorter: an AI chat is a stored document, and it gets copied to places you'll never see. Notion's advice is to rotate any password, API key or credential pasted into Notion AI. Same point as anything in the context can leave: don't hand it what it doesn't need.
The Tradeoff
Running classification in-house means owning a model deployment you didn't want to run, which is often why the job went to an inference vendor in the first place. Derived data loses detail: a category count won't tell you what users actually asked. And a per-flow inventory goes stale the week someone ships a feature, so it only works if adding a flow is a review step, not a spreadsheet nobody opens.
My read: a retention promise is worth having, but for data at a vendor you can verify very little of it. Minimize what leaves.