Definition
Data privacy in AI is the set of practices and rules governing how personal data is collected, used, stored and exposed across the life of an AI system: in the data used to train models, in the prompts and files people send to AI tools, in what those tools remember, and in what they say about people in their answers.
The risks sit at four points:
- Training data: models trained on web data or user conversations may include personal information, and can occasionally reproduce fragments of it.
- Inputs: prompts, uploaded documents and connected apps can contain customer records, health data or trade secrets, which may be stored by the provider or used for training depending on the plan and settings.
- Memory and logs: assistants increasingly remember past conversations, and agent features take screenshots or browse while signed in to accounts.
- Outputs: an AI answer can reveal or invent information about a person, and AI search engines can repeat personal details found on the web.
Provider policies differ by plan, and they change. Anthropic updated its consumer terms on August 28, 2025: Claude Free, Pro and Max users choose whether their chats and coding sessions may be used for training, with data kept up to five years if they opt in and 30 days if they do not (Anthropic's announcement). Commercial plans and the API are excluded. OpenAI lets ChatGPT users turn off 'Improve the model for everyone', deletes Temporary Chats after 30 days, and does not use Business, Enterprise, Edu or API data for training by default. Memory features add another layer: Claude's memory is on by default for Free, Pro and Max plans and off by default for Team and Enterprise, with sensitive topics excluded unless the user turns them on.
The legal framework is layered. In the EU, the GDPR applies whenever an AI system processes personal data, requiring a lawful basis, transparency, data subject rights and, for high-risk processing, a data protection impact assessment. The EU AI Act adds AI-specific duties on a staggered timeline: prohibited practices from February 2, 2025, obligations for general-purpose AI models from August 2, 2025, and transparency rules and enforcement from August 2, 2026. The AI Omnibus, in force since July 27, 2026, pushed the high-risk system rules back to December 2, 2027 for stand-alone systems and August 2, 2028 for AI embedded in regulated products (European Commission). In the US, state laws such as California's CCPA/CPRA and sector rules such as HIPAA and GLBA apply.
Organizations reduce risk with a few established controls: business plans with no-training terms and defined retention, zero-data-retention options for sensitive API workloads, data minimization and redaction before content reaches a model, access controls on connectors and agents, and self-hosted open-source models where data cannot leave their own infrastructure.
For marketing and GEO teams, privacy shows up in everyday work. Do not paste customer data into consumer AI tools to draft content or analyze feedback. Treat AI visibility and analytics tools like any other processor, checking what data they collect. Review what AI answers say about your company's people and customers, since assistants can surface outdated or personal details from the web. And remember the publisher side: what you publish can be crawled for training data, which TDM rights reservation and content signals let you address.
Examples of Data Privacy in AI
- A healthcare company moves clinical note summarization from a consumer chatbot to an enterprise API agreement with zero data retention and a signed business associate agreement.
- A law firm runs an open-weight model on its own servers for document review so client files never leave its infrastructure.
- A marketing team adds a redaction step that strips names and email addresses from customer reviews before sending them to an AI model for sentiment analysis.
- An EU retailer documents a data protection impact assessment before deploying an AI chatbot that can look up customers' order histories.
