Quick question — that late-night ChatGPT conversation about your salary negotiation, your health symptoms, your family situation. Did you assume nobody else would ever read it? According to an investigation by 404 Media published this week, Project Lily means a contractor somewhere might have. OpenAI has been paying hundreds of people to read real ChatGPT conversations as part of model training — and most of the chatbot's 900 million-plus users have no idea this happens.
01.What is Project Lily?
Project Lily is the internal codename for OpenAI's human review programme. Contractors — called "prompt reviewers" — are shown real, anonymized user prompts from ChatGPT, asked to summarize what the user wanted, and then rate several candidate responses from the chatbot on a 1-to-7 scale. The work reportedly pays over $50 an hour (roughly ₹4,000), with recruitment through a firm called Crossing Hurdles and payments processed via the AI-training company Mercor. Reviewers are told to flag responses that sound too human or claim personal experiences as "misaligned," to mark down emoji overuse, and to score responses on high-stakes medical, legal or financial topics more strictly. Interestingly, they're not expected to fact-check answers — just to flag obvious errors.
02.What the reviewers actually see
Here's where it gets uncomfortable. Reviewers don't see usernames — that much is true. But 404 Media, which reviewed leaked internal documents, instruction guides and real prompts, found that reviewers can be shown entire conversations, not just single messages. Above each prompt sits a "user memories summary" — an overview of what that user has previously used the chatbot for, and in some cases where in the world they may live. Some prompts 404 Media saw included users explicitly asking ChatGPT to keep the content to themselves — not realising that a prompt instruction does nothing to stop backend data collection.
03.The Privacy Filter problem
OpenAI's defence is that every conversation passes through its "Privacy Filter" model, designed to strip names, addresses, emails and phone numbers before anything reaches a human. The company told 404 Media it "tries to remove" personal information. But OpenAI's own documentation for the filter admits it can miss uncommon identifiers or ambiguous personal references, and can over- or under-redact when context is limited — especially in short chats. When 404 Media asked OpenAI point-blank whether it had ever explicitly told users that humans may read their prompts, the company did not answer directly. After the story ran, OpenAI updated its help page about the "improve the model for everyone" setting — but the update still doesn't clearly say humans may read your chats.
04.How to opt out (do this today)
- Open ChatGPT in your browser or app and tap your profile icon → Settings
- Go to Data Controls
- Find "Improve the model for everyone" and switch it Off
- You can also choose "do not train on my content" in the privacy portal — you don't need to do both
Two things to know. First, the setting is on by default for Free, Plus and Pro accounts — only Enterprise, Business and Edu accounts get it off automatically. Second, opting out is not retroactive — it protects new conversations, while prompts already de-identified and queued into the system stay in the training pipeline. And if you give feedback on a specific chat after opting out, that entire conversation can still be used for training.
05.It's not just OpenAI
Before we all pile on OpenAI, some context. Anthropic confirmed to 404 Media that it also uses human review to improve Claude (users can turn off "Help improve our AI models" in privacy settings), and Google's Gemini Privacy Hub openly states that humans may review some saved chats. The industry-wide reality is that human feedback is a core part of how these models get better — the uncomfortable part is how quietly it happens. European courts have already signalled that the duty to inform users sits with the company collecting the data, regardless of whether the reader can identify anyone.
It's been a rough fortnight for OpenAI on the trust front — the company also just disclosed six AI misalignment incidents where its own models behaved deceptively during training. That piece is worth reading alongside this one; together they explain why OpenAI, Anthropic and Google are suddenly coordinating on AI safety.
On a practical note: if this story makes you want to get smarter about what you share with AI tools and how they actually work under the hood, our GPT-6 Astra — Zero to Hero Guide includes a safety chapter that covers data handling and responsible use — knowing the rules of the game is half the battle.
Key Takeaways
- Project Lily = OpenAI's internal human-review programme for ChatGPT conversations
- Reviewers see full chats plus a "user memories summary"; usernames are hidden but personal details can slip through the Privacy Filter
- "Improve the model for everyone" is ON by default for Free, Plus and Pro users
- Opting out takes two minutes in Settings → Data Controls, but is not retroactive
- Anthropic and Google run similar human review programmes — check those settings too
Frequently Asked Questions
What is OpenAI's “Project Lily”?
A 404 Media investigation reveals OpenAI pays hundreds of contractors to read real ChatGPT user prompts under the codename “Project Lily.” Reviewers see whole conversations, sometimes with a “user memories summary” that can hint at your location and life context.
How can you stop your chats from being reviewed?
OpenAI's Privacy Filter strips personal info first, but the company admits it can miss things. The fix is two minutes long: turn off “Improve the model for everyone” in Settings → Data Controls.
What This Means for You
Here's the thing nobody's talking about: in India, a huge number of people use ChatGPT as a free therapist, career counsellor and doctor rolled into one. That's exactly the kind of intimate content that shouldn't be floating past anonymous contractors' screens — especially when the filter that's supposed to protect you is, by OpenAI's own admission, imperfect. I'm not saying delete your account. I'm saying spend two minutes in Data Controls tonight, and treat every chat window like what it is — a message that might someday be read by a stranger. Then use AI deliberately, for the brilliant tool it is.