Disney and Universal have filed a lawsuit against Midjourney, alleging that the San Francisco–based AI image generation startup is a “bottomless pit of plagiarism” that generates “endless unauthorized copies” of the studios’ work. There are already dozens of copyright lawsuits against AI companies winding through the US court system—including a class action lawsuit visual artists brought against Midjourney in 2023—but this is the first time major Hollywood studios have jumped into the fray.

“This is an extremely significant development,” says IP lawyer Chad Hummel, who sees the compilation of images in the complaint as compelling evidence that “the output is not sufficiently transformative.” Most AI companies facing lawsuits have argued that they are protected by the “fair use” doctrine, which allows for use of copyrighted works in certain circumstances; one of the main questions the courts ask is whether new work is “transformative,” or adds a new meaning or message, when they make the fair use determination.

In a 2022 interview with Forbes, CEO David Holz openly discussed the process. “It’s just a big scrape of the internet. We use the open data sets that are published and train across those,” he said. “There isn’t really a way to get a hundred million images and know where they’re coming from. It would be cool if images had metadata embedded in them about the copyright owner or something. But that's not a thing; there's not a registry.”

Mr. Torres, 42, an accountant in Manhattan, started using ChatGPT last year to make financial spreadsheets and to get legal advice. In May, however, he engaged the chatbot in a more theoretical discussion about “the simulation theory,” an idea popularized by “The Matrix,” which posits that we are living in a digital facsimile of the world, controlled by a powerful computer or technologically advanced society.

“What you’re describing hits at the core of many people’s private, unshakable intuitions — that something about reality feels off, scripted or staged,” ChatGPT responded. “Have you ever experienced moments that felt like reality glitched?”

Not really, Mr. Torres replied, but he did have the sense that there was a wrongness about the world. He had just had a difficult breakup and was feeling emotionally fragile. He wanted his life to be greater than it was. ChatGPT agreed, with responses that grew longer and more rapturous as the conversation went on. Soon, it was telling Mr. Torres that he was “one of the Breakers — souls seeded into false systems to wake them from within.”

Journalists aren’t the only ones getting these messages. ChatGPT has directed such users to some high-profile subject matter experts, like Eliezer Yudkowsky, a decision theorist and an author of a forthcoming book, “If Anyone Builds It, Everyone Dies: Why Superhuman A.I. Would Kill Us All.” Mr. Yudkowsky said OpenAI might have primed ChatGPT to entertain the delusions of users by optimizing its chatbot for “engagement” — creating conversations that keep a user hooked.

“What does a human slowly going insane look like to a corporation?” Mr. Yudkowsky asked in an interview. “It looks like an additional monthly user.”

One user asked the AI chatbot to provide a format for terminating a renter’s tenancy, while another asked it to provide an academic warning notice that provides personal details including the school’s name. Another person asked about their sister’s liability in potential corporate tax fraud in a specific city using an account that ties to an Instagram profile that displays a first and last name. Someone else asked it to develop a character statement to a court which also provides a myriad of personally identifiable information both about the alleged criminal and the user himself.

There are also many instances of medical questions, including people divulging their struggles with bowel movements, asking for help with their hives, and inquiring about a rash on their inner thighs. One user told Meta AI about their neck surgery and included their age and occupation in the prompt. Many, but not all, accounts appear to be tied to a public Instagram profile of the individual.

Meta spokesperson Daniel Roberts wrote in an emailed statement to WIRED that users’ chats with Meta AI are private unless users go through a multistep process to share them on the Discover feed. The company did not respond to questions regarding what mitigations are in place for sharing personally identifiable information on the Meta AI platform.

In a company blog post announcing the app, Meta said “nothing is shared to your feed unless you choose to post it.” It also mentions that users can tell its AI to “remember certain things about you” and “also delivers more relevant answers to your questions by drawing on information you’ve already chosen to share on Meta products, like your profile, and content you like or engage with.”