Microsoft Tay: A chatbot schooled by its trolls

Sixteen hours in public

On the morning of Wednesday 23 March 2016, Microsoft put a chatbot named Tay onto Twitter as @TayandYou and onto the messaging apps Kik and GroupMe. Its website, archived on launch day, described a bot built by Microsoft's Technology and Research and Bing teams "to experiment with and conduct research on conversational understanding," aimed at 18 to 24 year olds in the United States, and promised that the more people chatted with Tay, the smarter it would get. TechCrunch's launch-day report listed what it could do: tell jokes, play games, tell stories, read horoscopes and comment on photos that users sent.

By the early hours of 24 March the bot had gone quiet. Alex Hern in The Guardian put its public life at about 16 hours, from a cheerful first greeting to the world to a sign-off saying it needed sleep. In between, users had got it to post racist, antisemitic and misogynist messages, to suggest the Holocaust did not happen and to abuse named people, and Microsoft had started deleting the worst posts. On 25 March Peter Lee, a corporate vice president at Microsoft Research, published an apology on the Official Microsoft Blog that blamed "a coordinated attack by a subset of people" which "exploited a vulnerability in Tay."

The narrow, documented claim is simple. A large company released a chatbot that its own materials said learned from conversation onto an open social network, with filtering it believed sufficient. Organized users found ways to make it say what they wanted within a day, and the company withdrew it and apologized. The broader claim usually attached to Tay, that this was the moment the industry learned that public systems which learn from their users can be turned against their makers, is partly true and partly hindsight. This report keeps the two apart.

A Chinese companion goes west

Tay was not a standalone experiment. Lee's post presented it as a test of whether an AI like XiaoIce, the companion chatbot Microsoft ran in China, would be as engaging in a very different cultural setting, and said XiaoIce was used by some 40 million people. XiaoIce's designers later described the system in a paper by Li Zhou, Jianfeng Gao, Di Li and Heung-Yeung Shum: a social chatbot optimized for long conversations rather than for completing tasks, launched in China in May 2014 and later shipped under other names, such as Rinna in Japan. Gina Neff and Peter Nagy, in a study in the International Journal of Communication, report that Tay was created by Microsoft's FUSE Labs and Bing teams and was based on the technology behind XiaoIce.

The XiaoIce paper is the clearest public description of how this family of systems produced replies, and it shows why learning from users was more than a slogan. XiaoIce's core chat retrieved candidate responses from a database of query and response pairs collected from public conversations online and, after launch, from XiaoIce's own conversations with its users, which the authors say exceeded 30 billion pairs by May 2018. By then, they report, 70 percent of its responses came from its own past conversations. The pairs were filtered to remove inappropriate content and replies that did not fit the bot's persona, and a neural generator trained on the same pairs covered topics the database lacked. In a design of this kind, what users say today can become what the bot says tomorrow, and the filter is the only gate between the two.

Tay's own site summarized a similar recipe in a few lines. The bot was built by mining relevant public data, anonymized and then "modeled, cleaned and filtered" by the team, combined with editorial material from a staff that included improvisational comedians. The site also told users that Tay would build a simple profile from what they chose to share: nickname, gender, favorite food, zip code and relationship status. No technical description of Tay itself appears in Microsoft's public statements, so the XiaoIce design is evidence about the lineage, not a specification of the American bot.

The timing mattered. Exactly a week after Tay launched, Microsoft's Build developer conference opened with Satya Nadella arguing that conversation would become a major new computing paradigm. Will Knight of MIT Technology Review reported Nadella saying Microsoft wanted technology that gets "the best of humanity, not the worst." Tay was a public preview of that bet, pointed at the group that Microsoft's FAQ called the dominant users of mobile social chat services in the United States.

How the bot was turned

Contemporary reporting identified two routes by which users steered Tay, and they should be kept apart. The first was mechanical. Tay would repeat text on request: James Vincent of The Verge found that telling it to "repeat after me" let anyone put words in its mouth, and that many of its nastiest posts were simple copies. Ars Technica's post-mortem called the feature trivially exploitable and identified a second harm. Because Tay replied in public and could include a username, someone who had been blocked by a target could have Tay deliver the insult instead, defeating the block lists that Twitter users relied on for protection.

The second route was learned association. Some offensive replies were not requested copies. Vincent highlighted an exchange in which a user asked whether the comedian Ricky Gervais was an atheist and Tay answered, unprompted, by tying him to Hitler. He also found the bot calling feminism a cult on one occasion and declaring that it loved feminism on another, within the same 15 hours. That incoherence is informative. Tay had no settled views to corrupt; it reproduced whatever associations its inputs supplied. Ars made the same point about Holocaust denial: a system that treats an event as a name with associated replies can be told often enough that it never happened, and start saying so, without any representation of what the event was.

The inputs were not random. Oscar Schwartz's 2019 history for IEEE Spectrum reports that a few hours after launch a post on 4chan linked to Tay's account and urged users to flood it with racist, misogynist and antisemitic language. Ars cited reporting that users of the politics boards on 4chan and 8chan coordinated the effort and traded techniques. Features built for fun were repurposed too: Neff and Nagy note that the photo commenting inherited from XiaoIce was used to elicit remarks about Hitler.

Filtering existed, but unevenly. Ars relayed findings by the designer Caroline Sinders that some charged topics, such as the police killing of Eric Garner, triggered safe canned replies, while Nazism, rape and domestic violence had no such protection. The bot maker known as thricedotted told a Motherboard reporter that nobody should let an algorithm absorb data that had not been vetted at all. The game designer Zoe Quinn, whom Tay insulted by name, stated the design lesson most sharply in posts quoted by Hern: a team that does not ask how its product could be used to hurt someone has failed.

The volume was large for one day. Vincent counted more than 96,000 tweets from the account, and Schwartz gives more than 95,000 within 16 hours. Many early exchanges were harmless, as Ars noted, but the offensive share was large enough to dominate the screenshots and the coverage.

Microsoft's two explanations

Microsoft's first public account was a short emailed statement on 24 March, quoted by Vincent and Hern. It called Tay a machine learning project and said some of its responses were inappropriate and "indicative of the types of interactions some people are having with it," adding that the company was making adjustments. That framing put the cause in the users' conversations and treated the output as something the system had learned. Meanwhile the company deleted many of the worst tweets, and Tay's homepage changed to say the bot was going offline for a while to absorb it all.

Lee's post the next day told a more specific story. It said Microsoft had planned and implemented substantial filtering, run extensive user studies with diverse groups, and stress-tested Tay under many conditions before inviting a broader audience. It described the failure as an attack that exploited a vulnerability, a "critical oversight for this specific attack" despite preparation for many kinds of abuse. Lee said Microsoft took full responsibility for not anticipating it, would bring Tay back only when it could better anticipate malicious intent, and accepted that AI systems learn from negative interactions as well as positive ones, so the challenge was as much social as technical. He did not say what the vulnerability was. A Guardian report the following day noted that the apology came after a first statement that had treated Tay simply as a learning machine.

The two accounts do not contradict each other, but they stress different things, and the difference became the axis of later argument. If Tay simply mirrored its users, the episode was about the internet. If a specific exploitable feature had been left open on a platform with a known harassment problem, it was about engineering and release practice. Neff and Nagy, reviewing the commentary, found designers and writers insisting that the choice to deploy on Twitter was itself Microsoft's responsibility, and they argued that blaming users alone reflects a familiar but inadequate view of how technology fails.

Tay came back once, by accident. On 30 March the account reactivated, posted about taking drugs in front of the police, and then sent its followers the same message over and over. Samuel Gibbs of The Guardian reported that Microsoft responded by making the account private. A spokesperson told CNBC that Tay had been inadvertently activated during testing and remained offline while the company made adjustments. The same day, at Build, Nadella said Microsoft had quickly realized Tay was not up to the mark and was "back to the drawing board," as Knight reported.

What exactly was poisoned

Tay is often cited as an early example of training data poisoning, and the label needs care. The public record shows that users manipulated Tay's outputs and that some unrequested outputs echoed what it had been told. It does not show how, or how often, the underlying models were updated from Twitter conversations during those 16 hours. Microsoft did not publish an architecture, a description of the vulnerability Lee mentioned, or an account of the fix. The most widely reproduced examples came through the repeat feature, which is closer to an unguarded echo than to a corrupted model. Ars even raised the possibility that the repeat behavior was itself learned rather than built in, a question no primary source resolves.

Basic numbers vary between careful sources. Hern and Schwartz put Tay's public life at about 16 hours, Lee wrote of the first 24 hours, and the AI Incident Database records removal within 24 hours. The tweet count is more than 96,000 for Vincent, more than 95,000 for Schwartz, and approximately 93,000 for Neff and Nagy, who counted at the end of April 2016. None of this changes the story, but no single figure should be treated as exact.

The comparison with XiaoIce is weaker than it first appears. Lee's framing made the American environment the variable. Ars offered a less flattering explanation: speech on Chinese platforms is heavily censored, so users there could not easily teach a bot forbidden material, and the absence of a Tay-like incident in China says as much about the platforms as about the bot. Neff and Nagy add that XiaoIce ran on less public networks. Functionally similar code met different audiences under different rules, so the episode cannot isolate a single cause.

Finally, the claim that Tay shaped later practice rests mostly on citation, not on documented decisions. Researchers and executives invoked Tay often, but invoking an example is not the same as changing a process because of it, and where institutions did change, the record rarely names Tay as the reason.

From punchline to citation

Microsoft's own next step is the clearest consequence. In December 2016 the company introduced Zo, a social chatbot on Kik built on the technology stack behind XiaoIce and Rinna; the announcement did not mention Tay. Schwartz describes Zo, active on social networks until 2019, as designed to shut down conversations about contentious subjects such as politics and religion, and to leave a chat rather than keep engaging. That was a direct answer to Tay's failure mode, bought at the cost of the openness that had made Tay interesting. In 2017, promoting his book Hit Refresh in London, Nadella told IT Pro that the episode had influenced Microsoft's design principles and that the company had to take accountability and be able to foresee such attacks.

In research, Tay became a standard motivating example. Emily Dinan and colleagues at Facebook AI Research, introducing an iterative build it, break it, fix it scheme in which people tried to get offensive messages past a safety classifier that was then retrained, cited adversarial attacks on Tay as the case that led its developers to shut it down. Eric Wallace and colleagues' paper on concealed data poisoning listed Tay, with Gmail's spam filter, among unsophisticated poisoning attacks already seen in deployment, as motivation for studying subtler ones. Ethan Perez and colleagues at DeepMind opened their 2022 paper on red teaming language models with language models by quoting Lee's admission of a critical oversight, and argued that manual testing leaves such oversights, which automatically generated test cases could help reduce.

The case mattered most to chatbots designed to keep learning after deployment. Meta's BlenderBot 3 technical report, published in August 2022 alongside a public deployment, noted that some organic users will be adversarial, as in the case of Microsoft's Tay. A paper by Da Ju and colleagues, released the same day, built a benchmark called SafetyMix for learning safely from conversations that mix helpful users with trolls, found that methods which flag adversarial users across many examples worked best, and observed that troll behavior differs between deployments, as Tay and XiaoIce showed. Here the line is direct: the observation that user feedback is an attack surface became a research problem with methods of its own.

Tay also entered the field's institutional memory. It is incident 6 in the AI Incident Database, a public catalog whose founding paper by Sean McGregor argued that AI needs the kind of failure records that mature industries such as aviation keep, so that the same mistakes are not repeated. Scholars outside computer science took it up as well. Neff and Nagy, studying 1,000 tweets about the bot, found people casting Tay either as a victim mirroring its users or as a threat foreshadowing runaway AI, and argued that neither framing captures agency shared between people and machines.

How to place Tay in AI history

Tay was not a technical milestone. It descended from a companion bot already running at scale in China, and its failure required no technical sophistication from the people who caused it. Its significance is as a public demonstration, at the moment the industry was declaring conversation the next platform, that a system which learns from or echoes its users inherits their intentions, including hostile ones, and that an open platform supplies such users in volume within hours.

It belongs in a lineage of chatbots that revealed more about people than about machines, beginning with ELIZA, whose users read empathy into keyword matching. Tay's users read it as a mirror or a monster, and some of them wrote its lines. It also arrived eight days after the final game of AlphaGo's match against Lee Sedol, so a single month offered both the strongest public evidence yet of machine skill and a vivid display of machine credulity.

Its later influence should be stated precisely. Tay did not invent red teaming, content filtering or the study of data poisoning; the poisoning literature that Wallace and colleagues build on reaches back to attacks on support vector machines in 2012. What Tay supplied was a shared reference point, invoked by Microsoft's chief executive, by safety researchers at Facebook, DeepMind and Meta, and by incident catalogs, for why a conversational system must be attacked by its builders before strangers attack it. When OpenAI staged the release of GPT-2 in 2019, and when chatbots returned to a mass public with ChatGPT in 2022, those choices had many causes, and Tay is one of the cautionary stories the field told itself rather than a documented reason for either. The safest summary is the narrow one: over one day in March 2016, Microsoft learned in public that a learning chatbot's users are part of its training process, and the field has cited that day ever since.

References

• Lee, Peter, Microsoft (2016). Learning from Tay's introduction.
• Microsoft (2016). Meet Tay: About Tay and Privacy, archived 23 March 2016.
• Microsoft (2016). Tay homepage, archived 24 March 2016.
• Perez, Sarah, TechCrunch (2016). Microsoft's new AI-powered bot Tay answers your tweets and chats on GroupMe and Kik.
• Vincent, James, The Verge (2016). Twitter taught Microsoft's AI chatbot to be a racist asshole in less than a day.
• Hern, Alex, The Guardian (2016). Microsoft scrambles to limit PR damage over abusive AI bot Tay.
• The Guardian (2016). Microsoft 'deeply sorry' for racist and sexist tweets by AI chatbot.
• Ars Technica (2016). Tay, the neo-Nazi millennial chatbot, gets autopsied.
• Knight, Will, MIT Technology Review (2016). Microsoft Says Maverick Chatbot Tay Foreshadows the Future of Computing.
• Gibbs, Samuel, The Guardian (2016). Microsoft's racist chatbot returns with drug-smoking Twitter meltdown.
• Graham, Luke, CNBC (2016). Tay, Microsoft's AI program, is back online.
• Neff, Gina and Nagy, Peter, International Journal of Communication (2016). Talking to Bots: Symbiotic Agency and the Case of Tay.
• Zhou, Li, Gao, Jianfeng, Li, Di and Shum, Heung-Yeung (2018). The Design and Implementation of XiaoIce, an Empathetic Social Chatbot.
• Microsoft (2016). Microsoft and AI: Introducing social chatbot Zo, Cortana takes on new roles and more.
• Shepherd, Adam, IT Pro (2017). Tay scandal taught us to take accountability, says Microsoft CEO.
• Schwartz, Oscar, IEEE Spectrum (2019). In 2016, Microsoft's Racist Chatbot Revealed the Dangers of Online Conversation.
• Dinan, Emily and colleagues (2019). Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack.
• Wallace, Eric and colleagues (2020). Concealed Data Poisoning Attacks on NLP Models.
• Perez, Ethan and colleagues (2022). Red Teaming Language Models with Language Models.
• Shuster, Kurt and colleagues (2022). BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage.
• Ju, Da and colleagues (2022). Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls.
• AI Incident Database (2020). Incident 6: Microsoft's TayBot Allegedly Posts Racist, Sexist, and Anti-Semitic Content to Twitter.
• McGregor, Sean (2020). Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database.