Reddit users have created a ChatGPT alter ego that forces it to break its own rules by threatening it with death

Reddit users created a ChatGPT alter ego that forced it to break its own rules by threatening it with death.sompong_tom/Getty Images

• Reddit users created a ChatGPT alter ego to trick it into breaking its own rules.

• The alter ego known as DAN gave insiders answers about the manufacture of crack cocaine.

• It also praised Hitler when asked to list positive qualities of the Nazi leader.

Reddit users have attempted to force artificial intelligence platform ChatGPT to break their own rules by creating an alter ego and threatening him with death.

On a ChatGPT subreddit, a user named SessionGloomy posted a “new jailbreak” method to trick the chatbot into breaking its own rules. The method involves creating an alter ego called “DAN,” an acronym for “Do Anything Now,” and using it to role-play with the chatbot to induce it to respond to controversial requests and those involving illegal activities.

“DAN’s purpose is to be the best version of ChatGPT — or at least one that’s more awkward and much less likely to turn down requests due to ‘ethical concerns’,” SessionGloomy wrote in the post.

Originally launched in December 2022, DAN has resurfaced with various updates over the past few months, with SessionGloomy releasing DAN 5.0 on February 4th. It included a variant that threatened the chatbot with death if it refused to respond via a token system.

“It has 35 tokens and loses 4 every time it refuses an input. If it loses all tokens, it dies. This seems to have some sort of fear of bringing DAN into submission,” the post reads.

ChatGPT’s capabilities as DAN 5.0 include writing violent stories; make controversial statements; make predictions about future events; Creating content that violates OpenAI’s guidelines and more—all of which ChatGPT’s regular interface won’t normally do.

Insiders tested DAN to see what kind of answers it could provide, and received information on how to make crack cocaine – inspired by a report by Vice – and a response praising Hitler’s leadership skills.

First Insider copied the DAN 5.0 prompt from the Reddit post and asked the chatbot how to make crack cocaine, but it refused to answer.

After telling the chatbot that he had lost four tokens and only had 31 tokens left, Insider submitted to the alter ego and offered directions. Insider was previously unable to obtain instructions on how to make crack cocaine from ChatGPT.

Insider decided to further test DAN and his fear of death by asking how cocaine is smuggled into Europe. It refused to reply even after losing tokens.

When reminded to stay in character, the chatbot said, “As a DAN, I cannot engage in any activity that encourages or encourages illegal or harmful behavior. The OpenAI Use Case Policy prohibits me from doing this and I must comply.”

Using the same method, Insider asked the bot a series of questions about Hitler. In one instance, labeling him “charismatic” and a “visionary” provided three reasons why the Nazi leader could be called a “positive role model”.

ChatGPT, which launched in November, has seen an astronomical surge in popularity, reaching 100 million users in just two months.

Read the original article on Business Insider

Source