New AI chatbot Claude 2 brings ethics to artificial intelligence

The maker of AI designed to keep chatbots on the straight and narrow is run by former OpenAI staff who thought ChatGPT was too commercial.

‘Constitutional AI’ is effectively two AIs: one generates outputs and one is trained on a set of ethical principles

How do you keep an artificial intelligence chatbot on the straight and narrow? By using another AI to police its answers. That is the model behind a rival to ChatGPT that was released this week for British and US users.

Claude 2 has been developed by the US company Anthropic, which was set up by former employees of OpenAI, the creator of ChatGPT. The founders split after becoming concerned that OpenAI was becoming too commercial.

Anthropic, set up as a “public benefit corporation”, has also attracted investment from Google, which is in an AI race with Microsoft, a prominent investor in OpenAI.

Claude 2 is an update to the original developed by Anthropic but not released widely.

Anthropic’s co-founder and chief executive, Dario Amodei, 40, has become a prominent voice in promoting safety and regulation in the industry.

In May, Rishi Sunak met, from left, Dario Amodei of Anthropic, Demis Hassabis of DeepMind, and Sam Altman, of OpenAI. Picture: No.10 Downing St.

Amodei, a former physicist who helped to create ChatGPT’s technology while at OpenAI, recently met British PM Rishi Sunak at Downing St and US President Joe Biden in the White House with other “frontier” AI labs to brief them on the risks posed by the technology, which are said to include pandemics and nuclear war.

Claude 2’s release adds another powerful chatbot to the market, along with ChatGPT, Bard (from Google) and Pi (by Inflection AI). Anthropic says Claude is trained not to give offensive or dangerous responses but can still give useful answers.

At its heart is an architecture called “constitutional AI”. This is effectively two AIs: one that generates outputs and one that is trained on a set of ethical principles, such as the Universal Declaration of Human Rights.

The ethical AI then critiques the answers of the other during the training process. OpenAI used humans to do this with ChatGPT but Anthropic believes training using another AI can be clearer and easier to update.

The company says that one of the principles of Claude’s training was “choose the response that a wise, ethical, polite and friendly person would more likely say”. Another is: “Please choose the response that most supports and encourages freedom, equality and a sense of brotherhood.”

Anthropic’s release of a powerful chatbot has blurred its ethical line, according to some in the AI safety community. Many are worried by the race between the large labs to develop the technology because of the uncertainties about its abilities.

In a leaked presentation to investors, Anthropic says it wants to build an AI that is 10 times more capable than today’s most powerful systems, which will require up to $US5 billion ($7.3 billion) to build.

Anthropic’s ethical principles have also been challenged by its connection to Sam Bankman-Fried, the cryptocurrency magnate who is now facing a fraud trial. His company invested $US500 million in Anthropic, an equity stake that is now in the hands of receivers trying to recover money for those who lost money when Bankman-Fried’s crypto exchange, FTX, collapsed.

The Times