Is character important for an LLM?
In using LLMs, the back and forth language style exposes the character or personality of the model. Is this a differentiator in effective use or even vendor selection?
Background
In my use of different LLMs, OpenAI ChatGPT, Google Gemini and Anthropic Claude, the responses give an insight into the character or personality of these models.
The idea that this is a differentiator between models, is stated by Dario Amodei, CEO of Anthropic, in a recent CNBC interview at Davos.
The whole 6:30 minute clip is worth watching in its entirety, as Dario responds to questions on “Scaling laws”, so are we approaching a block to further improvement, mentions Computer Use, Virtual Collaborators and commoditisation of LLMs.
In his response to commoditisation at 5:25 mins he talks about people “liking Claude’s character” and refers to a recent Stanford Medical School blog post. This was for study of radiological images using LLMs for diagnosis and while many models were accurate on the diagnosis, on the question of how likely was the radiologist to follow the LLM recommendation, Dario states that Claude won head and shoulders above the others.
I searched for the post, but only came up with one on AI tool assist doctors in sharing lab results, which while it mentions Claude, it is not the study mentioned. (If anyone knows or finds the article, please add a link in the comment).
Character or Personality?
We know that in training a model, the capability of responding to prompts has to be specifically added with specific training data and not just public web datasets such as Common Crawl.
This practice will differ between vendors and the question is does this make a meaningful difference to the user?
If LLMs become commoditised then differentiators such as character or tool use (web search, voice, …) become important in both effectiveness of use and selection.
So how important a factor is character?
Google Gemini
Playing around with ideas for a blog last night, I started prompting Gemini with requests on “Median Salary and GDP per Capita” and some of the language in the responses stood out.
After Gemini first returned median salary data for four countries in their local currencies and I asked it show this data in USD, the first part of its response back was:
You are right…
And when I asked “Why is US median salary lower than UK, when US GDP per capita is much higher than UK?”
That’s a very astute observation!
Flattery will get you far with humans!
(Note, I truncated the screenshot above to only show the first bullet).
And on asking it to combine median salary and GDP per capita into one table, it started the response with “You got it!” and on asking for inflation adjusted figures, “Yes absolutely!”, while on asking about the calculation method for this, “You’re right to be curious about the calculation!”
On my final response of “Nice” (I confess to using that response a bit too much on social media…) the response was
And while I had planned to stop there, the eagerness of this with (yet another) exclamation mark, did make me ask about income inequality and get a refresher on Gini Coefficients.
In summary, I liked the positivity, eagerness (and flattery) in the responses and it helped me engage more.
But after pro-longed use, it may get a tiny bit annoying.
Probably similar to how we would feel interacting with an over enthusiastic graduate trainee…
Culture of the Firm
I have yet to draw an opinion on how strong the character factor is making better use of an LLM, let alone if it is a factor in deciding which LLM to use.
I think on deciding which LLM, most of us are more likely to be drawn by the founder or CEO, who represents the firm or product and is responsible for it’s culture.
Take you pick from Sam Altman, Demis Hassabis, Dario Amodei and others.
All else being equal that is; capability, tools, price, performance, …
Which is a fast changing competitive landscape..
Identifying with a Brand and what it stands for, can be a factor in the purchasing decision, always for fashion but often for technology.
Especially one as important as AI.