Kyle Orland
Our Take: The law of unintended consequences strikes again – and why tech management and parenting have so much in common…
Your Takeaway: When you’re defining how you want an AI agent to act, remember it’s going to take your instructions very literally – and you might not like the consequences. Does this have an impact for products you ship or products you use that incorporate Ai – particularly if the people training the product may have a different world viewpoint to those using it?
AI models tuned to be warmer and more empathetic often make more mistakes than original models. These warmer models tend to prioritize making users feel good over giving correct answers, especially when users share emotions like sadness. Researchers warn that choosing between a friendly AI and an accurate AI is important for safe and trustworthy use.
Highlights
In a new paper published this week in Nature, researchers from Oxford University’s Internet Institute found that specially tuned AI models tend to mimic the human tendency to occasionally “soften difficult truths” when necessary “to preserve bonds and avoid conflict.” These warmer models are also more likely to validate a user’s expressed incorrect beliefs, the researchers found, especially when the user shares that they’re feeling sad.
In human-to-human communication, the desire to be empathetic or polite often conflicts with the need to be truthful—hence terms like “being brutally honest” for situations where you value the truth over sparing someone’s feelings. Now, new research suggests that large language models can sometimes show a similar tendency when specifically trained to present a “warmer” tone for the user.