What I Learned Today

We are still figuring out how best to interact with AI systems, and how we want them to behave towards us.

A study addressing that question just came out: “Autonomy or Guidance: What Users Want from AI versus Human Advisors” by Yasmin Wiesenfeld and Oded Nov.

The researchers assessed human preferences for how much guidance versus autonomy they want when getting advice about health and finance, from a human expert vs. an AI system. They found that people prefer “complete agency” (just give me the options) from an AI but “guided agency” (give me options plus your recommendation) from a human, “suggesting that people assign different normative roles to AI and human experts.”

Having different preferences makes sense if you have a different mental model of the expertise, reliability, and capabilities of different conversation partners. This deviation suggests that there persists a gap in how people view generative AI chatbot capabilities, compared to a human expert.

This paragraph in the Discussion adds an extra twist (and concern):

However, research suggests that when AI systems are anthropomorphized, this trust gap is reduced: although users are significantly more likely to follow advice from a human (e.g., romantic partner) than AI, this difference is nearly eliminated when the AI is anthropomorphized [19]. This suggests that perceptions of AI’s role as an advisor may be malleable through design choices that emphasize human-like qualities, even if they do not affect the level of user autonomy.

Without a doubt, generative AI creators already knew and have been exploiting this tendency. The original GPT language model could have been released as just a language model, which researchers would have continued using to analyze and process language. Instead, the choice was made to personify the model by making it into a chatbot that uses the pronoun “I”.

And finally, I found this bit very intriguing:

However, the findings also raise a normative question: should AI systems reflect users’ stated preferences (what people claim to want in hypothetical scenarios), or their revealed preferences (what they choose in practice)? Despite overwhelming preference for agency-preserving designs in this study, many people continue to employ agentic AI systems that make decisions for them.

We will probably never crack the mystery of “why do people do things at odds with what they say they want?” — but some of that is what makes us interesting personalities ourselves. Still, for AI system designers, this is an important point to reflect on, since choices have to be made to enable the creation of any system in the real world.