Our assumptions about the power of words and data to convey knowledge can lead us to believe that AI promotes understanding when it doesn’t.
Knowledge has always depended on words and data. But it’s easy to assume that words or data themselves represent knowledge. That’s especially true as generative AI encourages users to think about knowledge in terms of natural language sentences.
Expertise, knowledge, and understanding are specific to the individual. Users want advice from credible experts. Yet such expertise can be a barrier to clear communication and interfere with user understanding.
Writers have long faced the challenge of bridging what they know with how they communicate that knowledge. When done poorly, it results in a chasm between the writer’s and the reader’s understanding of a topic. Experts are often lousy communicators.
Readers judge a source’s expertise by its mastery of factual details. If a person or system can answer detailed factual questions, then it has expert-level knowledge.
But knowledge isn’t a universal standard. The notion of a knowledge base can be deceptive because it implies a scope that is relevant to everyone. Knowledge is simply externalized understanding. We can talk about collective knowledge — the sum of available information that everyone has contributed. But the understanding attained will depend on the individual.
Clear writing does not necessarily translate into solid knowledge. For knowledge to be usable, it must be readable. But readability does not guarantee the information is useful to an individual.
Matthew Crawford, the author of Shopcraft as Soulcraft, draws a distinction between thinking-as-doing and thinking-as-writing. What’s written too often is divorced from what the user needs to do. He argues:
The writers of modern technical manuals are neither mechanics nor engineers but rather technical writers. This is a profession that is institutionalized on the assumption that it has its own principles that can be mastered without the writer being immersed in any particular problem; it is universal rather than situated. Technical writers know that, but they don’t know how.
The problem, as Crawford sees it, is that the writer does not necessarily have a firsthand understanding of the topic they write about. The writer has thought about the topic abstractly, but without direct experience. When confronted with explanations that don’t make sense, the user must:
render the gibberish coherent, and he can only do this by referring it to a model he had in his own mind of how the thing works.
The key to promoting understanding is the mental model framing the discussion. It must be accurate and intelligible to the user.
Writing generates a qualified understanding. The writing process can help writers understand a topic. It can help them clarify the relationships between issues and encourage them to develop precise terminology to describe concepts.
Writing is, without question, valuable for developing knowledge about complex issues. But the writing will only represent the knowledge of the writer, not necessarily the understanding that can be acquired by another reader. The knowledge articulated must be further revised to match the expectations and understanding of external readers.
The writer must decouple how they arrived at a conclusion from how they communicate that conclusion to others. The writer will have different background knowledge and motivations than the readers seeking to understand a topic. While in some fields (scientific research, for example), both the writer and the readers will follow a common, standard process for arriving at conclusions, this pattern is not the norm in most cases. Instead, the writer is presumed to have more expertise than the reader and will be concerned with factual and logical details that won’t be relevant to the non-expert.
With LLMs, we find that responses to user prompts often replicate the expert’s explanatory logic and terminology that was expressed in the source content, even though these are inappropriate for the user.
Explanations without rationales sound authoritarian. Users are exacting: they want expert opinions, yet don’t want to put up with expert-level explanations. At the same time, many users find explanations lacking rationales less credible. They question why they should believe what’s being said.
Public trust in professional experts has been eroding with the “democraticization” of information. People don’t like being told what to do when they feel they aren’t permitted to make the decision themselves. Users feel empowered and entitled to form their own conclusions. Yet LLMs don’t always provide an extensive rationale for their responses. The tendency of chatbots to provide “instant answers” can undercut their credibility.
Knowledge sources must be transparent and easy to trace. Users want to know where information comes from. AI solutions make tracing sources difficult.
The major generative AI platforms provide limited links to sources in their responses. A response might contain 5-8 links, often to obvious sources such as Wikipedia. At best, these links provide a shallow overview of available knowledge.
Dense knowledge is not necessarily clear information. Knowledge graphs represent an alternative paradigm for answering user questions. They can provide better traceability, but at the expense of poorer usability. They don’t promote deeper understanding.
Because of their complexity, knowledge graph implementations increasingly rely on natural language chatbot user interfaces. When using a chatbot to interact with knowledge graph data, information traceability is lost. Google’s Knowledge Panels, which aggregate information from multiple sources, require users to click links to perform new searches to trace the origins of the data shown.
Just because you can explain something doesn’t mean you understand it. It’s easy to make statements and to come up with reasons why readers should believe what you’ve said. But harder to justify statements with a sound rationale. Many assertions fail to withstand scrutiny.
Writers and AI platforms are both aware that credibility depends on providing a rationale. Writers have long used formulas such as “six reasons why….” to buttress their assertions. Professional writers might follow a template that outlines methods, procedures, and evidence before presenting conclusions. AI platforms reproduce these formulas in their responses.
It’s easy for users to see formulaic explanations as valid because they match expectations. But in many cases, especially with LLM responses, the justification is developed after the conclusion is generated. Chatbots explain things without understanding them.
A widespread myth about LLMs is that they can reason. Vendors promote the “Thought-Action-Observation Loop” technique and ReAct (Reasoning/Acting) prompting, suggesting that LLMs are reasoning. These techniques build on a prompt-engineering approach called Chain-of-Thought (CoT) reasoning, which supposedly shows the thought process of the LLM. However, more detailed research indicates that these measures are largely proformative. They slow down responses without yielding appreciable improvements in answer quality. On the contrary, the techniques can make answers less consistent.
The core problem is misplaced trust: CoTs can appear persuasive even when they do not faithfully reflect a model’s actual decision process
Explanations can be myths. To enhance perceived credibility, communications aim to sound expert and appear to be based on a comprehensive review of available information.
But mimicking common arguments does not make the arguments valid. Few people apply critical reading online, parsing claims, reasons, and evidence.
One of the most reliable forms of communication is the story. Stories often rely on reasoning by analogy. Chatbots have access to countless examples of material that are vaguely similar to the user’s prompt. We shouldn’t be surprised if we start seeing chatbot explanations including sports or TV show analogies.
— Michael Andrews