Our faith in the power of words and data to convey knowledge can lead us to believe that AI promotes understanding when it doesn’t.
Knowledge has always depended on words and data. But it’s easy to assume that words or data themselves encapsulate and transmit knowledge. That’s especially true as generative AI encourages users to approach knowledge evaluation and acquisition through the medium of natural language sentences.
Expertise, knowledge, and understanding are specific to the individual. Users want advice from credible experts. Yet such expertise can be a barrier to clear communication and interfere with user understanding.
Writers have long faced the challenge of bridging what they know with how they communicate that knowledge. When done poorly, it results in a chasm between the writer’s and the reader’s understanding of a topic. Experts are often lousy communicators.
Readers judge a source’s expertise by its mastery of factual details. If a person or system can answer detailed factual questions, then it has expert-level knowledge. They are credible.
But knowledge isn’t a universal standard. The notion of a knowledge base can be deceptive because it implies a scope that is relevant to everyone. Knowledge is simply externalized understanding. We can talk about collective knowledge — the sum of available information contributed by various experts. But the understanding attained from this body of knowledge will depend on the individual.
Clear writing does not necessarily create consumable knowledge. For knowledge to be usable, it must be readable. But readability does not guarantee the information is useful to an individual.
Matthew Crawford, the author of Shop Class as Soulcraft, draws a distinction between thinking-as-doing and thinking-as-writing. What’s written too often is divorced from what the user needs to do. He argues:
The writers of modern technical manuals are neither mechanics nor engineers but rather technical writers. This is a profession that is institutionalized on the assumption that it has its own principles that can be mastered without the writer being immersed in any particular problem; it is universal rather than situated. Technical writers know that, but they don’t know how.
The problem, as Crawford sees it, is that the writer does not necessarily have a firsthand understanding of the topic they write about. The writer has thought about the topic abstractly, but without direct experience.
When confronted with explanations that don’t make sense, the user must:
render the gibberish coherent, and he can only do this by referring it to a model he had in his own mind of how the thing works.
The key to promoting understanding is the mental model framing of the discussion. It must be accurate and intelligible to the user.
Writing generates qualified knowledge. The writing process can help writers understand a topic. It can help them clarify the relationships between issues and encourage them to develop precise terminology to describe concepts.
Writing is, without question, valuable for developing an understanding of complex issues. But the writing will only express the knowledge of the writer. It won’t necessarily convey understanding that another reader can acquire. The knowledge articulated must be further revised to match the expectations and understanding of external readers.
The writer must decouple how they arrived at a conclusion from how they communicate that conclusion to others. Their goal should not be to show their own thinking process, but to address how the reader is likely to evaluate the topic.
The writer will normally have different background knowledge and motivations than the readers seeking to understand a topic. While in some fields (scientific research, for example), both the writer and the readers will follow a common, standard process for arriving at conclusions, this pattern is not the norm in most cases. Instead, the writer is presumed to have more expertise than the reader and will be concerned with factual and logical details that won’t be relevant to the non-expert.
With LLMs, we find that responses to user prompts often replicate the expert’s explanatory logic and terminology that was expressed in the source content, even though these are inappropriate for the user.
Explanations or conclusions without rationales sound authoritarian. Users are exacting: they want expert opinions, yet don’t want to put up with expert-level explanations. At the same time, many users find explanations lacking rationales less credible. They question why they should believe what’s being said.
Public trust in professional experts has been eroding with the “democraticization” of information. People don’t like being told what to do when they feel they aren’t permitted to make the decision themselves. Users feel empowered and entitled to form their own conclusions. Yet LLMs don’t always provide an extensive rationale for their responses. The tendency of chatbots to provide “instant answers” can undercut their credibility.
Knowledge sources (justifications) must be transparent and easy to trace. Users want to know where information comes from. AI solutions make tracing sources difficult.
The major generative AI platforms provide limited links to sources in their responses. A response might contain 5-8 links, often to obvious sources such as Wikipedia. At best, these links provide a shallow overview of available knowledge.
Most chatbot responses have limited source citations. Source: Generative Pulse, survey of 25 million bot links
Dense knowledge is not necessarily clear information. Knowledge graphs represent an alternative paradigm for answering user questions. They can provide better traceability, but at the expense of poorer usability. They don’t facilitate deeper understanding.
Because of their complexity, knowledge graph implementations increasingly rely on natural language chatbot user interfaces. When using a chatbot to interact with knowledge graph data, information traceability is lost. Google’s Knowledge Panels, which aggregate information from multiple sources, require users to click links to perform new searches to trace the origins of the data shown.
Google Knowledge Panel: most links send users to another search, which might yield another knowledge panel. The answers and links raise more questions than they answer. To improve ease of use, traceability is sacrificed.
Just because the writer can explain something doesn’t mean they understand it. It’s easy to make statements and to come up with reasons why readers should believe what you’ve said. But harder to justify statements with a sound rationale. Many assertions fail to withstand scrutiny.
Writers and AI platforms are both aware that credibility depends on providing a rationale. Writers have long used formulas such as “six reasons why….” to buttress their assertions. Professional writers might follow a template that outlines methods, procedures, and evidence before presenting conclusions. AI platforms reproduce these formulas in their responses.
It’s tempting for users to see formulaic explanations as valid because they match expectations. But in many cases, especially with LLM responses, the justification is developed after the conclusion is generated. Chatbots explain things without understanding them.
A widespread myth about LLMs is that they can reason. Vendors promote the “Thought-Action-Observation Loop” technique and ReAct (Reasoning/Acting) prompting, suggesting that LLMs are reasoning. These techniques build on a prompt-engineering approach called Chain-of-Thought (CoT) reasoning, which supposedly shows the thought process of the LLM. However, more detailed research indicates that these measures are largely proformative. They slow down responses without yielding appreciable improvements in answer quality. On the contrary, the techniques can make answers less consistent.
The core problem is misplaced trust: CoTs can appear persuasive even when they do not faithfully reflect a model’s actual decision process
Explanations can be myths. To enhance perceived credibility, communications aim to sound expert and appear to be based on a comprehensive review of available information.
But mimicking common arguments does not make the arguments valid. Few people apply critical reading online, parsing claims, reasons, and evidence.
One of the most reliable and relatable forms of communication is the story. Stories often rely on reasoning by analogy. Chatbots have access to countless examples of material that are vaguely similar to the user’s prompt. We shouldn’t be surprised if we start seeing chatbot explanations including sports or TV show analogies.
Simplicity used to be a primary goal for the design of products and processes. That was before AI.
In recent years, people have given up on making tasks simpler.
Everyone used to be obsessed with removing unnecessary effort because complicated processes were confusing and slowed us down. Organizations focused on streamlining their processes. Individuals prioritized streamlined options. When processes were simplified, people could understand what was happening.
Now, AI makes any task seem effortless. The motivation to remove complexity is waning because people no longer have to experience the complexity themselves. A bot will take care of it.
We can hide from complexity only so long before it comes back to bite you. AI is having the perverse effect of generating even more complexity, which will sooner or later AI users will have to face. Our AI practices are creating environments that are too big to understand.
The curse of ‘AI agent sprawl’
The Wall St Journal’s “CIO Journal” column last week proclaimed a new worry in the enterprise: “AI agent sprawl.” Companies are choking on thousands of rule-based bots.
Agent sprawl resembles “shadow IT” in that users are making their own IT decisions in an uncoordinated manner. But unlike with shadow IT, users are responding to their leaders’ mandates to adopt AI as quickly as possible.
Screen cap: Wall Street Journal
The article points out that creating instructions for AI is so easy and seemingly cost-free that workers can’t refrain from writing more and more of them.
The proliferation of agents is causing chaos in organizations. The growing volume of complex instructions is becoming unmanageable. Some firms already have over 10,000 agents running and are expected to have over 100,000 agents within a couple of years.
AI agents have quickly become the latest form of technical debt. Who is reading all these instructions and deciding if they even make sense? Don’t expect bots to manage themselves.
Employees are strongly incentivized to build agents. They can quickly create new agents in the hope that offloading tasks will make their work easier. What’s less obvious are the costs of this behavior, both for their organization, and for themselves.
Perils of agentic cookbooks
Crafting instructions for AI agents is similar to creating a recipe, we’re told. Some AI platforms and frameworks even use the term recipes for instructions.
It turns out that cookbooks can teach us lots about why recipes can be more complicated and confusing than they first appear.
A few months ago, I gave my wife a cookbook called “On Vegetables” as a gift. It’s written by a celebrated chef, and subtitled “Modern Recipes for the Home Kitchen.”
Simple title, simple focus. And the recipes looked simple – the layout filled with calming white space. But it didn’t turn out to be so simple.
This recipe, “Brassicas a la catalan”, looks appetizing. The list of ingredients appears short, as do the directions. But this two-page spread is deceiving. This recipe has agentic properties: hidden subtasks that make it more complex.
First, the cook needs to obtain brassicas. If I were to ask my local grocery store, I would get a blank stare. There is no brassica sign in the grocery produce section. The recipe’s instructions are unclear about whether broccoli or cauliflower can be used.
But let’s assume we’ve figured out how to source the brassica component. We now have to prepare it.
The cookbook tells us to trim the brassica, following the instructions elsewhere in the book on pages 40-41. These hidden instructions aren’t obvious at first glance. And they require careful reading; the trimming instructions apply to vegetables other than brassicas.
At least there aren’t many other ingredients. Yet, one of the eight ingredients listed is Pine Nut Pudding. Where do you get that? You have to make it. The cook needs to flip to page 301 to learn how. The Pine Nut Pudding involves another half dozen ingredients and perhaps another hour of work. When you are finished preparing it, you can return to the main recipe to spoon it on top.
My wife is ready to donate the cookbook to our local library’s book sale, to benefit someone with more patience.
To be successful, recipes must be clear to users. Instructions require usability testing.
The recursive, nesting instructions in the cookbook are emblematic of how agents operate in practice. Each AI agent involves a series of instructions that often call upon other agents. They are difficult for humans to follow.
Simplicity in design
Simplicity has been a guiding principle in product design for many years. The celebrated designer Dieter Rams famously said, “Good design is as little design as possible.” Apple based its brand positioning on simplicity, striving to offer products that were intuitive and that communicated obvious value.
The design thinker John Maeda wrote a book about simplicity in design called “The Laws of Simplicity”, before he became a senior Microsoft executive promoting AI.
AI developments have displaced professional interest in simplicity as the focus has shifted away from traditional user interfaces. It’s worth revisiting why simplicity matters to users.
Simplicity brings clarity to the user. It makes the available functionality transparent. A simple design:
Focuses on the user’s main goals
Reduces the number of options to enhance the user’s focus of action
Tightens the connection between the execution of the task and the goals realized
When designers talk about making digital products easy to use, they are referring to cognitive effort, not physical effort. If designers wanted to reduce the number of clicks required, it is not out of a desire to eliminate the physical effort of clicking, but to improve how users connect their actions to outcomes.
Simplicity enhances efficiency. Reducing complexity allows one to use less effort while achieving the same outcomes. Again, the emphasis has been on the mental effort required to understand a process end-to-end.
Simplicity has always been hard to achieve. If simplicity were easy, it wouldn’t be cherished.
Pursuing the simple, design teams would focus on streamlining user flows, removing low-priority options, and adopting “lean” user-experience practices. We marvel at simple designs that just work and wince at gadgets with too many buttons and features. Yet despite its elusiveness, simplicity has been considered a worthy goal.
Simplicity helps both the designer and users focus on what really matters. It separates what’s necessary from what’s merely nice to have. It recognizes that having too much in the design detracts from its overall value.
As a goal, simplicity forces the designer to distill requirements to their essence and omit unnecessary details in the solution. The goal isn’t absolute universal perfection, but getting the basics perfect.
AI simplicity is superficial, not genuine. AI platforms promise users simplicity: just tell the bot what you want in natural language, and it will respond. Creating an instruction is as easy as writing a description, typing the steps to follow, and saving it in a file for future use.
But as AI platforms expanded their capabilities, compensating for their inadequacies and adding new features, the promised simplicity has disappeared.
AI tools offer a false sense of simplicity. They make it easy for a user to set up a rule for a bot to follow, but make managing these rules a burden to users.
John Maeda recommended that to promote simplicity, designers should subtract what’s obvious and highlight what’s meaningful. Doing so helps the user focus on what’s important. Yet bots need to be told the obvious, and struggle with the meaningful.
Overspecification is the antithesis of simplicity
Before AI, designers obsessed over constraints: costs, available space, and, most importantly, the user’s cognitive load.
After AI, constraints don’t matter anymore. Adding rules to agents seems cost-free. There are no worries about space. And bots have a limitless capacity to process rules.
AI users feel empowered to act like commissars issuing diktats to bots.
And as a result, bot instructions have been overwhelmed by bloat. It’s led to overspecification: too many instructions relative to the output sought. In other words, the value of the output is less than the cost of the instructions needed to generate it.
Rather than provide a clear focus, overspecification confuses and complicates. It strives to optimize minor issues that are of limited relevance to the task at hand.
Rules, assumptions, and examples all generate complexity. Even if bots can manage these additional instructions, it doesn’t follow that users can. As instructions grow, clarity for users gets murkier.
AI instructions keep growing because AI products lack a defined scope. AI users have trouble recognizing that their instructions are not useful in most situations, because they are preoccupied with a specific situation where the instruction might be used.
The accumulation of instructions is a symptom that bots do not understand the user. The need for instruction is minimal when the receiver has knowledge – an actual understanding of the situation. In the case of bots, even the basic requirements that would be obvious to a human need to be enumerated.
Cruft in AI instructions
Bots are clueless without instructions. AI platforms have developed a range of nomenclature for segmenting these instructions, such as projects (background knowledge), skills (procedural instructions), and subagents (domain-specific agents).
These setups can become Byzantine in their configuration and organization. The technical term is cruft.
Users create files filled with examples to follow, rules, intentions, and directives. While these statements and files vary in their degree of explicitness, they all provide instructions for bots to follow. Users can even add qualifying conditions for when to use an instruction or what threshold to apply.
The prevailing design philosophy for creating AI specifications is one of rule hoarding: stockpiling more and more rules to ensure the bot delivers what you hope for.
It’s easier to add rules than to delete them. Once you add a rule, and it starts being used, it isn’t clear what might happen if you later delete it. AI instructions become a black hole: instructions keep getting added and disappear into a mass of files in folders. No one remembers what is contained in them all.
I’ve watched this process at work with content instructions. Teams will develop hundreds of pages of instructions for how bots should write articles. It includes usage and formatting guidelines, templates, examples, word lists, and business rules to cover any situation that has arisen in the past or might arise in the future.
Teams are proud of their extensive documentation of AI instructions for content generation and their ability to wrangle bots to behave like human writers. But they have trouble acknowledging the absurdity of needing 500 pages of instructions to generate a 500-word article.
Bots don’t learn files of instructions and become suddenly wise; they must load these files every time they generate content. And most of the instructions will be irrelevant to the task at hand. Compare that to the human writer, who will only occasionally need to consult such documentation. Unlike the bot, the human has learned what’s necessary and knows what should be prioritized.
At the extreme end of bot triumphalism is tokenmaxxing, where using AI is the goal itself. Employees behave as if using bots intensively automatically improves outcomes. Firms promote the creed that AI usage is inherently productive, meaning that the more AI is used, the more productive employees are. Firms like Amazon publish employees’ AI usage data to pressure them to use AI even more, which encourages extraneous use of AI.
How AI is killing simplicity
AI responses are prone to slop – that ineffable quality that comes across as cringy. Left to their own devices, bots generate outputs that aren’t socially acceptable.
AI users lean on rules to control the slop. They may hate rules constraining their personal lives, but when it comes to bots, rules are wonderful. It’s rare to hear AI users talk about bad rules. Instead, their focus is on adding more rules.
AI has created an ironic paradox: it’s never been easier to create complexity.
Detailed instructions appear to save effort. But in practice, they only divert effort.
A bad reason to create an instruction is to spare yourself the annoyance of having to do something again that you didn’t enjoy doing before, without first asking how critical the task is. Defensive instructions can become a source of technical debt; specifications with little business value will clog processes.
When instructions are numerous and spread across multiple files, users can’t see or understand which instructions guide an AI agent.
AI users don’t need any special talents to make instructions complicated. They only need to focus narrowly on their individual goals (which may be unique to them) and have enough fear or imagination to write instructions about uncommon situations.
“Just in case” instructions are a common type of AI instruction. Creating them, on the surface, appears prudent. And the user will rarely be criticized for wasting AI resources. Organizations encourage employees to use AI; the more, the better.
The capability of agents to generate complexity is an example of what economists call the Jevons paradox: output “efficiency” drives up demand, making the process more resource-intensive because it seems “cheaper”. Microsoft’s CEO Satya Nadella said: “As AI gets more efficient and accessible, we will see its use skyrocket, turning it into a commodity we just can’t get enough of.”
In the case of AI agents, the efficiency claimed is the perceived ease of specifying instructions. By removing barriers to creating rules (compared to traditional programming), platforms encourage the creation of more instructions, which in turn generates more complexity for users navigating existing instructions.
The additional complexity seems cost-free. In reality, however, agents burn more tokens processing instructions addressing an ever-broader range of issues. And data centers burn more fuel to process these tokens.
What’s easy to do can undercut what’s wise. It’s easy to create a long list of rules, which, to a bot, are all equally important. Rules are created without testing their value to users or their criticality to the business. It requires much more thought and effort to create shrewd instructions that make a critical difference.
Keep rules simple
Rules are necessary to place constraints on bots. But it’s easy to overdo them. And it’s also easy for instructions to become constraints on users, who need to understand what past instructions say and do.
If instructions become so numerous that they are hard to understand, they’ll also be hard to maintain. It becomes less clear which instructions are necessary, and which ones are superfluous or redundant.
AI users deserve simplicity just like users of other digital applications. Yet AI platforms keep getting more complex, introducing new features that encourage users to configure yet more complicated setups.
How can simplicity be promoted? How can instructions be clear, efficient, intuitive, and focused?
Instructions for bots should be considered a user design issue. As with other user interfaces, achieving simplicity is not straightforward. It requires deep thinking and vigilance to prevent erroneous requirements from detracting from essential priorities.
Aim for simplicity in automation. Focus on requirements that are always true. Don’t try to automate issues with many nuances and edge cases. Only automate predictably repetitive tasks, not jaggy ones. Automation is elegant when it is simple. It’s klutzy when complex.
Focus on high-priority requirements in your instructions. Make sure instructions are clear and unambiguous so they get the most critical issues right. Ask whether the instruction is consequential for the user or the organization. Accuracy is more important than style. Prune those “nice to have” instructions; they tend to be subjective. If instructions risk making things confusing, skip using an agent; don’t try to perfect it.
Keep humans in the loop — and at top of mind. Bots make mistakes and will continue to do so for the foreseeable future. Plan for people to be involved in checking and adjusting outputs. It’s easier to troubleshoot mistakes when rules are simple rather than complex. Don’t be fooled into believing that more rules will prevent mistakes from happening.
Avoid end-to-end “solutioning”. It’s tempting to imagine the bot can flawlessly take over a complex task, but that’s rarely the case. The more complex the instructions and the wider their scope, the more brittle they will be.
Practice economy and transparency in instructions. Reveal what the agent will do in your instructions. Don’t hide instructions by subdividing tasks and creating subroutines. AI is artificially cheap now. Price hikes and usage limits are coming. Larger sets of instructions will either trigger higher monthly plans or max out existing plan limits. If you had to pay for this yourself, would the instructions make sense if charged $500 a month rather than $20 a month?
Don’t substitute rules for knowledge. Bots must be told everything that might be relevant. Many times, it’s inefficient or even foolhardy to try to recreate the user’s knowledge in the bot’s instructions. Leave complex tasks to people with the knowledge to make the right judgments.
For some AI users, I expect these guidelines may sound unsophisticated. They seem to limit the potential of AI capabilities. I am less concerned with what AI technology can do than I am with what humans can manage. From the human perspective, it’s possible to overuse AI.
