Many chatbot problems stem from their inability to understand context. In earlier posts in this series, I’ve discussed how aggregation flattens nuances in individual statements and how scraped content can disregard the timeframes for which the original source statements applied.
This post explores how user context affects the statements LLMs use to generate answers. It argues that essential context is routinely omitted from statements crawled by AI platforms and, as a result, is not included in chatbot responses. Notably, chatbots don’t consider the point of view of sources expressed in statements they draw upon.
AI platforms harvest online information that’s been stripped of its original context. Bots omit essential context by ignoring the role of the source posting the information.
Information accuracy is often highly contingent on its circumstances. While most online information was reasonably accurate at some point, it may be accurate only in specific circumstances. It can be described as “yes, it is (or was) true, but only if or when a particular circumstance is true.” These qualifications extend to who is making an assertion and what their role is. Although people do lie online, the bigger problem is that they misunderstand and miscommunicate. Bots struggle even more than humans with these ambiguities.
When assessing the credibility of information, readers must consider the circumstances of the person providing the information. They are interested not only in who said something, but also in their role.
We are accustomed to distinguishing between primary and secondary sources from years of schooling. We separate direct statements by people from indirect ones, where they are quoted or summarized. We focus on who said something.
Google recommends users search for information about sources they find online.
It’s important to look beyond the naive idea that sources have either a good or a bad reputation. Many platforms make simplistic assumptions about whether a source is trustworthy, without regard to the scope or domain of the topic. Contrary to SEO folklore, authority online isn’t an attribute of a website; it is intrinsically related to the topic of the content itself.
People and platforms should look more broadly at how information originates.
First-party and third-party information are similar to primary and secondary sources in that both concepts distinguish different categories of sources. But the concepts are slightly different. Instead of focusing only on who said something (the source), we also consider their authority to speak about what is said (the information).
In online forums, that rich source of advice, reviews and updates, first-person observations can be third-party information – someone’s interpretation. For example, John might post in an online forum that the IRS doesn’t allow a certain deduction because he wasn’t able to take it himself. But John doesn’t work for the IRS (which isn’t noted for posting helpful advice in online forums). He is only conveying his personal experience. The issue is not necessarily John’s credibility or knowledge – he’s candid about what he knows, as far as he knows it. And read carefully, John’s post may offer useful information for understanding how some taxpayers are able to take deductions or not. But John’s post can’t be taken as the universal truth.
First-hand statements are not first-party information unless they are made by someone who works for the organization that decides the information. An individual’s views can be first-hand and appear credible but not authoritative, as they involve interpretations, opinions, or experiences. Statements can be true as they relate to the individual’s circumstances, but not be correct if taken as global statements that apply to all situations.
Information provenance leads to an important qualification: eyewitness accounts are not the absolute truth.
This scepticism challenges the widely cherished idea that first-hand experiences provide the unvarnished truth. But in reality, experiences expressed online offer at best a limited truth that’s constrained by the circumstance of when, where, and who said it.
Chatbots can’t discern the context of the information they crawl. Even Google’s Gemini chatbot doesn’t follow Google’s guidelines for humans to investigate “why it’s sharing that info.” Gemini offers a blanket disclaimer, “AI responses may include mistakes.” It’s up to the human to figure out if the chatbot made mistakes and what those mistakes might be.
Chatbots have trouble distinguishing between third-hand and first-hand information. I’ll return to an example I raised in an earlier post in this series about finding a vegetarian restaurant while on vacation. Platforms scrape reviews, which can be misleading when someone mentions the word “vegetarian” in passing, even if it’s just a general comment. That’s an example of the unreliability of third-party information. The restaurant never made this claim.
Every time third-party information is used, someone else’s assumptions are being applied.
If platforms were scraping restaurants’ menus and could decipher which dishes were vegetarian, they would be relying on first-party information. If, however, the platform were deciding if the dish was vegetarian based solely on its name, we would be back to third-party information. The bot interprets menu names using third-party information to determine whether a dish is vegetarian. But many vegetable dishes have bacon or chicken stock in them, which won’t be apparent from the name of the dish. So even with first-hand information, the full context may be missing.
Textual declarations seldom explicitly qualify the limitations of a statement – the reader is expected to infer any limitations from the context in which the declaration is made. Bots, however, tend to decontextualize statements and make them into universal ones. Bot-generated statements derived from crowd-contributed content are often misleading.
Your experience may vary
The source’s identity will reflect their role: what matters to them and what they know about a situation. Various people can make statements that are inconsistent but nonetheless valid for them individually.
Online forums are where people share stories about themselves. A person will write in a forum about “what I did, and what worked for me”, with little initial consideration of how readers might be in different circumstances. Such egocentricity reflects the incentives and motivations of crowd-contributed forums. People enjoy talking about themselves and believe they are influencing others to emulate them. They enjoy getting praise and recognition when they post something deemed notable that hasn’t been seen before.
The individual posts that bots crawl contain sampling biases (the advice in each post is a sample of one). People write about what they did – what they considered and tried. Rarely do they write about having tried all possibilities and evaluated them. The information is selective.
When all parties view communication as a point-to-point exchange, each party strips out the context they deem unnecessary. They emphasize what they want to know rather than spending much time discussing what others may know. The information tends to be personal.
The writer of advice and the seeker of advice can have different preference profiles. The “best way” to do something depends heavily on the situation and individual preferences. For many tasks, determining the best approach can be challenging without understanding who, when, and why someone wants to undertake the task.
The challenges of human communication are magnified online, where distance in time and space makes clarification and qualification of statements much harder.
Even with these challenges, many forum participants want to help and may clarify statements in subsequent threads, especially when questions arise.
But bots crawl online forums with a more acquisitive agenda. They are indifferent to the discussion’s context. They simply want to harvest statements made. Whereas humans may engage in a close reading of the discussion, bots engage in a distant reading of it.
The problem is that much of the context shaping what’s said online is never explicitly stated, and if it is revealed, it may be noted later in the discussion.
Where context is omitted, gaps in understanding emerge. The writer’s context may not be transparent (even to the writer). The reader’s context – their preferences and circumstances – may be unknown to the writer. The bot, driven by its mission to scrape the discussion, is indifferent to the context.
The phantom of contextual AI
The omission of context in crawled online content poses a formidable challenge to the growth and development of AI.
The latest wave of AI development is focused on agents that use the Model Context Protocol. Context is essential for AI, but chatbots can’t supply the context needed.
There’s no simple fix for the omission of context in online information.
Content professionals often champion the importance of context in supplying relevant information. Many argue that contextual metadata should be added to source statements to enable bots to provide high-quality answers. Approaches such as GraphRAG are having a moment. Although commendable in principle, applying context to online content after it’s been written is difficult in practice.
Online content, particularly forum discussions, is not written for machines. People are writing for each other – in some cases, telling stories to themselves. The writer may be blissfully unaware of the limitations of their pronouncements and how those pronouncements reflect their personal biases.
Bots can’t detect the possibility that the facts of the matter may be specific to what the individual experienced in a given context. Omitted context can’t be auto-magically restored.
Yes, some context can be applied after the fact with automated tags. Yet, realistically, much of the context of online content requires close human reading to infer. Bots process text superficially, relying on relatively crude tools such as keyword and entity recognition, which are no match for the inherent ambiguity of most online discussions.
– Michael Andrews
