AI platforms promise instant answers, generated in real-time as you ask a question. Chatbots seem poised to render stale, out-of-date webpages a relic of the past.
Yet chatbot users aren’t getting fresh answers. They are getting “chatbot theater.”
Previously in this series, I have looked at how AI platforms can mislead users. In this post, I focus on a specific dimension of chatbot misinformation: how they present the recency of their answers.
The urgency for current information
Online readers more and more prioritize real-time information. Given the rapid pace and unpredictability of events in business and society, information published online risks being overtaken by events.
People require fresh information – old information can be misleading.
First-hand news is often fresh and immediate compared to third-hand accounts. For example, social media posts announce what people just saw or things that just happened to them. Traditional online publishers like corporate customer service departments or newspapers are slower to reflect changes, if they ever acknowledge them at all.
Despite the appeal of real-time information, most of us don’t receive information exactly when it’s published because we don’t need it at that moment. Only certain types of information (sports scores, stock prices) are suitable for a real-time feed. In most cases, our goal is to minimize the time gap between when we need information and when it is produced. We aim to maximize the recency of information that matches our interests.
Online forums are a place for breaking news
Online forums can be an ideal source for recent information in many situations.
Forums don’t just host consumer rants and raves. Forums have become the frontline of service for many businesses, serving as an essential platform for both customers and employees to get answers.
Customers rely on forums for product and service recommendations, as well as post-purchase support and self-service.
Businesses rely on forums to collect comments from customers and employees. Tools like ServiceNow and Slack capture the first-hand experiences users post online. Enterprises are exploring ways to integrate forums with AI for issue monitoring and resolution.
Forums play an overlooked role in online content ecosystems. They deal with unplanned and emergent information — the very kind of recent information people might need to know.
Forums disclose disruptions associated with change. Planned announcements introducing a new product or benefit can be broadcast in a press release. Forums, by contrast, tend to deal with unplanned changes.
For example, a software update might fix a problem – or create a new one. Supply chain changes might impact product reliability. New management might alter service support. Customers encounter many unannounced changes — changes that will only generate content once customers notice them.
Now, chatbots seek to replace online forums by offering real-time information generated as soon as people pose a question. It’s an enticing prospect, but unfortunately a deceptive one.
Disaggregating information origins and delivery
Information needs to come from somewhere. Let’s refer to the original source of information as first-hand news. It reflects what an individual with intimate knowledge of an event posts online.
First-hand news is not always accurate or complete, but when it’s first posted, at least it’s fresh.
But how do we get first-hand news (fresh information), and at what point does it become third-hand news (stale information)? The delivery channel shapes this process of revelation.
First, let’s look at how news arrives in an immediate delivery channel such as a feed or a notification. Here, people receive information as soon as it is posted. There is no difference between how fresh people perceive the information and the actual age of the information.
Next, let’s consider how a forum works. Some people post fresh news in forums, and readers may get a notification, which delivers a nearly real-time experience.
But more often, forums deal with questions and answers rather than announcements. One person asks a question, and another responds based on their experience and knowledge. Even if the question and answer exchange happens quickly and have the same posting dates, the timeframes of the question and answer can be quite different. The questioner typically will ask a question relevant to their current needs, while the answer reflects a past experience. The answer conveys a first-hand experience in the past: a situation the person encountered previously that seems relevant to the current question.
Q&A forums are hosted in an archive, which can be searched. In this situation, we introduce a third party, the searcher, who is drawing on the previous exchange between the question poster and the person answering. Rather than ask a question themselves (if they have that privilege), they try to determine if someone has already done so. It’s generally good practice (and socially expected behavior) not to ask a question in a forum that’s already been raised and answered.
When searching for answers in a forum, the searcher encounters two timeframes. They see a past Q&A exchange and tend to view the posting date “timestamp” as indicating when the information was current.
But in reality, the basis of the answer posted may be an even earlier experience. If someone asked how to do something, the answer may refer to the process used the last time it was done. If someone asks whether something is possible, the answer might note that the respondent tried it once in the past and how it worked out for them.
Each party (seacher, question poster, respondent) is associated with a different point of time. For example:
- (Now) The current information seeker looks for answers by doing a search
- (Last year) A similar question was posted in a forum in the past. It appears to the searcher as if this timestamp is the date of the information. But the answer is based on an earlier experience.
- (Two years ago) Respondent had a similar experience related to the question posed in the forum. The far past is the actual basis of the information.
We can see that the date the answer was posted is not the true age of the information.
Finally, let us consider how chatbots use this information.
Chatbots don’t (yet) have the privilege of asking questions directly of people in forums – they can only answer questions and often rely on previous forum answers to do so.
Chatbots generate answers that are essentially rewrites of previous answers.
From the questioner’s perspective, the chatbot appears to be generating real-time, up-to-date information. But in reality, the answer reflects old Q&A conversations. The underlying information could be based on first-hand experiences from the distant past. Yet, because the questioner does not see the provenance of the information, they are inclined to perceive it as current.
AI platforms obscure the age of information
AI platforms depend on the answers people have contributed in the past. But regrettably, they commonly fail to reveal the basis of their answers.
AI platforms confuse the picture by emphasizing an LLM’s “cutoff” date (they won’t know about events after the cutoff date). They imply that the crawl date is the primary factor in determining content recency.
Yet, bots now crawl the web frequently to update LLMs, unlike when they first launched. The crawl frequency creates a false impression that a chatbot will provide only the latest information.
Chatbots struggle to indicate a clear date as of when the information was current. Clarity of time depends not on the date of the last crawl, but whether LLMs can understand the temporal context of the information they crawl. Unfortunately, they can’t.
The root problem is that AI platforms position themselves as the source of information rather than the referrer of information from other sources. They conceal the source of the facts and thwart users from seeing the context of the original information.
Being dependent on legacy web content, chatbots are unable to generate fresh information. They are stuck rewriting existing information. But they make this rewriting seem as if it provides real-time information. In doing so, they undercut the credibility of the information they offer.
– Michael Andrews
