Categories
Creating content

Extending notations for writing composition

Our dominant mindset about technology is one of machine learning: people train computers, and computers learn from people. Eventually, computers will take over tasks that people do.

But suppose instead that humans could learn from how computers solve text issues to help them make better writing decisions, rather than deferring to an algorithm.

For computers to teach writers something of value, writers must decipher their workings, which are hidden in code. Notation in code offers a window into how text is handled as it evolves.

This post focuses on the role that notation plays in shaping text drafts. I’ll argue that notation can play a useful role in the writing process, provided it is not tightly coupled with specific media or tools.

Patterns and relationships in drafting

Writers imagine their relationship with computers, complex as it is, as one of teacher and pupil. Writers lead by example, computers learn. Writers hope to coax the computer to do what’s desired.

Yet writing is messy, and not an easily replicable process. Computers can only learn so much. That’s because writing involves many judgment calls, which only a human can make.

The writing process is a dialogue between the writer, the draft, and the imagined reader.

As the dialogue about the draft unfolds, fresh thoughts emerge. Writers and reviewers make notations in the draft for follow-up.

Editors apply blue pencils to typescripts, leaving proof-correction marks to indicate suggestions and changes. In computer manuscripts, reviewers layer ghost-like comments that hover in balloons.

Developing a text involves notational practices — letters, symbols, and marks. Writing notation is a kind of pattern language that evolves organically from conventions adopted by various groups over time.

Software applications often play the role of a third wheel in this dialogue, demanding attention, suggesting rewrites at inopportune moments. They aren’t seen as helpful contributors or respectful partners, but as an undisciplined force that needs taming.

Unfortunately, as computers have become more intertwined with the writing process, writers have become more dependent on them. The writing process becomes captured by the platform used.

While there’s no denying the influence of computers on writing, this development does not mean writers must become dependent on them. Writers should resist fatalism about the role of technology. Instead, they should consider how computers can inspire new practices, rather than mandate them.

Learning from customs in computer code

Does something that can often be annoying offer anything it can teach us?

I’ve been looking at how to appropriate some of the intentions behind software conventions to support the writing process. I seek to uncouple the underlying purpose of these protocols from their execution in random-access memory. I want to divorce the ideas expressed in the software from the systems that run it.

The code typed in a text editor incorporates a range of notations. These conventions evolved to solve specific problems in the computer domain — identifying and indexing snippets of text or strings of characters for further action without causing confusion.

The routines that computers follow are best geared to deterministic processes, not compositional writing. Yet they can be useful outside of the computer programs they were originally designed for.

Writing code has discernible similarities to writing prose. Both demand attention to the placement of and interaction between strings of letters.

Computer notations have evolved over time, with various computer languages and programs borrowing from one another. Software programming has existed long enough to establish its own traditions and customs. These languages express vernacular practices that writers can learn from.

Certain coding signposts can be useful for writers, too, even when writing by hand. Borrowing computer conventions does not turn writers into robot apprentices. Writers understandably don’t want to become even more beholden to computers.

Writers need notation practices that work with both handwriting and the keyboard, so that notetaking and drafting are independent of the platform, and so the writer stays in charge of the process. That hasn’t been the case with most writing tools.

The most popular notational systems for text, Markdown and similar formatting systems (Asciidoc, Typst), are designed to shape the layout of text on the screen, not to shape ideas. Too much emphasis is given to formatting already-composed text rather than to how to represent ideas and intentions while composing drafts. What’s needed is notation that facilitates the writing process.

Writing notation to help you to think through issues

The writer’s draft undergoes a long gestation as a work in progress. During this time, it is tentative and incomplete. It can be hard for the writer to track what is happening in the draft.

Notations are a way to indicate what remains to be done, or what themes are emerging. I’ve discovered five computer programming conventions are worth stealing for writing.

Placeholder for the right word

$variable Combining the $ sign with a word signals that the specific thing mentioned is not important. Rather, the statement around the variable is what matters.

Writers can use a variable to indicate that what is said is true, independent of context — or to test the consequences of alternative variables in a statement. For example: In the United States, $person has the right to due process.

The next step is to define the variable with words. In the sample sentence, does $person refer to anyone, US citizens, or some other description of a person? I find this exercise helps me distinguish whether it is the actor or their role that’s most important.

Another use of the $variable is to write about something before you have locked down the terminology you intend to use across a document. I find that writing about abstract concepts is difficult because any label for the concept is a metaphor that can be interpreted in more than one way. By not worrying about the final terminology, I can see how I write about a concept in different contexts, which helps me understand how the term I choose needs to work in different situations. If I were to start using random synonyms as I draft, the concept I am referring to might drift in its meaning.

Who said or did that?

@citation The “at” sign started as an accounting convention indicating “at a rate of.” It was adopted for use in email addresses to signify the addressee’s domain. In messaging apps, using the @ sign indicates a mention of someone’s name.

In writing, @ can indicate a citation of a writer’s work. For example, @Smith 1776 refers to Adam Smith’s Wealth of Nations, published 250 years ago. This citation can be embedded in text, such as: @Smith 1776 argued that it would be reasonable for the rich to pay higher house-rent taxes.

The @ symbol indicates a reference to a multi-line data entry, which readers don’t need to see while reading. The citation convention began with software called BibTeX, which created an entry type (e.g., @article, @book) followed by a “citation key” consisting of the author’s surname and publication date.

To align more closely with the “mentions” convention, some software uses only the key indicating who and when (the author’s surname and publication date), similar to inline references in research papers. I’m following this approach.

The @citation is succinct shorthand for noting who said something and when. But I also find it helpful when citing events driven by people, to build a timeline (you can add month and date as YYYY-MM-DD if needed). It can be applied to any kind of actor: a named individual, an organization, or a group. This usage again is similar to the “mentions” convention. You are citing someone’s activity rather than quoting them.

What are the key concepts?

#topic The hash, or octothorpe, is a mark used to indicate comments in computer code. It later came to signify a “tag” indicating the topic of content. The hash can be added to a keyword in a sentence to allow the sentence to be indexed by that keyword.

Hashtags are less common in social media posts these days, but they remain useful for notes. I find that adding a hash to a word is helpful when the word embodies an important concept I want to follow. The word might be a proper noun (#Silesia), a specialized term with a precise meaning (#prosopography), or a word that has a specific meaning for the author using it or in the context in which it is being used (for example, #winners).

An example of hash and at signs annotating notes in an application. This example is from the open source Databyss app, which I use to index notes.

Tags provide a thematic index of your draft. When embedded in the text, they can reveal how you discuss concepts. For example, do you repeat them across the text, or fan out the discussion to related topics? Do you drop discussion of a theme at a certain point, where it might fade from the reader’s awareness?

What needs doing?

/task The forward slash has become a common hotkey to invoke a menu of commands in applications, such as the WordPress blogging software I am currently writing with. In a text document, a forward slash, preceded by whitespaces, signals a change in path.

Drafts are full of loose ends. The writer can use the slash-task notation to indicate anything that needs more attention:

  • /factcheck
  • /getReference
  • /streamline
  • /updateInfo

In AI platforms, such commands are called “skills,” describing a set of procedures or routines to follow. They are not much different from a punch list or checklist a writer might follow to ensure they don’t forget anything. For example, you might maintain a list of questions that ask how the reader would understand or react to what’s been drafted.

Once finished with the task, you can remove it from the document: /getReference.

What’s missing?

{{insert content here}} Double braces (figuratively called “mustaches” or “antlers”) are a convention used to indicate a place where instructions or text are meant to be inserted.

They are common in templating systems, where supplemental or dynamic text is added to a base template. Some systems use single braces (“handlebars”) to indicate insertion, but double braces allow writers to use single braces for other kinds of notation, especially to indicate the grouping of items that span more than one line.

Double braces allow the writer to separate different sections of an article or document, to focus on different levels of detail.

The writer’s mind shifts focus between the big picture (the argument) and the details and nuances (the explanation or justification). Double braces are good for setting aside parts of writing you are not focused on.

I tend to develop an outline or skeleton of what I’m writing, but keep the details of each section as separate files. I can indicate in the outline the issues to elaborate within double braces. Initially, I will describe the section’s intent and goal within the braces. After I’ve drafted enough of the section to understand what I want to convey, I can label it with a more formal filename.

The shift from intent to file might look like this:

  • {{show three diverse examples}} –> {{examples.md}}
  • {{connect implications to next steps}} –> {{conclusions.md}}

Digital compatibility, not compliance

Writing practices should leverage computers without restricting the author’s freedom.

The motivation is to embrace innovations that computers have introduced in text annotation, without being forced to adopt a rigid, computer-defined process. The annotations can help you notice features in your draft, whether you scan the text visually or, if digital, use a basic “find” command.

Much writing advice promotes digital-only workflows, seeking to reduce the quantity of text that must be entered on a keyboard. That advice is useful for generating boilerplate (marketing copy, tech docs, etc.). But automation diminishes one’s cognitive engagement in the writing process. Compositional writing involves a set of tasks distinct from those in document production.

Writing shouldn’t be dependent on a specific tool or platform. The writing composition process remains poorly researched, and most recent research focuses on the technology-enablement of writing through networks, online collaboration, and AI augmentation — the features app vendors want to promote. Researchers assume that because technology is changing how writing is done, it’s important to learn to optimize it in the writing process.

But are the tools good for writing? Professional writers, such as Sven Birkerts and Cory Doctorow, among many others, have criticized the distractions caused by tech features in writing tools. “The last thing you want is your tool second-guessing you,” Doctorow wrote in 2009, a decade and a half before writing about internet enshittification.

Drafting quality versus drafting speed

Numerous writers insist that writing by hand helps them to think more deeply about a topic and choose words more deliberately. Handwriting is enjoying a mini-renaissance, from the teaching of cursive again in grade schools to the popularity of keyboardless devices like the Remarkable tablet.

Don’t underestimate the value of typing your handwritten notes: rereading, rearranging, and enhancing them. But if transcribing sounds like a chore, technology can accelerate the conversion to digital. Handwriting-to-text (HTT) is becoming more robust and widely available thanks to advances in AI, making paper-first a viable writing workflow.

For some other writers, a manual typewriter provides the friction to slow down and reflect before writing. While non-digital tools will never win in terms of speed or output quantity, they may foster higher-quality results for some tasks and some people. I can think faster than I write or type, but I find my thinking can be scattered when it’s a stream of consciousness. (For that reason, I never use voice dictation for non-trivial writing.) Now that AI can generate text faster than any human, speed is not the goal. Quality is.

I generally take notes by hand during the early stages of forming ideas. Handwriting helps me capture thoughts in a freeform manner, without deciding on their relationships yet. Handwriting frees you from making premature commitments. I can focus on concepts and ideas before worrying about how they fit into sentences and paragraphs. Computer screens, by contrast, force a linear ordering of material.

While I don’t typically draft complete articles by hand, I find it helpful in the early stages to generate options for the points I want to make. I consider handwriting more forgiving than typed text when drafting views in different ways, as I try to find the right phrasing.

Sometimes I struggle to spell a word when I am writing by hand, and I realize I don’t really understand the word’s roots and etymology. In contrast to my ugly handwriting, text typed on a computer screen looks finished, even when it is still raw. The neat characters call attention to stylistic issues that need fixing rather than the substantive ones. I’d rather learn tolerance for my half-formed ideas and nurture them.

When writing on a computer, choose a distraction-free editor — free of extraneous options and premature editorial feedback. All writers, not just those who struggle with ADHD, can find that using a tool that offers integrated access to everything online can be disruptive. It breaks your concentration and interferes with deep thinking about the meaning of words, statements, and arguments.

I prefer a document editor with a focus mode, so that I don’t have to see either menu options or text formatting markup. I don’t like writing in raw Markdown, as I find the formatting distracting. I appreciate Markdown as a portable, platform-independent file format. But I loathe it as a user interface for writing. Fortunately, Markdown editors exist that hide Markdown’s distracting ornamentation.

Keeping AI optional

AI chatbots have infiltrated almost every commercial application used for writing. It’s becoming harder to write on a computer without AI intruding, making suggestions you don’t want, hijacking your intentions. LLMs operate on the premise that “attention is all you need.” AI applies that mantra not just to web-scrapped text but to human users. It is now consuming our attention and seeking feedback in the same way social media has.

To ensure that the editor is truly AI-optional, rather than AI-dominant, look for an open-source tool, which will require you to use your own account if you want to connect to AI. Open source tools are free. The notion of having to pay an annual subscription licensing fee for the privilege of writing your own words is the opposite of freedom.

None of this advice is to imply you should never use AI. I recommend keeping your chatbot in a separate window from your editor during the drafting phase. If your text is typed in a computer file, you can always copy and paste those parts you’d like AI to provide feedback on. You can ask AI to critique the strengths and weaknesses of a snippet you’ve written, for example. But you should only do so when you’ve exhausted your internal ability to make that assessment.

There are downsides to allowing AI to second-guess what you want to convey. The writing becomes less yours. Your ability to compose and critique ideas atrophies.

Writers don’t need to run all their drafts through AI for a critique. Even the most capable AI platforms won’t necessarily understand the audience you are hoping to reach, or the point you are making — especially if the material is novel or presumes a lot of inside background knowledge.

Authorship is about the ownership of ideas. Its purpose is to express oneself authentically, even if imperfectly. One should always care about the reader’s priorities and strive to address them, but never assume a bot knows them better.

AI can be helpful for final editing, catching typos and wordiness. But even in the final stages of editing, don’t forfeit control. Bots only match patterns; they don’t understand meaning, despite the claimed semantic capabilities of newer LLMs. Bots can make your writing sound slick or mimic Hemingway’s style. But don’t let the bot make a choice you won’t make yourself. You stop being the author once that happens.

— Michael Andrews

Categories
Misinformation and AI

Ghost-busting Generative AI

People are increasingly tempted to outsource their thinking to chatbots. Chatbots seem to provide answers without the hassle of wading through individual articles.   

But unlike content written by human authors, chatbots aren’t accountable for what they write. Chatbots place a greater burden on readers to determine the veracity of online statements. Yet, for a variety of reasons, it is difficult for readers to assess where information in chatbot responses comes from, how complete it is, and whether it is reliable. 

This post explores the trust deficit inherent in chatbot responses. It will identify common weaknesses in the quality of responses and suggest countermeasures that users of AI tools can take to obtain more reliable information.

The rise of phantom authorship

If you read a best-selling book by someone famous, chances are that book was written by a ghostwriter, whose role is not acknowledged. To be clear: I’m not referring to books by a renowned author; few individuals become famous because of a book they’ve written.

Camera-preening media personalities seldom write books.  But their name recognition sells books. Actors, sports stars, politicians, TV hosts, influencers, and other celebrities license their names to books produced by others.  In such licensing deals, the party that develops the product is a mystery. The putative author pretends to have written the book, while the reader engages in “suspension of disbelief,” pretending that the story they are reading is genuine.

A similar phenomenon is happening with online content: it’s becoming unclear who wrote it.  News stories, research findings, regulatory notices, cooking recipes, and health advice get reinterpreted by AI robots as “responses” to user prompts.

An example of how pervasive chatbots are becoming is the new wave of health bots. Last month, both Microsoft and Perpexity announced health chatbots.

Generative AI is similar to ghostwriting: in both cases, the writer is anonymous and lacks firsthand knowledge of the topic.

In fact, GenAI is beginning to cannibalize traditional ghostwriting. “Today, ghostwriting websites must work to advertise why they could perform their writing-for-hire services better than a machine,” notes Emily Hodgson Anderson at USC.

Now, machines do the ghostwriting.  Online content is becoming dominated by phantom authorship.

While some users are willing to accept chatbot responses as genuine, many have nagging doubts about the trustworthiness of the outputs.

The growing supply and demand for machine-synthesized content

Ghostwriting and generative content are manifestations of a broader “explainer” economy, where one party makes money by explaining the work of others.  Explainers promise to improve clarity and reduce the time required to understand topics.  The benefit they sell is productivity.

Explainers are not new.  Students have long used Cliff Notes to get the essence of classic novels — without the effort of reading them. More recently, a range of apps, such as Blinkist and Shortform, promise 15-minute summaries of current bestsellers.

Generative AI has made it possible to turn any content — audio, video, PDFs, PowerPoints, web pages, Word docs — into a short explanation.

Talks, interviews, legal documents, scientific studies, blog posts, and academic articles, each created by individuals or small teams of collaborators, can all be regenerated by AI chatbots.

Generative AI rewrites and repackages original content into various products:

  • As a summary of a single source from a single author
  • As a retelling of discrete cases, anecdotes, or incidents into a common story arc
  • As the synthesis of multiple sources from different authors

The supply of synthesized content is now endless.  It’s easy, cheap, and profitable to repackage original content. GenAI tools rely on “word spinners” that rephrase and restructure the original content to make it more “engaging” and avoid plagiarism and copyright infringement.

And users can’t seem to get enough of this effort-saving convenience.  Historically, summaries such as abstracts were descriptions used to preview content that might be read in full.  Now, AI summaries have become the end goal.

An anonymous AI-generated summary increasingly acts as a substitute for reading the original source, such as an interview transcript, research document, or a book. The summary will often be more convenient to read or access, especially in the browser or editor the user relies on.  The original source content is no longer necessary.

So, what’s the problem?

The little “labs” icon on the right is a clue

Trust in machine-generated responses

Because of the impersonal quality of online communications, trust has always been top of mind for users.  AI-generated content adds another complication to users’ concerns about trust because it is even more impersonal than articles on websites.

Traditionally. written content embodied a social contract between the writer and their readers.  The writer addressed readers, who, in turn. expected the writer to reveal themselves through their prose.  Both sides aspired to believe they knew and understood each other on some level.

Corporate online content, being more prosaic, often lacks a byline that identifies an individual as the author. In such cases, the company is promoted to the role of author.  The brand’s reputation guarantees the accuracy of the content.  Ideally, the web page includes a link or contact information so customers can reach a real person if they need information clarified.  While users didn’t know who specifically wrote the content, they at least knew a live human being was accountable for it.  The involvement of a biological entity having a heartbeat was presumed — until the advent of chatbots.

Responses generated by AI platforms lack any discernible connection to a real person.  Readers no longer have a relationship with anyone.  Users are left to imagine what is driving the dynamic between them and the responses they counter.

These anonymous responses seem suspect and untrustworthy because no person seems accountable for what’s said.  It’s unclear who’s behind the statements; asking the bot doesn’t clarify the issue.  Readers are haunted by nagging doubts. Who is saying this, really: the bot’s vendor or the firm that the customer wants to do business with?  Is the message spin or manipulation — is it generated to make me feel a certain way or to placate me?

Chatbots can seem evasive: hard to pin down, too obsequious to trust, alternately vague or over-confident.

And when users feel forced to rely on a bot, they can feel their time is being wasted.  As one comment overheard in an online forum noted: “If it wasn’t worth a human’s time or effort to write, it’s not worth the time or effort for a human to read.”

Such hesitations are not limited to bots.  People are generally spooked by things that happen that can’t be attributed.  For example, many are unnerved by the pervasive influence of so-called “dark money” funding of political campaigns and operations in the United States by anonymous donors.

With bots, missing accountability is not a choice or a bug but an inherent feature of the product.  Few users understand the mechanics of bot training, but they can sense that no one wants to take the blame if the bot behaves badly.

While chatbots can trigger low trust, they can also induce false confidence. Some users will take bot responses at face value, to their detriment.

Boobytraps in AI-generated content

A boobytrap is an apparently harmless object that is no such thing.  Despite its benign exterior, it can exact painful results.  AI responses can be boobytraps. 

AI responses appear reassuring, with friendly wording, each sentence dedicated to a single idea. The responses seem the antithesis of the convoluted double-talk of human rhetoricians. The wording of responses is optimized to disarm skepticism.

I don’t want to make people paranoid about bot responses.  Many are genuinely useful. Yet, it’s prudent not to take them at face value and question their consequences and intent.

Let’s look at some less obvious boobytraps.

Boobytrap 1: Untraceable original information

With authored content, any information not cited as originating from someone else is presumed to be developed by the author. But with AI-generated content, it can be unclear who contributed the information and who is accountable for it.

Even when phantom authors credit sources, it does not follow that they haven’t added original content of their own, or misstated original content.   There’s the possibility of embellishments — statements that sound credible but aren’t entirely accurate. AI-generated responses are like a movie that’s “based on a true story.” The user can’t be sure if the whole package is true.

The concern is not necessarily obvious hallucinations involving wholesale fabrications. The problems may involve real information that doesn’t belong in the context because it isn’t related to the sources or topic covered.  A common occurrence is when generated content conflates people or events because it “borrows” information from extraneous sources.

Boobytrap 2: Inaccurate interpretations

When you can’t read the original content, a summary’s faithfulness to the original meaning becomes important.  Translators who render the words of an author into another language have a responsibility for fidelity.  But AI-generated content can take liberties with phrasing and argumentation.

Chatbot responses often reflect a fast-casual writing style.  Friendly short-form outputs convey familiarity and reassurance, even though the concepts involved may be more complex and nuanced than portrayed.  Chatbots behave as if they can explain nuclear physics to a ten-year-old.

Chatbots are especially prone to misrepresenting original ideas in two areas.

First, chatbots will adopt simplified terminology that can erase the distinctive meaning of the original terminology. Words have specific meanings in particular contexts, but chatbot responses tend to use common words that may not precisely reflect the intent of the original terminology. The facile substitution of word-spinning can destroy the meaning of concepts discussed in the original sources.

Second, chatbot responses tend to disaggregate and nullify the argumentation used in original sources.  Instead of maintaining the sequencing and transitions of explanatory arguments, chatbot responses tend to break thematic ideas into discrete sentences, often strung together as a list of bullets.  To the user, the laundry list of statements reads like a wall of declarations, devoid of coherence.

Responses that seem easier to read are not necessarily easier to understand or more informative.

Boobytrap 3: Third-hand information 

LLMs don’t convey knowledge.  They merely repeat messages in a modified form.  And LLMs can’t distinguish primary sources from secondary ones.  All text is of equal value, regardless of its provenance. 

By the time a user sees an AI chatbot response, they are seeing third-hand information. The chatbot is regenerating text from other sources.  Most of this text will be secondary sources that explain and summarize what subject-matter experts know.  Only a small portion of the texts that LLMs rely on are primary sources written by the experts themselves.  LLMs depend on large quantities of content, even if the quality of that content is suspect. 

It’s easy to see how informational accuracy degrades as a secondary source explains what an expert says, then a chatbot reinterprets what the secondary source says. 

Yet the myth persists that AI chatbots can “reason” and, in doing so, verify knowledge.  But the reality is that, while AI chatbots explain, they have no understanding of what they explain. 

Given their parroting tendencies, LLMs are prone to reproducing at scale the phenomenon known as truthiness — statements that sound true but are substantively empty or even false.  This happens with memes and other folk wisdom that many people believe to be true, and that is widely disseminated online. 

Boobytrap 4: Hidden agendas of unnamed writers

Human readers notice an author’s point of view and their voice. They detect an author’s agenda, which is often spelled out explicitly in a forward or introduction. The author wants to convince you of something and crafts an argument or narrative to that end.

Sometimes the author’s agenda is not explicit, but it can be inferred. If the author is employed by someone or invests in some financial enterprise, we infer they have a vested interest in promoting those interests.

The phantom writer’s agenda is hidden. Human ghostwriters are hired to burnish the image of celebrity clients.  Chatbots are also expected to advance their funders’ goals.

Bots are designed to appear free of self-interest, presenting themselves as loyal, tireless servants of the user. At times, bots complement the user on their questions. Yet, as anyone with a dog will know, even a loyal companion has an agenda of its own. Man’s best friend learns to manipulate its owner to obtain desired rewards.

Unlike a dog, lifeless AI bots lack emotions and a sense of right and wrong.  LLMs are amoral; at most, they have ethical guardrails introduced by developers. Guardrails can also be imposed to censor LLM outputs, as shown in several authoritarian countries.

The agenda of AI platforms is largely commercial and not necessarily aligned with the user.  Like social media and gaming applications, AI chatbots are designed to offer users feedback and rewards to encourage continued use.  Chatbots make users feel good through flattery and convenience. Users marvel at how much more productive and accomplished they’ve become.  Mastery of generative AI tools is a status reward, separating the workers of the future from those left behind.

But the promotion of engagement by AI platforms is subtly different from early waves of online products.  AI platforms envision themselves managing everything in your life and, consequently, are motivated to make users dependent on their products, if for no other reason than the worry that users might start using a competitor.  Platforms are battling in a winner-takes-all competition.

AI capabilities are everywhere because firms want users to get into the habit of relying on AI tools as their default behavior.  As this happens, the user’s ability to use alternative AI products, or none at all, becomes restricted.

While platforms promote user dependence on their products, they also seek to maximize revenues and profits.  The pricing of AI usage is opaque, and users may not be aware how their experience with AI responses is shaped by the platform’s financial objectives. Users should never assume that the response they get is the best one possible.

Several platforms are exploring advertising in chatbots.  Whether such marketing promotion seeps into chatbot responses remains to be seen.  One can easily imagine product placements, such as those that routinely occur in movies, appearing in chatbot responses.

The value of the user for the platform shapes how many tokens the platform is willing to expend to generate a response.  If the user isn’t a high-value prospect, the platform will truncate the response by offering the fastest and easiest response to generate, rather than the most complete one.

Countermeasures to phantom authorship

Despite the problems associated with phantom authorship, I’m not advocating that you refrain from using AI.  Chatbots can be a valuable tool when used judiciously.

GenAI requires a new kind of information literacy, one that builds on best practices of the search era, yet extends them to address both new problems and opportunities.

The most basic principle to keep in mind is that chatbot responses are not answers or data.  They are best thought of as pointers to actual information rather than as vetted knowledge.

Here are some tips to counter bots’ tendency to sound unjustifiably authoritative. 

Countermeasure 1: Authenticate the source

When reviewing a response, start with the question: Who said that? Track down where the information supposedly comes from and look at the source directly.

Some AI platforms provide links to the sources used — following these links helps provide more context for what the response has pulled.  You can check what exactly was said and the framing of the original discussion.  You can evaluate the source’s authority and credibility on the topic, dimensions that chatbots don’t evaluate.  Some platforms have been known to generate “ghost references” of sources that don’t exist.

If a link isn’t provided in the response, note if any source document is mentioned, and search to locate that document.  If the source document is behind a paywall, an archival version may be available. 

Sometimes the source document is overwhelming in its size or complexity. In such cases, don’t be afraid to dig deeper into the content.  Browsers now have built-in AI chatbots that let users explore the source material directly, rather than relying on an AI platform’s initial response that synthesized multiple sources. 

A potential red flag is when AI responses don’t cite anyone specifically for a statement.  Maybe the response reflects a consensus opinion, which could be either right or wrong. Or maybe the AI platform only provided a superficial response to a complex issue.  Oversimplification is a common boobytrap in AI responses.  AI platforms ration the tokens used to generate responses. 

Unless the question is strictly factual, the issue may involve nuances that aren’t reflected.  It’s a good idea to ask more specific questions to drill into potential nuances, and to seek alternative responses from other AI platforms to ensure you are covering all relevant perspectives.

Countermeasure 2: When scanning for information, try more than one platform

AI platforms don’t want people to browse the way they used to, by scanning links in search results.  Platforms synthesize information from diverse sources so users don’t have to.

But diversity of information is a good thing, not a burden. It provides a richer picture of a topic.

Don’t feel pressured to stick with a single platform — don’t be captured by a subscription plan. Be a showroom shopper who is “just looking” at the responses that a platform has to offer, and be willing to check out other platforms.

AI platforms want an exclusive relationship with you.  But for now at least, users have many suitors.  Most platforms offer at least a basic tier for free, allowing users to ask the same question on different platforms.  While doing so will yield some overlap in responses, it will also surface new avenues to explore. 

Countermeasure 3: Ask the core question multiple ways

Chatbots are known for providing slightly different responses to the same prompt. It’s useful to leverage this property to your advantage. Embrace the indeterminacy of bots to widen the range of responses to get useful ones.

There’s no such thing as a perfect prompt, especially when exploring a topic you are unfamiliar with.  You might not be sure how source content discusses a topic, or how the bot will interpret your prompt.  Experimentation helps to clarify these relationships.

Chatbots can be highly sensitive to the terminology used in prompts.  If the bot seems to misinterpret what you are seeking, try different terminology.  Sometimes, more concrete terminology helps; other times, more abstract or general terms work better.

Similarly, broadening and narrowing the requirements expressed in prompts can change the utility of responses. 

Countermeasure 4: Be mindful of the changeability of a topic

Users don’t know what content the LLM was trained on, or how old it is.

Chatbot responses can be biased toward reflecting the dominant views of topics, especially when the prompts are general. What view is dominant will depend on how much it is discussed within the corpus of text on which the LLM was trained.

Some topics are anchored in old information.  A plethora of dated information crowds out nascent information that hasn’t been written about extensively. The old consensus dominates. 

Other topics aren’t covered because they are too old.  Most LLMs are trained primarily on post-1995 web pages, meaning that pre-Internet content is unknown to them. (Anthropic is an exception, as they have scanned millions of paper books to feed into their training.)  Even internet content more than a decade old has often been taken down and has disappeared forever. 

Internet content, upon which LLMs depend, is sensitive to fads and fashions.  Publishers create content based on current reader interests, not long-term interests.  Many topics and perspectives lack comprehensive coverage because they were never part of a popularity wave. 

Be sure to include the timeframe of information you are seeking. Decide what timeframe will yield the most useful insights. If you know that medical guidelines have changed recently, be sure to ask for the most recent guidelines. But in other cases, prioritizing the most recent information will skew responses toward the latest controversies on a topic that may be of little importance to the user. 

Ask how people viewed an issue prior to a certain date, or what the major issues were at different time frames relating to a topic.  Doing so can bring additional perspectives and overcome the recency bias in chatbot responses. It might also reveal what the LLM can’t address, which is also valuable feedback. 

Countermeasure 5: Probe terminology

The closer the response draws from sources developed by experts, the more accurate and reliable the information is likely to be. But experts, while knowledgeable, tend to speak in jargon.

Chatbot responses tend to translate both the experts’ information and their wording simultaneously, which can result in a loss of detail or inaccuracies. For example, if a word represents a fundamental concept in a domain like law or medicine, but a synonym is used, the specific meaning of the underlying concept might not be conveyed. 

Rather than have the chatbot translate both the experts’ information and their wording simultaneously, break these into separate steps.   Ask how the expert would describe the issue, then follow up by asking what unfamiliar terms mean in that context. 

LLMs have been trained on an astronomical amount of text that sheds light on what words mean in specific contexts.  Ask bots what a term means in a specific context and why a concept is important in a specific context.

Ghost-busting Generative AI

Andrej Karpathy, co-founder of OpenAI, has said: “Today’s LLMs are like ghosts.” 

Because so much about chatbot mechanics is invisible, users must infer how responses are generated. They shouldn’t take responses at face value. 

Users need to bring a skeptical mindset to AI tools and be prepared to challenge their responses. 

— Michael Andrews