Categories
User centered AI

Setting boundaries for bots

Let’s talk about our relationship with AI.  Is it a healthy one?  How might it be more satisfying?

Setting boundaries is one of the most discussed topics in relationship advice. Such advice distinguishes healthy boundaries from unhealthy ones and explains how violating boundaries leads to controlling behavior. It counsels people to take action rather than be passive in relationships. They must set limits on what’s permitted and not permitted. Boundaries don’t exist until they are communicated to others.

These same concepts are relevant to how we use AI. Our use of AI involves hidden power dynamics, which operate behind the scenes and are not explicitly articulated. 

Computer users should challenge dominant AI practices to ensure they serve their needs – and not harm their interests.  The notion of boundaries will be central to gaining this control.

Boundaries are options. But they are not mentioned in any user guide to AI platforms.  They are choices that individuals need to make beyond the options available in tools. 

Why boundaries matter when computers impersonate people

When I studied human-computer interaction (HCI) in graduate school a quarter of a century ago, computers were still aliens, with their own lingo and ways of behavior. The challenge was getting them to act more like people.

In the AI era, the situation has reversed: computers impersonate humans.  AI platforms give their bots human names and describe their capabilities using human terms, promoting anthropomorphism. Platforms like Anthropic hire storytellers and content designers (with compensation that can exceed $500,000) to make chatting with bots seem indistinguishable from talking to a person. Platforms want you to believe their products offer all the benefits of a trusted confidant, without the drama. 

Screencap. Source: Wired. Anthropic recently introduced features it referred as “dreaming” and “memories” (plural), alluding to human qualities.

The challenge now is to maintain awareness that bots aren’t people. The roles of the human and the bot are intentionally fused together in a blurry mind meld.  AI platforms would like users to see bots as active collaborators rather than as machines to control at arm’s length. Bots are generous giving users the credit. Let’s not worry whose idea is being discussed in the chat.

As a form of hype, anthropomorphism is shown to exaggerate AI capabilities and performance by attributing human-like traits to systems that do not possess them. As a fallacy, anthropomorphism is shown to distort moral judgments about AI, such as those concerning its moral character and status, as well as judgments of responsibility and trust.

Anthropomorphism in AI: hype and fallacy

It’s a mistake to view susceptibility to AI harms as a personality vulnerability. I approach this issue as an analyst of human-computer interaction, not as a therapist. I see systemic risks in AI platforms that affect everyone.

Boundaries are necessary in social situations and when using technology.  They are important for clarity of understanding and safety. 

Computer applications constantly challenge our boundaries.  Modal popups and notifications ask us: Allow device sharing? Use your login credentials from another platform? Share your profile? Share your data/files?  Applications are always testing our limits, pushing us to grant them permission. 

AI is moving from opt-in to opt-out.  AI features now show up in the operating systems of our devices, in our online search results, and in everyday applications like email and word processors.  These AI-enabled features appear without our asking, and they often displace previous functionality we’ve been accustomed to using.  

The hidden pressures to use AI

Whether you like AI or not, you face pressure to use it.  This pressure comes from two sources:

  • Social pressure
  • Platform pressure

AI has become part of our social fabric. The more that your social contacts use AI, the more pressure you will encounter to do so as well.  

Screen cap of Google’s AI summary of an article from the American Association of Retired Persons. The summary emphasizes how engaging AI for seniors, providing companionship and stimulation while requiring little skill. The actual article is more cautious.

In some respects, bots are displacing social media. Users chat with bots in lieu of remote people online. Bots give users feedback and praise.  And they provide material to discuss in real life with friends, just like updates about grandchildren (minus the cute photos). Bots reward users with bragging rights for what they did with AI, or with conversation starters about what AI said. People use bots to have conversations and be involved in conversations.

Social learning is another vector. The boundary between work life and home life has been blurring for some time.  For people who work in offices, AI is often already a constant companion, and workplace ways of doing tasks transfer to the home, even though the context is different. While the employee uses AI in organizationally agreed ways where the organization assumes the risks, the same person at home must decide what AI use is appropriate and bear the risks themselves. 

Social validation pressure, used extensively in marketing and social media, is finding its way to AI platforms. Countless influencers tout online their achievements using AI, earning extra money or finding the perfect vacation. Are you missing out? 

Platforms encourage AI use through subtle manipulation. It’s hard to ignore the nudging of bots in your application. They signal that a new feature is available you should try. They helpfully suggest you rewrite that sentence. Or they volunteer to write it for you.  The chatbot appears at the bottom of the screen when you visit a bank or online store, greeting you and asking how it can help you. If you don’t have any questions to ask, the chatbot will suggest some questions it can answer for you. It will also offer to teach you how you can use the bot.

This unsolicited advice can wear users down, and many surrender. But bot designers know they can’t rely on pressure alone. They need for bots to offer users emotional rewards.

The alluring attractions of bot delegation

Bots create the sensation that they are taking care of the user. 

Why are people enticed by bots? Because they believe that bots are better than they are. They decide bots are competent, and unburden themselves.  They believe they are in the bot’s good hands. Competence is a perception, not an objective benchmark.  

A study of over a million ChatGPT prompts reveals that users expect bots to provide guidance, information, and help expressing themselves — activities that people, until recently, would want to do themselves.

ChapGPT usage study from the National Bureau of Economic Research

Users find bots appealing for several key reasons. One is objectively true, while others are more subjective. 

Users believe bots have three virtues.  They see bots as being:

  • Faster
  • Easer
  • Better

What more could you want? Each of these benefits is plausible, but they deserve scrutiny. 

Bots are generally faster.  Bots deliver speed by removing clicks. They can provide responses and complete most non-trivial tasks faster than humans.  Rather than the user having to plow through web pages and web forms, the bot does the legwork. Users now wait for the bot. Instead of patiently awaiting a response, some users focus on other tasks, including asking another bot to do something else. People can work faster because bots work faster.

Bots seem easier, even if they create problems later.  What is easy is more subjective. To learn about a topic, watching a video might seem easier than pinging a chatbot back and forth.  Tasks seem easier when users can avoid doing tasks they find unpleasant. Such tasks might be reading explanations, filling out forms, or making judgments about which option is best. But as we’ll see, even if bots offer to address complexity, they don’t make the issue’s inherent complexity go away.

Most users would agree that accessing AI platforms has become easier. AI platforms have reduced the friction associated with adopting them, starting with account setup. 

Screencap: Anthropic options in Firefox. No dedicated Claude login is required.

Platforms have emphasized their ease of access over their long-term benefits. For consumers, chat interactions generally last a single session.  It’s a shallow, transactional relationship. It requires work by the user to guide and build on the platform’s responses from previous sessions. In consumer-grade AI, there may be limited persistence of prior user activity. 

The short-term focus means that bots highlight immediate advantages. 

Bots appear smarter – but ask why.  The bot acts smarter than you, or at least it has access to more information. They seem to perform better than humans on high memory tasks that require consulting and considering many facts at once. But they can still struggle at times with simple counting and arithmetic that are easy for humans.  And they don’t infer implicit information that humans would understand as common sense.  

Bots have paradoxical properties: they outperform most humans on many tasks, but can be naive and clueless on basic ones.  Chatbot error rates vary widely, but errors in responses can range from 10% to over 60%.

Users are encouraged to trust the model.  Over the past year, as the newest models have grown in size, prompting advice has shifted.  Now, vendors recommend that users just tell the bot the outcome they want and not bother detailing the process.  Trust the bot to make the right choices to get you what you want. OpenAI tells users: “Shorter, outcome-first prompts usually work better than process-heavy prompt stacks.”  Elaborate prompts are no longer necessary, or even desirable. Such prompts interfere with the optimization in the large model. It’s easier than before to rely on bots for complex issues, but the user’s agency in shaping the response is diminished. 

What could go wrong? Identifying risks when using AI

Most concerns about AI have focused on its societal risks and on whether governments or industry bodies should regulate its use (e.g., no bomb-making advice allowed).  As important as such discussions are, they don’t seem to be resulting in any meaningful protections for individual users. The political power of AI platforms and the money they have to influence politics has prevented meaningful regulation. 

Users must assume that no organization will protect them from using AI in the wrong way.  On the contrary, platforms may offer various incentives that encourage individuals to use AI in ways that jeopardize their interests. 

The risks of using AI fall into five main categories:

  1. Financial
  2. Legal
  3. Security
  4. Health
  5. Mental health

These dimensions involve different hazards for users. But they are similarly ambiguous about who’s to blame when AI triggers a nasty surprise. In each case, AI platforms are likely to hold the user responsible for any unpleasant outcome. 

From the platform’s perspective, it’s the user’s fault if they misuse AI. Platforms insist they don’t want to tell users what they can and can’t do. 

Each of these risks deserves detailed elaboration, but for now, let’s look at some examples for each.

Financial risks of bot delegation

Platforms are aggressively looking to monetize the usage of their products to recoup the billions they are investing. The financial pressures on platforms are escalating, and these firms seek opportunities where bots can play an intermediary role. 

Users find that big purchases and investments can be complicated decisions and transactions.  They are attractive targets for bot interventions. Sales and financial advisory agents are becoming available.  Shopping bots are also emerging, promising to take on the routine chores of buying goods.

Loss aversion is a major driver of human behavior. Unsurprisingly, AI platforms don’t want to highlight to users the financial risks associated with their products. 

For users, bots heighten financial risks. Bots lack transparency and make fast decisions. Users need visibility and not to be rushed on money matters.

The biggest financial risks to users are making suboptimal choices and losing money.

Bots can facilitate suboptimal choices relating to pricing or investment returns.  Bots might not show users the best deal available.  They may encourage users to make a decision prematurely, before knowing all the facts.

When bots don’t behave as users expect or deliver unexpected outcomes, they can be implicated in the loss of funds. Users might be surprised by deals that turn out to be worse than they thought or by investment returns that are less than expected. 

Legal risks of bot usage

Legal advice is expensive and often unavailable to people, so chatbot responses are a tempting substitute for a lawyer, if not always reliable.  Bots are already doing the work of junior attorneys in law firms. And consumers are already turning to bots for legal advice.  Bot-delivery of consumer-facing legal advice seems destined to become common.

Bots even pose legal risks to users when they don’t act as legal advisors. They proffer advice of all kinds, any of which can generate legal risks. Bots are known to give bum advice.  

Users have a disadvantaged relationship with AI platforms. By signing up for an AI platform, users surrender their rights to the platform. They agree to binding arbitration for any dispute; the arbiter is appointed by the platform. 

Users delegate their rights as individuals to bots. The bot acts as the user’s proxy. Bots have no liability because they can’t be held accountable for their actions. Good luck trying to sue the company behind the bot if things blow up.

Security risks of bots

The security risks of using bots are hard for users to gauge because AI can access all kinds of data about users.  As AI moves in an agentic direction (discussed below), it will become even more interdependent with the user’s online ecosystem, multiplying the potential vulnerabilities.

Bad actors are now using bots to find vulnerabilities in other bots. The payment platform Stripe notes: “Fraudulent actors can deploy agents to test stolen credentials or probe checkout logic at scale.”

Among the biggest worries is that AI could enable a breach, allowing access to:

  • Bank accounts, brokerage accounts, credit cards, or digital wallets
  • Personal data, information about family members, or religious and political affiliations
  • Credentials for government services, access to facilities, or for identification verification

At the extreme end of AI threats is the popular OpenClaw agent, which takes over the user’s machine. 

Although AI platforms are developing security protocols, their reliability is open to question. Several blue-chip firms have endured embarrassment over security breaches of their AI implementations. Security researchers warn of an arms race between expanding AI capabilities and the opportunities bad actors have to hack them using AI. The lone user needs to be careful in this unstable environment.

The other security risk involves the user’s misplaced trust in the AI chatbot. AI platforms have porous privacy policies. 

Your personal information might end up as “training data.” Unfortunately, many people are giving bots their most personal details about mental or financial problems they face because they are too embarrassed to discuss them with fellow humans.  But AI platforms don’t guarantee this information stays private. How platforms might collect, store, and use these details is unclear. Personally identifying information (PII) could be publicly leaked or obtained by data brokers.  

Health dangers of bot advice

People’s bodies are complex organisms that undergo many changes over a lifetime. Illnesses can be difficult for humans to diagnose. Bots are ready to cut through the complexity.  But the scope for inappropriate advice is great. 

While online health advice has long been available, bots change the dynamics by offering advice that seems personally tailored to an individual. Bot-generated advice seems more credible and actionable than generic online health explanations. 

OpenAI notes that more than 40 million people worldwide use ChatGPT daily for health questions, accounting for more than 5% of all prompts. To capitalize on this demand, OpenAI is introducing ChatGPT Health. 

Microsoft is also building a health chatbot called Copilot Health. Microsoft notes: “Long waits, clinician shortages, and uneven access to medical care lead many people turning to online sources for help.”  Yes, the health system is broken.  But are bots the answer, or just a symptom of the brokenness?

Screencap: Microsoft Copilot Health announcement. The pro forma disclaimer contrasts with the announcement’s emphasis.

Microsoft offers a standard disclaimer that Copilot Health is not intended to diagnose, even though it accesses your medical data. 

Perplexity drops the pretense that it doesn’t diagnose:  

Perplexity Health tracks metrics and trends over time across biomarkers and activity data through a personalized dashboard. Ask a health question and the answer draws from your medical records, lab results, and wearable data at once.

Without doom-mongering, it’s prudent to foresee risks as they are already present in legacy online health information. Personalized chatbot responses might lead to a misdiagnosis of a serious condition, since many relatively benign symptoms are superficially similar to life-threatening ones. They might suggest an ineffectual treatment – or even a dangerous one. 

The stakes for health bots are high.  Users must be highly confident they are accurate, which is possible only when they know they are highly reliable. The scope for error is non-existent. 

Mental health hazards of bot reliance

The bot’s ability to tell a story makes it believable  — and dangerous. Bot usage can be bad for mental health because bots generate dependency – the feeling that bots are necessary to decide an issue – which can result in feelings of helplessness. 

Because bots offer quick, polished responses, often with a rationale, they can seem credible even when they aren’t. The imbalance between the slow, uncertain user and the powerful bot can sap the person’s confidence and undermine reflection, seeding self-doubt.

Even if the user remains vigilant about the bot’s responses, feelings of helplessness can arise.  The user is often not sure how sound the bot’s response is. 

Bots can trigger in users with a range of bad emotions, from annoying to worrying:

  • Frustration at bot responses, such as when they don’t reflect the user’s intent accurately
  • Anxiety about the soundness of bot choices, and whether all options were thoroughly explored and considered
  • Regret about a bot decision, such as a definitive-sounding answer that’s counterproductive

Types of AI boundaries 

Computer marketing tends to emphasize the power of connectivity. The more relationships there are, and the more open they are, the better. Platforms promote a vision of a world without boundaries.

Users are finding this boundary-less technology intrusive. They need ways to keep it at bay.  

How should users think about healthy boundaries in their relationship with AI?

Boundaries in human relationships provide an obvious source of inspiration, since people are half of the relationship, and the other half, while a machine, acts as though it were a human. In popular psychology, concepts such as toxic relationships and codependency describe situations when appropriate boundaries are missing.

In the world of machine-to-machine (M2M) interaction, boundaries are also essential, and they point to another source of inspiration. 

Computer practices rely on clear boundaries to prevent system conflicts. Computer systems have firewalls, data storage may be partitioned, and data may be quarantined.  

In computers, a fundamental concept is the separation of concerns. As a matter of principle, applications shouldn’t interfere with or intrude on the decisions for which other applications are responsible.  They should stay in their swim lane. 

AI needs to stay in its swim lane, too.

Setting boundaries isn’t about being anti-AI, but being a smart AI user, rather than a native one.

Boundaries fall into two main categories:

  • Around when and where the bot is available for the user
  • Around what decisions can bots make 

Boundaries around the availability of AI

Tech firms often talk about “creating a moat” to keep other firms from poaching their business. Users of AI tools need to create moats of their own to keep AI tools from encroaching on their lives uninvited.

Tech firms recognize the benefits of boundaries, even if they don’t encourage their customers to apply them. It’s instructive to watch what they do, rather than what they say. 

Screencap: Yondr

The tech firms building our AI platforms set boundaries on their employees’ use of technology.  Many companies make personal devices unavailable at work. Amazon, Google, and Apple deploy Yondr pouches that lock up employee smartphones and make them inaccessible. Yondr says such restrictions “create a more focused and secure work environment that encourages productivity, protects sensitive information, and prioritizes the well-being of your employees.” Only when outside an office or conference room can the pouch be unlocked. 

Yet for consumers, tech firms promote the “always on, always available” paradigm. Each software update seems to install new AI features on your device. These features are often enabled by default.  But this on-by-default is not in the best interest of many users. 

Claude seems to preemptively change defaults for applications you may not have even installed yet, this article from The Register says, likening it to spyware.

Many people feel too tied to their phones, distracted by their constant pull.  And AI is becoming available on phones as well as desktops.

Despite these stimuli, users can place boundaries on the availability of AI tools.

The first boundary is to opt out of having AI always on.

  • Users can choose not to be logged in to AI accounts all the time.
  • They can keep AI tools from accessing data and other applications without express permission. 
  • They should avoid installing AI applications on their desktop or other devices. 

Many AI developers actively mitigate these risks. They use separate computers (Mac minis are a favorite) to run AI applications and keep them away from their personal data.  For users, if they don’t want a dedicated AI device, they can keep AI usage restricted to a specific browser.

Users can also choose what data to allow AI to access by using data curation.  For example, rather than have a bot consider information from any source, users can ask bots to consider only certain sources, such as a folder of PDFs the user has already screened and deemed relevant. 

AI Tools can set boundaries around how data gets accessed

If you do install AI on your device, you can limit how it behaves.  As mentioned, you can install AI on a secondary device so that it doesn’t interfere with your routine online activities and data.  You can also be selective about what AI applications to install.

AI tools can support healthy boundaries through:

  • Privacy-first designs
  • Local-first setups 

Most AI platforms prioritize data gathering over privacy. But IT firms in Europe have been concerned with data sovereignty in recent years, and several offer AI options that emphasize privacy. 

A handful of AI tools aim to be privacy-first. Lumo by Proton is a privacy-first chatbot that employs local data storage, encryption, and zero logging. 

Local first setups can support privacy by preventing models from training on your data.  It’s possible to download open-source LLMs such as Ollama to a local computer and run AI “on-prem” (on the user’s own premises, rather than in the “public” cloud).  LM Studio offers a GUI for using locally hosted models, including Google’s open-source Gemma. This approach may appeal to the computer-savvy user, but it remains challenging for mainstream users. 

NextCloud, a German open-source data storage and app vendor supporting on-prem solutions, has introduced the NetCloud AI Assistant, which it claims is “the first open-source AI assistant that is hosted where you want it to be,” including local hosting. The AI bot can also access the data locally as well.  The chatbot allows the user to “manually define the scope and even limit it to a specific folder or file for more precision.”  

Screencap: Nextcloud Assistant chatbot working with locally stored content

Boundaries around allowing AI to make decisions

The expected next AI tsunami will be agentic AI, in which bots make decisions on your behalf. So far, agentic AI is mostly a topic of discussion, but agentic features are starting to emerge.  Last year, Amazon introduced a “Buy For Me” feature on its website.  

A recent Fast Company article says that agentic commerce is just around the corner: “The commerce leads at Google and OpenAI, the two biggest players in the space, say that we’re months—not years—away from a tipping point where agentic commerce really will become commonplace.”

The payments processor Stripe outlines how agentic commerce will work

  • “The user gives the agent a goal and constraints: A sample instruction might read, ‘Buy me a replacement filter for my air purifier—same brand if possible, under $40, delivered by Thursday.’ Those constraints then govern the agent’s decisions.”
  • Consumers can set up “event-triggered purchases” that happen automatically when certain events happen.
  • Purchases will be made by “payment without a human at checkout. This requires tokenized payment credentials, delegated authorization, or wallet-level integrations.”

Eventually, firms want to force customers to use AI agents. Lendi, an Australian mortgage lender, expressed this vision as “agents managing humans.”

Granting AI agents the autonomy to make purchases on your behalf involves a major delegation of responsibilities. 

Deciding what to delegate to bots

Bots promise to tackle challenging issues. These same issues often involve hidden risks.

Warning: Bots are especially tempting to use when they promise big payoffs but entail big risks.

Why do bot risks often increase in proportion to the rewards they offer?

Bots can produce a temporal asymmetry in outcomes. Bots deliver immediate benefits that have delayed costs. Users won’t appreciate the risks or experience their consequences until they finish their bot session. 

Users are motivated to delegate tasks to bots when the problem to solve is: 

  • Time-intensive 
  • Procedurally complex, requiring sustained attention
  • An unfamiliar topic, where advice is expensive to obtain 

These factors are related. Procedurally complex tasks tend to be time-intensive. They are also difficult for novices to understand.  

Imagine having a bot choose your mortgage. Using a bot promises to save you hours of research, avoiding the pain of wading through details, and the anxiety of deciding on the right choice. But since you don’t know what the bot considered and what it didn’t, you don’t know whether those saved hours were worth it. 

When issues are time-intensive, procedurally complex, and outside an individual’s expertise, the incentive to use a bot is strong. These kinds of knotty issues are the ones most likely to trigger a nasty surprise.  The danger is that the user has placed trust entirely in the bot. The user hasn’t investigated the problem themselves. 

Don’t forfeit due diligence. Even though going through the tedium of a time-intensive, complex task is unappealing, it does help an individual understand the topic and allows them to make a better-informed decision. It boosts the person’s knowledge so they can evaluate the situation. That effort doesn’t mean they’ll necessarily make a better decision than a bot – and that’s the inherent uncertainty.   

When delegating unfamiliar topics to bots, you don’t fully know what you don’t know.  And you also don’t know what the bot doesn’t know, or has chosen to deprioritize. You have no basis to evaluate the bot’s responses.

Alternatively, you can make a decision by doing your own research.  If you’re still uncertain, you can ask a bot to explore the best solution independently of your investigation, then compare your choice with theirs. 

Always be clear who owns the problem and is responsible for the solution.

Boundary problems arise when roles conflict. Both parties believe they have control over what is being done.  With AI platforms, it is difficult for users to explicitly direct bot behavior, since bots reinterpret prompts when generating responses. Users have very limited visibility into what bots are authorized to do, especially as bot capabilities are upgraded continually.

It’s easy to have misaligned expectations. The user may be disappointed that the bot didn’t do something because the bot didn’t have authorization to do. But more likely, the bot will take actions that were authorized by default that the user wasn’t expecting.

With platform technologies, you are not unambiguously the customer. You are also the product. AI platforms generate responses. You generate data that platforms learn from and leverage. It’s a two-sided relationship, even if it seems like the user is directing the platform.

Bots have programmatic agendas that are distinct from the user’s. Bots have biases in what sources they consult, how thoroughly they assess information, and how they make decisions. These behaviors are often not aligned with the user’s intentions or interests.

Delegating to a bot is different from delegating to a trusted advisor. Your advisor has a fiduciary responsibility to look after your interests. A bot, in contrast, effectively has legal indemnity due to the T&Cs you signed. If you are unhappy, you only have the option of mandatory arbitration.

What if your advisor uses a bot?  The situation is different. The advisor still has the fiduciary responsibility to you.  And they also have familiarity with the material the bot is working on and are therefore better able to evaluate its accuracy and the value of bot responses.

While AI will be used more prevalently in the future, that trend doesn’t imply bots are the right option for everyone in every situation. 

The right boundaries for bots depend on whether their use is appropriate for a given situation

Delegating knowledge ownership

What are you comfortable having a bot decide for you? 

When you decide to let bots decide, you are assuming bots understand the situation as well as you do, or maybe better than you.

Surrendering ownership of situational understanding changes the nature of the relationship. The bot is no longer a client.  It is in charge.

The bossy bot is becoming normalized. Bots are prone to presumption. Think about wearable devices that buzz you when they decide you haven’t moved enough.  Now imagine bots dealing with all aspects of your life, love-bombing you with friendly messages telling you to do something. 

Platforms are positioning bots as “coaches.” Users let bots decide what you should do and when. No decision is too big for a bot to offer its opinion. They presume to have sufficient knowledge about highly nuanced issues, including the user’s personal goals, abilities, and preferences.

Delegating task ownership

Bots now want to help you find love. The dating app “Bumble is launching a new AI assistant, Bee, within its app to help users create and optimize their profiles.” Bots want to play matchmaker. The next logical step is having a bot set up a date for you. 

The upcoming evolution in bots – agentic AI – will reset our boundaries further.  After telling you what to do, their next mission will be to complete tasks themselves without your involvement.   

AI platforms want to inject agents into all aspects of your life, such as setting up appointments for you, sending messages on your behalf, or organizing activities for your family.  

User-centric workflows for agentic AI have yet to be designed. AI platforms treat users as a bit players in agentic scenarios, and AI engineers so far have not discussed how users can express their needs and preferences. The presumption seems to be that the bot can read the user’s mind. The user will simply give a one-sentence command, and the bot will do the rest.

Despite the inattention given to users thus far, it’s clear which variables that a user-centered workflow for bots will need to cover:

  • What tasks to delegate to agents
  • What constraints should be placed on agents
  • What checks to impose

The bad news: the use of agentic AI will place a bigger onus on the user. They must specify in great detail what they don’t want the bot to do. Even then, the bot might screw up and cause headaches.  For many tasks, the effort and risks involved in delegating tasks to bots would not seem worth it.

Bots are watching you – are you monitoring them?

Boundaries require asserting control

Online platforms have long logged data on the user’s behavior to make their products stickier and boost “engagement”  —  the amount of time you spend using them.

AI platforms take this user data harvesting a step further. Because chatbots are inherently conversational, they can seamlessly ask questions that are motivated by the platform vendor’s business interests, rather than by the user’s personal goals. 

Not only do AI platforms have unprecedented access to data about your interests and objectives in your chats, they are deploying agents at scale to ask you about topics you aren’t chatting about.  Anthropic, for example, has created the “Anthropic Interviewer” bot to ask customers questions.  Customers are being tasked by bots to write answers to the bot’s questions. The human is now the bot’s client.

Screencap: How Anthropic uses the Anthropic Assistant bot the gather responses to questions it poses to Claude users

The guiding principle of user-centered design is that the user is always in control.   AI platforms are dismantling these notions. Racing to surpass competitors, AI platforms are like the Wild West.

Users must be proactive and take options not offered. They have power over how to set up AI tools, when to use them, and how. 

– Michael Andrews

Categories
Creating content

Extending notations for writing composition

Our dominant mindset about technology is one of machine learning: people train computers, and computers learn from people. Eventually, computers will take over tasks that people do.

But suppose instead that humans could learn from how computers solve text issues to help people to make better writing decisions, rather than deferring to an algorithm.

For computers to teach writers something of value, writers must decipher their workings, which are hidden in code. Notation in code offers a window into how text is handled as it evolves.

This post focuses on the role that notation plays in shaping text drafts. I’ll argue that notation can play a useful role in the writing process, provided it is not tightly coupled with specific media or tools.

Patterns and relationships in drafting

Writers imagine their relationship with computers, complex as it is, as one of teacher and pupil. Writers lead by example, computers learn. Writers hope to coax the computer to do what’s desired.

Yet writing is messy, and not an easily replicable process. Computers can only learn so much. That’s because writing involves many judgment calls, which only a human can make.

The writing process is a dialogue between the writer, the draft, and the imagined reader.

As the dialogue about the draft unfolds, fresh thoughts emerge. Writers and reviewers make notations in the draft for follow-up.

Editors apply blue pencils to typescripts, leaving proof-correction marks to indicate suggestions and changes. In computer manuscripts, reviewers layer ghost-like comments that hover in balloons.

Developing a text involves notational practices — letters, symbols, and marks. Writing notation is a kind of pattern language that evolves organically from conventions adopted by various groups over time.

Software applications often play the role of a third wheel in this dialogue, demanding attention, suggesting rewrites at inopportune moments. They aren’t seen as helpful contributors or respectful partners, but as an undisciplined force that needs taming.

Unfortunately, as computers have become more intertwined with the writing process, writers have become more dependent on them. The writing process becomes captured by the platform used.

While there’s no denying the influence of computers on writing, this development does not mean writers must become dependent on them. Writers should resist fatalism about the role of technology. Instead, they should consider how computers can inspire new practices, rather than mandate them.

Learning from customs in computer code

Does something that can often be annoying offer anything it can teach us?

I’ve been looking at how to appropriate some of the intentions behind software conventions to support the writing process. I seek to uncouple the underlying purpose of these protocols from their execution in random-access memory. I want to divorce the ideas expressed in the software from the systems that run it.

The code typed in a text editor incorporates a range of notations. These conventions evolved to solve specific problems in the computer domain — identifying and indexing snippets of text or strings of characters for further action without causing confusion.

The routines that computers follow are best geared to deterministic processes, not compositional writing. Yet they can be useful outside of the computer programs they were originally designed for.

Writing code has discernible similarities to writing prose. Both demand attention to the placement of and interaction between strings of letters.

Computer notations have evolved over time, with various computer languages and programs borrowing from one another. Software programming has existed long enough to establish its own traditions and customs. These languages express vernacular practices that writers can learn from.

Certain coding signposts can be useful for writers, too, even when writing by hand. Borrowing computer conventions does not turn writers into robot apprentices. Writers understandably don’t want to become even more beholden to computers.

Writers need notation practices that work with both handwriting and the keyboard, so that notetaking and drafting are independent of the platform, and so the writer stays in charge of the process. That hasn’t been the case with most writing tools.

The most popular notational systems for text, Markdown and similar formatting systems (Asciidoc, Typst), are designed to shape the layout of text on the screen, not to shape ideas. Too much emphasis is given to formatting already-composed text rather than to how to represent ideas and intentions while composing drafts. What’s needed is notation that facilitates the writing process.

Writing notation to help you to think through issues

The writer’s draft undergoes a long gestation as a work in progress. During this time, it is tentative and incomplete. It can be hard for the writer to track what is happening in the draft.

Notations are a way to indicate what remains to be done, or what themes are emerging. I’ve discovered five computer programming conventions are worth stealing for writing.

Placeholder for the right word

$variable Combining the $ sign with a word signals that the specific thing mentioned is not important. Rather, the statement around the variable is what matters.

Writers can use a variable to indicate that what is said is true, independent of context — or to test the consequences of alternative variables in a statement. For example: In the United States, $person has the right to due process.

The next step is to define the variable with words. In the sample sentence, does $person refer to anyone, US citizens, or some other description of a person? I find this exercise helps me distinguish whether it is the actor or their role that’s most important.

Another use of the $variable is to write about something before you have locked down the terminology you intend to use across a document. I find that writing about abstract concepts is difficult because any label for the concept is a metaphor that can be interpreted in more than one way. By not worrying about the final terminology, I can see how I write about a concept in different contexts, which helps me understand how the term I choose needs to work in different situations. If I were to start using random synonyms as I draft, the concept I am referring to might drift in its meaning.

Who said or did that?

@citation The “at” sign started as an accounting convention indicating “at a rate of.” It was adopted for use in email addresses to signify the addressee’s domain. In messaging apps, using the @ sign indicates a mention of someone’s name.

In writing, @ can indicate a citation of a writer’s work. For example, @Smith 1776 refers to Adam Smith’s Wealth of Nations, published 250 years ago. This citation can be embedded in text, such as: @Smith 1776 argued that it would be reasonable for the rich to pay higher house-rent taxes.

The @ symbol indicates a reference to a multi-line data entry, which readers don’t need to see while reading. The citation convention began with software called BibTeX, which created an entry type (e.g., @article, @book) followed by a “citation key” consisting of the author’s surname and publication date.

To align more closely with the “mentions” convention, some software uses only the key indicating who and when (the author’s surname and publication date), similar to inline references in research papers. I’m following this approach.

The @citation is succinct shorthand for noting who said something and when. But I also find it helpful when citing events driven by people, to build a timeline (you can add month and date as YYYY-MM-DD if needed). It can be applied to any kind of actor: a named individual, an organization, or a group. This usage again is similar to the “mentions” convention. You are citing someone’s activity rather than quoting them.

What are the key concepts?

#topic The hash, or octothorpe, is a mark used to indicate comments in computer code. It later came to signify a “tag” indicating the topic of content. The hash can be added to a keyword in a sentence to allow the sentence to be indexed by that keyword.

Hashtags are less common in social media posts these days, but they remain useful for annotations. I find that adding a hash to a word is helpful when the word embodies an important concept I want to follow. The word might be a proper noun (#Silesia), a specialized term with a precise meaning (#prosopography), or a word that has a specific meaning for the author using it or in the context in which it is being used (for example, #winners).

An example of hash and at signs annotating notes in an application. This example is from the open source Databyss app, which I use to index notes.

Tags provide a thematic index of your draft. When embedded in the text, they can reveal how you discuss concepts. For example, do you repeat them across the text, or fan out the discussion to related topics? Do you drop discussion of a theme at a certain point, where it might fade from the reader’s awareness?

What needs doing?

/task The forward slash has become a common hotkey to invoke a menu of commands in applications, such as the WordPress blogging software I am currently writing with. In a text document, a forward slash, preceded by whitespaces, signals a change in path.

Drafts are full of loose ends. The writer can use the slash-task notation to indicate anything that needs more attention:

  • /factcheck
  • /getReference
  • /streamline
  • /updateInfo

In AI platforms, such commands are called “skills,” describing a set of procedures or routines to follow. They are not much different from a punch list or checklist a writer might follow to ensure they don’t forget anything. For example, you might maintain a list of questions that ask how the reader would understand or react to what’s been drafted.

Once finished with the task, you can remove it from the document: /getReference.

What’s missing?

{{insert content here}} Double braces (figuratively called “mustaches” or “antlers”) are a convention used to indicate a place where instructions or text are meant to be inserted.

They are common in templating systems, where supplemental or dynamic text is added to a base template. Some systems use single braces (“handlebars”) to indicate insertion, but double braces allow writers to use single braces for other kinds of notation, especially to indicate the grouping of items that span more than one line.

Double braces allow the writer to separate different sections of an article or document, to focus on different levels of detail.

The writer’s mind shifts focus between the big picture (the argument) and the details and nuances (the explanation or justification). Double braces are good for setting aside parts of writing you are not focused on.

I tend to develop an outline or skeleton of what I’m writing, but keep the details of each section as separate files. I can indicate in the outline the issues to elaborate within double braces. Initially, I will describe the section’s intent and goal within the braces. After I’ve drafted enough of the section to understand what I want to convey, I can label it with a more formal filename.

The shift from intent to file might look like this:

  • {{show three diverse examples}} –> {{examples.md}}
  • {{connect implications to next steps}} –> {{conclusions.md}}

Digital compatibility, not compliance

Writing practices should leverage computers without restricting the author’s freedom.

The motivation is to embrace innovations that computers have introduced in text annotation, without being forced to adopt a rigid, computer-defined process. The annotations can help you notice features in your draft, whether you scan the text visually or, if digital, use a basic “find” command.

Much writing advice promotes digital-only workflows, seeking to reduce the quantity of text that must be entered on a keyboard. That advice is useful for generating boilerplate (marketing copy, tech docs, etc.). But automation diminishes one’s cognitive engagement in the writing process. Compositional writing involves a set of tasks distinct from those in document production.

Writing shouldn’t be dependent on a specific tool or platform. The writing composition process remains poorly researched, and most recent research focuses on the technology-enablement of writing through networks, online collaboration, and AI augmentation — the features app vendors want to promote. Researchers assume that because technology is changing how writing is done, it’s important to learn to optimize it in the writing process.

But are the tools good for writing? Professional writers, such as Sven Birkerts and Cory Doctorow, among many others, have criticized the distractions caused by tech features in writing tools. “The last thing you want is your tool second-guessing you,” Doctorow wrote in 2009, a decade and a half before writing about internet enshittification.

Drafting quality versus drafting speed

Numerous writers insist that writing by hand helps them to think more deeply about a topic and choose words more deliberately. Handwriting is enjoying a mini-renaissance, from the teaching of cursive again in grade schools to the popularity of keyboardless devices like the Remarkable tablet.

Don’t underestimate the value of typing your handwritten notes: rereading, rearranging, and enhancing them. But if transcribing sounds like a chore, technology can accelerate the conversion to digital. Handwriting-to-text (HTT) is becoming more robust and widely available thanks to advances in AI, making paper-first a viable writing workflow.

For some other writers, a manual typewriter provides the friction to slow down and reflect before writing. While non-digital tools will never win in terms of speed or output quantity, they may foster higher-quality results for some tasks and some people. I can think faster than I write or type, but I find my thinking can be scattered when it’s a stream of consciousness. (For that reason, I never use voice dictation for non-trivial writing.) Now that AI can generate text faster than any human, speed is not the goal. Quality is.

I generally take notes by hand during the early stages of forming ideas. Handwriting helps me capture thoughts in a freeform manner, without deciding on their relationships yet. Handwriting frees you from making premature commitments. I can focus on concepts and ideas before worrying about how they fit into sentences and paragraphs. Computer screens, by contrast, force a linear ordering of material.

While I don’t typically draft complete articles by hand, I find it helpful in the early stages to generate options for the points I want to make. I consider handwriting more forgiving than typed text when drafting views in different ways, as I try to find the right phrasing.

Sometimes I struggle to spell a word when I am writing by hand, and I realize I don’t really understand the word’s roots and etymology. In contrast to my ugly handwriting, text typed on a computer screen looks finished, even when it is still raw. The neat characters call attention to stylistic issues that need fixing rather than the substantive ones. I’d rather learn tolerance for my half-formed ideas and nurture them.

When writing on a computer, choose a distraction-free editor — free of extraneous options and premature editorial feedback. All writers, not just those who struggle with ADHD, can find that using a tool that offers integrated access to everything online can be disruptive. It breaks your concentration and interferes with deep thinking about the meaning of words, statements, and arguments.

I prefer a document editor with a focus mode, so that I don’t have to see either menu options or text formatting markup. I don’t like writing in raw Markdown, as I find the formatting distracting. I appreciate Markdown as a portable, platform-independent file format. But I loathe it as a user interface for writing. Fortunately, Markdown editors exist that hide Markdown’s distracting ornamentation.

Keeping AI optional

AI chatbots have infiltrated almost every commercial application used for writing. It’s becoming harder to write on a computer without AI intruding, making suggestions you don’t want, hijacking your intentions. LLMs operate on the premise that “attention is all you need.” AI applies that mantra not just to web-scrapped text but to human users. It is now consuming our attention and seeking feedback in the same way social media has.

To ensure that the editor is truly AI-optional, rather than AI-dominant, look for an open-source tool, which will require you to use your own account if you want to connect to AI. Open source tools are free. The notion of having to pay an annual subscription licensing fee for the privilege of writing your own words is the opposite of freedom.

None of this advice is to imply you should never use AI. I recommend keeping your chatbot in a separate window from your editor during the drafting phase. If your text is typed in a computer file, you can always copy and paste those parts you’d like AI to provide feedback on. You can ask AI to critique the strengths and weaknesses of a snippet you’ve written, for example. But you should only do so when you’ve exhausted your internal ability to make that assessment.

There are downsides to allowing AI to second-guess what you want to convey. The writing becomes less yours. Your ability to compose and critique ideas atrophies.

“After submitting their essays, people in the AI group were unable to quote from their essays, and several felt they had no ownership over the work.”

AI chatbots could be making you stupiderBBC

Writers don’t need to run all their drafts through AI for a critique. Even the most capable AI platforms won’t necessarily understand the audience you are hoping to reach, or the point you are making — especially if the material is novel or presumes a lot of inside background knowledge.

Authorship is about the ownership of ideas. Its purpose is to express oneself authentically, even if imperfectly. One should always care about the reader’s priorities and strive to address them, but never assume a bot knows them better.

AI can be helpful for final editing, catching typos and wordiness. But even in the final stages of editing, don’t forfeit control. Bots only match patterns; they don’t understand meaning, despite the claimed semantic capabilities of newer LLMs. Bots can make your writing sound slick or mimic Hemingway’s style. But don’t let the bot make a choice you won’t make yourself. You stop being the author once that happens.

— Michael Andrews