metadata Archives - Story Needle

Readers online deserve to know who or what is responsible for the content they see. AI transparency is both necessary and possible.

AI is shaping more online content, from tweaking human-authored content to providing entirely machine-authored outputs. Readers may notice that text sounds AI-written, but in most cases they can’t detect how much AI was used.

The perils of AI fatalism

The position of the AI industry and countless AI users is “get over it.” AI is used everywhere now; society needs to accept it and stop worrying about AI’s inevitable role. Such AI fatalism ignores genuine concerns about AI accuracy and fidelity, problems arising from unclear or absent accountability with generated outputs.

AI fatalism also dismisses the possibility of solutions to AI attribution problems. AI platforms have largely been uninterested in developing solutions. But their lack of interest does not mean solutions aren’t possible.

Defining human-authorship in the AI age

As bots write more content, readers aren’t surprised to learn it was machine-generated. On the contrary: they will often presume that it is. Online readers face fake reviews written by bots, spammy AI marketing, and vague bot-delivered customer service assurances. Trust in information is eroding quickly.

Simply flagging content as AI-generated may not provide much useful information if nearly all content is designated so. Readers want to know how involved humans were in drafting the content. Did they tweak a bot-developed draft, or did they develop the draft themselves?

An authorship question surfaced in a recent editorial in The Wall Street Journal. The editorial, written by the billionaire investor Stanley Druckenmiller, criticized his former mentee, the current US Treasury Secretary. Readers noticed that the editorial sounded like it was drafted by AI. Druckenmiller admitted getting AI help (“I’m kind of proud of using it”), but insisted that the article sounded like him, since he’d been saying the same things for the past 15 years. Of course, it’s possible he asked a bot to draft something new based on arguments he’d made in the past. That’s what LLMs do: generate new text derived from prior text.

AI doppelgangers are becoming more common, further challenging content originality. Some Harvard Business School professors have cloned themselves digitally to deliver an online course. They may look like the original, but what they say won’t be what the real people would have said. It will be an AI-generated interpretation.

Relying on the “voice” of content is not sufficient to determine whether it is original material or was developed with active human oversight.

Readers want to know how much of the content the individual who purportedly created it actually wrote. People want a reliable measure of human authorship. Human authors are accountable: either they are lying to you, or not. Machine-generated content is ownerless — even the putative authors may disavow it if a mistake is spotted after it’s public.

School teachers deter cheating by making students show their work

School teachers face widespread AI cheating by students. Polished term papers are suddenly everywhere. Yet, unlike many consumers, teachers don’t roll over and accept that they can’t control AI.

When a student submits an essay in Google Docs, teachers look at the version history, which provides a complete listing of various iterations of a doc and the differences between them. If no changes are visible, the student likely pasted an AI chatbot output into an empty Google Doc.

Teachers require students to “show their work.” The same principle should apply to online content available to users.

AI content needs a blockchain ledger

AI vendors are starting to watermark outputs generated by their tools. Anthropic, Google, Meta, Adobe, and others are inserting watermarks into outputs. While a start, the watermarking doesn’t provide a complete version history of the role of AI in the development of items.

The Content Credentials embraced by Adobe provide a “manifest” for an item, describing the tools used to create it. Each item has a unique ID.

What’s needed for AI outputs is a blockchain ledger that shows what happened to the item at different stages of its development: who made changes and what tools were used.

Blockchain ledgers are already used to trace the provenance and authenticity of many goods. Blockchain is decentralized, so that parties using different tools or platforms can all follow a common versioning system.

There’s no reason it couldn’t be used to version digital content that’s co-developed with AI. Doing so would help establish the true authorship of items and allow publishers to pinpoint where in the publishing process human- or AI-originating errors were introduced.

Authorship matters: supply chain transparency is crucial

AI-generated content is a commodity. Readers value it less than human-authored content. Human authors are surrounded by AI tools. They deserve credit for their work. And readers want to be able to trust that work.

Those who argue that AI is now the default and that human authorship no longer matters are misreading the situation. Public skepticism of AI has exploded in the past year. Vendors need more robust solutions that provide transparency into AI’s role.

— Michael Andrews

Your customers aren’t that into your website anymore. Most websites have noticed a drop in traffic as users query bots and bots supply answers. Bots generate few clicks to web pages, and the proportion of referral clicks seems to be falling.

Web publishers are aware of the existential threat they face. So far, they’ve tried to make themselves more lovable for bots. They scheme to get noticed by bots (GEO – generative engine optimization). Or they try to make their pages “friendlier” for bots (Google’s WebMCP is the latest example). The legacy thinking still frames the problem as one of visibility — getting noticed in a crowd.

Yet bots aren’t people, and don’t need to be wooed. The old psychology of wooing is no longer relevant. If bots need something, they will take it from your website, whether you invite them or not. In many cases, they will take content even if you don’t want them to.

The problem websites must solve now is how to ensure bots extract the right content from your site. And your existing HTML content, built for web browsers and surfers, isn’t what bots need, if your organization cares about ensuring the accuracy and relevance of what bots provide. JavaScript, the foundation of websites, is a liability for bots.

AI platforms are evolving quickly. They are pivoting away from indiscriminate web scraping for “training” and towards RAG, where they search first for information before generating answers. AI platforms have also embraced the Model Context Protocol (MCP) standard, which, when enabled, allows them to access enterprise content directly. Already, third-party MCP platforms such as Scite and Tollbit have emerged to connect content publishers with AI platforms.

Publishers will continue to publish webpages for human readers, but they need to ensure that AI platforms access the right content for bot users. The best practices for doing this are still emerging, and several initiatives are underway to define protocols and standards.

What’s becoming apparent is that MCP will play an important role in controlling bot access and content governance. The diagram below illustrates a potential content pipeline for a scholarly publisher. A similar pipeline might be adopted by a website publisher — but some additional steps are needed to transform HTML-centric content into bot-ready content.

Example pipeline. Source: Scholarly Kitchen

How are publishers getting ready? Let’s look at how Tollbit helps web publishers. Tollbit works with the Associated Press and other publishers to make their content ready for AI platforms.

The first task is to “clean” the web content to remove material that’s not relevant or canonical. This can be done through DOM filtering to exclude certain classes of content, such as navigation text, promotional assets, or customer comments.

Additional filtering can be done by excluding pages or directories that are procedural or administrative rather than substantive in focus.

Next, the content should be transformed by removing clunky HTML tags to convert the content into a bot-readable format. Many organizations opt to convert content into Markdown, which preserves heading hierarchies (useful for bots) while striping away extraneous markup that bots don’t need.

Bots benefit from metadata, but need help identifying it. The content transformation process should address metadata that’s not visible to human readers. This includes descriptive metadata (such as schema.org) about the content for external systems like search engines, and internal administrative and technical metadata (such as geolocation coordinates) used for web page delivery. This conversion, known as re-serialization, makes the metadata queriable. The metadata can be “hydrated” into the bot’s payload.

AI platforms, ever motivated to increase the sophistication of their products, will take advantage of these content enhancements.

Getting content “bot-ready” will become crucial as AI platforms expand their agentic capabilities. Publishers will need to define access rights and permissions. What materials can bots read, re-publish, or process?

Publishers will shape these affordances through both explicit statements and implicit decisions that influence the ease with which bots can perform actions.

— Michael Andrews