Category Archives: Content Engineering

Sorting to Prioritize the Content Experience

The act of sorting seems so familiar you may think little about it.  We sort our possessions to organize them.  Some people even sort the sox in their drawer according to color or occasion.  We sort to organize things, and more fundamentally, to prioritize our content. When considered in relation to IT, sorting is the programmatic prioritization of content using a simple ordering procedure. Simple sorting routines can offer much value, even if more sophisticated techniques to prioritize content are available.

Most discussion of sorting focuses on its technical dimensions — the rules for sorting correctly.  Developers study various algorithms to optimize sorting.  Editors must follow detailed rules to correctly sort entries appearing in indexes.  Interaction designers focus on how to implement sorting options on user interfaces.  Sorting is also utilized in statistical operations, though the strict criteria applied in statistical analysis is different from how people will think about sorting content.

In contrast to its technical dimensions, the experiential dimensions of sorting receive less attention. Besides considering how to sort items, we must also consider why audiences want items sorted in different scenarios.  Every day, when we write lists, we make decisions that reflect our understanding of the purpose of sorting .  Do we present the list as unordered bullets, or do we number the list items?  If we number the items, what do the numbers represent?  All of our content faces such existential dilemmas — indicating to others how items are prioritized.

Interaction designers often assume that users will want to sort items appearing in a list.   Such an assumption confuses a want with a need.  In many cases users don’t want to interact to get specific views of content.  They may need different views, but don’t want unnecessary work to get those views.  Consider the below screenshot, from a website devoted to user interface design patterns. A spreadsheet paradigm is imposed on web content.  Audiences are given the option to sort content on any criteria, but have no guidance about which criteria are important.

A sorting UI pattern. Source:
A sorting UI pattern. Source:

This pattern is widely implemented. The user interface presents the illusion of control, but offers little to help audiences understand insights in the content. It presents unnecessary work for audiences, and provides an uncertain payoff.  Screens like this continue getting made because sorting functionality is ubiquitous and easy to implement. Adding it seems cost-free. The publisher never did the hard work of asking why the audience wanted the information sorted.  What value does a sort offer the audience?  How can computers provide such sorting without requiring the audiences to specify it manually?

Sorting tends to be discussed as being a widget and labeling issue, debating the options that are possible, and the user confusion that can result from having those options.  Instead of worrying about the clarity of the widgets and labels, a better approach is to make editorial decisions that remove that complexity, and focus on the value sorting can offer.

Sorting practices can be explored in terms of five goals they support:

  1. Sorting to Locate
  2. Sorting to Rank
  3. Sorting to Sequence
  4. Sorting to Sample
  5. Sorting to Profile and Evaluate

While these approaches differ in emphasis, they share a common goal of prioritizing content by making a judgment about what’s important to the audience. Users prefer sorted content because it reduces the amount of content they need to view and consider.  Sorting doesn’t remove content: it highlights certain content, allowing other content to be ignored.  Sorting should make using content easier for audiences, which is why that task needs to be delegated to computers whenever possible.

Prioritizing by Index Value: Sorting to Locate

Indexes are markers used to locate content.  With indexes, the chief role of sorting is “findability”.

Alphabetic sorting is the most widely used form of sorting.  Though clearly useful, alphabetic sorting’s value is sometimes presumed when it has none.  In many situations, alphabetic sorting merely provides a semblance of order without providing any true value beyond psychological comfort.  The chief value of alphabetic sorting arises when the user already knows about an item and expects to find it on the list.  It can help locate the item, and confirm its inclusion. Developers have use for reverse alphabetic sorting, but audiences rarely benefit from reverse alphabetic ordering.   Except in rare cases, it makes no sense to offer audiences a choice of sorting either in ascending or descending alphabetical order.

Another kind of locational sorting involves sorting items into nominal (named) categories.  For example, the fact checking website PoliFact identifies statements made by American politicians to see if they have “flipped” their previous position.  One can sort statements according to whether is a statement is: 1. Partially (half) flipped; 2. Fully “flopped”; or 3. Not flipped.  Such sorting helps audiences locate statements they might be interested in, to judge if a view is simply getting new attention, or whether it is a new position.

An item sorting according to category "Fully flopped". Source: PolitiFact
An item sorting according to category “Full Flop”. Source: PolitiFact

Prioritizing by Frequency: Sorting to Rank

Sorting is especially useful with quantitative values.  Sorting can rank items on the basis of numeric values associated with the item. Sorting on the quantity determines the ordinal ranking.

In contrast to alphabetic sorting, descending order (from high to low) is frequently useful for quantitative values. People are often looking for content relating to the highest rated item, or best performing one.

In addition to sorting by explicit quantitive values such as price, less obvious applications of rank-based sorting exist that utilize hidden or implicit information.  Behavioral values can summarize activity relating to the content, such as when articles are sorted by number of comments.  The most common form of behavioral values are consumption-related popularity values, These come in many forms, and  are widely used to sort content.  Examples included sorting by:

  • Best selling
  • Most viewed/played
  • Most recommended
  • Most downloaded
  • Most frequently bought together

Another use of quantitative sorts is to create “buckets” that rank content in summary form.  Buckets are derived data that are not explicitly visible in the content, where clustering (summarizing frequency) is performed in tandem with sorting.  Publishers create buckets covering ranges of values to summarize how frequently items appear.  Items are sorted into buckets defined by ranges (e.g., Number of items “below $100,” “$101 – 499,” “$500 and above”).  When applied to the sorting of content, these intervals don’t need to be equal in size.

In addition to ranking by a single criterion, publishers can provide different ranking perspectives that consider alternative criteria.  Alternative-criteria rankings can be complex, so care is needed to hide the complexity from audiences.  Imagine that content addresses five products: A, B, C, D, and E.  A consumer ratings website might sort-rank items according to different criteria:

  • Best picks for budget buyers: A, D
  • Best picks for power users: E , C
  • Best picks for novices: A, B

In some respects, this kind of sorting is similar to the sorting of columns in a spreadsheet.  In a spreadsheet, one can rank  alternative criteria by selecting different columns to reorder rows, to see how rankings change when various criteria are considered.  What’s different here is that editorial choices are being made instead of forcing the reader to decide what’s important to focus on.  Each category represents a theme rather than formal attribute.  For example, what’s best for power users might be a combination of the number of features a product offers, and the extent the performance of key features are above average.  That kind of score can be computed behind the scenes.  Once items are ranked, only the top two items are presented, to keep the focus on what’s best in each category.

An alternate-criteria sort, where items are ranked according to best overall and best value. Source: Consumer Reports, via
An alternate-criteria sort, where items are ranked according to best overall and best value. Source: Consumer Reports, via

Another ranking pattern is the “Top N by category” pattern.  Many publishers will curate content according to how the content ranks within different categories.  A news site might present a list of top five articles in sports, in health, in politics and in business.  The curatorial decisions relate to what categories are most important in the larger body of content, how many items (N) to show, and on what basis (number of views, comments, etc.)

A rank sorting can be applied to any content that involves a scale.  While most often rankings are based on numeric scores, they can also be used with qualitative scales such as good, better, and best.  Sorts based on qualitative ranking are known as enumerations.

Prioritizing by Time: Sorting to Sequence

Much content has a time dimension, and can be compared to other content according to various time frames.  These comparisons are made through sequence-related sorting.

Chronological sorts use dates to sort content.  They can be valuable in either ascending (earlier to later) or descending (recent to older) order.  Ascending chronological order is useful when content items reference and build on each other.  For example, I find live blogged stories easier to follow when they are in chronological order, since many statements will depend on what was said in prior statements.  Many live blogs choose reverse chronological order, however.  The benefit of reverse chronological order is to highlight the newest information, which is assumed to be more important than older information.  Reverse chronological order works best when each item is independent of the others, so some live bloggers try to make their statements be understandable without referring to other content items.

An important way to sort content is by how stale or fresh it is.  Computer scientists refer to stale content as LRU or “Least Recently Used,” where LRU content is considered obsolete and is purged from the computer’s cache memory. The concept of freshness is captured by a term borrowed from accounting known as LIFO or “Last In, First Out”.  Content that is new or changed is generally more valuable than older content.  People look for new content, or content that’s been updated.

LIFO is useful as a way to sort behavioral data.  In many situations, the content someone most recently viewed will be the most likely content they will want to use again.  Imagine someone constantly checking some pages relating to the stock performance of different companies they own or are considering buying.  Because they routinely check these pages, the sorting would present these pages as a list ordered according to how recently they were viewed, with the most recently viewed at the top of the list.  LIFO behavioral sorting is dynamic, changing as audience interests do, so that new items get added immediately, fading interests soon disappear from the list.  It is more flexible, and less work, than having the audience create a custom list.

Sequences are a special kind of list where the order is predefined.  Content relating to a sequence should be automatically sorted.  Sequences can take various forms:

  • Procedural with antecedent dependencies, such as “Step 1, Step 2”
  • Time-defined, such as “Phase 1, Phase 2”
  • Life-cycle or life-stage based, e.g., “R&D stage, Trials stage, Approval stage, Marketing stage”

A big use of sequences is to position content in time, providing a context for the sorted content.  Three kinds of content sequences are:

  1. Before/After sequences
  2. Now/Next sequences
  3. Lower/Higher sequences

Before/After sequences position a topic within a spectrum of time.  When items are classified according to their stage, they can be compared.  Items at the same stage are similar, and one can locate content about events that preceded that stage, and identify content about other entities that are in a more advanced stage.  An example of such a sequence sorting would be articles about a class of new drugs, where different companies are in different stages of market introduction.

Now/Next sequences are similar to Before/After, but are more focused on content relating to a single person or entity, rather than a range of entities.  Content can be sorted according to what matches the current context, and list content that will be relevant to the subsequent stage.  An example of Now/Next sorting is content about repairing a product.

Lower/Higher sequences define time in terms of proficiency required.  The content is ranked according to the abilities of the reader.  Publishers frequently classify content according to the level of expertise needed to understand the content.  Content might be classified as Beginner, Basic, Intermediate, Advanced, or Expert.   The sequence associated with those labels is generally easy to understand.  Alternatively, the publisher could rank the difficulty of the content according to a color code:

  • White belt (= Beginner)
  • Yellow belt (= Basic)
  • Blue belt (= Intermediate)
  • Purple belt (= Advanced)
  • Black belt (= Expert)

Such rankings can be useful provided the audience knows the level they are at.  They might graduate from one tier after viewing all content in that tier.  The content sorting might identify and show all Yellow belt content that the reader has not yet seen.

Prioritizing by Novelty: Sorting to Sample

While sorting is generally thought of as involving either ascending or descending order, publishers can also sort items in a statistically random order. This ensures that items presented are unique each time a page loads.

Random sort offered by a content management system. Source: Webflow
Random sort offered by a content management system. Source: Webflow

For audiences, random sorting provides novelty, offering something that may not have been encountered previously.  Random sorting and selection can make content viewing more interesting provided the item pool represents content of potential interest.

Random presentation of content can be used to discover if certain content receives greater than expected attention.  Publishers might discover through random promotion of content that audiences are interested in topics that previously did not receive much attention when prioritized by frequency of views.

Prioritizing by Relationships: Sorting to Profile and Evaluate

Sometimes audiences need to sort content to see the relationships between items.  One example would be to see what’s more general and what’s more specific.  For example, Wikipedia uses a hierarchy based on categories, subcategories, and pages.  The entry for a category will be broader in scope than an entry for a page that’s not a category.

For many topics, audiences understand which items are broader than others.  But for more specialized fields, automated sorting of topics from broader to narrower is useful.  Suppose someone encountered content relating to enzymes.  They see content on the following topics:

  • Acid preparations
  • Digestives
  • Betaine hydrochloride

The list has no order.  Unless the reader is a specialist, they would not know which topic is the most specific.  The sorting order from broader to narrower would be Digestives > Acid preparations > Betaine hydrochloride.  Hierarchies provide context for the content, indicating what is background and what is detail.

Websites typically present hierarchies to audiences as an input into a task to complete.  The website will require audiences to assess visual relationships in a menu, and select the narrower option in a manual process of “drilling down”.   Alternatively, publishers can incorporate such sorting into an automated presentation of content, where the content order is predetermined for the audience.  Two common patterns are to show broader topics followed by narrower examples, and to show a narrow example then present the broader topic it represents.  Instructional content often utilizes non-visible, programmatic hierarchies to guide presentation of content, to ensure comprehension, or to encourage the review of key concepts.

Nested sorts involve sorting on two or more criteria, such as sorting on a qualitative category and a numeric value together.  They are difficult for users to specify themselves, so are best offered pre-packaged.  Nested sorts are useful for dynamic content that will change frequently.  PolitiFact, the Pulitzer-Prize winning website, rates statements by politicians according to their veracity.  A nested sort allows the audience to see how many statements by a politician were associated with different categories of truthfulness:

  • True
  • Mostly True
  • Half-True
  • Mostly False
  • False
  • Pants on Fire!
Nested sort of statements made by Tim Kane. Source: PolitiFact
Nested sort of statements made by Tim Kane. Source: PolitiFact

Automated Editorial Sorting

Content designers and content engineers should consider how computers can order lists to deliver the greatest audience benefits.  This approach can be described as automated editorial sorting.  It is automated, in the sense that a computer algorithm performs the sort without requiring user interaction.  And it is editorial in the sense that prioritization delivered by the sort reflects a judgment concerning what information is most valuable to highlight.

Sorting should not be treated as generic functionality that can be applied indiscriminately to any kind of content.  Sorting  should provide context for audiences. To be valuable, sorting should surface the content that audiences consider to be their highest priority.  It is not enough to give audiences tools to dig out that information themselves.  Audiences expect publishers to anticipate what they need, and present content to them in a well ordered manner.

— Michael Andrews

Format Free Content and Format Agility

A core pillar supporting the goal of reusable modules of content is that the content should be “format free”.  Format free conveys a target for content to attain, but the phrase tends to downplay how readily content can be transformed from one state to another.  It can conceal how people need to receive content, and whether the underlying content can support those needs.

I want to bring the user perspective into the discussion of formats.  Rather than only think about the desirability of format neutrality, I believe we should broaden the objective to consider the concept of format readiness.  Instead of just trying to transcend formats, content engineers should also consider how to enable customized formats to support different scenarios of use.  Users need content to have format flexibility, a quality that doesn’t happen automatically.   Not all content is equally ready for different format needs.

The Promise and Tyranny of Formats

Formats promise us access to content where we want it, how we want it. Consider two trends underway in the world of audio content.  First, there is growing emphasis on audio content for in-car experiences.  Since staring at a screen while driving is not recommended, auto makers are exploring how to make the driving experience more enriching with audio content.  A second trend goes in the opposite direction.  We see a renewed interested in a nearly dead format, the long playing record disc, with its expressive analog sensuality.  Suddenly LPs are everywhere, even in the supermarket.  The natural progression of these trends is that people buy a record in the supermarket, and then play the record in their car as soon as they reach the parking lot. An enveloping sonic experience awaits.

Playing records in your car may sound far fetched.  But the idea has a long pedigree.  As Consumer Reports notes: “A new technology came on the market in the mid-1950s and early 1960s that freed drivers from commercials and unreliable broadcast signals, allowing them to be the masters of their motoring soundtrack with their favorite pressed vinyl spinning on a record player mounted under the dash.”

Highway Hi-Fi record player. Image via Wikipedia.
Highway Hi-Fi record player. Image via Wikipedia.

In 1956, Chrysler introduced Highway Hi-Fi, an in-dash record player that played special sized discs that ran at 16 ⅔ rpms — half the speed of regular LPs, packing twice the playtime.  You could get a Dodge or DeSoto with a Highway Hi-Fi, and play records such as the musical the “Pajama Game.”  The Highway Hi-Fi came endorsed by the accordion playing taste maker, Laurence Welk.

Sadly playing records while driving in your car didn’t turn out to be a good idea.  Surprise: the records skipped in real-world driving conditions.  Owners complained, and Chrysler discontinued the Highway Hi-Fi in 1959.  Some hapless people were stuck with discs of the Pajama Game that they couldn’t play in their cars, and few home stereos supported 16 ⅔ play.  The content was locked in a dead format.

Format Free and Transcending Limitations

Many people imagine we’ve solved the straight jacket of formats in the digital era.  All content is now just a stream of zeros and ones.  Nearly any kind of digital content can be reduced to an XML representation.  Format free implies we can keep content in a raw state, unfettered by complicating configurations.

Format free content is a fantastic idea, worth pursuing as far as possible.  The prospect of freedom from formats can lead one to believe that formats are of secondary importance, and that content can maintain meaning completely independently of them.

The vexing reality is that content can never be completely output-agnostic.  Even when content is not stored in an audience-facing format, that doesn’t imply it can be successfully delivered to any audience-facing format. Computer servers are happy to store zeros and ones, but humans need that content translated into a form that is meaningful to them.  And the form does ultimately influence the substance of the content.  The content is more the file that stores it.

Four Types of Formats

In many cases when content strategists talk about format free content, they are referring to content that doesn’t contain styling.  But formats may refer to any one of four different dimensions:

  1. The file format, such as whether the content is HTML or PDF
  2. The media format, such as whether the content is audio, video, or image
  3. The output format, such as whether the content is a slide, an article, or a book
  4. The rendered formatting, or how the content is laid out and presented.

Each of these dimensions impacts how content is consumed, and each has implications for what information is conveyed.  Formats aren’t neutral.  One shouldn’t presume parity between formats.  Formats embody biases that skew how information is conveyed.  Content can’t simply be converted from one format to another and express the content in the same way.

Just Words: The Limitations of Fixed Wording

Let’s start with words.  Historically, the word has existed in two forms: the spoken word, and the written word.  People told stories or gave speeches to audiences.  Some of these stories and speeches were written down.  People also composed writings that were published.  These writings were sometimes read aloud, especially in the days when books were scarce.

Today moving between text and audio is simple.  Text can be synthesized into speech, and speech can be digitally processed into text.  Words seemingly are free now from the constraints of formats.

But converting words between writing and speech is more than a technical problem.  Our brains process words heard, and words read, differently.  When reading, we skim ahead, and reread text seen already.  When listening, we need to follow the pace of the spoken word, and require redundancy to make sure we’ve heard things correctly.

People who write for radio know that writing for the ear is different from writing for a reader.  The same text will not be equally effective as audio and as writing. National Public Radio, in their guidebook Sound Reporting, notes: “A reader who becomes confused at any point in [a] sentence or elsewhere in the story can just go back and reread it — or even jump ahead a few paragraphs to search for more details.  But if a listener doesn’t catch a fact the first time around, it’s lost.”  They go on to say that even the syntax, grammar and wording used may need to be different when writing for the ear.

The media involved changes what’s required of words.  Consider a recipe for a dish.  Presented in writing, the recipe follows a standard structure, listing ingredients and steps.  Presented on television, a recipe follows a different structure.  According to the Recipe Writers Handbook, a recipe for television is “a success when it works visually, not when it is well written in a literary, stylistic, or even culinary sense.”  The book notes that on television: “you must show, not tell; i.e., stir, fry, serve…usually under four minutes.”  Actions replace explicit words.  If one were to transcribe the audio of the TV show, it is unlikely the text would convey adequately how to prepare the dish.

The Hidden Semantics of Presentational Rendering

For written text, content strategists prudently advise content creators to separate the structure of content from how it is presented.  The advice is sensible for many reasons: it allows publishers to restyle content, and to change how it is rendered on different devices. Cascading Style Sheets (CSS), and Responsive Web Design (RWD) frameworks, allow the same content to appear in different ways on different devices.

Restyling written content is generally easy to do, and can be sophisticated as well.  But the variety of CSS classes that can be created for styling can overshadow how rudimentary the underlying structures are that define the meaning of the text.  Most digital text relies on the basic structural elements available in HTML.  The major elements are headings at different levels, ordered and unordered lists, and tables.  Less common elements include block quotes and code blocks.  Syntaxes such as Markdown have emerged to specify text structure without presentational formatting.

While these structural elements are useful, for complex text they are not very sophisticated.  Consider the case of a multi-paragraph list.  I’m writing a book where I want to list items in a series of numbered statements.  Each numbered statement has an associated paragraph providing elaboration.  To associate the explanatory paragraph with the statement, I must use indenting to draw a connection between the two.  This is essentially a hack, because HTML does not have a concept of an ordered list item elaboration paragraph.  Instead, I rely on pseudo-structure.

When rendered visually, the connection between the statement and elaboration is clear.  But the connection is implicit rather than explicit.  To access only the statement without the elaboration paragraph, one would need to know the structure of the document beforehand, and filter it using an XPath query.

Output Containers May Be Inelastic

Output formats inform the structure of content needed.  In an ideal world, a body of structured content can be sent to many different forms of output.  There’s a nifty software program called Pandoc that lets you convert text between different output formats.  A file can become an HTML webpage, or an EPUB book, or a slide show using Slidy or DZSlides.

HTML content can be displayed in many containers. But those containers may be of vastly different scales.  Web pages don’t roll up into a book without first planning a structure to match the target output format.  Books can’t be broken down into a slide show.  Because output formats inform structure required, changing the output format can necessitate a restructuring of content.

The output format can effect the fidelity of the content. The edges of a wide screen video are chopped off when displayed  within the boxy frame of an in-flight entertainment screen.  We trust that this possibility was planned for, and that nothing important is lost in the truncated screen. But information is lost.

The Challenges of Cross-Media Content Translation

If content could be genuinely format free, then content could easily morph between different kinds of media.  Yet the translational subtleties of switching between written text and spoken audio content demonstrate how the form of content carries implicit sensory and perceptual expectations.

Broadly speaking, five forms of digital media exist:

  1. Text
  2. Image
  3. Audio
  4. Video
  5. Interactive.

Video and interactive content are widely considered “richer” than text, images and audio.  Richer content conveys more information.  Switching between media formats involves either extracting content from a richer format into a simpler one, or compiling richer format content using simpler format inputs.

The transformation possibilities between media formats determine:

  • how much automation is possible
  • how usable the content will be.

From a technical perspective, content can be transformed between media as follows.

Media format conversion is possible between text and spoken audio.  While bi-directional, the conversion involves some potential loss of expressiveness and usability.  The issues become far more complex when there are several speakers, or when non-verbal audio is also involved.

Various content can be extracted from video.  Text (either on-screen text, or converted from spoken words in audio) can be extracted, as well as images (frames) and audio (soundtracks).  Machine learning technologies are making such extraction more sophisticated, as millions of us answer image recognition CAPTCHA quizzes on Google and elsewhere.  Because the extracted content is divorced from its context, its complete meaning is not always clear.

Transforming interactive content typically involves converting it into a linear time sequence.  A series of interactive content explorations can be recorded as a non-interactive animation (video).

Simple media formats can be assembled into richer ones.  Text, images and audio can be combined to feed into video  content.  Software exists that can “auto-create” a video by combining text with related images to produce a narrated slide show.  From a technical perspective, the instant video is impressive, because little pre-planning is required.  But the user experience of the video is poor, with the content feeling empty and wooden.

Interactive content is assembled from various inputs: video, text/data, images, and audio formats.  Because the user is defining what to view, the interaction between formats needs to be planned.  The possible combinations are determined by the modularity of the inputs, and how well-defined they are in terms of metadata description.

translation of content between formats
Translation of content between formats

Atomic Content Fidelity

Formats of all kinds (file, output, rendering, and media) together produce the form of the content that determines the content experience and the content’s usability.

  • File formats can influence the perceptual richness (e.g., a 4k video verses a YouTube-quality one).
  • Rendition formatting influences audience awareness of distinct content elements.
  • Output formats influence the pacing of how content gets delivered, and how immersive content the content engagement will be.
  • Media formats influence how content is processed cognitively and emotionally by audiences and viewers.

Formats define the fidelity of the content that conveys the intent behind the communication.  Automation can convert formats, but conversion won’t necessarily preserve fidelity.

Format conversions are easy or complex according to how the conversion impacts the fidelity of the content.  Let’s consider each kind of content format in turn.

File format conversions are easy to do, and any loss in fidelity is generally manageable.

Rendition format conversions such as CSS changes or RWD alternative views are simple to implement.  In many cases the impact on users is minimal, though in some cases contextual content cues can be lost in the conversion, especially when  a change in emphasis occurs in what content is displayed or how it is prioritized.

Output format conversion is tricky to do.  Few people want to read an e-book novel on their Apple Watch.  The hurdles to automation are apparent when one looks at the auto-summarization of a text.  Can we trust the software to identify the most important points? An inherent tension exists between introducing structures to control machine prioritization of content, and creating a natural content flow necessary for a good content experience.  The first sentence of a paragraph will often introduce the topic and main point, but won’t always.

Media format conversion is typically lossy.  Extracting content from a rich media format to a simpler one generally involves a loss of information.  The automated assembly of content rich media formats from content in simpler formats often feels less interesting and enjoyable than rich formats that were purposively designed by humans.

Format Agility and Content as Objects

We want to transcend the limitations of specific formats to support different scenarios.  We also want to leverage the power of formats to deliver the best content experience possible across different scenarios.  One approach to achieve these goals would be to extend some of the scenario-driven, rules-based thinking that underpins CSS and RWD, and apply it more generally to scenarios beyond basic web content delivery.  Such an approach would consider how formats need to adjust based on contextual factors.

If content cannot always be free from the shaping influence of format, we can at least aim to make formats more agile.  A BBC research program is doing exciting work in this area, developing an approach called Object Based Media (OBM) or Object Based Broadcasting.  I will highlight some interesting ideas from the OBM program, based on my understanding of it reading the BBC’s research blog.

Object-Based Media brings intelligence to content form.  Instead of considering formats as all equivalent, and independent of the content, OBM considers formats in part of the content hierarchy.  Object Based Media takes a core set of content, and then augments the content with auxiliary forms that might be useful in various scenarios.  Content form becomes a progressive enhancement opportunity.  Auxiliary content could be subtitles and audio transcripts that can be used in combination with, or in leu of, the primary content in different scenarios.

During design explorations with the OBM concept, the BBC found that “stories can’t yet be fully portable across formats — the same story needed to be tailored differently on each prototype.” The notion of tailoring content to suit the format is one of the main areas under investigation.

A key concept in Object-Based Media is unbundling different inputs to allow them to be configured in different format variations on delivery.  The reconfiguration can be done automatically (adaptively), or via user selection.  For example, OBM can enable a video to be replaced with an image having text captions in a low bandwidth situation.  Video inputs (text, background graphics, motion overlays) are assembled on delivery, to accommodate different output formats and rendering requirements.  In another scenario, a presenter in a video can be replaced with a signer for someone who is hearing impaired.

The BBC refers to OBM as “adjustable content.”  They are looking at ways to allow listeners to specify how long they want to listen to a program, and give audiences control over video and audio options during live events.

Format Intelligence

In recent years we’ve witnessed remarkable progress transcending the past limitations that formats pose to content.  File formats are more open, and metadata standards have introduced more consistency in how content is structured.  Technical progress has enabled basic translation of content between media formats.

Against this progress taming idiosyncrasies that formats pose, new challenges have emerged.   Output formats keep getting more diverse: whether wearables or immersive environments including virtual reality.  The fastest growing forms of content media are video and audio, which are less malleable than text.  Users increasingly want to personalize the content experience, which includes dimensions relating to the form of content.

We are in the early days of thinking about flexibility in formats that give users more control over their content experience — adjustable content.  The concept of content modularity should be broadened to consider not only chunks of information, but chunks of experience.  Users want the right content, at the right time, in the right format for their needs and preferences.

— Michael Andrews