Categories
Intelligent Content

Content Variables in Content Engineering

Content engineering is an important new approach that can offer greater precision and flexibility in how content is delivered to audiences.   It defines how content is developed, and how it is made available to audiences in different situations.  Much of the flexibility and precision of content engineering comes of its use of content variables.  However, the topic of content variables has received limited discussion thus far.  I want to offer some ideas to advance thinking about this important dimension.

Prevailing Concepts of Variables in Content

Discussions of variables in content tend to treat variables as a list of items that can be reused.  For example, according to MadCap software, a popular tool for managing technical content: “Variables are pre-set terms that you can use in your project over and over.” They elaborate: “Variables are used for brief, non-formatted pieces of content (such as the name of your company’s product or your company’s phone number).”  This explanation assumes the term is “pre-set” and implies a simple substitution of a word or phrase in a location. The problem with looking at a variable as a “term” is that is doesn’t distinguish between what something is called, and what its value is.

Another common form of substitution is to swap out big sections of content to create different versions.  But treating an entire section as a single variable falls short of the precision and flexibility promised by content engineering.

The function of variables in content management systems can be opaque to authors.  In Drupal, for example, some variables may be unique to particular templates, while others are common to all.  Authors don’t have control over where variables can be used in many CMSes. Variables are often administrative in character. A variable might represent “last updated,” which is supplied by the system. There may not be a clear distinction between which variables are strictly backend variables, and which are audience-facing.

Discussion of variables has largely focused on making things simpler for authors, relieving them from having to recreate snippets of content.  While that’s a solid motivation, it focuses on business efficiency rather than on business growth.  The goal should be to consider the business value of a content variable, and offer specific content that offers the highest value to customers and the brand.

photo of bird
This bird is called a variable oystercatcher.  I would see them when I lived in Wellington.  Seemed fitting to have a bird named “variable” looking to catch oysters to inspire us to poke around to find something valuable.  Photo by Tony Wills via Wikipedia.

Application of Content Variables

Content engineering evaluates different content needs, and translates these needs into requirements and designs that support the goals of both audiences and brands. Designated variables reflect the needs of the audience, the business, and the design of the content incorporating the variable. The specification of variables is a core ingredient of content engineering.

  • Audience requirements: From an audience perspective, variables can highlight aspects of content that are engaging.  Our visual perception is drawn to what’s changed, because it might be interesting, or be of concern.  We want content that seems relevant, which is often related to dimensions of time, geography, or frequency.  We desire content that’s personalized, tailored to our specific needs.
  • Business requirements: Brands want to present different content in different situations.   They don’t want their content to be wooden: it needs to respond both to whom the audience is, and to different operational needs of the brand.  Variables enable optimization of content.  With variables, content can change dynamically based on usage, or customer and business analytics.  The variables can reflect — and express — the brand’s business model.
  • Content design: Content design translates the audience and business requirements into specifications that are implemented to deliver content as desired.  These specifications define elements, and associated metadata and business rules, to display different content elements in various circumstances. Variables in these specifications delineate what precise content appears within what section of content.

Variables in Declarative and Imperative Content Objects

Rather than viewing variables as simple text snippets, the content engineering approach considers variables as living content objects, a bit like a Tamagotchi.  Content professionals can benefit by leveraging ideas about variables from computer programming.  While there are differences between how backend systems process variables and how they need to be displayed to audiences, there are broad similarities. Because I want to locate useful common ground, I am going to highlight these similarities in a simplified manner.

Computer programs typically rely on either declarative statements (a statement of what happens) or imperative statements (a statement of what should happen).  Ideally, variables will support not just declarative content (e.g., place content with a given mark up in this place, such as <place author’s name here>.) It will also support imperative content — content exhibiting some novelty when generated through the use of a sequence flow, a test condition, and a calculation.

Declarative statements specify what information to show, rather than how to develop information.  It says “SELECT  {item}… FROM {set of items}….WHERE {qualifying criteria}…”   The qualifying criteria tends to be simple, using an attribute that already exists.  Sometimes there is a compound criteria (e.g., include this but not that), but the filtering still uses known criteria. Declarative programs are typically simpler than imperative ones, and are used when the kind of information that is available is already known. The best examples are SQL database queries, or instructions for accessing content available through an API.

Imperative statements specify how to develop the information to show. Most programming languages that use a sequence of instructions rely on imperative programming. Unlike declarative statements, imperative ones produce derived variables.  Such scripts use content variables to deliver fresh information to audiences. A simple example of a derived variable is a list of the top five most popular articles for a given week. The list represents an imperative variable, telling the computer how to create the list, but not knowing in advance what the information will be.  The information is recalculated each week: the content itself subject to change. A more complex example would be showing what articles are popular with your neighbors.  The list has to respond to an abstract concept of a neighbor, which might be based on visitors to an app or site whose IP address or GPS location indicate they are within 20 kilometers of your current location.

Scripts support business goals as well. Content variables can work with marketing automation and predictive analytics technologies. Digital content is becoming more dynamic, optimized based on known and inferred preferences and predicted behaviors. Such calculation requires imperative programs, and will be essential for personalization of content in the future.

Discussions of structured content, particularly within the technical writing community, have focused on declarative content rather than imperative content. In addition to supporting complex but static publishing of multiple versions of a document, content engineering also produces imperative content to address an infinite range of scenarios, and to provide dynamic, realtime information. We need to be able to support both forms of content: content that’s predetermined, as well as content that’s a more fluid and adaptive. Declarative statements are easier to implement, but imperative ones enable more flexibility and customization.

How can content engineers, who are responsible for developing audience and business requirements, identify needed content variables?  Concepts in programming suggest a framework for thinking about audience uses of variables that can be broadly related to the needs of backend systems.  How backend systems need to handle variables efficiently will be different from how audiences think about them, and even among developers and between programming languages, terminologies can vary. The terrain at times can seem confusing, but I believe the foundations center on three elements.

Framework for Content Variables: Labels/Values/Context

Variables need to be considered in three parts:

  1. The labels used to describe them
  2. The values associated with them
  3. The context in which they are used.

The diagram below introduces a content variable framework addressing the relationship between labels, values, and context.

diagram: content variables
A framework for considering variables in content engineering

Variables are a combination of a label (an attribute identifier) and a value.   The label is the variable’s name. The term label is more helpful than name, because it captures how audiences perceive the identity of what they see.

Values are bound to labels.  The nature of the binding can be subtle: some values can change, others can’t.  Labels for an attribute can potentially change as well.  And differing contexts can redefine both labels and values.

To understand the difference between what something is called, and what its value is, consider two situations.  In one case, people are interested in knowing how many times a given thing appears — counting the number of instances of an item that is labeled in a certain way. For example, how many articles have been written by a given author? In another case, audiences are interested in knowing how many different items (items having distinctly different labels) share the same value.  For example, how many other items have been shared on Facebook as frequently as this article? These examples  are admittedly trivial, but one can highlight interesting and meaningful relationships when content items have multiple attributes.

Range: How Variables Vary

Some attributes have persistence, while other attributes are fluid.  We describe the difference in character as attributes with fixed values, and attributes with mutable (changeable) values.  Knowing whether the range of values for a variable is fixed, or mutable, is a central question.

Fixed variables have values that are frozen. They are identifiers — any ID-type value needs to be fixed. They express discrete (non-continuous) properties. Fixed variables are predefined, rather than being dynamically defined. Text variables are generally fixed: think of a value of a state or province. Other textual variables, the word that describe colors of clothing, or the name of a conference event, are fixed as well. Predefined numeric variables such as postal codes or phone numbers are also fixed — they don’t fluctuate over time. Due to their role as identifiers, fixed values often have rules for what values are considered valid.  Fixed variables are constrained according to these rules, and entries may be validated.  Because their value doesn’t vary over time, they can be counted, and one can count the number of instances of a variable, and determine sets of content items having these defined characteristics. Fixed values are handy for declarative variables: show items with a value of X that does not have a value of Y.

Fixed values can be “keys” that can be mapped to other content. If a content item has an author of a given name, that name is a fixed value.  The author’s name can be mapped to other articles she has written. If there are several fixed values associated with the content, one can find a set of items having common keys. Some fixed values, such as an ISBN number for books, are unique, allowing the attribute to identity the larger item with which it is associated.

Another kind of fixed value is an ordered sequence. The order will never change: the forth U.S. President will always be number four on the list.

Mutable variables are the opposite of fixed ones: the value mutates over time. Mutable variables are dynamic — we expect the value to change. Suppose you are looking for content about vacation destinations that are warm, or stocks that are undervalued. The values associated with those attributes are in constant flux, depending on the weather or the stock market. Many content variables fluctuate in relation to other variables, responding to interactions with other content and with audiences.

Mutable variables are often numeric, representing a continuous range of values. They may relate to the content itself: a count of content items, or analytical data about the audience’s use of the content. Alternatively, the values can represent real world properties that can change, such as prices or inventory levels.

The other major type of mutable content is lists. Lists are collections of items, and items can be added, deleted and reordered. The total number of items will fluctuate. List variables can contain text items that are described with fixed values, but the collection of values in the list itself is open to change.

Mutation Verses Reassignment

Time now to address a technical aspect of variables that can cause confusion: the difference between the assignment of values, and the behavior of values.  It may sound somewhat contradictory to speak of a fixed variable: what varies if the value is fixed?  The answer is the assignment of the fixed value to the label of the attribute.  I have a phone number with a fixed value, a series of digits.  It is different from your phone number, so it is a variable.  But my phone number does not change from day to day. It is not mutable.

Now, suppose I move house. I might speak about changing my phone number, but strictly speaking I am not changing the number itself (its value). I am having a different number assigned to me. My old phone number has a fixed value, and continues to exist, ready to be assigned to someone else. Meanwhile, I acquire a new phone number because an existing number has been reassigned to me from an inventory of numbers not being used.

When two things are related to each other, and both have fixed identities, to make a change, they must be reassigned.

Because of the significant difference between reassignment and mutation, it is best to avoid speaking generically about “changing” a variable.

Scope: Where Variables Can Be Applied

Another key aspect of variables is where they can be used, and what precisely they refer to.  Some variables are comprehensive variables: they refer to the brand as a whole, and all people who interact with the brand. Other variables are more limited in their scope. Scope determines both where a variable can be used, and how a variable is presented. We can think of variable scope in three buckets;

  • Global variables
  • Behaviorally framed variables
  • Local variables

Global variables are the simplest to understand.  They can be used anywhere, and wherever they appear, they mean the same thing. So if an article is listed as the most visited article, it is the most visited article for all platforms, the most visited by all audiences. Everyone sees the same information, regardless of who they are, or what platform they are viewing.

Behaviorally framed variables are the most complex. They are variables where the value is dependent on conditional characteristics, such as time period, audience segment, promotional campaign, or other behavior that is at least partly extrinsic to the content item. The variable is placed within a context: the time frame, the person, or reference group the variable relates to. Consider the seemingly straightforward variable of price. The price shown of an airline ticket can depend on many factors, external and internal. It may depend on timing: different times of year command different prices.  Externally defined variables may have different business rules governing them, reflecting demand and supply.  But other factors are less impersonal. The value presented can depend on who the person is: people who have certain characteristics such as past buying behavior or presumed willingness to pay a given price will be offered different prices.  People who use certain browsers, or who are based in certain locations, may be presumed to pay more or less for an item, and accordingly be shown a different price. The variation in content values can be extended well beyond literal values such as prices to more experientially-focused content variations (such as what order of a list of items to present) that depend on a host of behavioral characteristics. For now, I just want to introduce the notion that a variable relates not just to the label it is associated with, but also the context in which it is viewed. That context may not be explicitly identified to the person viewing the variable.

Finally, local variables apply to how and where content is shown in different content types. For simplicity, let’s consider content types as being a distinct page section, widget, card, or tile, that displays content. The values shown in each of these may be specific to these elements, such as when you present the “most popular <content type>.” Local variables can be more impactful when used to populate specific content types with specific variable content. Suppose you designate some content as a “special promotion for this week.” That designation may trigger a discount (a behaviorally framed adjustment) and also invoke a rule concerning where the content should appear. Designated items might be displayed in a special section of tiles dedicated to promotions, in addition to its default content type location. Content that meet certain criteria can be promoted in various content types to amplify how often it is seen. Viewed from a pull perspective, a content type will include content variables that satisfy certain criteria.

Label Morphing: Getting Microcopy Right

We like to think of the name of our variable as being reliable.  While information architects and copywriters understand the importance of having clearly worded labels, there hasn’t been much attention given to how labels should change in different circumstances. Labels need to adapt to two factors: values, and context. There are two types of label adaption:

  1. Label – value agreement
  2. Label aliasing

Label-value agreement specifies how the label needs to change to accommodate differences in values associated with the label.  The most common example is pluralizing a label, though sometimes the label reads better when the entire label wording changes: for example, one person changes into two people. Some developers squawk at the extra effort required for such label changes, and copywriters lacking an understanding of variables have defaulted to work-arounds that avoid using words that trigger agreement changes. But as personalized, dynamic content becomes more ubiquitous, it is important to build the capability to have labels adapt dynamically with changes in value. Although English is less complex than some languages when it comes to rules for pluralization, it does have numerous special cases: words ending in -o, ending in -y, or ending in -f, proper nouns, and compound nouns. Rather than try to develop a complex rule set to generate plurals imperatively, it is easier to approach the problem declaratively by stating that content should display the proper singular or plural form of a countable noun, and supply the exact wording for each.

Another dimension is how labels may need to adapt to time. For labels involving status-related verbs, the wording needs to change dynamically according to the period of time specified. Differences between past tense and past particle need to be indicated. “You gave $100 in 2012” needs to be rewritten “You have given $200 since 2012” when the user changes her query.  This example highlights how labels themselves can be interactive elements.

Aliases are necessary when an attribute needs to be referred to by different names in different contexts. While consistent labeling is beneficial for helping people learn and understand quickly what you say, rigidly consistent labels are often a hindrance. Adaptive content implies that content labels should adapt to different contexts. The same value, representing the same attribute may be used in many different contexts, and needs to be referred to in slightly different ways. The most obvious example is the limited space on a small mobile device: even if a longer label might be ideal, such a label would interfere with the presentation of content. Tables often use short labels or even abbreviations. Labels adapt not only to space constraints, but also to address the degree of potential ambiguity. Labels for commonly recurring variables can be shorter, as they will be familiar.

Alias names used may be longer or shorter or abbreviated, and may be more formal or less formal.  For example, a label saying “vehicle” is shorter, but more formal, than one saying “cars and trucks.” Consider the scope in which a variable will appear, and develop appropriate alias options for different content types.

Inter-relationships

The three elements of variables — labels, values, and context — work in concert. How something is labeled can depend on its value and the context in which it appears. Dynamic, mutable values are often likely to be behaviorally framed, considering the context of who is viewing the variable, or changing according to a when the variable is viewed.  Such mutable values also need to have labels that agree with the values presented.

There are many more dimensions to content variables, particularly prioritization of content and the dynamics of whether to show variables, or not, depending on audience behaviors — issues of sequencing and level of detail. In practice, personalization is accomplished through how a collection of variables are presented together.

Because the topic of content variables is not well-developed, it is especially tricky to get the terminology right, making it understandable to content creators and software developers alike. I’ve tried my best to avoid existing jargon that carries diverse meaning or no meaning to different stakeholders.  I would welcome suggestions for improvements.

— Michael Andrews

Categories
Intelligent Content

Types of Content Structure

Paradoxically, even though content strategists frequently speak of the advantages of semantic structure for content, well structured digital content is far from the norm. Most published digital content has far less structure than might benefit it. Parties who work with content in an IT or marketing capacity can hold different ideas about what structure is. People may favor structure, but focus on different benefits. Structure is a woolly, abstract word that sounds solid.

As a practical matter, no one-size-fits-all solution can provide content with structure in all situations successfully. To advocate a single solution is to limit the adoption of structured content and its benefits. Different degrees of structure are possible for content. Structure can enable many different things. It is beneficial to know how much structure is needed, and what each threshold offers. Structured content deserves structured thinking.

It may be tempting to advocate for all content to be structured with the most robust metadata possible. Some people may even suggest that one can’t have too much structure: it’s bound to be useful sooner or later. But we must consider the costs of structuring content. Structure is expensive. What outcome does the structure accomplish? The task of content strategists is to make all content useful, within the constraints of what exists and what changes are possible. By recognizing these constraints one can develop an appropriate level of structure for a project.

Rather than simply advocate for structure, we need to be able to answer:

  • What content needs to be structured, and what is less critical?
  • What kinds of structure are best to apply in what situations?

Implementation Diversity

Content strategists typically discuss content structure in one of two ways. They may talk about structure in a generic way, without getting into any specifics about how to implement structured content. They may even suggest how you do it is less important than that you do it. Alternatively, they may focus exclusively on one specific implementation approach, such as the DITA format for technical communication, and give the impression that their favored implementation approach will satisfy any and all requirements. Both these approaches, the generic “just do it” and the specific “do it this way,” tend to minimize the diversity of content structure and its nuances.

Structure is a continuum. Each of the following content components may appear together when presented to audiences:

  1. Unstructured content, blobs that can contain anything without restrictions, such as a user comment.
  2. Semi-structured content, where there is a blob that has some selective structural description either framing it, or embedded within part of it, such as the body of an article that marks up the key people and things that are mentioned.
  3. Fully structured content, where all content elements are validated, such as a fact box showing realtime information.

Structure is defined through metadata, which allows humans and machines to understand what the content is about. Metadata can support different functions and convey different degrees of exactness. Together these factors determine what can be done with the content.

How to implement structure can be challenging to discuss comprehensively, because it involves four major syntaxes that are different from each other:

  1. HTML elements, which can include microformat extensions
  2. XML format
  3. JSON format
  4. The catalog and schema in a relational database

A content resource may be composed of items that come from multiple repositories or applications that use different syntax to structure the content. Different syntaxes favor different approaches to structuring content. They reflect differences in how content is stored, accessed and queried. Content described by different syntaxes needs to be able to work together.

Degrees of Structure: From Implicit to Explicit

Structure may be implied, or highly formal. The progression from lesser to greater rigor involves increasing costs (levels of effort) and benefits (data accuracy, interoperability, and functional capabilities). A summary of the requirements and benefits are illustrated in the chart below.

metadata levels

At the most basic level, content can have an implied structure. The meaning of content can be inferred through its proximity and regular patterns of presentation. Tabular data often has implied structure: a regular layout, with either column or row headings offering a quasi-metadata description of the content. While implied structure is not optimized to be machine readable, it can be consumed by machines. Google recently announced structured snippets where Google is able to infer through machine learning the key facts embedded in content presented within a table. Another example of implied structure is open data contained in a CSV format: machines can read the content, but it needs to have formal metadata applied to the tabular data in order to be useful.

A level above is content that is marked up for internal purposes. HTML, being a generic standard, allows elements of content to be marked up in various ways to support a range of uses. Content elements may be given an ID and be assigned to a class. These identifiers may support the styling of the content or the presentation and behavior of elements with CSS and Javascript. Such IDs may imply (to humans at least) items with similar characteristics, but don’t convey semantic information about the meaning of the items. It may be possible to use the structure to extract elements of content identified by common markup, but it requires human intervention to try to understand the patterns and relationships of interest.

Semantic meaning is conveyed through markup that identifies entities associated with the content. Entities have attributes, and semantic metadata indicates whether content describes the whole (the parent) or specific aspects (child elements). Historically, structured content was produced with the aim of creating a comprehensive and complete record in either a relational or XML datastore. More recently, content is being described in a more ad hoc manner, indicating where it fits in a bigger hierarchy without requiring all related content to be described at the same time. Content that is marked up may use a locally defined vocabulary, or adopt a vocabulary developed by others. Semantic markup is not necessarily validated. Microformats and microdata allow authors to add descriptions within the context of use. Such in-line markup within text is becoming more common.

A potentially important development in HTML5 metadata is something called Custom HTML Elements. These allow publishers to create their own elements (much like with XML) instead of having to rely on the limited range of predefined ones. There is no mention of linking Custom Elements to a schema in the current draft of these recommendations. Whether Custom Elements will result in another form of ad hoc markup, or become the basis for content to be described in greater semantic detail, remains to be seen.

The more complete the semantic description — the more attributes that are described, and the intricate the connections between parent and child elements — the more important that validation of these declarations becomes. Validation is the process of evaluating the declarations to confirm they comply with rules. For example, you may need to make sure that numbers are expressed as digits in a certain format, rather than allowing authors to enter numbers as words. A schema provides the rules for validating the content, including what elements are required. A publisher that creates a schema to validate content makes an additional investment of effort in return for having more reliable metadata that the publisher can reuse elsewhere.

The most explicit form of metadata is when a publisher decides to adhere to an open metadata standard and use that schema to validate its content. This allows other parties to locate and use the content. Other parties know how to ask for the content, and know it will be returned in an expected format. This degree of explicitness is central to the vision of linked data, where many parties share common data sets. The level of effort can be greater because publishers loose some flexibility in their markup (e.g., metadata description precision), and need to make sure their content works for all parties, not just their own needs (e.g., potential metadata syntax translation).

Roles and Applications of Structure

There are also different roles that metadata play, which influence what degree of structure may be needed.

Metadata supports the movement of content between the publishing platform and audiences, and plays a critical role in the dynamic delivery of content within the publishing platform. Metadata helps audiences discover content when using search engines. Publishers can use metadata in many ways, from aggregating similar items and rank ordering them, to tracking the use and performance of specific items of content.

Different kinds of metadata supports different aspects of structure and functionality.
Different kinds of metadata supports different aspects of structure and functionality.

The boundaries between descriptive, structural and administrative metadata are becoming more fluid as we move away from traditional ideas of fixed documents. On a conceptual level, the distinctions between metadata roles remains valuable to help understand what functions the metadata supports. Descriptive metadata aids the discovery of content, which is increasingly granular. People may only need to locate a fragment of content, not the larger whole. Descriptive metadata may be embedded within the body of the content, rather than only in the header. Structural metadata defines the structure of compound content objects. These objects increasingly change according to context: audience, location, device, and prior behavior. It is also becoming more common to retrieve specific, detailed content items without retrieving the structural container in which they are presented. Google is isolating facts that are embedded in documents, and presenting these outside of the document context. In HTML5, descriptive metadata in the body of content is called flow content (represented by inline elements), while structural metadata is referred to as section content.

Metadata in Conditional Content Output

Metadata values that support conditional content output may not be apparent to audiences, especially when business rules are involved. Many decisions about the content are processed by servers well upstream from the delivery of content to audiences, and are not discernible when viewing the markup of the source content. Administrative metadata describing content use and management plays a role determining what content elements are shown and to whom. New visitors may see different content than repeat visitors. Administrative metadata generally supports internal content decisions, and not external facilitation. Some content values are calculated, dependent on multiple criteria. The price of an item, for example, may vary according to the user’s past interaction with the site, the user’s geographic location, whether a cookie indicates the user has visited a competitor site, etc. Such dynamic content output means that there is not one fixed value associated with the metadata description, but it involves the calling of a function that may consider several items of data stored in various places.

Structured content is broader than structural metadata — the lego blocks of content we generally think about. Content structure is shaped by any metadata that impacts the audience experience. For example, we don’t know if the date published (administrative metadata) will necessarily be visible to the audience, but it can certainly impact other content, so it needs to be included a discussion of structure.

Expressed and Derived Metadata

Another dimension of metadata is whether it is human generated, or machine generated. Provided the values are validated, human entered metadata will often be more accurately classified, since humans can understand language nuances and author intentions. However, human entered metadata can be problematic when the format is not validated, or values are entered inconsistently. Inconsistent values may be ones that are incomplete, or when terminology is applied inconsistently: for example, when descriptors are too broad or narrow, or not defined in a manner that authors understand uniformly.

Machine generated metadata can describe events relating to the content (administrative metadata such as author name or time published), or can describe the characteristics of the content — certain descriptive metadata. Machine generated descriptive metadata is derived from features of the content. A simple, common example is when software extracts the important colors from a photograph. These colors then are classified by either a color swatch or name, and the descriptive metadata can be used to support search and filtering. A more involved kind of machine generated descriptive metadata is creating subject tags using named entity recognition. Such an approach is most suited to factually oriented content, and requires some supervision of the results. Machine generated metadata is generally uniform, but may not be entirely accurate semantically if there is scope for misinterpretation.

auto tag examples
Examples of color extraction metadata and Named Entity Recognition. Screenshots from Rijksmuseum and Europeana respectively.

 

How Audiences Differ from Brands and Machines

Whether expressed by humans, or generated by machine, metadata serves two functions: it provides information to help humans understand the content, and to help machines act on the content.

Humans care about metadata information because

  • the information itself is of interest (e.g., the color of a shirt)
  • the information is a tool to getting content of interest (e.g., show newest first).

Machines care about metadata because

  • they rely on it to determine the presentation of content to audiences
  • they need it to support specific content interactions with the user
  • they need it to support business rules that are important to the brand.

When comparing the respective needs of audiences and brands, it appears that brands have more need for explicit, validated metadata than do audiences. Many audience needs for interacting with content can be satisfied through the use of intelligent realtime search capabilities that work with more loosely defined content.

Audiences are largely indifferent to administrative metadata. They care about descriptive metadata to the extent they must rely on it to find what they seek amidst the volumes of content brands make available. If the exact content they want were to magically appear when they wanted it, descriptive metadata would be largely irrelevant. Audiences rely on structural metadata to navigate through detailed content, but are not generally interested in the compositional structure of content, except where it isn’t serving their needs. Content structure supports the audience’s content experience, but is mostly invisible to them.

Brands need metadata for many reasons. They need to ensure that the right items of content are reaching the right audiences. The more that audiences must work to locate the content they seek, they less likely they are to persevere to find it. Brands rely on metadata to ensure the accuracy of content, the effective use of content across lines of business, and critically, that content is presented in a way that maximizes potential business value for the brand. The more that the content delivered is based on business rules linked to product pricing strategies, CRM data on customers, and analytics performance optimization, the more important that underlying metadata is unambiguous and accurate.

Impressionistic and Precise Descriptions

In recent years, text analysis approaches have emerged as an important way to understand and manage unstructured and semi-structured content. Text analysis involves the indexing of words, and performing various natural language processing operations to discover patterns and meaning. For some tasks, these new approaches seem to obviate the requirement for highly structured and validated metadata. It is important to understand the relative strengths and weaknesses of text analysis compared to formal metadata.

Let’s consider two kinds of audience interactions with brand content. First, audiences need to discover the content, which often happens through a search engine such as Google. Brands are increasingly marking up their content using Schema.org metadata to help make their content more discoverable. But the key words audiences enter as search terms are not necessarily the exact words marked up in Schema. Behind the scenes, Google is applying its Knowledge Graph and linguistic technology to interpret the intent of the search, and determine how relevant the meaning of the brand’s content is to that intent. Interestingly, we don’t see stories of brands using the Schema.org markup to support internal content management decisions. The brands don’t use the structure they are adding to support their own needs. Their motivation appears entirely to support the needs of Google, which uses Schema to improve the effectiveness of its text mining and data analysis. Ironically, most articles these days about semantic content are written by SEO consultants who reveal little knowledge of how to structure content, or the different roles of metadata.

Second, audiences may want to submit comments on brand content. While brands may be able to leverage portable audience metadata associated Facebook account logons, the audience is not likely to contribute metadata as part of the their comments. Metadata that must be manually supplied by users is laborious, which is why the structure of user generated content is often limited to a simple rating of stars, or a thumbs up or down. The richest content, opinions expressed in comments, is unstructured. Managing such unstructured text requires text analysis to identify themes and patterns in the content.

Fuzzy Metadata via Schemaless Structure

Brands can benefit by using text analysis in lieu of highly structured metadata to support some audience-facing content management. Audiences will often have less precise needs when navigating content than brands have managing content. Audiences may have only a general sense of what they seek, and may not be comfortable specifying their needs with formal precision.

Fuzzy metadata exists when content records are selectively structured and have variable descriptions. Much of the metadata for content that is published is not tightly managed and has quality issues. Perhaps fields are sometimes filled in, but not always. The terms used in descriptions may vary. Even with these quality issue, the metadata is still valuable. Text analysis provides tools to help identify items of interest. The tools typically work with schemaless document databases (or sometimes called semi-structured data ) that embed the structure within the document. The appeal of the approach is that diverse content items can be searched together, and the structure of content does not need to be planned in advance, but can evolve. There are many limitations as well, but I’ll focus on benefits for now.

Perhaps the most interesting text analysis application is the open source text database elasticsearch. In contrast to traditional databases, elasticsearch is built around indexing concepts from information retrieval. It supports a variety of fuzzy queries, making it well suited for locating meaning in large volumes of text. It has out of the box features that:

  • Perform natural language indexing, such as stemming (word roots to account for word variants) and stop words (common words that create noise)
  • Consider word similarity matching
  • Consider synonyms
  • Analyze NGrams (commonly co-occurring words) and word order
  • Support numeric range queries
  • Calculate relevance based on how rare a term is in a corpus (a body of content), or how frequently it appears in a document
  • Provide “more like this” recommendations
  • Offer autocomplete suggestions for user queries.

Much of this semantic legwork is done in realtime, rather than in advance.

Brands such as SoundCloud use elasticsearch to help audiences find content of interest. Elasticsearch also includes features that allow the aggregation of data that may not be described in precisely the same way, and so it is good for internal data management tasks such as content analytics (used by the Guardian) and customer relationship analysis. Elasticsearch can evaluate content metadata by different criteria to score content. It can also evaluate customer data as part of scoring to determine how to prioritize the display of content: what content to promote in what situations. Livingsocial, a local deals site, uses elasticsearch to rank order which tiles to present on landing pages based on a combined scoring of content and customer metadata.

While capable, elasticsearch can’t do certain things well. According to developers who work with it, elasticsearch is not good for transactional data, or for canonical data. It doesn’t include a formal data model that validates the data, so business-critical data needs to be entered and stored elsewhere, especially when it is transactional. Relational databases provide more accurate reporting, while schemaless databases offer accessible pattern finding abilities. Metadata that has not been validated can result in duplications, or inconsistencies that even fuzzy searches cannot identify.

The Economics of Structure

Structure helps brands realize greater value from their content. But rigorous structure is not always appropriate for all content. Creating metadata can be expensive, as is the process of validating these records. Against this, poorly managed metadata can carry hidden costs. When organizations have content they decide they need to manage more precisely, they must perform an involved process of cleaning and reconciling the data.

Organizations need to decide how much risk they are prepared to accept regarding the quality of their metadata. The goals organizations have for their content vary widely, so it is difficult to generalize concerning the risks of poor quality metadata. But one lens to consider is the user’s journey, and their goals.

In general, less rigorous metadata can be acceptable for audience interactions in the early stages of a customer journey. Applications such as elasticsearch can provide good functionality to support audience browsing. When customers are narrowing their decisions, and are presented with an important call-to-action, it becomes critical to have more rigorous metadata associated with relevant key content. Brands need to be confident what they present to audiences who are ready to make a decision is accurate and reflects all the relevant concerns they may have. Approximation is not acceptable if it results in customers abandoning their journey, and it may be hard to diagnose specific problems when relying on general algorithmic approaches. Data quality can influence customer decisions (e.g., the revenue impact of typos or confusing wording), and so it is important to identify and trace any data anomalies. Valid and accurate data is also important for any conditional content that could effect a decision: for example, whether to present an up-sell message based on a specific content path. Finally, rigorous content structure is important for content that must be authoritative and not exist in multiple versions, such as the wording of terms and conditions, or high visibility content that impacts brand integrity and perception of brand value.

Approaches to metadata are becoming more diverse. New standards and software techniques provide growing a range of options. Content strategists will need to consider the short and long term consequences of decisions concerning metadata implementation.

—Michael Andrews

Categories
Intelligent Content

Content Structure and JavaScript

How audiences view content has radically changed since the introduction of HTML5 around five years ago.  JavaScript is playing a significant role in how content is accessed, and this has implications for content structure.  Content is shifting from being document centric to application centric.  Content strategy needs to reconsider content from an applications centric perspective.

The Standards Consensus: Separate Content Structure from Content Behavior

In the first decade of the new millennium, the web community formed a consensus around the importance of web standards.  Existing web standards were inadequate, so solid standards were needed.  And a widely accepted idea was that content structure, content behavior, and content presentation should all be separate from each other.  This idea was sometimes expressed as the “separation of concerns.”  As a practical matter, it meant making sure CSS and JavaScript doesn’t impact the integrity of your content.

“Just like the CSS gurus of old taught us there should be a separation of layout from markup, there should be a separation of behavior from markup. That’s HTML for the content and structure of the document, CSS for the layout and style, and Unobtrusive JavaScript for behavior and interactivity. Simple.”

— Treehouse blog January 2014

The advice to keep content structure separate from content behavior continues today.   The pillars of separating behavior from structure are unobtrusive JavaScript, and progressive enhancement.  A W3C tutorial advises: “Once you’ve made these scriptless pages you have created a basic layer that will more or less work in any browser on any device.”

Google says similar things: “If you’re starting from scratch, a good approach is to build your site’s structure and navigation using only HTML. Then, once you have the site’s pages, links, and content in place, you can spice up the appearance and interface with AJAX. Googlebot will be happy looking at the HTML, while users with modern browsers can enjoy your AJAX bonuses.”  Google’s advice here considers JavaScript as supporting presentation, rather than affecting content.

Microsoft writer argues: “The idea is to create a Web site where basic content is available to everyone while more advanced content and functionality are accessible to those with more capability, more bandwidth or more advanced tools.”   While it’s not clear what the distinction is between basic and advanced content, the core idea is similar: that important content shouldn’t be dependent on JavaScript behavior.

The web standards consensus was driven by an awareness that browsers varied, that JavaScript was sometimes unreliable, and that separation meant that persons using assistive technology were not disadvantaged.  That consensus is now eroding.  Some developers argue that it no longer matches the reality of current technical capabilities, and that the evolution of standards is solving prior issues that necessitated separation.  These developers are fusing content behavior and structure together.

The New Reality: JavaScript Driven Content

“The separation of structure, presentation and behavior is dead. It has been dead for a while. Still, this golden rule of web design sticks around. It lives on like Elvis and we need to address it.”

— Treehouse blog January 2012

Over the past five years, the big change in the web world has been the adoption of HTML5, with its heavy focus on applications, in contrast to the more document focused XHTML it replaced.  The emphasis among developers has been more about enhancing application behavior, and less about enhancing content structure.   HTML5 killed the unpopular proposed XHTML2 spec that emphasized greater structure in content, and developers have been seeking ways to remove XML-like markup where possible.

Silicon Veteran David Rosenthal, an Internet engineer at Stanford, describes the change this way: “The key impact of HTML5 is that, in effect, it changes the language of the Web from HTML to JavaScript, from a static document description language to a programming language.”  He notes: “The communication between the browser and the application’s back-end running in the server will be in some application-specific, probably proprietary, and possibly even encrypted format.”  And adds: “HTML5 allows content owners to implement a semi-effective form of DRM for the Web.”

The emphasis on applications behavior has resulted in new interaction capabilities and enhanced user experiences.  Rather than view a succession of webpages, users can interact with content continuously.  This has resulted in what’s called the Single Page Application, where “the web page is constructed by loading chunks of HTML fragments and JSON data.”

This shift has also been referred to as the “app-ification” of the web, where “a single page app typically feels much more responsive to user actions.”  “Single Page Applications work by loading a single HTML page to the user’s browser and subsequently never navigating away from this page. Instead, content, functional buttons, and actions are implemented as JavaScript actions.”

People are now thinking about content as apps.  An article entitled “The Death of the Web Page” declares: a “Single Page can produce much slicker, more customized and faster experiences for content consumption just as it can for web apps.”

JavaScript increasingly shapes the web’s building blocks.   Even semantic markup identifying the meaning of pieces of content, which has customarily been expressed in XML-flavored syntax (eg, RDF), is now being expressed through scripts.  JSON-LD, an implementation of the JavaScript Object Notation that is being used for some Schema descriptions of web content, relies on an embedded script, rather than markup that’s independent of the browser.

Risks Associated with Content On Demand

The rise of the Single Page Application is the most recent stage in the evolution of an approach I’ll call content on demand.

Content on demand means that content is hidden from view, and can only be discovered through intensive interrogation.  JavaScript libraries such as AngularJS determine the display of content in the client’s browser.  Server side content decisions are also being guided by browser interactions.  Even prior the rise of the current generation of Single Page Applications, the use of AJAX meant that users were specifying many parameters for content, especially on ecommerce sites.  “Entity-oriented deep-web sites are very common and represent a significant portion of the deep-web sites. Examples include, among other things, almost all online shopping sites (e.g., ebay.com, amazon.com, etc), where each entity is typically a product that is associated with rich information like item name, brand name, price, and so forth. Additional examples of entity-oriented deep-web sites include movie sites, job listings, etc” noted a team of Google researchers.  Such sites are hard for bots to crawl.

Google may not know what’s on your website if a database needs to return specific content.  If you have a complex system of separate product, customer and content databases feeding content to your visitors, it’s possible you are not entirely certain what content you have.  The Internet Archive’s Wayback Machine has trouble archiving the growing amount of content that is dependent on JavaScript.  There are now companies specializing in the crawling and scraping of “deep web” content to try to figure out what’s there.

Content on demand can sometimes be fragmented, and hard to manage.  Traditional server driven ecommerce sites manage their content using product information management databases, and can run reports on different content dimensions.  The same isn’t true of newer Single Page Applications, which may talk to content repositories that have little structure to them. JavaScript often manipulates content based on numeric IDs that may be arbitrary and do not represent semantic properties of content.  Content with idiosyncratic IDs obviously can’t be reused in other contexts easily.

Dynamic, constantly refreshing content can be relevant, and engaging for users.  But it doesn’t always meet their needs.  Especially when the implementing technology assumes audiences will want the existing paradigm exactly as it is.

JavaScript rendered content presumes the use of browsers for audience interaction.  That’s a good bet for many use cases, but it’s not a safe bet.   Audiences may choose to access their content through a simple API — perhaps a RSS feed or an email update sent to Evernote — that doesn’t allow them to interrogate the content.   In practice, the proportion of content being delivered through traditional browsers seems to be declining as new platforms and channels emerge.

Forcing users to interrogate content consistently could pose problems with the emerging category of multimodal devices.  To access content, audiences may depend on different input types such as gestures, speech recognition and voice search.    Content needs to be available in non-browser contexts on phones and handheld devices, home appliances, intelligent autos, and medical devices.  But input implementations are not uniform, and can be often proprietary.  Consider the hottest new form of interaction: speech input.  Chrome allows speech input, but other browsers can’t use Google’s proprietary technology, and x-webkit-speech only supports speech interaction for some form input types.

When viewable content is determined by a sequence of user interactions, it can become an exercise in “guess what’s here” because content is hidden behind buttons, menus and gestures.   Often, the presence of these controls only provides the illusion of choice.  In older page-based systems, users might choose many terms and be lead to pages with different URLs that had the same content.  Now, with “stateless” content, users might not even be sure of how they got to what they are seeing, and have no way to backtrace their journey through a history or bookmarks.

The risk of the content on demand approach is that content is may loose its portability when it is optimized for certain platforms.  We might want to believe that everyone is now following the same standards, but that wouldn’t be wise.  While tremendous progress has been made harmonizing standards for the web, the relentless innovations mean that different players such as Google, Apple, and Microsoft are being pulled in different directions.  Even Android devices, all nominally following the same approach, implement things differently, so that the browser on an Amazon Kindle will not display the same as a browser on a Samsung tablet.  The more JavaScript embedded in one’s content, the less easily it can be adapted to new platforms and services.

Some kinds of content hidden in the Deep Web (via Wikipedia)

  • Dynamic content: dynamic pages which are returned in response to a submitted query or accessed only through a form, especially if open-domain input elements (such as text fields) are used; such fields are hard to navigate without domain knowledge.

  • Contextual Web: pages with content varying for different access contexts (e.g., ranges of client IP addresses or previous navigation sequence).

  • Scripted content: pages that are only accessible through links produced by JavaScript

Know Your Risks, Prioritize What’s Important

My goal is not to criticize the app-ification of the web.  It has brought many benefits to audiences and to brands.  But it is important not to be intoxicated by these benefits to the point of underestimating associated costs.

Google, which has a big interest in the rise of JavaScript rendered content, recently noted:

“When pages that have valuable content rendered by JavaScript started showing up, we weren’t able to let searchers know about it, which is a sad outcome for both searchers and webmasters. In order to solve this problem, we decided to try to understand pages by executing JavaScript. It’s hard to do that at the scale of the current web, but we decided that it’s worth it. We have been gradually improving how we do this for some time.”

It’s fair to say JavaScript-rendered content is here to stay.  But if it’s hard for a bot to click on every JavaScript element to find hidden content, think about the effort it takes for an ordinary user.  Just because content is rendered quickly doesn’t mean the user doesn’t have to do a lot of work to swipe their way through it.  My advice: use JavaScript intelligently, and only when it really benefits the content.

Functionality should support choices significant to the user, and not mandate interactions.  There is an unfortunate tendency among some cosmetically focused front-end developers to provide gratuitous interactions because they seem cool.  Rather than be motivated by the goal of reducing friction, they present widgets for their spectacle effects rather than their necessity to support the user journey.  Is that slider really necessary, or was too much content presented to begin with which required the filtering?

We should consider limiting the number of parameters for dynamic content.  In the name of choice, or because we don’t know what audiences want, we sometimes provide them with countless parameters they can fiddle with.   Too many parameters can be overwhelming to users, and make content unnecessarily complex.  When Google studied ecommerce sites several years ago, they discovered that the numerous different results returned by searching product databases actually aligned to a limited number of product facets.  The combination of these facets represented “a more tractable way to retrieve a similar subset of entities than enumerating the space of all input value combinations in the search form.”    In other words, instead of considering content in terms of user selected contingencies, one can often discover that content has inherent structure that can be worked with.

A big consideration with content on demand is understanding what entities have an enduring presence.  As content moves toward being more adaptive and personalized, it is important to know and manage the core content.  There can be a danger when stringing together various and changing HTML fragments via continuous XMLHttpRequests on a single page that neither the audience nor the brand can be sure what was presented at a given point in time.  This is not just a concern for the legal compliance officer working at a bank: it’s important to all content owners in all organizations.  For audiences, it is hugely frustrating to be unable to retrieve content one has seen previously because you are unable to recreate the original sequence of steps that produced that view.

A core content entity should be a destination that is not dependent on a series of interactions to reach it.   Google has long advocated the use of canonical URLs instead of database generated ones.  But stateless app-like web pages lack any persistent identity.  Is that really necessary?  The BBC manages a vast database of changing content while providing persistent URLs.  Notably, they use specific URIs for their core content that allow content to be shared and re-used.  They do this without requiring the use of JavaScript.  To me, it seems impressive, and I encourage you to read about it.

What’s the Future of Structure?

An approach that decomposes content into unique URIs could provide the benefits of dynamic content with the benefits of persistence.  Each unique entity gets a unique URI, and entities are determined through the combination of relevant facets.  URIs are helpful for linking content to content hosted elsewhere.  One could layer personalization or modifications around the core content, and reference these through a sub path linked to the parent URI.  Such an approach requires more planning, but would enable content to be ready for any device or platform without scripting dependencies.  I can’t speak authoritatively concerning the effort required, any implementation limitations, or how readily such an approach could be used in different context.  This kind of approach isn’t being done much, but it leverages thinking from linked data about making content atoms that communicate with each other.  I would like to see developers review and explore the practicalities  of URI-defined content as content strategists think through the organizational and audience use cases.

Content strategists often advocate XML-like markup for structure, but I see few signs that is gaining widespread traction in the developer world, where XML is loathed.  XML markup seems to be in retreat in the web world, while JSON is king. How do we express structured content in the context of a programming language rather than a documentation language?  We need collectively to figure out how to make structure the friend of development, rather than a hinderance.

Content strategists can no longer presume content will be represented by static html pages that are unaffected by JavaScript behavior.  JavaScript rendered content is already a reality.   The full implications of these changes are still not clear, and neither are realistic best practices.  We need to discover how to balance the value of persistent content having a coherent identity, with the value of dynamic adaptive and personalized content that may never be the same twice.

— Michael Andrews