As websites publish more metadata, publishers need ways to audit what they’ve published. This post will look at a tool called jq that can be used to audit metadata.
Metadata code is invisible to audiences. It operates behind the scenes. To find out what metadata exists entails looking the source code, squinting at a jumble of div tags, css, javascript and other stuff. Glancing at the source code is not a very efficient way to see what metadata is included with the content. Publishers need easy ways for their web teams to find out what metadata they’ve published.
This discussion will focus on metadata that’s serialized in the JSON-LD format. One nice thing about JSON-LD is that it separates the metadata from other code, making it easier to locate. For those not familiar with JSON-LD, a brief introduction. JSON-LD is the latest format for encoding web metadata, especially the widely-used schema.org vocabulary. JSON-LD is still less pervasive than microdata and RDFa, which are described within HTML elements. But JSON-LD has quickly emerged as preferred the syntax for many websites. It is more developer-friendly than HTML syntaxes, and shares a common heritage with the widely-used JSON data format.
According to statistics, around 225,000 websites are using JSON-LD. That’s about 21% of all websites globally, and is nearly 30% of English language websites. Some major sites using JSON-LD for metadata include Apple, Booking.com, Ebay, LinkedIn, and Yelp.
Why Audit Metadata?
I’ve previously touched on the value of auditing metadata in my book, Metadata Basics for Web Content. For this discussion, I want to highlight a few specific benefits.
For those who work with SEO, the value of knowing what metadata exists is obvious: it influences discovery through search. But content creators will also want to know the metadata profile of their content. It can yield important insights useful for editorial planning.
Metadata provides a useful summary of the key information within published content. Reviewing metadata can provide a quick synopsis of what the content is about. At the same time, if metadata is missing, that means that machines can’t find the key information that audiences will want to know when viewing the content.
Auditing can reveal:
- what key information is included in the content
- if any important properties are missing that should be included
Online publishers should routinely audit their own metadata. And they may decide they’d benefit by auditing their competitor’s metadata as well. Generally, the more detailed and complete the metadata is, the more likely a publisher will be successful with their content. So seeing how well one’s own metadata compares with one’s competitors can reveal important insights into how readily audiences can access information.
How to Audit JSON-LD metadata
Metadata is code, written for machines. So how can members of web teams, whether writers or SEO specialists, get a quick sense of what metadata they have currently? Since I have mission to evangelize the benefits of metadata to all content stakeholders, including less technical ones, I’ve been looking for light-weight ways to help all kinds of people discover what metadata they have.
For metadata encoded in HTML tags, the simplest way to explore it is using XPath, a simple filter query that searches down the DOM tree to find the relevant part containing the metadata. XPath is not too hard to learn (at least for basic needs), and is available within common tools such as Google Sheets.
Unfortunately, XPath can’t be used for metadata in JSON-LD. But happily, there is an equivalent to XPath that can be used to query JSON-based metadata. It is called jq.
The first step to doing an audit is to extract the JSON-LD from the website you want to audit. It lives within the element <script type= application/ld+json></script>. Even if you need to manually extract the JSON-LD, it is easy to find in the source code (use CTR-F and search for ld+json). Be aware that there may be more than one JSON-LD metadata statement available. For example, when looking at the source code of a webpage on Apple’s website, I notice three JSON-LD script elements representing three different statements: one covering product information (Offer), one covering the company (Organization), and another covering the website structure (BreadcrumbList). Some automated tools have been known to stop harvesting JSON-LD statements after finding the first one, so make sure you get them all, especially the ones with information unique to the webpage.
Once you have collected the JSON-LD statements, you can begin to audit them to see what information they contain. Much like a content audit, you can set up a spreadsheet to track metadata for specific URLs.
Exploring JSON-LD with jq
jq is a “command line” application, which can present a hurdle for non-developers. But an online version of it exists called jq Play that is easy to use.
Although jq was designed for filtering ordinary plain JSON, it can also be used for JSON-LD. Just paste your JSON-LD statement in jq Play, and add a filter.
Let’s look at some simple filters that can identify important information in JSON-LD statements.
The first filter can tell us what properties are mentioned in the metadata. We can find that out using the “keys” filter. Type keys and you will get a list of properties at the highest level of the tree. Some of these have an @ symbol, indicating the are structural properties (for example "@context", "@id", "@type"). Don’t worry about those for now. Others will resemble words and be more understandable, for example, “contactPoint”, “logo”, “name”, “sameAs”, and “url”. These keys, from Apple’s Organization statement, tells us the kinds of information Apple includes about itself on its website.
Let’s suppose we have JSON-LD for an event. An event has many different kinds of entities associated with it, such as a location, the event’s name, and the performer. It would be nice to know what entities are mentioned in the metadata. All kinds of entities use a common property: name. Filtering on the name property can let us know what entities are mentioned in the metadata.
Using jq, we find out entities by using the filter ..|.name? which provides a list of names. When applied to a JSON-LD code sample from the schema.org website, we get the names associated with the Event: the name of the orchestra, the auditorium, the conductor, and the two symphonic works.
The filter was constructed using a pattern ..|.foo? (foo is a jibberish name to indicate any property you want to filter on.) JSON-LD stores information in a tree that may be deeply nested: entities can refer to other entities. The pattern lets the filtering move through the tree and keep looking for potential matches.
Finally, let’s make use of the structural information encoded with the @ symbol. Because lots of different entities have names, we also want to know the type of entity something is. Is the “Chicago Symphony” the name of a symphonic work, or the name of an orchestra? In JSON-LD, the type of entity is indicated with the @type property. We can use jq to find what types of entities are include in the metadata. To do this, the filter would be ..|."@type"? . It follows the same ..|.foo? pattern, except that structural properties that have a @ prefix need to be within quotes, because ordinary JSON doesn’t use the @ prefix and so jq doesn’t recognize it unless it’s in quotes.
When we use this filter for an Event, we learn that the statement covers the following types of entities:
- “MusicEvent”
- “MusicVenue”
- “Offer”
- “MusicGroup”
- “Person”
- “CreativeWork”
That one simple query reveals a lot about what is included. We can confirm that the star of the show (type Person) is included in the metadata. If not, we know to add the name of the conductor.
Explore Further
I’m unable here to go into the details of how JSON-LD and schema.org metadata statements are constructed — though I do cover these basics in my book. To use jq in an audit, you will need some basic knowledge of important schema.org entities and properties, and know how JSON-LD creates objects (the curly braces) and lists (the brackets). If you don’t know these things yet, they can be learned easily.
The patterns in jq can be sophisticated, but at times, they can be fussy to wrangle. JSON-LD statements are frequently richer and more complex than simple statements in plain JSON. If you want to extract some specific information within JSON-LD, don’t hesitate to ask a friendly developer to help you set up a filter. Once you have the pattern, you can reuse it to retrieve similar information.
JSON-LD is still fairly new. Hopefully, purpose-built tools will emerge to help with auditing JSON-LD metadata. Until then, jq provides a light weight option for exploring JSON-LD statements.
— Michael Andrews