Response format

The search API, StoredSearch and SafeFeed return articles in the same format. This page describes it once, field by field.

Which fields a response has depends on:

• The parameters of the request – some data you have to ask for, like the article text or topics and entities. A field that a parameter turns on lists it under Params.
• Your license – readership data, for instance, is only returned if your license includes it. These fields are marked Requires license.
• The API – a few fields are only returned by one of them, and are marked with its name; see differences between the APIs.

Fields with no data for an article may be left out, so code reading the response should not expect every field to be there.

Structure

The response has a single searchresult, with data about the result as a whole, like where to continue from to get more articles. Its document array holds the articles, one document each, with the article's text, url, source, time and other metadata.

JSON

A search result with one document, shortened:

{
  "searchresult": {
    "documents": 2,
    "count": 3,
    "context": "534348478:1:0;TOTDOC=2",
    "document": [
      {
        "id_site": 131,
        "id_article": 2122020,
        "unix_timestamp": 1613135182,
        "url": "https://www.bbc.com/news/uk-wales-56041927",
        "language": {
          "encoding": "iso-639",
          "text": "en"
        },
        "countrycode": "GB",
        "header": {
          "text": "Stonehenge: Did animals pull stones from Wales to England?"
        },
        "summary": {
          "text": "One of Britain's biggest and oldest stone circles has been found in Wales - and could be the original building blocks of Stonehenge."
        },
        "body": {
          "text": "<p> Archaeologists uncovered the remains of the Waun Mawn site in Pembrokeshire's Preseli Hills.</p> <p> They believe the stones could have been dismantled and …"
        },
        "similarweb": {
          "domain": "bbc.com",
          "readership": {
            "P": 0.012741,
            "value": 2162922
          }
        }
      }
    ]
  }
}

XML

All three APIs can also return XML. The fields have the same names as in JSON, but some are attributes:

• Simple fields of the search result, like documents and context, are attributes of <searchresult>.
• id_site and id_article are attributes of <document>, and fields like language and header have their extra values (encoding, matches) as attributes.
• A field whose JSON value is an object with a text member has the text in a <text> element, and the body is split into <p> paragraphs.
• Arrays are repeated elements: the documents are <document> elements directly under <searchresult>.

The same article in XML:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<searchresult documents="2" count="3" context="534348478:1:0">
    <document id_site="131" id_article="2122020">
        <unix_timestamp>1613135182</unix_timestamp>
        <url>https://www.bbc.com/news/uk-wales-56041927</url>
        <language encoding="iso-639">en</language>
        <countrycode>GB</countrycode>
        <header matches="false">
            <text>Stonehenge: Did animals pull stones from Wales to England?</text>
        </header>
        <summary matches="false">
            <text>One of Britain's biggest and oldest stone circles has been found in Wales - and could be the original building blocks of Stonehenge.</text>
        </summary>
        <body matches="false">
            <p>
                <text>Archaeologists uncovered the remains of the Waun Mawn site in Pembrokeshire's Preseli Hills.</text>
            </p>
            <p>
                <text>They believe the stones could have been dismantled and …</text>
            </p>
        </body>
        <similarweb>
            <domain>bbc.com</domain>
            <readership P="0.012741">2162922</readership>
        </similarweb>
    </document>
</searchresult>

How to ask for XML differs between the APIs: the search API uses the Accept: application/xml header, StoredSearch the format=xml parameter and SafeFeed doc_format.

Errors and corrections

If the search term has minor mistakes, such as a missing closing parenthesis, the server guesses what was meant and runs the search. The corrections are reported in the debug field of the search result.

If the search can't be run, the search result has an errors field instead of documents:

{
  "searchresult": {
    "host": "euler3.opoint.com",
    "compiledate": "Aug 14 2020 14:02:47",
    "errors": "Solr could not handle the query"
  }
}

If there is neither an errors nor a debug field, the search was well formed. How to fix search terms is described under errors in the search term.

Differences between the APIs

The documents are the same from all three APIs. The search result differs in how you continue to the next batch of articles, and each API adds a few fields of its own.

Search API

The search API returns the search result fields below. Pass context back to get the next batch, or search_start as oldest to get only newer articles later.

StoredSearch

The StoredSearch feed adds these:

• searchresult.next_from: a Unix timestamp to pass as from in the next request. It is there even when there are no new articles, but only when the request used interval.
• searchresult.covered: the number of seconds from from to next_from.
• stored_search_id on each document: the id of the StoredSearch the article matched.

SafeFeed

SafeFeed continues from an id instead of a time:

• In XML, the root element is <OpointResult> instead of <searchresult>, and the id to continue from is in an <opoint_search_start> element.
• In JSON, the same id is in searchresult.search_start. Unlike in the search API, it is not a timestamp.

Pass the id as lastid in the next request. The SafeFeed page has full examples in both formats.

Search result

The search result has data about the result as a whole. The articles are in its document array, described in the next section.

Document

Each document is one article, in the document array of the search result.