Search response

Responses from the different search APIs share a common base structure.

The data in the response is found in the searchresult field. It contains data about the search as a whole, like pagination parameters, in addition to adocument array, containing data about each individual article, like it's header, url, various metadata, and so on.

Although the basic structure is always the same, the exact format of the response depends on multiple factors:

  • The parameters supplied in the search – some data you have to explicitly ask for.
  • The search service – the StoredSearch API will for instance add an id for you to locate the search.
  • Your licenses and what are you allowed to view – readership data is for instance only accessible to user with the correct permissions.

Examples of concrete search responses in JSON and XML formats can be found on the examples page.

Syntax errors

The server will do its best to parse the search term if there are some minor mistakes in the syntax, such as a missing closing parenthesis, by guessing the users intention. In this case, the corrections are reported in the debug field in the response.

In cases where the syntax errors are too grave, the response will contain an errors field, indicating that something went wrong: 

{
  "searchresult": {
    "host": "euler3.opoint.com",
    "compiledate": "Aug 14 2020 14:02:47",
    "errors": "Solr could not handle the query"
  }
}

If neither an errors field, nor a debug field is present, the search was well formed.

Search result

The search result object contains data about the search as a whole, while data for the individual documents are found in the document field, and is described in a separate section below.

  • link documents int

    Number of documents in the result, that is the length of the document list.

  • link document list of objects

    Articles matching search.

  • link debug object or missing

    Information about search term corrections. Whenever there are minor mistakes in the seach term, e.g. a missing closing quote, the parser attempts to correct it.

    • link debug.count int or missing

      Number of lines.

    • link debug.lines list of objects or missing
      • link lines[].state int or missing

        1 if the line had an error. 0 otherwise.

      • link lines[].query string or missing

        What the query was interpreted as, if the line had any errors corrected.

  • link first_timestamp int

    Unix timestamp of the first document matching the search also in the response. Matches are found going backwards in time, so it will also be the document published the latest.

  • link last_timestamp int

    Unix timestamp of the last document matching the search also in the response. Matches are found going backwards in time, so it will also be the document published the earliest.

  • link context string

    String value used for pagination, representing where the current search ended. For use by the context parameter in the request.

    If the value is empty there are no more articles to fetch.

    Params:
    context
  • link search_start int or missing

    Unix timestamp a new search can start from were it to get newer content. Thus, it can be passed for oldest. Will never be larger than current time.

  • link range_count int or missing

    Estimate on the total number of documents matching the search in the range given by range_start and range_end.

  • link range_start int or missing

    Unix timestamp of the start of the range used for computing range_count.

  • link range_end int or missing

    Unix timestamp of the end of the range used for computing range_count. A range_end of 0 means that the end of the range is now.

  • link wikinames object or missing

    A dictionary mapping entity ids to entity names.

    Params:
    textrazor
  • link wikidescriptions object or missing

    A dictionary mapping entity ids to entity descriptions.

    Params:
    textrazor
  • link search_matches list of objects or missing
    Params:
    main.matches
    • link search_matches[].type string or missing
    • link search_matches[].color int or missing
    • link search_matches[].text string or missing
  • link matches_table object or missing
    Params:
    main.matches
    • link matches_table.rows int or missing
    • link matches_table.cols int or missing
    • link matches_table.row object or missing
      • link row.match_item object or missing
        • link match_item.rowspan int or missing
        • link match_item.colspan int or missing
        • link match_item.type string or missing
        • link match_item.color int or missing
        • link match_item.active boolean or missing
        • link match_item.length int or missing
        • link match_item.removed int or missing
        • link match_item.text string or missing
  • link watch_id int or missing

    Id of watch process.

    Params:
    watch_id
  • link count int
  • link cacheage int
  • link cputime int
  • link host string

    Server having processed the request.

  • link compiledate string
  • link branch string

    Name of the current release of the search system. Not in use at the moment. Returns NOT ACTIVE for all requests.

  • link notimeout boolean
  • link sort_groups object or missing
    Params:
    categories
    • link sort_groups.group int or missing

      Id the group was filtered on. 0 if it's the group containing the articles not matching any group.

    • link sort_groups.weight int or missing
    • link sort_groups.count int or missing

      Number of articles in group.

    • link sort_groups.name string or missing

      Name of the group.

Document

Each document object contains data about individual articles. It is found in the document array in the searchresult field in the response.

  • link header object or missing

    The articles header. Does not contain HTML.

    Params:
    main.header compare_header
    • link header.text string or missing
    • link header.matches boolean or missing

      Whether matches should be marked in the header, i.e. if main.header is set to 2.

    • link header.compare int or missing

      Edit distance between the compare_header parameter and the header text.

  • link summary object or missing

    The articles lede. Does not contain HTML.

    Params:
    main.summary max_article_length compare_text
    • link summary.text string or missing
    • link summary.matches boolean or missing

      Whether matches should be marked in the summary, i.e. if main.summary is set to 2.

    • link summary.compare int or missing

      Edit distance between the compare_text parameter and the combined summary and body text.

    • link summary.truncated boolean or missing

      Whether the field was truncated or not.

  • link body object or missing

    The articles body text. Typically contains HTML.

    Params:
    main.text max_article_length
    • link body.text string or missing
    • link body.readmore boolean or missing
    • link body.matches boolean or missing

      Whether matches should be marked in the body text, i.e. if main.text is set to 2.

    • link body.truncated boolean or missing

      Whether the field was truncated or not.

  • link author string or missing

    The articles authors.

  • link word_count int
  • link short_header object or missing
    Params:
    main.shortheaderlength
    • link short_header.text string or missing
    • link short_header.matches boolean or missing
    • link short_header.length int or missing
    • link short_header.removed int or missing
  • link short_summary object or missing
    Params:
    main.shortsummarylength
    • link short_summary.text string or missing
    • link short_summary.matches boolean or missing
    • link short_summary.length int or missing
    • link short_summary.removed int or missing
  • link short_body object or missing
    Params:
    main.shortbodylength
    • link short_body.text string or missing
    • link short_body.matches boolean or missing
    • link short_body.length int or missing
    • link short_body.removed int or missing
  • link similarweb object or missing
    error_outline Requires license

    Traffic data for the domain and readership estimates.

    Note: Will only be present if there is traffic data to return for the article.

    Params:
    readership
    • link similarweb.readership object or missing

      Site level readership estimates.

      • link readership.P number or missing
      • link readership.value int or missing
    • link similarweb.article_readership object or missing

      Article level readership estimates.

      • link article_readership.value int or missing

        Estimated number of readers of the article so far.

      • link article_readership.DeltaPValue int or missing

        Predicted number of additional readers in the next hour, assuming its visibility does not change.

      • link article_readership.ExpectedPValue int or missing

        Predicted number of people that will eventually read the article.

      • link article_readership.StdExpectedPValue int or missing

        Standard deviation of ExpectedPValue. Higher values indicate lower confidence in the final value.

        For recently published articles, this value can be high, since little is know about how long it will stay visible etc.

      • link article_readership.P number or missing

        Estimated fraction of the sites readers having read the article so far.

      • link article_readership.DeltaP number or missing

        Predicted fraction of the sites readers that will read the article in the next hour.

      • link article_readership.ExpectedP number or missing

        Predicted fraction of the sites readers that will eventually read the article.

      • link article_readership.StdExpectedP number or missing

        Standard deviation of ExpectedP.

  • link site_rank object or missing
    • link site_rank.rank_global int or missing
    • link site_rank.rank_country int or missing
  • link topics list of objects or missing

    Information about the articles subjects, such as language, country and the medium it was published in. Articles are assigned subjects based on the site and source it was found.

    • link topics[].id int

      Subject id.

    • link topics[].text string

      Subject name.

    • link topics[].linktype_max int or missing
    • link topics[].linktype_now int or missing
    • link topics[].metacat int or missing

      Meta category, an integer representing the type of the subject.

    • link topics[].metacat_level int or missing

      Meta category abstraction level, ordered from 0 and up.

  • link topics_and_entities object or missing
    error_outline Requires license

    Topics and entities found in the article.

    Note: Will only be present if there are topics and entities data to return for the article.

    Params:
    textrazor
    • link topics_and_entities.topics object or missing

      IPTC Media Topics found in the article.

      • link topics.label string or missing

        IPTC Media Topic name. Topic names are hierarchically structured, each level separated by a >, and with the levels ordered from the most general to the most specific.

      • link topics.mediatopic_id string or missing

        IPTC Media Code.

      • link topics.score number or missing

        Confidence in the article revolving around the topic.

    • link topics_and_entities.entities object or missing

      Entities found in the article. There are three kinds of entities registered: locations, organizations and persons.

      • link entities.location object or missing
        • link location.entity string or missing

          Name of the location.

        • link location.wikidata_id string or missing

          Id in the Wikidata knowledge graph.

        • link location.confidence_score number or missing

          Confidence score for the entity recognition.

        • link location.relevance_score number or missing

          Relevance score indicating how relevant the entity is to the article.

        • link location.crunchbase string or missing

          Name in the crunchbase platform.

        • link location.permid string or missing

          links to Thomson Reuters Open PermID. PermID provides comprehensive identification across a wide variety of entity types including organizations, instruments, funds, issuers and people.

        • link location.lei string or missing

          Legal Entity Identifier (or LEI) is a international identifier made up of a 20-character identifier that identifies distinct legal entities that engage in financial transactions.

        • link location.figi string or missing

          The Financial Instrument Global Identifier (FIGI) (formerly Bloomberg Global Identifier (BBGID)) is an open standard, unique identifier of financial instruments.

      • link entities.organization object or missing
        • link organization.entity string or missing

          Name of the organization.

        • link organization.wikidata_id string or missing

          Id in the Wikidata knowledge graph.

        • link organization.confidence_score number or missing

          Confidence score for the entity recognition.

        • link organization.relevance_score number or missing

          Relevance score indicating how relevant the entity is to the article.

        • link organization.crunchbase string or missing

          Name in the crunchbase platform.

        • link organization.permid string or missing

          links to Thomson Reuters Open PermID. PermID provides comprehensive identification across a wide variety of entity types including organizations, instruments, funds, issuers and people.

        • link organization.lei string or missing

          Legal Entity Identifier (or LEI) is a international identifier made up of a 20-character identifier that identifies distinct legal entities that engage in financial transactions.

        • link organization.figi string or missing

          The Financial Instrument Global Identifier (FIGI) (formerly Bloomberg Global Identifier (BBGID)) is an open standard, unique identifier of financial instruments.

      • link entities.person object or missing
        • link person.entity string or missing

          Name of the person.

        • link person.wikidata_id string or missing

          Id in the Wikidata knowledge graph.

        • link person.confidence_score number or missing

          Confidence score for the entity recognition.

        • link person.relevance_score number or missing

          Relevance score indicating how relevant the entity is to the article.

        • link person.crunchbase string or missing

          Name in the crunchbase platform.

        • link person.permid string or missing

          links to Thomson Reuters Open PermID. PermID provides comprehensive identification across a wide variety of entity types including organizations, instruments, funds, issuers and people.

        • link person.lei string or missing

          Legal Entity Identifier (or LEI) is a international identifier made up of a 20-character identifier that identifies distinct legal entities that engage in financial transactions.

        • link person.figi string or missing

          The Financial Instrument Global Identifier (FIGI) (formerly Bloomberg Global Identifier (BBGID)) is an open standard, unique identifier of financial instruments.

  • link unix_timestamp int

    Unix timestamp of when the article was published.

  • link local_time object

    Local time the article was published.

    • link local_time.text string

      ISO 8601 formatted time.

    • link local_time.GMT int
  • link local_rcf822_time object

    Local time the article was published.

    • link local_rcf822_time.text string

      RFC 822 formatted time

  • link countryname string or missing
  • link countrycode string or missing
    Params:
    allsubject
  • link language object
  • link orig_url string or missing

    Url where the article was found.

  • link url_common string or missing

    Domain name of the site where the article was found.

  • link url string or missing

    Link to tool displaying the crawled article, with options for highlighting matches etc.

    Params:
    use_short_links
  • link compare int or missing
    Params:
    compare_url
  • link articleimages object or missing

    The articles images.

    • link articleimages.count int

      Number of images, that is the length of articleimage.

    • link articleimages.articleimage list of objects

      Data about each image. Note that it could be that there is a caption, but for some reason there were no url.

      • link articleimage[].url string or missing

        Url of the image.

      • link articleimage[].caption object or missing
        • link caption.matches boolean or missing
        • link caption.text string or missing
  • link caption object or missing

    All the images' captions concatenated.

    • link caption.text string or missing
  • link screenshots list of objects or missing
    • link screenshots[].id_type int
    • link screenshots[].width int
    • link screenshots[].height int
    • link screenshots[].text string

      Url

Internal id system

Each article is published on a site, for instance in a local newspaper or on a website. Each of these sites have an associated id. The article also has an id, but this id is only unique for the site it was published on.

The pair id_site and id_article is globally unique.

In addition to sites, there are also sources. A source is a page where links to articles are harvested. One site can have multiple sources. A news website can have a front page and a separate page with sports content. Source ids are globally unique.

  • link id_site int

    Id of the site the article was published.

  • link id_article int

    Site specific article id. Unique under each site.

  • link first_source object

    Information about the source the article was first observed on.

    • link first_source.id int

      Id of the source.

    • link first_source.name string

      Name of the source.

    • link first_source.sitename string

      Name of the site.

    • link first_source.url string

      Url of the source.

    • link first_source.siteurl string

      Url of the site.

  • link sources object or missing

    Information about sources the article was observed on.

    Params:
    main.othersources
    • link sources.source object or missing
      • link source.id int

        Id of the source.

      • link source.linktype_max int or missing
      • link source.linktype_now int or missing
      • link source.name string

        Name of the source.

      • link source.sitename string

        Name of the site.

      • link source.url string

        Url of the source.

      • link source.siteurl string

        Url of the site.

  • link sort_group_ref int or missing

    Reference to categorization group.

  • link mediatype object or missing

    Description of the origin of the article.

    • link mediatype.timemap boolean

      Whether timemap is present.

    • link mediatype.duration int or missing

      Length of timemap, if present, in seconds.

    • link mediatype.clip boolean
    • link mediatype.hastext boolean
    • link mediatype.paywall boolean

      Whether article was fetched from behind a paywall.

    • link mediatype.fulltext boolean
    • link mediatype.text string
  • link distribute_conditions string or missing
  • link circulation int or missing
  • link adprice int or missing
  • link reach int or missing
  • link content_protected int or missing
  • link position int

    Index of the document in the list. One-based.

  • link hidden boolean or missing
  • link sort_group_ref int or missing

    Reference to categorization group.

  • link shareable boolean or missing
  • link stored_search_id int or missing

    Id of the stored search the article matched, if requested through the StoredSearch feed.

  • link profile_id int or missing
  • link profile_name string or missing
  • link tracking object or missing

    If the document is a tracking match, details about the upload it matched and additional data regarding the relationship between the upload and the document.

    • link tracking.partner_article_id string

      Partner specific id of the upload the document matched.

    • link tracking.score int

      Tracking score, a measure of how likely it is that the document is linked to the uploaded article.

    • link tracking.percent_used int, between 0 and 100

      A percentage estimate of how much of the uploaded article that were referenced in the matching article.

    • link tracking.percent_of_match int, between 0 and 100

      A percentage estimate of how much of the matching article that stems from the uploaded article.