Search syntax

How to write the search term: which articles count as a match.

The search term goes in the searchterm field of the search request. Which data to return for the matching articles is set by params, and covered in the search guide.

If the search term has mistakes, the response says so; see errors in the search term.

The basics

Words and phrases

Based on the search string, each article is either considered a match or not. As a start, we can search for articles containing a single word. Adding another word, the default behaviour is to match articles having all words somewhere in the text, but not necessarily in consecutive order. However, multiple words can be treated as a single phrase by enclosing them in quotes.

• spotify

  Will match articles with spotify mentioned somewhere in the text.

• spotify music

  Will match articles containing spotify somewhere and music somewhere.

• 'Michael Jordan'

  Will match articles containing Michael followed directly by Jordan.

The API also supports more advanced string matching, like common wildcards and proximity search. For instance, to search for articles mentioning Trump and Biden close by, we can use the proximity search operator ~:

• 'Trump Biden'~5

  Will match articles mentioning Trump and Biden at most five words apart.

Searching in a section of the article

We can also be more specific about which part of the article we search in. Each article is divided into a header, a summary (the lede), and a body, along with image captions, if present. With a prefix, the phrase will only be matched against content in a particular section. Multiple sections can be searched by separating them with a comma.

• header:spotify

  Will match articles with spotify in the header.

• summary:spotify

  Will match articles with spotify in the lede.

• header:'Trump Biden'~5

  Will match articles mentioning Trump and Biden close by in the header.

Without any prefix, the search is performed against all the associated content, both the header, body, summary and any image captions. Phrases, wildcards and other search operators can be combined freely with the restrictions.

Note that a prefix applies to a single term or phrase. In header:(spotify OR football), the prefix is dropped without an error, and the whole article is searched; the response's debug field shows the search term as it was understood. Write (header:spotify OR header:football) instead.

Filters

Filters narrow the results based on e.g. language, publisher and popularity of the site. They are added to the search string by first mentioning the type, and then the value following a colon, just as when restricting the search to a section of the article.

• header:tax lang:en

  Will match articles mentioning tax in the header, and restrict the language to English.

• header:tax geo:1372

  Will match articles mentioning tax in the header, with content from the state of New York.

Most filters take ids rather than names, like the geo filter above, to make them language agnostic: France is called Frankrike in Norwegian, but it is still the same country. To find the id for a name, see finding ids.

Combining terms

The search engine supports boolean operators for more complex search queries. You can add AND, OR and ANDNOT operators between any of the terms. Instead of ANDNOT, a hyphen can be added as a prefix. Regular precedence rules apply.

• football AND (url:theguardian.com OR url:bbc.com)

  Will match articles mentioning football from The Guardian or the BBC.

• header:football -header:'premier league'

  Will match articles having football in the header, but not Premier league.

• a AND b OR c AND d

  Is equivalent to (a AND b) OR (c AND d).

Terms without an operator between them are joined with AND, because the API wraps the search string in ANDMODE[ ]. Filters of the same type are the exception: they are joined with OR, since an article can't match two of them at once. ORMODE[ ] joins terms with OR instead.

• ANDMODE[ a b OR c d ]

  Is equivalent to a AND b OR c AND d.

• ORMODE[ a b AND c d ]

  Is equivalent to a OR b AND c OR d.

• ANDMODE[ a lang:no lang:sv ]

  Is equivalent to a AND (lang:no OR lang:sv).

Full syntax

Every term below can also be negated, to exclude content, by prefixing it with ANDNOT.

Basic terms and phrases

• spotify

  Search for spotify somewhere in all the text, i.e. either in the header, the summary, the body, or in any of the captions.

• music spotify

  Search for articles containing spotify somewhere and music somewhere.

• Spotify

  Searching for a term with an upper case letter will match words with the same casing. Using only lower case letters matches any casing. While spotify will match both Spotify and spotify, Spotify only matches Spotify.

• ANDNOT spotify

  Search for articles not mentioning spotify.

• 'Michael Jordan'

  Search for Michael followed immediately by Jordan, i.e. the phrase 'Michael Jordan'.

• "Michael Jordan"

  You can also use double quotes to denote phrases.

• 'Jeanne d\'Arc'

  If the phrase contains a quote, the quote must be escaped, otherwise 'Jeanne d' would be interpreted as the phrase. Adding \ before the quote means the ' character will be interpreted as being part of the phrase.

• "Jeanne d'Arc"

  Alternatively, if there are only single quotes in the phrase, you can use double quotes around it. In this case escaping is not needed. Conversely if there are double quotes in the phrase, you can use single quotes around it.

Searching in specific sections

By default all content is matched against, but this can be changed by prefixing the term or phrase by the name of a section:

• header:spotify

  Search for spotify in the header.

• summary:spotify

  Search for spotify in the summary.

• body:spotify

  Search for spotify in the body text.

• caption:spotify

  Search for spotify in any of the captions.

• author:johnson

  Search for johnson in authors' names.

• domain:bbc

  Search for articles containing bbc in its domain name.

• url:bbc.co.uk/news

  Search for articles containing bbc.co.uk/news in its url.

Exact, prefix and wildcard matching

There is no support for prefix wildcards, and there must be at least three characters before any * or ?. Phrases and restrictions, however, as detailed above, can be used freely.

• domain:=bbc.co.uk

  Search for the exact phrase, in this case articles under exactly bbc.co.uk, not our-bbc.co.uk.

• spot*

  Search for words beginning with spot.

• harbo*r

  Search for words beginning with harbo and ending with r. There can be zero or more characters in between.

• visuali?ation

  Search for words with any character at the position of the question mark. This will match both visualization and visualisation.

• header:^London

  Search headers beginning with London.

Proximity

There are several ways to match based on terms occurring together in the text:

• 'Trump Biden'~5

  Search for Trump and Biden with at most 5 words between them.

• SPAN/5(Trump Pence, Biden Obama)

  Search for either Trump or Pence with either Biden or Obama at most 5 words apart. This is a generalized version of the proximity search, ~.

  The order of the words does not matter, Obama can come before Trump in the text, or it could be the other way around.

• OSPAN/5(Trump Pence, Biden Obama)

  An ordered version of SPAN, where Trump or Pence must occur before Biden or Obama in the text.

• SPAN/5(Trump Pence, Biden Obama, election)

  A three-argument span search. Search for either Trump or Pence with either Biden or Obama and the word election, all with at most 5 words between any of the terms.

Frequency

Matches can also be done based on how often a term occurs:

• London[3..]

  Match articles where the word London occurs at least 3 times.

• London[..7]

  Match articles where the word London occurs no more than 7 times.

• London[3..7]

  Match articles where the word London occurs at least 3 times, but no more than 7 times.

Ranks

Sites are given both a global and country based rank of their importance as a source of news, where the most important site is given a ranking of 1, the second most important a ranking of 2, and so on.

• globalrank<200

  Match only articles among sites ranked within the top 200 globally.

• globalrank>200

  Match only articles among sites ranked outside of the top 200 globally.

• countryrank<50

  Match only articles among sites ranked within the top 50 in their country.

• countryrank>50

  Match only articles among sites ranked outside of the top 50 in their country.

Language and subjects

It is possible to filter on other attributes of the article than their text content, such as the language it is written in, and metadata set on the site level, for instance the country of the publisher.

Except for the language codes, these filters apply to what has historically been named subjects. A subject is a piece of metadata: there is for instance a subject for the location Norway, and another for the media type print. The subjects are divided into categories, like geo for locations, containing for instance Norway and Sweden. Sites, and so their articles, are assigned subjects, and those are searchable.

For example, 1203 is the id of the subject Norway, so searching for geo:1203 only returns articles on sites where the publisher is from Norway, such as VG.

The ids are looked up with the suggestion server, which also gives the names of the subjects in the user's language, for display.

• lang:en

  Filter on a language code.

• geo:1372

  Filter on the geographical area represented by the id.

• site:318

  Filter on a site.

• media:575

  Filter on the media type represented by the id.

• content:501

  Filter on the content provider type represented by the id.

• rank:11

  Filter on the rank category represented by the id.

  A newer, more fine-grained alternative is to use the countryrank and globalrank operators. The rank subjects can be considered deprecated.

• subject:3333

  Filter on a generic subject id, irrespective of the category.

Topics and entities

• ent:1000291644

  Filter on entity represented by id.

• topic:111000000

  Filter on topic represented by id.

• crunchbase:123456

  Filter on entities using their Crunchbase ID.

• permid:123456

  Filter on entities using their PermID.

• lei:12345678901234567890

  Filter on legal entities using their LEI (Legal Entity Identifier).

Sites, sources and articles

Each article is published on a site, for instance in a local newspaper or on a website. Each site has an id. The article also has an id, but it is only unique within the site it was published on.

Specific articles must therefore be accessed using both a site id and an article id. This combination is unique.

• site:11

  Match on a site. Id can be fetched from the suggestion server.

• site:11,14,110

  Match on several sites. Ids can be fetched from the suggestion server.

• ANDNOT site:11

  Exclude a site. Id can be fetched from the suggestion server.

• article:11-123

  Match on a specific article under the site. The site id comes first, then the article id. In this case 11 is the site id, and 123 is the article id.

• article:123

  Match on an article id. Combine with site id to target unique articles.

• article:2,12,45,65

  Match on any of the article ids. Articles having an id in the list will be matched. Combine with site id to target many unique articles.

  If searching only based on ids, it is more efficient to use the articles param.

• id:11_123

  Match on a specific article under the site. The site id comes first, then the article id. In this case 11 is the site id, and 123 is the article id.

  If searching only based on ids, it is more efficient to use the articles param.

In addition to sites, there are sources. A source is a page where links to articles are harvested, and one site can have several: a news website can have a front page and a separate page with sports content. Source ids are globally unique.

• source:233

  Match a source.

• firstsource:8758

  Match based on the first source the article was observed on.

Profiles

Searches that are stored as profiles can be used as a building block in the search term.

• profile:123343

  Expands to search stored in profile with id 123343.

• profile:123343:1680159851

  Expands to search stored in a previous version of profile 123343, where the second number is the version timestamp, that can be found using the profile history endpoint in the API.

• profile:123343 ANDNOT profile:123343:1680159851

  Return articles matching the current version of the profile but not the older version of it.

• profile:123343:1680159851 ANDNOT profile:123343

  Conversely, return articles matching the old version of the profile but not the new version. This way, you can see hits that you have filtered away.

Misc

• charcount<2000

  Match only articles where the number of letters in the header, summary and body text combined amounts to less than 2000.

• charcount>500

  Match only articles where the number of letters in the header, summary and body text combined, exceeds 500.

Boolean expressions

Use boolean operators to build complex expressions.

• football AND england

  Both terms must match.

• football england

  By default, listing multiple terms is equivalent to an AND operator. All the terms must match.

• (url:telegraph.co.uk OR url:bbc.co.uk)

  Matches either term. It is usually a good idea to be explicit with parentheses and operator precedence.

• football ANDNOT tottenham

• football -tottenham

  A shorter form of the negation operator is available. Use a dash before the term.

• ANDNOT site:11

  The ANDNOT operator can also be used on its own.

Finding ids

The suggestion server is a search engine for filters, and translates names and urls to ids the API understands.

Whenever you want to only search for content in a particular language, on a given news site or maybe show news only from print media, you want to reference those entities by their names, like France and AP News. Since the API is language agnostic, and only accepts ids, the suggestion server is there to help you translate from something human-readable to a format the API understands, taking search terms in the language of your choice.

Ids of sites

To demonstrate, if we want to use the search API to get content only from AP News, we could do that by adding a site filter with an id of 215324:

• header:tax site:215324

  Articles from AP News with tax in the header.

However, we would need to get the id from somewhere. The suggestion server can perform this translation from names to ids. If we search for apnews among sites only, we get back a filter for the AP News website:

{
  "results": [
    {
      "id": "215324",
      "name": "AP News",
      "url": "https://www.apnews.com",
      "accessGroup": 1,
      "access": "all",
      "type": "site",
      "nl": false,
      "meta": "us",
      "mo": "2",
      "match": "2",
      "hlu": "12-17",
      "weight": 1540,
      "typeName": "Source"
    }
  ],
  "search": "apnews",
  "hits": 1
}

The type and id fields contain the values that can be sent to the search API.

Some news providers have several sites, each with its own id, for example separate web and print editions. The suggestion server then returns all of their ids, separated by commas: for Al Jazeera, 14681,328456. Pass them as they are, site:14681,328456, and articles from any of the sites will match.

Other filters

The suggestion server supports a selected number of filters, and the search terms will match the names and urls of whatever they refer to. Language filters match names of languages, geo filters match geographical regions. Without site:0 in the url, all filter types are searched: a search for France returns the geo filter for France, geo:1103, followed by sites like France 24:

Looking at what the various filters refer to, thus gives some idea of what to expect in the results:

• site

  Site filters match particular providers of content, like The Guardian.

  Since the providers often have a website, and a corresponding url, the url can also be matched, like theguardian.com.

• lang

  Language filters match languages, like English.

• geo

  Geographic filters match regions, like a continent, a country like England, and sometimes even at county level.

• rank

  Ranks are categories ordered 1 through 5. The rank is a function, among other things, of a sites popularity and how often they publish articles, 1 being the most reputable.

• media

  Media filters match media types, like print and web.

• content

  Content filters match the kind of content provider, like the government, an individual, editorial media or a local newspaper.

• timePeriod

  A time interval.

• ent

  An entity id.

• topic

  A topic id.

The request

The suggestion server is queried at the suggest endpoint:

https://suggest.api.opoint.com/suggest/<locale>/<mode>/<results_count>/<offsets>/0/<access_group>/<selected_filters>/<search_term>

For example, to get the top five suggestions for France:

curl "https://suggest.api.opoint.com/suggest/en_GB_1/single/5/0/0/2147483647/nometa/france"

It is the locale parameter that determines the language the search term is interpreted in, and the language of the names in the response. Changing the locale to no_NO_1 would not change the ids, but the name and the typeName would be different. It would also search with different names, for instance Frankrike instead of France.

In addition to a search_term, the suggest endpoint takes its own list of filters, selected_filters. Using the geo filter for New York, geo:1372, the suggested sites will be sites found in the state of New York. When a user is building a search, any of the filters already chosen can be passed to the suggestion server, in order to add filters with more precision. Here, without a search term, for the top sites in the state of New York:

curl "https://suggest.api.opoint.com/suggest/en_GB_1/multiple/5/0/0/2147483647/geo:1372/" | jq '.site'

For single mode the list of filters are added to the results field. For multiple mode the results are added to fields named after the filter types.

From ids to names

Typically, we want the id of a named entity. But sometimes we would like to go the other way, maybe obtain the name of a site we only know the id of. This can be done with the getids method:

https://suggest.api.opoint.com/suggest/<locale>/getids/type:id;type:id

The response is a list of objects of the same format as when searching.

Passing suggestions as filters

The suggestion server is designed to be used interactively, and it is particularly apt to use it in conjunction with a search field in a UI. The user can start to type the name of a site or a country, and a dropdown menu can show the sites and countries best matching the search term. The response from the server contains useful information such as names in a given language for display, and highlighting information about any of the matches.

The filters are then added to the rest of the search string, and used for making a search request to the API. Since the suggestion server returns JSON objects with all the data about the filters, the search API accepts a slightly more verbose, but in this case more convenient, format:

{
    "expressions": [
        {
            "linemode": "R",
            "searchline": {
                "searchterm": "header:tax",
                "filters": [
                    {
                        "type": "site",
                        "id": "215324"
                    }
                ]
            }
        }
    ],
    "params": {
        "requestedarticles": 10
    }
}

The searchterm field is changed to a list of expression objects. The linemode determines how multiple expression objects are interpreted, but can be set to R for now. The searchline field contains a searchterm, as before, but also a list of filters that will be prepended to the searchterm by the API.

This might seem like more work, but the response from the suggestion server already contains objects with type and id fields, and can therefore be relayed to the filters field directly, without having to parse it into a type:id form.

Errors in the search term

Minor mistakes, like a missing closing parenthesis, are corrected by guessing what was meant. The search then runs as normal, and the corrections are reported in a debug field in searchresult.

If the search term can't be understood, the response has an errors field instead of articles:

{
  "searchresult": {
    "host": "euler3.opoint.com",
    "compiledate": "Aug 14 2020 14:02:47",
    "errors": "Solr could not handle the query"
  }
}

If there is neither an errors nor a debug field, the search term was understood as written.