Examples
Search responses
Note: The responses depends on parameters used for the search and the licenses you have.
JSON response
{
"searchresult": {
"documents": 2,
"first_timestamp": 1613135182,
"last_timestamp": 1613135169,
"search_start": 2147483647,
"context": "534348478:1:0;TOTDOC=2",
"count": 3,
"range_count": 8,
"range_start": 1612393200,
"range_end": 0,
"cacheage": -1,
"cputime": 30,
"host": "euler3.opoint.com",
"compiledate": "Feb 2 2021 16:40:04",
"branch": "NOT ACTIVE",
"notimeout": true,
"netsprintindextimer": 21,
"search_matches": {
"search_match": {
"type": "word",
"color": 4,
"text": "Stonehenge"
}
},
"matches_table": {
"rows": 1,
"columns": 1,
"row": {
"match_item": {
"rowspan": 1,
"colspan": 1,
"type": "word",
"color": 4,
"active": true,
"length": 59,
"removed": 0,
"text": "Stonehenge"
}
}
},
"document": [
{
"id_site": 131,
"id_article": 2122020,
"hidden": false,
"position": 1,
"countrycode": "GB",
"countryname": "United Kingdom",
"similarweb": {
"domain": "bbc.com",
"readership": {
"P": 0.012741,
"value": 2162922
},
"article_readership": {
"P": 0.000101,
"DeltaP": 0.000204,
"DeltaPValue": 34631,
"ExpectedP": 0.021335,
"ExpectedPValue": 3621847,
"StdExpectedP": 0.050802,
"StdExpectedPValue": 8624188,
"value": 17149
}
},
"site_rank": {
"rank_global": 26,
"rank_country": 3
},
"unix_timestamp": 1613135182,
"mediatype": {
"timemap": false,
"clip": false,
"hastext": true,
"haslogo": false,
"paywall": false,
"fulltext": true,
"text": "WEB"
},
"circulation": 10000,
"stimestamp": 534348465,
"stimestamp_index": 534348465,
"internal_search_reply": {
"id_site": 131,
"id_article": 2122020,
"text": "(otb++) 10:537860106:537126922: (otb--)"
},
"local_time": {
"GMT": 1,
"text": "20210212T14:06:22+0100"
},
"local_rcf822_time": {
"text": "Fri, 12 Feb 2021 14:06:22 +0100"
},
"url": "http://redir.opoint.com/?url=https://www.bbc.com/news/uk-wales-56041927&OpointData=00224944d8c163c6dc90fd9e27cc7c3eJmlkX3NpdGU9MTMxJmlkX2FydGljbGU9MjEyMjAyMCZpZF91c2VyPTIyNDIyJmlkX2FwcGxpY2F0aW9uPTEwMDA0MTcmaWRfcGFydG5lcj0xJmxhbmc9ZW4=",
"orig_url": "https://www.bbc.com/news/uk-wales-56041927",
"url_common": "bbc.com",
"distribute_conditions": "",
"content_protected": 0,
"language": {
"encoding": "iso-639",
"text": "en"
},
"word_count": 119,
"first_source": {
"id": 85181,
"name": "BBC News - Wales: South West Wales",
"sitename": "BBC News",
"url": "https://www.bbc.com/news/wales/south_west_wales",
"siteurl": "http://www.bbc.com"
},
"header": {
"matches": true,
"text": "<match color='4'>Stonehenge</match>: Did animals pull stones from Wales to England?"
},
"summary": {
"matches": true,
"text": " One of Britain's biggest and oldest stone circles has been found in Wales - and could be the original building blocks of <match color='4'>Stonehenge</match>."
},
"body": {
"matches": true,
"text": "<p> Archaeologists uncovered the remains of the Waun Mawn site in Pembrokeshire's Preseli Hills.</p> <p> They believe the stones could have been dismantled and rebuilt 150 miles (240 km) away on Salisbury Plain, Wiltshire.</p> <p> Professor Michael Pearson said the remains found in Pembrokeshire were the exact same size as the ditch around <match color='4'>Stonehenge</match>, and the type of stone matched.</p> <p> Prof Pearson said the remains of a cow, which was found at the site, suggested animals may have helped to pull the stones to their resting spot in England.</p>"
},
"articleimages": {
"count": 1,
"articleimage": [
{
"url": "https://ichef.bbci.co.uk/images/ic/400xn/p096tb52.jpg"
}
]
},
"caption": {
"text": " "
},
"matches": [
{
"type": "word",
"color": 4,
"text": "Stonehenge"
}
],
"matches_table": {
"rows": 1,
"columns": 1,
"row": {
"match_item": {
"rowspan": 1,
"colspan": 1,
"type": "word",
"color": 4,
"active": true,
"length": 59,
"removed": 0,
"text": "Stonehenge"
}
}
},
"quotes": {},
"screenshots": [],
"exist_in_basket": "",
"tags": []
},
{
"id_site": 165150,
"id_article": 86382,
"hidden": false,
"position": 2,
"countrycode": "GB",
"countryname": "United Kingdom",
"similarweb": {
"domain": "bbc.co.uk",
"readership": {
"P": 0.028655,
"value": 2173410
},
"article_readership": {
"P": 0.000107,
"DeltaP": 0.000292,
"DeltaPValue": 22176,
"ExpectedP": 0.000215,
"ExpectedPValue": 16272,
"StdExpectedP": 0.0,
"StdExpectedPValue": 0,
"value": 8135
}
},
"site_rank": {
"rank_global": 3426,
"rank_country": 126
},
"unix_timestamp": 1613135169,
"mediatype": {
"timemap": false,
"clip": false,
"hastext": true,
"haslogo": false,
"paywall": false,
"fulltext": true,
"text": "WEB"
},
"stimestamp": 534348478,
"stimestamp_index": 534348478,
"internal_search_reply": {
"id_site": 165150,
"id_article": 86382,
"text": "(otb++) 10:268683274:537569290: (otb--)"
},
"local_time": {
"GMT": 1,
"text": "20210212T14:06:09+0100"
},
"local_rcf822_time": {
"text": "Fri, 12 Feb 2021 14:06:09 +0100"
},
"url": "http://redir.opoint.com/?url=https://www.bbc.com/news/uk-wales-56041927&OpointData=36be70015ee0d6adc8966c753f648968JmlkX3NpdGU9MTY1MTUwJmlkX2FydGljbGU9ODYzODImaWRfdXNlcj0yMjQyMiZpZF9hcHBsaWNhdGlvbj0xMDAwNDE3JmlkX3BhcnRuZXI9MSZsYW5nPWVu",
"orig_url": "https://www.bbc.com/news/uk-wales-56041927",
"url_common": "bbc.co.uk/radioulster",
"distribute_conditions": "",
"content_protected": 0,
"language": {
"encoding": "iso-639",
"text": "en"
},
"word_count": 119,
"first_source": {
"id": 3020091,
"name": "BBC",
"sitename": "BBC",
"url": "https://www.bbc.co.uk/radioulster",
"siteurl": "https://www.bbc.co.uk/radioulster"
},
"header": {
"matches": true,
"text": "<match color='4'>Stonehenge</match>: Did animals pull stones from Wales to England?"
},
"summary": {
"matches": true,
"text": "One of Britain's biggest and oldest stone circles has been found in Wales - and could be the original building blocks of <match color='4'>Stonehenge</match>."
},
"body": {
"matches": true,
"text": "<p>Archaeologists uncovered the remains of the Waun Mawn site in Pembrokeshire's Preseli Hills.</p> <p>They believe the stones could have been dismantled and rebuilt 150 miles (240 km) away on Salisbury Plain, Wiltshire.</p> <p>Professor Michael Pearson said the remains found in Pembrokeshire were the exact same size as the ditch around <match color='4'>Stonehenge</match>, and the type of stone matched.</p> <p>Prof Pearson said the remains of a cow, which was found at the site, suggested animals may have helped to pull the stones to their resting spot in England.</p>"
},
"matches": [
{
"type": "word",
"color": 4,
"text": "Stonehenge"
}
],
"matches_table": {
"rows": 1,
"columns": 1,
"row": {
"match_item": {
"rowspan": 1,
"colspan": 1,
"type": "word",
"color": 4,
"active": true,
"length": 59,
"removed": 0,
"text": "Stonehenge"
}
}
},
"quotes": {},
"screenshots": [],
"exist_in_basket": "",
"tags": []
}
],
"wikinames": {},
"wikidescriptions": {}
}
}
XML response
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<searchresult documents="2" first_timestamp="1613135182" last_timestamp="1613135169" search_start="2147483647" context="534348478:1:0" count="3" cacheage="-1" cputime="34" host="euler3.opoint.com" compiledate="Feb 2 2021 16:40:04" branch="NOT ACTIVE" notimeout="true" netsprintindextimer="26">
<search_matches>
<search_match type="word" color="4">Stonehenge</search_match>
</search_matches>
<matches_table rows="1" columns="1">
<row>
<match_item rowspan="1" colspan="1" type="word" color="4" active="true" length="59" removed="0">Stonehenge</match_item>
</row>
</matches_table>
<document id_site="131" id_article="2122020" hidden="false" position="1">
<countrycode>GB</countrycode>
<countryname>United Kingdom</countryname>
<similarweb>
<domain>bbc.com</domain>
<mvisit>449415759</mvisit>
<unique_mobile_visit>125968088</unique_mobile_visit>
<unique_desktop_visit>43792708</unique_desktop_visit>
<unique_visit>169760796</unique_visit>
<page_per_visit>1.990107</page_per_visit>
<bounce_rate>0.644187</bounce_rate>
<time_on_site>156</time_on_site>
<category>News_and_Media</category>
<sanitize>1.000000</sanitize>
<readership P="0.012741">2162922</readership>
<article_readership P="0.000105" DeltaP="0.000204" DeltaPValue="34631" ExpectedP="0.021335" ExpectedPValue="3621847" StdExpectedP="0.050802" StdExpectedPValue="8624188">17784</article_readership>
</similarweb>
<site_rank>
<rank_global>26</rank_global>
<rank_country>3</rank_country>
</site_rank>
<unix_timestamp>1613135182</unix_timestamp>
<mediatype timemap="false" clip="false" hastext="true" haslogo="false" paywall="false" fulltext="true">WEB</mediatype>
<circulation>10000</circulation>
<stimestamp>534348465</stimestamp>
<stimestamp_index>534348465</stimestamp_index>
<internal_search_reply id_site="131" id_article="2122020">(otb++) 10:537860106:537126922: (otb--)</internal_search_reply>
<local_time GMT="1">20210212T14:06:22+0100</local_time>
<local_rcf822_time>Fri, 12 Feb 2021 14:06:22 +0100</local_rcf822_time>
<url>http://redir.opoint.com/?url=https://www.bbc.com/news/uk-wales-56041927&OpointData=00224944d8c163c6dc90fd9e27cc7c3eJmlkX3NpdGU9MTMxJmlkX2FydGljbGU9MjEyMjAyMCZpZF91c2VyPTIyNDIyJmlkX2FwcGxpY2F0aW9uPTEwMDA0MTcmaWRfcGFydG5lcj0xJmxhbmc9ZW4=</url>
<orig_url>https://www.bbc.com/news/uk-wales-56041927</orig_url>
<url_common>bbc.com</url_common>
<distribute_conditions></distribute_conditions>
<content_protected>0</content_protected>
<language encoding="iso-639">en</language>
<word_count>119</word_count>
<first_source id="85181">
<name>BBC News - Wales: South West Wales</name>
<sitename>BBC News</sitename>
<url>https://www.bbc.com/news/wales/south_west_wales</url>
<siteurl>http://www.bbc.com</siteurl>
</first_source>
<header matches="true">
<match color="4">Stonehenge</match>
<text>: Did animals pull stones from Wales to England?</text>
</header>
<summary matches="true">
<text> One of Britain's biggest and oldest stone circles has been found in Wales - and could be the original building blocks of </text>
<match color="4">Stonehenge</match>
<text>.</text>
</summary>
<body matches="true">
<p>
<text> Archaeologists uncovered the remains of the Waun Mawn site in Pembrokeshire's Preseli Hills.</text>
</p>
<text></text>
<p>
<text> They believe the stones could have been dismantled and rebuilt 150 miles (240 km) away on Salisbury Plain, Wiltshire.</text>
</p>
<text></text>
<p>
<text> Professor Michael Pearson said the remains found in Pembrokeshire were the exact same size as the ditch around </text>
<match color="4">Stonehenge</match>
<text>, and the type of stone matched.</text>
</p>
<text> </text>
<p>
<text> Prof Pearson said the remains of a cow, which was found at the site, suggested animals may have helped to pull the stones to their resting spot in England.</text>
</p>
</body>
<articleimages count="1">
<articleimage>
<url>https://ichef.bbci.co.uk/images/ic/400xn/p096tb52.jpg</url>
</articleimage>
</articleimages>
<caption>
<text> </text>
</caption>
<matches>
<match type="word" color="4">Stonehenge</match>
</matches>
<matches_table rows="1" columns="1">
<row>
<match_item rowspan="1" colspan="1" type="word" color="4" active="true" length="59" removed="0">Stonehenge</match_item>
</row>
</matches_table>
<quotes></quotes>
<screenshots></screenshots>
<exist_in_basket></exist_in_basket>
<tags></tags>
</document>
<document id_site="165150" id_article="86382" hidden="false" position="2">
<countrycode>GB</countrycode>
<countryname>United Kingdom</countryname>
<similarweb>
<domain>bbc.co.uk</domain>
<mvisit>609102659</mvisit>
<unique_mobile_visit>50563837</unique_mobile_visit>
<unique_desktop_visit>25283654</unique_desktop_visit>
<unique_visit>75847491</unique_visit>
<page_per_visit>3.198875</page_per_visit>
<bounce_rate>0.469738</bounce_rate>
<time_on_site>417</time_on_site>
<category>News_and_Media</category>
<readership P="0.028655">2173410</readership>
<article_readership P="0.000112" DeltaP="0.000292" DeltaPValue="22147" ExpectedP="0.000215" ExpectedPValue="16307" StdExpectedP="0.000000" StdExpectedPValue="0">8522</article_readership>
</similarweb>
<site_rank>
<rank_global>3426</rank_global>
<rank_country>126</rank_country>
</site_rank>
<unix_timestamp>1613135169</unix_timestamp>
<mediatype timemap="false" clip="false" hastext="true" haslogo="false" paywall="false" fulltext="true">WEB</mediatype>
<stimestamp>534348478</stimestamp>
<stimestamp_index>534348478</stimestamp_index>
<internal_search_reply id_site="165150" id_article="86382">(otb++) 10:268683274:537569290: (otb--)</internal_search_reply>
<local_time GMT="1">20210212T14:06:09+0100</local_time>
<local_rcf822_time>Fri, 12 Feb 2021 14:06:09 +0100</local_rcf822_time>
<url>http://redir.opoint.com/?url=https://www.bbc.com/news/uk-wales-56041927&OpointData=36be70015ee0d6adc8966c753f648968JmlkX3NpdGU9MTY1MTUwJmlkX2FydGljbGU9ODYzODImaWRfdXNlcj0yMjQyMiZpZF9hcHBsaWNhdGlvbj0xMDAwNDE3JmlkX3BhcnRuZXI9MSZsYW5nPWVu</url>
<orig_url>https://www.bbc.com/news/uk-wales-56041927</orig_url>
<url_common>bbc.co.uk/radioulster</url_common>
<distribute_conditions></distribute_conditions>
<content_protected>0</content_protected>
<language encoding="iso-639">en</language>
<word_count>119</word_count>
<first_source id="3020091">
<name>BBC</name>
<sitename>BBC</sitename>
<url>https://www.bbc.co.uk/radioulster</url>
<siteurl>https://www.bbc.co.uk/radioulster</siteurl>
</first_source>
<header matches="true">
<match color="4">Stonehenge</match>
<text>: Did animals pull stones from Wales to England?</text>
</header>
<summary matches="true">
<text>One of Britain's biggest and oldest stone circles has been found in Wales - and could be the original building blocks of </text>
<match color="4">Stonehenge</match>
<text>.</text>
</summary>
<body matches="true">
<p>
<text>Archaeologists uncovered the remains of the Waun Mawn site in Pembrokeshire's Preseli Hills.</text>
</p>
<text>
</text>
<p>
<text>They believe the stones could have been dismantled and rebuilt 150 miles (240 km) away on Salisbury Plain, Wiltshire.</text>
</p>
<text>
</text>
<p>
<text>Professor Michael Pearson said the remains found in Pembrokeshire were the exact same size as the ditch around </text>
<match color="4">Stonehenge</match>
<text>, and the type of stone matched.</text>
</p>
<text> </text>
<p>
<text>Prof Pearson said the remains of a cow, which was found at the site, suggested animals may have helped to pull the stones to their resting spot in England.</text>
</p>
</body>
<matches>
<match type="word" color="4">Stonehenge</match>
</matches>
<matches_table rows="1" columns="1">
<row>
<match_item rowspan="1" colspan="1" type="word" color="4" active="true" length="59" removed="0">Stonehenge</match_item>
</row>
</matches_table>
<quotes></quotes>
<screenshots></screenshots>
<exist_in_basket></exist_in_basket>
<tags></tags>
</document>
</searchresult>
Search requests
Return article text
In order to also return headers and text from the matching articles, the corresponding fields in the main parameter must be set to non-zero values.
{
"searchterm": "'artificial intelligence'",
"params": {
"requestedarticles": 10,
"main": {
"header": 1,
"summary": 1,
"text": 1
}
}
}Filter on publishing time
You can filter the result by publishing time using the oldest and newest parameters.
{
"searchterm": "'artificial intelligence'",
"params": {
"requestedarticles": 10,
"oldest": 1609455600,
"newest": 1612134000,
"main": {
"header": 1,
"summary": 1,
"text": 1
}
}
}List and highlight search matches
To return phrases matching the search term, main.matches must be set to true.
If the main.header, main.summary and main.body fields in main is also set to 2, those fields will wrap any matches in <match> tags. This can be used for highlighting.
{
"searchterm": "'artificial intelligence'",
"params": {
"requestedarticles": 10,
"main": {
"header": 2,
"summary": 2,
"text": 2,
"matches": true
}
}
}Group identical articles
Some articles are classified as identical. It could be a column delivered to several outlets, or a plain news article published by different papers, but with the same source.
By default, articles deemed identical are included, but not grouped. Set groupidentical to true, in order to only return one article in the group at the top level, and include the rest in a identical_articles field under that article.
Setting identical to inherit, means that the identical articles share the same content settings, header, summary, text etc., as the top level document. This setting should be set.
Note: Grouping articles can be slightly expensive if the result set is very large.
{
"searchterm": "'artificial intelligence'",
"params": {
"requestedarticles": 10,
"main": {
"header": 1,
"summary": 1,
"text": 1
},
"groupidentical": true,
"identical": {
"inherit": true
},
"includeidentical": true
}
}Get articles with specific ids
We can get a specific set of articles by listing their site and article ids in the articles parameter.
When only getting articles with specific ids, there is no need for the system to parse the search string, it is not used anyways. By setting select_by_ids to true, the search string is disregarded altoghether. This results in much more efficient lookups.
Make sure that the requestedarticles param is at least as high as the number of ids, otherwise, the resulting list is capped.
{
"params": {
"requestedarticles": 2,
"articles": [
{
"id_site": 11,
"id_article": 887677
},
{
"id_site": 11,
"id_article": 887676
}
],
"select_by_ids": true,
"main": {
"header": 1,
"summary": 1,
"text": 1
}
}
}Paginate results
We can control the size of the result set with requestedarticles. But what if dont know exactly how many articles we want, for instance if we are fetching all articles from the last week that match the search term?
This is when the context parameter is useful. With it, you can send the same search again, but instead of receiving the same articles, you tell the search server that you want the next batch of articles.
The first search is constructed as normal, but with the context parameter empty. This tells the search server that we want the most recent content.
{
"searchterm": "'artificial intelligence'",
"params": {
"requestedarticles": 1000,
"context": "",
"oldest": 1626152097,
"main": {
"header": 1,
"summary": 1,
"text": 1
}
}
}The response will contain a context field. If it is non-empty, there are more results matching the search, but which could not be returned due to a limit on the number of articles in the response.
By passing the context value from the response as the context parameter in the next search, we match articles in the same way, but tell the server to look for articles that are older than those in the last reponse.
{
"searchterm": "'artificial intelligence'",
"params": {
"requestedarticles": 1000,
"context": "540062299:1:0;TOTDOC=10",
"oldest": 1626152097,
"main": {
"header": 1,
"summary": 1,
"text": 1
}
}
}This way we can loop until we have gotten all the articles matching the search, and do so with a reasonble amount of articles in each response.
Only receive new hits
Repeatedly querying the search endpoint for the
But as integrating with StoredSearch requires some effort, we offer a lighter version, where you mark your search as an
{
"searchterm": "Stoltenberg",
"params": {
"update_search": true
}
}When encountering a new search with the parameter set, we will register it as an update search. For a few minutes there is an initial startup phase where the endpoint will return results as it normally would. After that initial warm up phase is over, any subsequent calls to the endpoint will only return new hits to the search, excluding the articles that you've gathered before. You can reuse the same search request for all calls.