StoredSearch

With the StoredSearch API you register searches that we match against newly crawled articles. The search hits are provided in a near real time feed.

It is important to understand that although you can register multiple searches, all the hits are delivered in a single feed. Each StoredSearch is assigned an id, and every document in the feed contains a stored_search_id, indicating which of the StoredSearches it matched.

All requests are authorized with your API token, as described on the front page.

Register your searches

To register a search through the API, send its search string to /storedsearch/. It uses the same search syntax as the search API:

curl "https://api.opoint.com/storedsearch/" \
    -H "Authorization: Token yourtoken" \
    -H "Content-Type: application/json" \
    -d '{"search": "\"Novo Nordisk\" AND lang:en"}'

The response contains the StoredSearch's id, which is the stored_search_id you will see in the feed.

StoredSearches are matched against newly crawled articles. Set get_old_articles to also retrieve up to 25 000 articles from the past, and max_age to how many seconds back to search.

Check the search string first

The endpoint doesn't validate the syntax of the search string. Before registering it, send it as the searchterm of a request to the search API. If the response has an errors field, or a debug field with corrections, fix the search string first; see errors in the search term. The search also shows which articles the StoredSearch would match.

Or use the Opoint Portal

The Opoint Portal handles the whole process: build the search with the query builder, check what it matches, and add it to your feed. We recommend doing this for more complex searches.

Pull the feed

Instead of pulling the feed, you can also have the matches pushed to you; see pull or push.

The feed returns the articles matching any of your StoredSearches, starting at the from parameter, searching forward in time:

curl "https://api.opoint.com/storedsearch/feed/?from=1767225600&interval=1200" \
    -H "Authorization: Token yourtoken" \
    -H "Accept: application/json"

All parameters of the feed are described in the API reference.

If you are continuously pulling data from the feed, a good strategy is to first pick an initial timestamp from where to get data. Then, make a request, passing the from and interval parameters. From the response, get the next_from field, and use it as the from parameter in the next request, to get the next batch of articles.

Set interval to about twice the time between your requests: if you request the feed every 600 seconds, set it to 1200. The feed never searches past the current time, so a larger interval costs nothing, but it lets you catch up after an outage. The interval can be between 30 and 1800 seconds.

import time

import requests

FEED_URL = "https://api.opoint.com/storedsearch/feed/"
HEADERS = {"Authorization": "Token yourtoken", "Accept": "application/json"}

# Poll every 10 minutes, with an interval twice as long, so a missed
# request or a short outage is caught up on the next one.
POLL_EVERY = 600
INTERVAL = 1200

next_from = int(time.time()) - POLL_EVERY

while True:
    response = requests.get(
        FEED_URL, headers=HEADERS, params={"from": next_from, "interval": INTERVAL}
    )
    response.raise_for_status()
    searchresult = response.json()["searchresult"]

    for article in searchresult.get("document", []):
        print(article["stored_search_id"], article["header"]["text"])

    next_from = searchresult["next_from"]
    time.sleep(POLL_EVERY)

Add topics and entities

Many want topics and entities, or readership data, on the articles. Add them with the textrazor and readership parameters in the url. textrazor takes the same value as in the search API, and both require that your license includes the data:

curl "https://api.opoint.com/storedsearch/feed/?from=1767225600&interval=1200&textrazor=30975&readership=1" \
    -H "Authorization: Token yourtoken" \
    -H "Accept: application/json"

Use interval, not to

The feed also accepts a to parameter, but you should generally not use it. With to, the response has no next_from, so you have to keep track of the time windows yourself, and a to in the future is answered with 429 Too Many Requests. Use from and interval, and pass on next_from.

Leave num_art at the default

num_art limits the number of articles per StoredSearch in each request, and is 25 000 by default. Articles beyond the limit are not returned later, since next_from moves past them, so don't set it lower than one of your StoredSearches can match in one interval.

The response

The articles are in searchresult.document, newest first, in the same format as from the search API, with the article text included. In addition, each document has a stored_search_id: the id of the StoredSearch it matched.

searchresult.next_from is where to continue from, and is there even when there are no new articles. Add format=xml to get XML instead of JSON. Metadata like topics and entities or readership is added with the textrazor and readership parameters.

Manage your searches

/storedsearch/ lists your StoredSearches, and /storedsearch/{id}/ retrieves, updates or deletes one of them. Every endpoint and parameter is described in the API reference.