# Pushing data to the Crawler

Before starting to push urls to the Crawler, you first need to create a new Crawler one here.

To push urls to be crawled by the Crawler, you must use the Crawling API with two additional parameters:

  • You must append &callback=true
  • You must append &crawler=YourCrawlerName using the name of the crawler you created here.

In response to your crawler push, the API will send back a JSON representation with a unique request identifier RID. This RID is unique and will help you identify the request at any point in the future.

Example of push response:

{ "rid": "1e92e8bff32c31c2728714d4" }

By default, you can push up to 30 urls each second to the Crawler.

# Sending additional data

Optionally, you can receive custom headers to your callback if you use the callback_headers parameter. That is great for passing additional data for identification purposes at your side.

The format is the following: HEADER_NAME:VALUE|HEADER_NAME_2:VALUE_2|etc. And it must be encoded properly.

Example for headers and values MY_ID 1234, some_other 4321


Those headers will come back in the webhook post request.