crawler FAQ

How many times are requests re-tried?

By default a request is retried 110 times for a period of 48 hours from the first time it starts to be processed.

What happens in the case of a permanent failure?

You will always receive a callback in case of success or failure, please check the pc_status and original_status to know the status code.

What if my webhook endpoint is down?

If your Crawler callback is down, you are notified by email, your crawlers get paused and your last failed request due to downtime at your endpoint, is set to be retried. Your crawlers get resumed when your endpoint becomes available automatically. Our monitoring system checks your endpoint every minute.

Live monitor wordings

"Waiting" means that your requests are in your crawler queue waiting to be processed. "Concurrent crawlers" are the requests that are being crawled at the same time. Concurrent crawlers gets increased by our system if you have many pages to crawl, we also monitor crawlers and increase or decrease the concurrency depending on the pool. "Sets to be retried" are your requests that failed for any reason, they land in your crawler retry queue and are processed with a retry rate up until maximum 110 retries.

Where can I get the API keys?

You can the API Key either in the Normal Request Tokens or the Javascript Request Tokens.

There is a limit of 30 URLs push each second. We are looking to crawl for a huge number of URL. Is there a way to increase the limit?

For LinkedIn crawls, the limit is 30 push per second. But for other URLs, we can increase the limit on a case-to-case basis

Start crawling the web today
Try it free. No credit card required. Instant set-up.