What if my webhook endpoint is down?
If your Crawler callback is down, you are notified by email, your crawlers get paused and your last failed request due to downtime at your endpoint, is set to be retried. Your crawlers get resumed when your endpoint becomes available automatically. Our monitoring system checks your endpoint every minute.
Live monitor wordings
"Waiting" means that your requests are in your crawler queue waiting to be processed. "Concurrent crawlers" are the requests that are being crawled at the same time. Concurrent crawlers gets increased by our system if you have many pages to crawl, we also monitor crawlers and increase or decrease the concurrency depending on the pool. "Sets to be retried" are your requests that failed for any reason, they land in your crawler retry queue and are processed with a retry rate up until maximum 110 retries.
Where can I get the API keys?