# The Crawling API in minutes
We've created an API that will make integrating ProxyCrawl in your crawling project very easy.
# Free trial
The first 1,000 requests are free of charge.
Make sure to fully use the free trial!
# Rate limit
The API is rate limited to a maximum of 20 requests per second, per token (rate limit can be increased upon request).
This means that you can send up to 20 requests every second, which means around 51 million requests per month, regardless of the number of threads they use.
The API will respond with
429 status code when the rate limit is exceeded.
Note: Some specific websites might have lower limits. If you require higher limits on those, please contact support (opens new window).
# API response times
The average API response time is between 4 to 10 seconds, but we recommend setting a timeout for the calls of at least 90 seconds.
# Success vs fail
# Other notes
- If you prefer to use a library to integrate ProxyCrawl, you can see available API libraries here (opens new window).
- Using the Accept-Encoding gzip header is recommended.
- If you use Scrapy for python, make sure to disable the DNS cache (opens new window).
You will need authentication tokens to use the API.
# Your first API call
All API URLs start with the following base part:
Therefore making your first call is as easy as running the following line in the terminal.
Go ahead and try it!
Go ahead and try JS crawling!
Note that POST requests are also supported. Please go here for more information or continue below with the API parameters.