# Proxy API Mode (Beta)

If your application is not designed to work with an API to crawl and scrape websites like the Crawling API, we have designed an intelligent rotating proxy API that forwards your requests to the Crawling API. You simply use it as a normal proxy in your application.

All Proxy calls should go to http://proxyapi.proxycrawl.com and port 8000 using your access token as a proxy username.

Therefore making your first call is as easy as running the following line in the terminal. Go ahead and try it!

curl -x "http://[email protected]:8000" -k "http://httpbin.org/ip"

# How it works?

When you send a request to the Proxy API, the proxy will authorize your request using your proxy authorization username which is your normal or javascript access token, check your tokens below. It will then redirect your request to the Crawling API and then return the response to your application. If you require to use the extra features of the Crawling API in this mode, you will need to send the HTTP header ProxyCrawlAPI-Parameters and send the options you require to use. Check the examples section below for real examples.

Normal token

_USER_TOKEN_

Javscript token

_JS_TOKEN_

# Important Note

It is important to disable SSL verification when using the Crawling API in proxy mode, otherwise we will not be able to redirect your requests to the Crawling API. Therefore you should skip verifying certificates.

# Proxy API Mode Rate Limit

By default the Crawling API in proxy mode is rate limited to 20 requests per second (1.728M req/day). If your proxy management solution is working with concurrent requests/threads instead of requests per second, its important to note that 20 requests per second converts to much more concurrent requests in general. As an example, If you are crawling Amazon with ProxyCrawl, the average request takes about 4 seconds, therefore 20 requests per second converts to 80 concurrent threads. If the website you are crawling responds quick then you need less concurent requests. If you hit the limit of concurrent requests, please contact support (opens new window) with your usecase to increase your concurrency.

# Proxy API Mode request example

  • curl
  • ruby
  • node
  • php
  • python
  • go
// GET request

curl -x "http://[email protected]:8000" -k "http://httpbin.org/ip"

// GET request with custom UserAgent and returning original headers

// Make use of the API parameters, like store=true, device=mobile, get_cookies=true and etc.

// Example of using a custom user agent and storing a copy of the result in ProxyCrawl Cloud storage.

curl -H "ProxyCrawlAPI-Parameters: get_headers=true&store=true&user_agent=Mozilla%2F5.0%20%28Windows%20NT%2010.0%3B%20Win64%3B%20x64%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Chrome%2F91.0.4472.124%20Safari%2F537.36" \
-x "http://[email protected]:8000" \
-k "http://httpbin.org/headers"

// GET request with headless browser (Javascript Enabled)

curl -x "http://[email protected]:8000" \
-k "http://httpbin.org/anything"

// POST form request
curl -H 'Content-Type: application/x-www-form-urlencoded' \
-F 'param=value' \
-X POST \
-x "http://[email protected]:8000" \
-k "http://httpbin.org/anything"

// POST JSON request
curl -H "accept: application/json" \
--data '{"key1":"value1","key2":"value2"}' \
-X POST \
-x "http://[email protected]:8000" \
-k "http://httpbin.org/anything"