Want to use OxyLabs' residential proxies? Then you can't miss this article. It is the most comprehensive OxyLabs residential proxies user guide.

Quick Start

This is by far the fastest way to start using Residential Proxies. You will make a request to https://ipinfo.io using a residential proxy from a random location through our Backconnect Entry. Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME:PASSWORD" https://ipinfo.io

Let's make a request using a proxy from France. All you need to do is add cc-FR to your username string. Below is an example in cURL, and here you can find code samples in other languages.

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-cc-FR:PASSWORD" https://ipinfo.io

Here is one more example of a request that goes through a proxy from London, United Kingdom. Adding cc-GB-city-london to username string enables you to do just that. Click here to get more information on how to master city-level proxy targeting.

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-cc-GB-city-london:PASSWORD" https://ipinfo.io

What if you want to keep the same proxy for more than one request? Session control enables that. Just add sessid-abc12345 to your username string, where abc12345 can be any random string, and as long as you keep sending requests with this string (session ID), we will try our best to give you the same proxy IP.

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-sessid-randomString123:PASSWORD" https://ipinfo.io

Check out our integration tutorials with the most popular 3rd party tools, such as Multilogin, ShadowRocket, or SwitchyOmega.

If you have any additional questions, please contact your account manager or our support team at support@oxylabs.io.

To find out your usage statistics, generated traffic, request count and more, please login to dashboard.oxylabs.io. You can also add proxy users (subusers) and track their statistics separately.

Backconnect Entry Node

pr.oxylabs.io:7777

The single backconnect proxy enables to choose a specific country or city proxy via additional parameters in the username. This approach also supports session control. Below is a sample credentials structure:

customer-USERNAME-cc-US-city-CITY-sessid-abcde12345:PASSWORD

If you add more than one location parameter in the username, the system will ignore all but one. Therefore it's important to understand which parameters get prioritized. The hierarchy is as follows:

1. city
2. state
3. cc

For example, if your query has these parameters customer-username-cc-US-city-paris-st-us_california, you will get a proxy from Paris, US.

Entry Node for China

https://a81298871.com:8000 - Hong Kong

https://cnt9t1is.com:8000 - Beijing

For better connectivity from China, we have configured entry nodes in Hong Kong and Beijing regions. Note that it is an HTTPS proxy node. All username parameters are identical to the regular entry node.

Please note that some 3rd party tools currently do not work with Hong Kong and Beijing entry nodes. 3rd party tools that work with said entry nodes are:

1. 1. 1. Oxylabs Proxy Manager
      2. SwitchyOmega
      3. Shadowrocket
      4. Postern

We are currently working on fixing the rest of the integrations on which HTTPS protocol will be supported.

If you are connecting to our residential network from China and still have issues with provided entry nodes, please get in touch at support@oxylabs.io and we will figure out a solution together.

Basic Query

Basic query only requires to pass username and password. No other parameters are needed. Such query will result in the request being made from a random IP address (proxy). Every new request will use a different proxy.

In this example a query to ipinfo.io is performed from a random IP:

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME:PASSWORD" https://ipinfo.io

Select Country

Adding cc flag to the authorization header enables to specify which country IP to use to process the request. The value of this parameter is a case insensitive country code in 2-letter 3166-1 alpha-2 format. For example, DE for Germany, GB for the United Kingdom, TH for Thailand proxy. See examples for more details.

In this example a query to ipinfo.io is performed from a random IP address from Germany:

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-cc-DE:PASSWORD" https://ipinfo.io

Select State

To get proxy from a specified US state, add st flag to the authorization string. For example, us_california, us_illinois. Full list of supported cities in TXT format can be downloaded here.

In this example a query to ipinfo.io is performed from a random IP address from California, USA:

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-st-us_california:PASSWORD" https://ipinfo.io

Select City

To narrow down to a city-level targeting, a city parameter needs to be added. For example, cc-DE-city-munich means that a proxy from Munich, Germany, will handle the query. We support every city in the world, but we do not guarantee we will have proxies there. Most popular cities are well covered, and will have many proxies to choose from.

Here are a few examples of valid combinations of cc and city parameters:
cc-US-city-los_angeles
cc-IT-city-rome
cc-TH-city-bangkok
cc-JP-city-tokyo
cc-AU-city-sydney
cc-ES-city-barcelona

In this example a query to ipinfo.io is performed from a random IP address from Munich, Germany:

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-cc-DE-city-munich:PASSWORD" https://ipinfo.io

Session Control

Adding sessid parameter with session ID string, for example sessid-abc12345, enables session control. This means that the proxy will not change with the following requests. The example below shows how session control works:

The system keeps the same IP associated with a particular session ID as long as there is some activity (requests). After 60 seconds of inactivity, the IP is automatically changed to a different one.

For example: * Your initial query with sessid-random123 got assigned proxy IP address 1.1.1.1 * As long as you keep sending new requests with the same session ID and that IP address is online and available, the system will route your queries through 1.1.1.1 * If you stop sending requests or the IP address is no longer online, the system will assign a new proxy IP after 60 seconds. * This means that your next query with sessid-random123 will be routed through a different IP address.

Standard session time is 10 minutes, but it can be increased. Contact your account manager to learn more.

German IP will be chosen for the first request, and then the same IP will be kept with new queries (session control):

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-cc-DE-sessid-abcde12345:PASSWORD" https://ipinfo.io

Country Specific Entry Nodes

Random Proxy Entry Nodes

Oxylabs residential network has coverage in almost every country in the world. Country-specific random proxy entry point will return new IP with every new request.

You do not need to pass any additional parameters, just customer-username:password. If you have whitelisted IPs, you do not need to pass login credentials. Country-specific ports do not support city-level results.

Please see the dashboard for entry point information or download a full list of random entry points in XLS here.

In this example a query to ipinfo.io is performed from a random Turkey IP:

curl -x tr-pr.oxylabs.io:30000 -U "customer-USERNAME:PASSWORD" https://ipinfo.io

pr.oxylabs.io:7777
us-pr.oxylabs.io:10000
ca-pr.oxylabs.io:30000
gb-pr.oxylabs.io:20000
de-pr.oxylabs.io:30000
fr-pr.oxylabs.io:40000
es-pr.oxylabs.io:10000
it-pr.oxylabs.io:20000
se-pr.oxylabs.io:30000
gr-pr.oxylabs.io:40000
pt-pr.oxylabs.io:10000
nl-pr.oxylabs.io:20000
be-pr.oxylabs.io:30000
ru-pr.oxylabs.io:40000
ua-pr.oxylabs.io:10000
pl-pr.oxylabs.io:20000
il-pr.oxylabs.io:20000
tr-pr.oxylabs.io:30000
au-pr.oxylabs.io:40000
my-pr.oxylabs.io:10000
th-pr.oxylabs.io:20000
kr-pr.oxylabs.io:30000
jp-pr.oxylabs.io:40000
ph-pr.oxylabs.io:10000
sg-pr.oxylabs.io:20000
cn-pr.oxylabs.io:30000
hk-pr.oxylabs.io:40000
tw-pr.oxylabs.io:10000
in-pr.oxylabs.io:20000
pk-pr.oxylabs.io:30000
ir-pr.oxylabs.io:40000
id-pr.oxylabs.io:10000
az-pr.oxylabs.io:20000
kz-pr.oxylabs.io:30000
ae-pr.oxylabs.io:40000
mx-pr.oxylabs.io:10000
br-pr.oxylabs.io:20000
ar-pr.oxylabs.io:30000
cl-pr.oxylabs.io:40000
pe-pr.oxylabs.io:10000
ec-pr.oxylabs.io:20000
co-pr.oxylabs.io:30000
za-pr.oxylabs.io:40000
eg-pr.oxylabs.io:10000
sa-pr.oxylabs.io:44000
dk-pr.oxylabs.io:19000
ao-pr.oxylabs.io:40000
cm-pr.oxylabs.io:41000
cf-pr.oxylabs.io:42000
td-pr.oxylabs.io:43000
bj-pr.oxylabs.io:44000
et-pr.oxylabs.io:45000
dj-pr.oxylabs.io:46000
gm-pr.oxylabs.io:47000
gh-pr.oxylabs.io:48000
ci-pr.oxylabs.io:49000
ke-pr.oxylabs.io:10000
lr-pr.oxylabs.io:11000
mg-pr.oxylabs.io:12000
ml-pr.oxylabs.io:13000
mr-pr.oxylabs.io:14000
mu-pr.oxylabs.io:15000
ma-pr.oxylabs.io:16000
mz-pr.oxylabs.io:17000
ng-pr.oxylabs.io:18000
sn-pr.oxylabs.io:19000
sc-pr.oxylabs.io:20000
zw-pr.oxylabs.io:21000
ss-pr.oxylabs.io:22000
sd-pr.oxylabs.io:23000
tg-pr.oxylabs.io:24000
tn-pr.oxylabs.io:25000
ug-pr.oxylabs.io:26000
zm-pr.oxylabs.io:27000
af-pr.oxylabs.io:28000
bh-pr.oxylabs.io:29000
bd-pr.oxylabs.io:30000
am-pr.oxylabs.io:31000
bt-pr.oxylabs.io:32000
mm-pr.oxylabs.io:33000
kh-pr.oxylabs.io:34000
ge-pr.oxylabs.io:36000
iq-pr.oxylabs.io:37000
jo-pr.oxylabs.io:38000
lb-pr.oxylabs.io:39000
mv-pr.oxylabs.io:40000
mn-pr.oxylabs.io:41000
om-pr.oxylabs.io:42000
qa-pr.oxylabs.io:43000
sa-pr.oxylabs.io:44000
vn-pr.oxylabs.io:45000
tm-pr.oxylabs.io:46000
uz-pr.oxylabs.io:47000
ye-pr.oxylabs.io:48000
al-pr.oxylabs.io:49000
ad-pr.oxylabs.io:10000
at-pr.oxylabs.io:11000
ba-pr.oxylabs.io:13000
bg-pr.oxylabs.io:14000
by-pr.oxylabs.io:15000
hr-pr.oxylabs.io:16000
cy-pr.oxylabs.io:35000
cz-pr.oxylabs.io:18000
dk-pr.oxylabs.io:19000
ee-pr.oxylabs.io:20000
fi-pr.oxylabs.io:21000
hu-pr.oxylabs.io:23000
is-pr.oxylabs.io:24000
ie-pr.oxylabs.io:25000
lv-pr.oxylabs.io:26000
li-pr.oxylabs.io:27000
lt-pr.oxylabs.io:28000
lu-pr.oxylabs.io:29000
mt-pr.oxylabs.io:30000
mc-pr.oxylabs.io:31000
md-pr.oxylabs.io:32000
me-pr.oxylabs.io:33000
no-pr.oxylabs.io:34000
ro-pr.oxylabs.io:35000
rs-pr.oxylabs.io:36000
sk-pr.oxylabs.io:37000
si-pr.oxylabs.io:38000
ch-pr.oxylabs.io:39000
mk-pr.oxylabs.io:40000
bs-pr.oxylabs.io:41000
bz-pr.oxylabs.io:42000
vg-pr.oxylabs.io:43000
cr-pr.oxylabs.io:44000
cu-pr.oxylabs.io:45000
dm-pr.oxylabs.io:46000
ht-pr.oxylabs.io:47000
hn-pr.oxylabs.io:48000
jm-pr.oxylabs.io:49000
aw-pr.oxylabs.io:10000
pa-pr.oxylabs.io:11000
pr-pr.oxylabs.io:12000
tt-pr.oxylabs.io:13000
fj-pr.oxylabs.io:14000
nz-pr.oxylabs.io:15000
bo-pr.oxylabs.io:16000
py-pr.oxylabs.io:17000
uy-pr.oxylabs.io:18000
ve-pr.oxylabs.io:19000
kw-pr.oxylabs.io:20000

Sticky Proxy Entry Nodes

Country-specific sticky proxy entry point will return the same IP with every new request while you will use the same port. IP stickiness works for up to 10 minutes. After that, the IP is replaced with a new one.

You do not need to pass any additional parameters, just customer-username:password. If you have whitelisted IPs, you do not need to pass login credentials. Country-specific ports do not support city-level results.

Please see the dashboard for entry point information or download a full list of sticky entry points in XLS here.

In this example, a query to ipinfo.io is performed from Turkey IP, and the same IP will stay for up to 10 minutes with each request. We have chosen port 30001, however, Turkey sticky entry points cover ports from 30001 to 39999.

curl -x tr-pr.oxylabs.io:30001 -U "customer-USERNAME:PASSWORD" https://ipinfo.io

pr.oxylabs.io:10000 - 49999
us-pr.oxylabs.io:10001 - 19999
ca-pr.oxylabs.io:30001 - 39999
gb-pr.oxylabs.io:20001 - 29999
de-pr.oxylabs.io:30001 - 39999
fr-pr.oxylabs.io:40001 - 49999
es-pr.oxylabs.io:10001 - 19999
it-pr.oxylabs.io:20001 - 29999
se-pr.oxylabs.io:30001 - 39999
gr-pr.oxylabs.io:40001 - 49999
pt-pr.oxylabs.io:10001 - 19999
nl-pr.oxylabs.io:20001 - 29999
be-pr.oxylabs.io:30001 - 39999
ru-pr.oxylabs.io:40001 - 49999
ua-pr.oxylabs.io:10001 - 19999
pl-pr.oxylabs.io:20001 - 29999
il-pr.oxylabs.io:20001 - 29999
tr-pr.oxylabs.io:30001 - 39999
au-pr.oxylabs.io:40001 - 49999
my-pr.oxylabs.io:10001 - 19999
th-pr.oxylabs.io:20001 - 29999
kr-pr.oxylabs.io:30001 - 39999
jp-pr.oxylabs.io:40001 - 49999
ph-pr.oxylabs.io:10001 - 19999
sg-pr.oxylabs.io:20001 - 29999
cn-pr.oxylabs.io:30001 - 39999
hk-pr.oxylabs.io:40001 - 49999
tw-pr.oxylabs.io:10001 - 19999
in-pr.oxylabs.io:20001 - 29999
pk-pr.oxylabs.io:30001 - 39999
ir-pr.oxylabs.io:40001 - 49999
id-pr.oxylabs.io:10001 - 19999
az-pr.oxylabs.io:20001 - 29999
kz-pr.oxylabs.io:30001 - 39999
ae-pr.oxylabs.io:40001 - 49999
mx-pr.oxylabs.io:10001 - 19999
br-pr.oxylabs.io:20001 - 29999
ar-pr.oxylabs.io:30001 - 39999
cl-pr.oxylabs.io:40001 - 49999
pe-pr.oxylabs.io:10001 - 19999
ec-pr.oxylabs.io:20001 - 29999
co-pr.oxylabs.io:30001 - 39999
za-pr.oxylabs.io:40001 - 49999
eg-pr.oxylabs.io:10001 - 19999
sa-pr.oxylabs.io:44001 - 44999
dk-pr.oxylabs.io:19001 - 19999

Additional Information

Country, State, City Mapping

We use MaxMind GeoIP2 database to assign our exit nodes cc, st and city values. This database is updated weekly.

Some other public IP databases may show different locations. In cases where location mismatch is crucial, we encourage to double-check if the location of the assigned proxy is the same on the IP database used by the target website.

Response Codes

These are the most common response codes that you can get while using our residential proxy pool. If you receive any other response code, get in touch with your dedicated account manager for more information.

Backconnect Entry IPs

If you want to use your own backconnect entry node DNS name, you can do that by pointing your subdomain/domain to our entry node. To do that, you need to add a DNS CNAME record that has pr.oxylabs.io as the target. This is how such setup looks on Cloudflare DNS manager for entry subdomain:

For more information, please contact support@oxylabs.io or your account manager.

Restricted URLs

To avoid abuse and unclear activities, some websites are restricted on our residential proxy network. The list includes, but is not limited to websites such as all Google domains including Play, linkedin.com, sonyentertainmentnetwork.com, all Apple domains, including iTunes, netflix.com and others. For more information please contact our support staff at support@oxylabs.io.

Public API

To manage proxy users via Public API, please refer to this documentation in OpenAPI format. The API allows to create, delete, and modify proxy users, set their limits and see current usage, disable and enable them.

3rd Party Integrations

Please note that you must use an HTTPS protocol when accessing China entry nodes for 3rd party tools to work correctly.

Oxylabs Proxy Manager

Oxylabs Proxy Manager is a proxy management extension for Chrome. It is not only free but also works with any proxy provider of your choice. Enable your proxy without having to jump through settings and menu options. The extension has an additional feature specifically designed to use with Oxylabs Residential Proxies, which enables to easily change Oxylabs Residential Proxy session without editing your username. Download the extension from Chrome Web Store here

• Click the extension icon to open the application
• If you haven't added any proxies yet, you will see Add new proxy button. Click it to continue.

• Add proxy profile. Provide a name for the profile under Enter name. Enter pr.oxylabs.io as Proxy server IP and 7777 as Port. Fill in your Username and Password. Do not forget to add customer- before your username. Finally, click Save changes.

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Proxy server IP and 10001 under Port, you will receive US exit node with sticky session.

If you have whitelisted your IP, you can skip this step.

• Open the extension and click Connect. That's it, you are now using proxies.

• Chrome is notorious for caching proxy credentials, and on many proxy management extensions sticky sessions via username string does not work. If you are using sessions with main entry node (pr.oxylabs.io:7777), we created a special feature that allows rotating session much more easily. First, enter your username with sessid parameter, like this:

• Then, open the extenion window. Connect to the proxy and you will see a new button – Refresh session. Click it to change session ID and get a new proxy.

SwitchyOmega

SwitchyOmega is a powerful and reliable proxy manager that works with many popular browsers. First you need to get the plugin. For Chrome, click here, for Firefox click here. Once you have it installed, you should:

• Open Switchy Omega options.

• Click New profile….

• Give this new profile a name, choose Proxy Profile type and click Create

• Change Protocol to HTTP. Under Server enter our backconnect entry node pr.oxylabs.io. Under Port add 7777

• You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Server and 10001 under Port, you will receive US exit node with sticky session.
• Next, click the lock button on the right. Fill in your Username and Password. Do not forget to add customer- before your given username. Finally, click Save changes.

If you have whitelisted your IP, please skip this step.

• Click Apply changes

• For the final step, click on SwitchyOmega icon and choose your newly configured proxy. You are now ready to go!

Shadowrocket

If you want to use our Residential Proxies with Shadowrocket, you can easily do that by following the instructions below.

• Open Shadowrocket app.
• Click Add Server.

• In the following window click on Type.

• Choose HTTP and go back to the previous screen.

• Enter proxy and authentication information. Under Host type in pr.oxylabs.io. Port is 7777 to use our backconnect entry. Do not forget your crendentials under User and Password. Do not forget to add customer- before your given username. Once you are done, go back to the previous screen by pressing Back arrow.

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Host and 10001 under Port, you will receive US exit node with a sticky session.

• You need to turn proxy on. Do that by activating a toggle under Not Connected

• If that is your first time using Shadowrocket, you will get an iOS prompt to add Shadowrocket to VPN configurations. Click Add. You will need to enter your iPhone passcode to confirm.

• That is it. A toggle at the top of the app should now be activated.

• You can test if the proxy is working by visiting icanhazip.com.

Proxifier

If you want to use our Residential Proxies with Proxfier, you can easily do that by following the instructions below.

• Open Proxifier app.
• Add proxies. Click on Proxies (1). Click Add(2).

• In the following window enter required information. Address(1), which is pr.oxylabs.io, Port(2) which is 7777, select HTTPS(3) protocol. In authentication box click Enable(4) and enter provided Username(5) and Password(6). Do not forget to add customer- at the beginning of the username. If you have whitelisted your devices, do not click on Enable, your authenticiation will be done without username/password. If you wish, you can enable Appear as Web Browser option. Click OK.
• You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Address(1)* and **10001 under Port(2), you will receive US exit node with sticky session.

• You will be prompted with a window asking if you want Proxifier to use this proxy by default? This means that this setup will be user for all of your browsers. It is up to you to decide. If you select Yes, you are done setting up. If you wish to use proxies on a particular browser, follow the next steps.
• Click on Rules(1) and then click Add…(2).

• Click +(1) button to add applications that you want to use proxy settings. Select the browser with which you want to use these proxy settings and click on it. Finally, you have to change Action(2). Click on it and select your created proxy.

• This is it, all of your requests, while using browser which you defined in your rules of Proxifier, will be done from proxies.

Multilogin

Our proxies also work with Multilogin app. Setting up is easy, just follow the instructions below.

• Open Multilogin app.
• Add proxies. Click on Create new under Browser profile.
• In the following window enter Browser profile name (for example Oxylabs). Next, click Edit proxy settings.
• Choose HTTP proxy as Connection type. Now fill in the required details: under New address and Port insert pr.oxylabs.io and 7777. Under Login enter your username, and Password. Do not forget to add customer- at the beginning of the username. If you have whitelisted your devices, you do not need to provide the login credentials.
• You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under New address and 10001 under Port, you will receive US exit node with a sticky session.
• Click Check proxy. If everything is fine, you should get a response indicating External IP.
• Confirm changes by clicking Create profile.
• This is it, you are good to go.

Postern

Postern is a very popular proxy/VPN management application on Android. You can get it on Google Play Store. Here is a step-by-step guide how to configure our Residential Proxies on Postern:

• Open the app.
• Click Add Proxy.

• In the following window enter required proxy information. Server Name is Oxylabs, Server Address is pr.oxylabs.io. Under Server Port enter 7777. Click Server Type and choose HTTPS/HTTP CONNECT. Next, enter your login credentials under Username and Password. For this to work, you must type customer- before you type in your username, e.g., customer-username.Finally, click Save.

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Server Address and 10001 under Server Port, you will receive the US exit node with sticky session.

• Now, open the app menu and go to Rules.

• Click Add Rule

• Under Match Method select Match All. Choose Proxy/Tunnel as the Rule. Proxy/Proxy Group should automatically fill up with Oxylabs – pr.oxylabs.io:7777″. Click Save.

• Now you need to turn on the VPN. Open the app menu and click on VPN Off to activate the connection. That is it, you are now using proxies.

GoLogin

GoLogin is a multi-login browser management application for Windows, Linux and Mac OS. Get it on its official website here. Here is a step-by-step guide how to configure our Residential Proxies on GoLogin for Windows:

• Open the app.
• Click Create Your First Profile.

• In the following window enter the required proxy information. Under Profile Name write Oxylabs. Next, choose Connection Type as HTTP Proxy. Under Proxy Host and Port enter pr.oxylabs.io and 7777. Next, enter your login credentials under Username and Password. Finally, click Create Profile.

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io and 10001 under Proxy Host and Port, you will receive the US exit node with a sticky session. To fully utilize all available Residential Proxies features such as sessions and country, state, city geolocation parameters, please reffer to the corresponding sections in the documentation.

• All set, now simply click Run and wait for the browser to load.

• • If everything is fine, a website http://myip.gologin.app will open and show the IP information.

Kameleo

Kameleo is a multi-login browser management application for Windows and Android. Get it on its official website here. Here is a step-by-step guide how to configure our Residential Proxies on Kameleo for Windows:

• Open the app.
• Click APPLICATION SETTINGS.

• In the following window enter the required proxy information under Global proxy settings. For Upstream Proxy Type choose Http. Next, fill in Proxy IP or hostname with pr.oxylabs.io. Under Proxy Port enter 7777. Finally, enter your login credentials under Username and Password. Close the window to save settings.

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Proxy IP or hostname and 10001 under Proxy Port, you will receive the US exit node with a sticky session.

• That is it, you are ready to go. Either create a NEW PROFILE or click QUICK PROFILE to launch the browser which now has proxy enabled.

ClonBrowser

ClonBrowser is another multi-login browser management application for Windows and Mac. Get it on its official website here. Here is a step-by-step guide how to configure our Residential Proxies on ClonBrowser for Windows:

• Open the app.
• Click Create a New browser profile.

• Click Setting next to Proxy settings.

• In the following window choose Http as the Connection type. Under IP or host enter pr.oxylabs.io. Fill in 7777 for Port. Finally, enter your login credentials under Username in customer-username format and Password.

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Proxy IP or hostname and 10001 under Proxy Port, you will receive the US exit node with a sticky session.

You can check if the proxy works by clicking Connection test. If everything is fine, click Create profile.

• That is all, you can now launch the browser by clicking Start under Option.

AdsPower

AdsPower is a multi-login browser management application for Windows and Mac. You get it on its official website here. Here is the step-by-step guide how to configure our Residential Proxies on AdsPower for Windows:

• Open the app.
• Click Upload..

• Under Proxy Type choose http. Now fill in the form bellow. Enter pr.oxylabs.io for Proxy Host and 7777 for Proxy Port. Enter your login credentials customer-username under Proxy User and your password for Password.

You can also use country specific entries. For example, if you put us-pr.oxylabs.io under Proxy Host and 10001 under *Proxy Port, you will receive US exit node with sticky session.

You can check if the proxy works by clicking Check Proxy. If everything is fine, click Create profile.

• That's all, you can now launch the browser by clicking Open under Account Management.

VMLogin

Virtual Multi Login, or VMLogin, is a tool for creating virtual browser profiles that help control browser fingerprinting, setup business workflows, and develop web automation, among other things. To use VMLogin with Oxylabs Residential Proxies, download the latest version of VMLogin here.

• Launch VMLogin and create a new browser profile. You can either click Get random profile (1), or select your settings.
• Click on Setting proxy server (2)

• Create a new Residential Proxy user.

• Click on Setting proxy server and choose HTTP Proxy as the Connection type. Under IP or host enter pr.oxylabs.io. Fill in 7777 for Port. Finally, enter your login credentials under Username (in customer-username format) and Password. You can check if the proxy works by clicking Test Proxy. If everything is fine, save profile and launch the browser.

• You can also use country specific entries. For example, if you put us-pr.oxylabs.io under Proxy IP address and 10001 under Proxy Port, you will receive a US exit node with a sticky session.

• This is all, you are ready to use Residential Proxies with VMLogin.

Undetectable

Firstly, you will need to get Undetectable. To do that, simply go to undetectable.io, sign up, and download the browser.

Then, open the application and click “+” to create a new profile.

In the following window, under New Profile, enter the new profile name. Choose the settings (OS, browser, configurations, and screen) of how you want your browser to be recognized by websites.

Also, you can select the main and additional languages, while geo-location will be determined by your proxy.

Go to the Proxy tab and select the “New proxy” option in the box.

You can choose either SOCKS5 or HTTP (select HTTP proxy as a connection type). Now fill in proxy and authentication details: type pr.oxylabs.io for your Host and use 7777 for Port. Enter your credentials in the Login and Password spaces. Do not forget to add customer- at the beginning of your username. If you have whitelisted your devices, you do not need to provide login credentials.

You can also use country-specific entries. For example, if you put us- pr.oxylabs.io under Host and 10001 under Port, you will receive US exit node with a sticky session. Please check out our complete list of country-secific entry nodes.

Click the Check button. If everything is fine, you should get a response indicating External IP. Press the Save Proxy button to confirm the changes.

Now save the profile by pressing the Save button, and you will see it appear on the left side menu. Edit your profiles easily by simply clicking on them.

That's it! You have successfully set up your proxies with Undetectable.io.

Puppeteer

Before getting started with Puppeteer, you’ll need to install some basic tools: Node.js and a code editor of your choice. After that, create a Node.js project and install the required packages. You may find a detailed guide on how to install and run Puppeteer on our blog post.

Once everything is set up, we can move on to the next part – Oxylabs Residential Proxies integration with Puppeteer.

• Within Puppeteer, fill in the value, for example:

pr.oxylabs.io:7777

• Under ‘page.authenticate’ input your Oxylabs proxy sub-user username in the ‘username’ value and your password.
• The example of a code looks like this:

const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch({
    headless: false,
    args: ['--proxy-server=pr.oxylabs.io:7777]  
});
  const page = await browser.newPage();
    await page.authenticate({
        username: 'USERNAME',
        password: 'PASSWORD'
    });
    await page.goto('https://ip.oxylabs.io');
    await page.screenshot({path: 'example.png'});
    await browser.close();
})();

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under ‘host’ and 10001 under ‘port’, you will receive a US exit node with a sticky session. Please check out our documentation for a complete list of country-specific entry nodes.

And that’s it! You’ve successfully integrated Oxylabs Residential proxies with Puppeteer.

References,

https://docs.oxylabs.io/residential/index.html

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Want to use OxyLabs' datacenter proxies? Then you can't miss this article. It is the most comprehensive OxyLabs datacenter proxies user guide.

Quick Start

Before we begin, please make sure you have received a username, password, and all URLs linking to your IP list from a Sales Representative, Dedicated Account Manager, or Support Team.

There are two ways you can get to your IP list. You can either access the IP list directly from your browser by opening the URL you received (e.g.: https://proxy.oxylabs.io/all). You will then be asked to enter your login credentials (HTTP Basic Auth).

Or you can retrieve the list by making a GET request to the URL you received (for example: https://proxy.oxylabs.io/all). Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl -u "USERNAME:PASSWORD" https://proxy.oxylabs.io/all

curl -x 1.2.3.4:60000 -U "USERNAME:PASSWORD" http://ip.oxylabs.io

Proxy Lists

Basic

You should receive a URL to your proxy list from your Dedicated Account Manager or our Support and it should look something like this:

https://proxy.oxylabs.io/key/a27c5343ce4c4637a85a9297ddf1cace

Simply open this URL in your browser (or any other prefered software). You will then be asked to enter your login credentials (HTTP Basic Auth):

Once you do that, the list of proxies will be shown in a plain text.

If you have multiple lists and wish to get all IPs assigned to your account, you can use this URL:

https://proxy.oxylabs.io/all

You can also include ?showCountry or ?showCity at the end of your URL to get IPs with their corresponding locations.

RESTful

Showing All Available Proxy Lists

With this endpoint, you can retrieve proxy lists that belong to your user:

GET https://api.oxylabs.io/v1/proxies/lists

curl -U username:password "https://api.oxylabs.io/v1/proxies/lists"

Sample output:

[
{
"uuid": "123e4567-e89b-12d3-a456-426655440000",
"key": "12345678",
"updated_timestamp": "2016-09-28T13:03:50+0300",
"ips_count": 2000,
"href": "http://api.oxylabs.io/v1/proxies/lists/123e4567-e89b-12d3-a456-426655440000"
}
]

Retrieving a Specific Proxy List

With this endpoint, you can retrieve a specific proxy list:

GET https://api.oxylabs.io/v1/proxies/lists/{uuid}

Request parameters:

curl -u username:password "https://api.oxylabs.io/v1/proxies/lists/{uuid}"

Sample output:

[
{
"ip": "127.0.0.1",
"port": "60000",
"country": "US"
},
{ 
"ip": "127.0.0.2",
"port": "60000",
"country": "GB"
}
]

Making Requests

By default, Oxylabs proxies and Proxy Rotator use a basic HTTP authentication that requires you to provide a username and password. You can get your credentials by contacting your Dedicated Account Manager or our Support Team at support@oxylabs.io.

We also support authentication based on whitelisted IP addresses. See Whitelisting IPs.

Depending on your authentication method, you might need to change the proxy port:

Proxy

If you want to use Datacenter Proxies via user/pass authentification method:

curl -x 1.2.3.4:60000 -U user1:pass1 http://ip.oxylabs.io

If you want to use Datacenter Proxies via Whitelisted IP authentification method:

curl -x 1.2.3.4:65432 http://ip.oxylabs.io

Whitelisting IPs

Dashboard

We also support authentication based on an IP address. If you are using proxies directly (without proxy Proxy Rotator), you can whitelist your IPs by yourself. Just log into https://stats.oxylabs.io and on the sidebar at the bottom you will see a section named “Add Whitelist”. After clicking on the link you will be navigated to self-whitelist page which looks like this:

Here you can manage your whitelisted IPs that you can use to access proxies. You can whitelist individual IPs, as well as up to 24 subnets. If you would like to whitelist a larger subnet, please contact your Dedicated Account Manager or Support Team at support@oxylabs.io.

If you are using proxy Proxy Rotator or SOCKS proxies, you cannot whitelist your IP address by yourself. Instead, you will have to give us a list of IP address you want to use to connect to proxies. We will whitelist them (add them to firewall exception rules). After doing so, you will be able to access your proxies without login credentials.

Bear in mind, that if you are using whitelisted IPs to access your proxies, you will need to use port 65432. For proxy Proxy Rotator, the port remains 60000.

RESTful

You can also whitelist IPs by sending queries directly to our endpoints. Please note there is a 5-minute cooldown for confirming any changes to your whitelisted IP list.

Getting Whitelisted IPs List

GET https://api.oxylabs.io/v1/whitelisted_ips

curl -U username:password "https://api.oxylabs.io/v1/whitelisted_ips"

Sample output:

{
"data": [
{
"id": 9765,
"address": "127.0.0.1"
},
{
"id": 9766,
"address": "127.0.0.2"
}
]
}

Adding a Whitelisted IP

POST https://api.oxylabs.io/v1/whitelisted_ips

curl -X POST -u username:password "https://api.oxylabs.io/v1/whitelisted_ips"
--header "Content-Type: application/json" --data '{"address": "127.0.0.1"}'

{
"error": {
"message": "This is not a valid IP address.",
"code": 422
}
}

Sample output:

{
"id": 9767,
"address": "127.0.0.1"
}

Removing a Whitelisted IP

DELETE https://api.oxylabs.io/v1/whitelisted_ips/{id}

curl -X DELETE -u username:password "https://api.oxylabs.io/v1/whitelisted_ips/{id}"

Saving Changes (5min Cooldown)

POST https://api.oxylabs.io/v1/whitelisted_ips/upload_to_servers

curl -X POST -u username:password "https://api.oxylabs.io/v1/whitelisted_ips/upload_to_servers"

Datacenter Proxy API

Replacing subnet IPs

POST https://api.oxylabs.io/v1/proxy-list/rest/proxy-lists/{PROXY_LIST_UUID}/by-uuid/network-modification-requests/replace

For you to replace IPs from a specific subnet in your existing proxy list while using Proxy Management API, you will need to make a POST request to the endpoint above with specific subnets you want to replace. To make it work correctly, you must insert a proxy list UUID that can be queried from the API (as described here).

curl -X POST -H "Content-Type: application/json" -d '{"subnets": ["x.x.x.0/24","y.y.y.0/24"]}' https://api.oxylabs.io/v1/proxy-list/rest/proxy-lists/{PROXY_LIST_UUID}/by-uuid/subnet-replacement-requests -u user:pass

Sample output:

{
    "id": "1", 
    "status": "processing", 
    "message": None, 
    "type": "replace", 
    "created_at": "2021-01-01T00:00:00+00:00", 
    "link_to_list_plain": "https://proxy.oxylabs.io/key/4f34fe5862603138c1bc5ce28b181aef", 
    "link_to_list_json": "https://api.oxylabs.io/v1/proxies/lists/8f11274c-a900-11eb-b910-901b0ec4424b"
}

Checking replacement status

GET https://api.oxylabs.io/v1/proxy-list/rest/subnet-replacement-requests/{INITIATED_REPLACEMENT_PROCESS_ID}

To confirm that the subnet replacement was a success, you must make a GET request to the endpoint above while inserting Initiated replacement process ID from the replacement process initiated before.

curl -u user:pass https://api.oxylabs.io/v1/proxy-list/rest/subnet-replacement-requests/{INITIATED_REPLACEMENT_PROCESS_ID}"

Sample output:

{
    "created_at": "2021-01-01T00:00:00+00:00",
    "id": "1",
    "link_to_list_json": "https://api.oxylabs.io/v1/proxies/lists/8f11274c-a900-11eb-b910-901b0ec4424b",
    "link_to_list_plain": "https://proxy.oxylabs.io/key/4f34fe5862603138c1bc5ce28b181aef",
    "message": None,
    "status": "completed",
    "type": "replace"
}

Listing replaced and new IPs

GET https://api.oxylabs.io/v1/proxy-list/rest/network-modification-requests/{INITIATED_REPLACEMENT_PROCESS_ID}/ips

When the replacement request has been processed, removed IPs can be found by making a GET request to the endpoint above while inserting initiated replacement process ID from the replacement process initiated before.

Only 100 replaced IPs will be displayed with the initial request, and pagination metadata will be in the response. In order to fetch the remaining list of replaced IPs, pagination must be used. For pagination, please format the endpoint below:

GET https://api.oxylabs.io/v1/proxy-list/rest/network-modification-requests/{INITIATED_REPLACEMENT_PROCESS_ID}/ips?fields=metadata.total&after={metadata_after}

curl -u user:pass "https://api.oxylabs.io/v1/proxy-list/rest/network-modification-requests/{INITIATED_REPLACEMENT_PROCESS_ID}/ips"

Sample output:

{
  "items": [
    {
      "id": "1",
      "ip": "127.1.0.1",
      "subnet": "127.1.0.0/24",
      "type": "redundant"
    },
    {
      "id": "2",
      "ip": "127.1.1.2",
      "subnet": "127.1.1.0/24",
      "type": "new"
    }
  ],
  "metadata": {
    "after": "'100'",
    "before": "'None'",
    "total": 100
  }
}

Proxy Rotator – Optional

This service is optional and is not included in subscription. Instead of connecting to individual IPs, we provide you with a single endpoint to your assigned proxy list. With every request, the endpoint fetches a different IP. To learn how to use the endpoint as a proxy, please see the example on the right.

Note: Proxy Rotator should be used only with port 60000.

curl --proxy vm.oxylabs.io:60000 http://ip.oxylabs.io

Session Control Using Proxy Rotator

It is possible to keep the same IP address with the Proxy Rotator. Firstly, you will need to know how many proxies your Proxy Rotator has. Then simply add --proxy-header "Proxy-Server: sXXX" header to your request, where sXXX is the number of the proxy, for example, s5 or s2541.

user1@machine:~$ curl -x vm.oxylabs.io:60000 -U user:pass http://ip.oxylabs.io --proxy-header "Proxy-Server: s10"
1.2.30.40
user1@machine:~$ curl -x vm.oxylabs.io:60000 -U user:pass http://ip.oxylabs.io --proxy-header "Proxy-Server: s10"
1.2.30.40
user1@machine:~$ curl -x vm.oxylabs.io:60000 -U user:pass http://ip.oxylabs.io --proxy-header "Proxy-Server: s10"
1.2.30.40

Response Codes

These are the most common response codes that you can get while using our Datacentre Proxies. If you receive any other response code, please get in touch with your Dedicated Account Manager for more information.

3rd Party Integrations

Oxylabs Proxy Manager

Oxylabs Proxy Manager is a proxy management extension for Chrome. It is not only free but also works with any proxy provider of your choice. Enable your proxy without having to jump through settings and menu options. Download the extension from Chrome Web Store here

• Click the extension icon to open the application
• If you haven't added any proxies yet, you will see Add new proxy button. Click it to continue.

• Add proxy profile. Provide a name for the profile under Enter name. Enter on of the proxies you have purchased from us as Proxy server IP and 60000 as Port if you are using username:password authentication method, and 65432 if you are using whitelisted IPs.

Fill in your Username and Password. Finally, click Save changes. If you have whitelisted your IP, you don't need to do this step at all.

• Open the extension and click Connect. That's it, you are now using datacenter proxies.

SwitchyOmega

SwitchyOmega is a powerful and reliable proxy manager that works with many popular browsers. From our own experience, it is one of the best proxy plugins out there. We highly recommend using SwitchyOmega with our proxies. To do that, you will need to get the plugin. For Chrome, click here, for Firefox click here. Once you have it installed, you should:

• Open Switchy Omega options.

• Click New profile…

• Give the new profile a name, choose Proxy Profile type and click Create

• Change Protocol to HTTP. Under Server enter one of the proxies you have purchased from us. Under Port add 60000 if you are using username:password authentication method, and 65432 if you are using whitelisted IPs.

• Next, click the lock button on the right. Fill in your Username and Password. Click Save changes.

If you have whitelisted your IP, you don't need to do this step at all.

• Click Apply changes

• For the final step, click on SwitchyOmega icon and choose your newly configured proxy. You are now ready to go!

FoxyProxy

FoxyProxy is a simple proxy manager that works with many popular browsers. First, you will need to get FoxyProxy. For Chrome, click here, for Firefox click here. Once you have it installed, you should:

• Open FoxyProxy Options.
• Click Add.
• In IP address, DNS name, server name fields enter one of the proxies.
• In Port field enter port. If you use user/pass authentication port is 60000. If you have whitelisted your IP the port is 65432.
• In Username (optional) field you should enter your username. If you have whitelisted your IP, leave this field blank as the plugin will authorize through it and ignore everything you write in this field.
• In Password (Optional) field you should enter your password. If you have whitelisted your IP and wish to authorise through it, leave this field blank.
• After filling in all fields press Save.
• In order to get all of your requests running through this setup, you must press on the FoxyProxy plugin icon (1) and click on Use proxy Proxy for all URLs (ignore patterns) (2).

(1)

(2)

• That is it, you are done!

Proxifier

To use Datacenter Proxies with Proxfier, simply follow these steps:

• Open Proxifier app.
• Add proxies. Click on Proxies (1). Click Add(2).

• In the following window, enter the required information. IP Address(1), Port(2) which for the username/pass authentication is 60000 and for whitelisted IP authentication – 65432 , select HTTPS(3) protocol. In the authentication box click Enable(4) and enter provided Username(5) and Password(6).

Do not forget to add customer- at the beginning of the username. If you have whitelisted your devices, do not click on Enable, your authenticiation will be done without username/password. If you wish, you can enable Appear as Web Browser option. Click OK.

• You will be asked Do you want Proxifier to use this proxy by default? This means that this setup will be used for all of your browsers. It is up to you to decide whether you want that or not. If you select Yes, you are done setting up. If you wish to use proxies just on a particular browser, follow these next steps.
• Click on Rules(1) and then click Add…(2).

• Click +(1) button to add applications that you want to use the proxy settings with. Select a browser that you want to use these proxy settings with and click on it. Finally, you have to change Action(2). Click on it and select your created proxy.

• That is it. Now all of your requests (while using a browser) will be made via proxies.

References

https://docs.oxylabs.io/dc/index.html

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Did you heard of the Next-Gen Residential Proxies by Oxylabs? This article will show you how to use it in detail.

Quick Start

Next-Gen Residential Proxies are built for heavy-duty data retrieval operations. They enable effortless web data extraction without any delays or errors. The product is as customizable as a regular proxy, but at the same time it guarantees a much higher success rate. Custom headers and IP stickiness are both supported, alongside reusable cookies and POST requests.

If you have ever used regular proxies for data scraping, integrating Next-Gen Residential Proxies will be a breeze. The only difference is that we require to accept our certificate, or ignore it altogether with -k or --insecure cURL flags (or an equivalent expression in the language of your choice).

To make a request using Next-Gen Residential Proxies, we need to use ngrp.oxylabs.io:60000 endpoint. Below is an example in cURL. You can find code samples in other languages here or full code examples on our GitHub.

curl -k -x ngrp.oxylabs.io:60000 -U "USERNAME:PASSWORD" http://ip.oxylabs.io

If you have any questions not covered by this documentation, please contact your account manager or our support staff at support@oxylabs.io.

Making requests

GET ngrp.oxylabs.io:60000

The easiest way to start is to send us a simple query without any custom options. We will add all standard headers on our end, pick the fastest proxy and deliver you the response body.

To utilize desired functionalities of Next-Gen Residential Proxies such as setting up proxy geolocation or reusing the same IP via session control, we need to send additional headers with the request.

Here's the full list of supported functionalities and headers:

Query parameters

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io"

Session

If you want to use the same proxy to make multiple requests, you can do that by adding the X-Oxylabs-Session-Id header with a randomly-generated string for the session ID. We will assign a proxy to this ID and keep it for up to 10 minutes. After that a new proxy will be assigned to that particular session ID.

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "X-Oxylabs-Session-Id: 123randomString"

Geo-Location

Some websites will not serve content if accessed from unsupported geo-locations. You can specify in your request which country you want to access the target from. Just add the X-Oxylabs-Geo-Location header with the value set to a country name, for example, Germany for Germany or United States for the United States.

Next-Gen Residential Proxies also support city-level targeting. If you want to add a city, simply add the city name before the country, e.g,: Munich, Germany.

The full list of supported geo-location parameter values can be found here.

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "X-Oxylabs-Geo-Location: Munich,Germany"

Headers

If you know more than we do about a particular target, you can add your own headers to your request. It can be both standard headers, such as User-Agent or Accept-Language, and something completely custom and target-specific.

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "Your-Custom-Header: interesting header content" -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36" -H "Accept-Language: en-US"

Cookies

The system also allows setting custom cookies to target website. With your initial request we will return all response headers and cookies. You can modify them on your end and send them back to our system with the next request. It may be a good idea to reuse the cookies if you use the same IP address to make a few consecutive requests (see Session for more)

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "Cookie: NID=1234567890; 1P_JAR=0987654321"

Custom Status Code

By default we assume that the request is successful as long as it returns a 2xx or a 4xx status code. However, sometimes websites return the required content together with a non-standard HTTPS status code. If one of your targets does that, you can indicate which status codes are acceptable and actually valuable for you. Simply add X-Oxylabs-Status-Code header with all HTTP response codes that actually work for you. Please note that 2xx and 4xx will still be automatically marked as successful.

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "X-Oxylabs-Status-Code: 500,501,502,503" 

POST Requests

Next-Gen Residential Proxies don't only support sending GET requests, but also let you POST to a web endpoint of your choice. This means that you can send data to a target website, which can then make the website return a different result.

curl -X POST -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -d "@/path/to/file.json"

JavaScript rendering

Enable JavaScript rendering. Use when the target requires JavaScript to load content. There are two available values for this parameter: html (get raw output) and png (get a Base64-encoded screenshot).

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "X-Oxylabs-Render: html"

Adaptive parsing

Adaptive parsing is capable of parsing the most important fields from any e-commerce product page. To enable adaptive parsing for e-commerce product pages it is required to send two additional headers: X-Oxylabs-Parser-Type: ecommerce_product to select the parser type and X-Oxylabs-Parse: 1 to parse the retrieved page.

Fields that can be parsed by Next-Gen Residential Proxies Adaptive Parsing for e-commerce product pages:

• Title
• Regular price
• Offer price
• Currency
• Product description
• Product IDs
• Image URLs
• URLs from the page

curl -v -k -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "X-Oxylabs-Parser-Type: ecommerce_product" -H "X-Oxylabs-Parse: 1"

Sample output:

{
    "results": [
        {
            "content": {
                "url": "https://ip.oxylabs.io/product/product_example.html",
                "body": {
                    "price": 11.99,
                    "title": "Example product title",
                    "currency": "$",
                    "old_price": 15.99,
                    "description": null,
                    "image_links": [
                        "https://ip.oxylabs.io/img/product_example.png"
                    ],
                    "ids_from_url": [],
                    "simple_links": [
                        {
                            "link": "https://ip.oxylabs.io/product/another_example.html",
                            "description": "Another product example"
                        }
                    ],
                    "ids_from_html": [
                        {
                            "Product number": "14158288"
                        }
                    ],
                    "price_range_lower": null,
                    "price_range_upper": null
                },
                "meta": {
                    "title": "Example product meta title",
                    "keywords": [],
                    "description": "Example product description"
                },
                "parse_status_code": 12000
            },
            "created_at": "2020-01-01 10:00:00",
            "updated_at": "2020-01-01 10:00:07",
            "id": 29964797,
            "page": 1,
            "url": "https://ip.oxylabs.io/product/product_example.html",
            "job_id": "6699272813062145025",
            "status_code": 200
        }
    ]
}

Usage Statistics

You can find your usage statistics by visiting our Dashboard or by querying the following endpoint:

GET https://data.oxylabs.io/v1/stats

By default the API will return all time usage statistics. Adding ?group_by=month will return monthly stats, while ?group_by=day will return daily numbers. Traffic is being displayed in bytes.

his query will return all time statistics. You can find your daily and monthly usage by adding either ?group_by=day or ?group_by=month. Traffic is being displayed in bytes.

curl --user user:pass1 'https://data.oxylabs.io/v1/stats'

Sample output:

{
    "meta": {
        "group_by": null,
        "date_from": null,
        "date_to": null,
        "source": null
    },
    "data": {
        "sources": [
            {
                "results_count_all": "1482",
                "results_count": "0",
                "realtime_results_count": "0",
                "super_api_results_count": "12777",
                "render": "0",
                "geo_location": "0",
                "average_response_time": 2.18,
                "request_traffic": "6629", 
                "response_traffic": "17850",
                "title": "universal"
            }
        ]
    }
}

Sample Response

This is what the response looks like. To see headers and cookies, Verbose mode must be enabled.

HTTP/1.1 200 OK
X-Job-Id: 1234567890123456
X-Session-Id: 123randomString
cf-cache-status: DYNAMIC
cf-ray: 55c2ab837eddcba8-VIE
content-encoding: gzip
content-length: 72657
content-type: text/html
date: Tue, 1 Jan 2020 00:00:01 GMT,expect-ct: max-age=604800,report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
last-modified: Tue, 1 Dec 2020 00:00:00 GMT
server: cloudflare
status: 200
strict-transport-security: max-age=31536000
X-DNS-Prefetch-Control: off
Set-Cookie: NID=1234567890; expires=Wed, 29-Jul-2020 10:56:21 GMT
Set-Cookie: 1P_JAR=0987654321; expires=Wed, 29-Jul-2020 10:56:21 GMT
x-cache: MISS from localhost
x-cache-lookup: MISS from localhost:3129content here

References

https://docs.oxylabs.io/next-gen-residential-proxies/index.html#quick-start

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Do you know how to use OxyLabs Real-time Crawler for Google? This is the most comprehensive introduction from OxyLabs official.

Quick Start

Real-Time Crawler is built for heavy-duty data retrieval operations. You can use Real-Time Crawler to access various Google pages, including regular search, hotel availability and Google Shopping. It enables effortless web data extraction from search engines without any delays or errors.

Real-Time Crawler for Google uses basic HTTP authentication that requires sending username and password.

This is by far the fastest way to start using Real-Time Crawler for Google. You will send a query adidas to google_search using Realtime integration method. Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "google_search", "domain": "com", "query": "adidas"}'

If you have any questions not covered by this documentation, please contact your account manager or our support staff at support@oxylabs.io.

Postman

Download and import this Postman collection to try out all of the Google crawler functions and data delivery methods documented on this page.

Integration Methods

Real-Time Crawler for Google supports three integration methods which have their unique benefits:

• Push-Pull. Using this method it is now required to mainain an active connection with our endpoint to retrieve the data. Upon making a request, our system is able to automatically ping users server when the job is done (see Callback). This method saves computing resources and can be scaled easily.
• Realtime. The method requires user to maintain an active connection with our endpoint in order to get the results successfully when the job is completed. This method can be implemented into one service while Push-Pull method is a two step process.
• SuperAPI. This method is very similar to Realtime but instead posting data to our endpoint, user can use HTML Cralwer as a proxy. To retrieve the data, user must set up a proxy endpoint and make GET request to a desired URL. Additional parameters must be added using headers.

Our recommended data extraction method is Push-Pull.

Push-Pull

This is the most simple yet the most reliable and recommended data delivery method. In Push-Pull scenario you send us a query, we return you job id, and once the job is done you can use that id to retrieve content from /results endpoint. You can check job completion status yourself, or you can set up a simple listener that is able to accept POST queries. This way we will send you a callback message once the job is ready to be retrieved. In this particular example the results will be automatically uploaded to your S3 bucket named YOUR_BUCKET_NAME.

You can also try and see how Push-Pull method works via Postman. Download this file to get started.

Single Query

The following endpoint will handle single queries for one keyword or URL. The API will return a confirmation message containing job information, including job id. You can check job completion status using that id, or you can ask us to ping your callback endpoint once the scraping task is finished by adding callback_url in the query.

POST https://data.oxylabs.io/v1/queries

You need to post query parameters as data in JSON body.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "google_search", "domain": "com", "query": "adidas", "callback_url": "https://your.callback.url", "storage_type": "s3", "storage_url": "YOUR_BUCKET_NAME"}'

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "callback_url": "https://your.callback.url",
  "client_id": 5,
  "context": [
    {
      "key": "results_language",
      "value": null
    },
    {
      "key": "safe_search",
      "value": null
    },
    {
      "key": "tbm",
      "value": null
    },
    {
      "key": "cr",
      "value": null
    },
    {
      "key": "filter",
      "value": null
    }
  ],
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "query": "adidas",
  "source": "google_search",
  "start_page": 1,
  "status": "pending",
  "storage_type": "s3",
  "storage_url": "YOUR_BUCKET_NAME/12345678900987654321.json",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:01",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Check Job Status

If your query had callback_url, we will send you a message containing link to content once the scraping task is done. However, if there was no callback_url in the query, you will need to check job status yourself. For that you need to use the URL in href under rel:self in the response message you received after submitting your query to our API. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321.

GET https://data.oxylabs.io/v1/queries/{id}

Querying this link will return job information, including its status. There are 3 possible status values:

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321'

The API will respond with query information in JSON format, by printing it in response body. Notice that job status has been changed to done. You can now retrieve content by querying http://data.oxylabs.io/v1/queries/12345678900987654321/results.

You can also see that the task has been updated_at 2019-10-01 00:00:15 – the query took 14 seconds to complete.

{
  "client_id": 5,
  "context": [
    {
      "key": "results_language",
      "value": null
    },
    {
      "key": "safe_search",
      "value": null
    },
    {
      "key": "tbm",
      "value": null
    },
    {
      "key": "cr",
      "value": null
    },
    {
      "key": "filter",
      "value": null
    }
  ],
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "query": "adidas",
  "source": "google_search",
  "start_page": 1,
  "status": "done",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:15",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Retrieve Job Content

Once you know the job is ready to retrieved either by checking its status or receiving a callback from us, you can GET it using the URL in href under rel:results in either our initial response or in callback message. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321/results.

GET https://data.oxylabs.io/v1/queries/{id}/results

The results can be automatically retrieved without periodically checking job status by setting up Callback service. User needs to specfy the IP or domain of the server where the Callback service is running. When our system completes a job, it will send a message to the provided IP or domain and the Callback service will download the results as described in the Callback implementation example.

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321/results'

The API will return job content:

{
  "results": [
    {
      "content": "<!doctype html>
        CONTENT      
      ",
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "page": 1,
      "url": "https://www.google.com/search?q=adidas&hl=en&gl=US",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

Callback

A callback is a POST request we send to your machine, informing that the data extraction task is completed and providing URL to download scraped content. This means that you no longer need to check job status manually. Once the data is here, we will let you know, and all you need to do now is retrieve it.

# Please see code samples in Python and PHP.

Sample callback output

{  
   "created_at":"2019-10-01 00:00:01",
   "updated_at":"2019-10-01 00:00:15",
   "locale":null,
   "client_id":163,
   "user_agent_type":"desktop",
   "source":"google_search",
   "pages":1,
   "subdomain":"www",
   "status":"done",
   "start_page":1,
   "parse":0,
   "render":null,
   "priority":0,
   "ttl":0,
   "origin":"api",
   "persist":true,
   "id":"12345678900987654321",
   "callback_url":"http://your.callback.url/",
   "query":"adidas",
   "domain":"com",
   "limit":10,
   "geo_location":null,
   {...}
   "_links":[
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321",
         "method":"GET",
         "rel":"self"
      },
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321/results",
         "method":"GET",
         "rel":"results"
      }
   ],
}

Batch Query

Real-Time Crawler also supports executing multiple keywords, up to 1,000 keywords with each batch. The following endpoint will submit multiple keywords to the extraction queue.

POST https://data.oxylabs.io/v1/queries/batch

You need to post query parameters as data in JSON body.

The system will handle every keyword as a separate request. If you provided callback URL, you will get a separate call for each keyword. Otherwise, our initial response will contain job ids for all keywords. For example, if you sent 50 keywords, we will return 50 unique job ids.

Important! query is the only parameter that can have multiple values. All other parameters are the same for that batch query.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries/batch' -H 'Content-Type: application/json'
 -d '@keywords.json'

keywords.json content:

{  
   "query":[  
      "adidas",
      "nike",
      "reebok"
   ],
   "source": "google_search",
   "domain": "com",
   "callback_url": "https://your.callback.url"
}

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "queries": [
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678900987654321",
      {...}
      "query": "adidas",
      "source": "google_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678901234567890",
      {...}
      "query": "nike",
      "source": "google_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678901234567890/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "01234567899876543210",
      {...}
      "query": "reebok",
      "source": "google_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/01234567899876543210/results",
          "method": "GET"
        }
      ]
    }
  ]
}

Get Notifier IP Address List

You may want to whitelist the IPs sending you callback messages or get the list of these IPs for other purposes. This can be done by GETing this endpoint: https://data.oxylabs.io/v1/info/callbacker_ips.

curl --user user:pass1 'https://data.oxylabs.io/v1/info/callbacker_ips'

The API will return the list of IPs making callback requests to your system:

{
    "ips": [
        "x.x.x.x",
        "y.y.y.y"
    ]
}

Upload to Storage

By default RTC job results are stored in our databases. This means that you will need to query our results endpoint and retrieve content yourself. Custom storage feature allows you to store results in your own cloud storage. The advantage of this feature is that you don't have to make extra requests in order to fetch results – everything goes directly to your storage bucket.

We support Amazon S3 and Google Cloud Storage. If you would like to use a different type of storage, please contact your account manager to discuss the feature delivery timeline.

Amazon S3

To get your job results uploaded to your Amazon S3 bucket, please set up access permissions for our service. To do that, go to https://s3.console.aws.amazon.com/ > S3 > Storage > Bucket Name (if don't have one, create new) > Permissions > Bucket Policy

You can find bucket policy in this JSON or in code sample area on the right. Don't forget to change bucket name under YOUR_BUCKET_NAME. This policy allows us to write to your bucket, give access to uploaded files to you, and know bucket location.

Google Cloud Storage

To get your job results uploaded to your Google Cloud Storage bucket, please set up special permissions for our service. To do that, please create a custom role with the storage.objects.create permission and assign it to the Oxylabs service account email oxyserps-storage@oxyserps-storage.iam.gserviceaccount.com.

Usage

To use this feature, please specify two additional parameters in your requests. Learn more here.

The upload path looks like this: YOUR_BUCKET_NAME/job_ID.json. You will find job ID in response body that you receive from us after submitting a request. In this example job ID is 12345678900987654321.

{
    "Version": "2012-10-17",
    "Id": "Policy1577442634787",
    "Statement": [
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
        },
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
        }
    ]
}

Realtime

The data submission is the same as in Push-Pull method, but Realtime case we will return the content on open connection. You send us a query, the connection remains open, we retrieve the content and bring it to you. The endpoint that handles that is this:

POST https://realtime.oxylabs.io/v1/queries

There is a timeout limit of 150 seconds for open connections, therefore in rare cases of heavy load we may not be able to ensure the data gets to you.

You need to post query parameters as data in JSON body. Please see example for more details.

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "google_search", "domain": "com", "query": "adidas"}'

Example response body that will be returned on open connection:

{
  "results": [
    {
      "content": "
      CONTENT
      "
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "id": null,
      "page": 1,
      "url": "https://www.google.com/search?q=adidas&hl=en&gl=US",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

SuperAPI

If you ever used regular proxies for data scraping, integrating SuperAPI delivery method will be a breeze. All that needs to be done is to use our entry node as proxy, authorize with Real-Time Crawler credentials, and ignore certificates. In cURL it's -k or --insecure. Your data will reach you on open connection.

GET realtime.oxylabs.io:60000

SuperAPI only supports a handful of parameters since it only works with a Direct data source where full URL is provided. These parameters should be sent as headers. This is a list of accepted parameters:

If you need help setting up SuperAPI, drop a line at support@oxylabs.io.

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 -H "X-OxySERPs-User-Agent-Type: desktop_chrome" -H "X-OxySERPs-Geo-Location: New York,New York,United States" "https://www.google.com/search?q=adidas"

Content Type

Real-Time Crawler can return either raw HTML, or structured (parsed) JSON. Bear in mind that not all data sources can be returned structured. An icon under each data source in this documentation will indicate whether we are able to parse it, or we can only return raw HTML.

Please see Parsed Data to see which fields we return with each Data Source.

Data Sources

There are multiple approaches how to retrieve data from Google using Real-Time Crawler. You can give us full URL via Direct, or you can specify parameters via specifically built data sources, such as Search, Shopping Product or Images.

Technically not a content type, but Real-Time Crawler is able to render JavaScript when scraping. This is necessary in some Google pages, such as Flights and Patents. A checkmark under Render JS will indicate whether a particular data source can be scraped with JavaScript enabled.

If you are unsure which way to choose, drop us a line at support@oxylabs.io or contact your account manager.

Direct

google source is designed to retrieve content of direct URLs of various Google pages. This means that instead of sending multiple parameters, you can provide us with a direct URL to required Google page. We do not strip any parameters or alter your URLs in any other way.

This data source also supports parsed data (Parsed JSON), as long as the URL submitted is for Google Search (SERP page). If we are unable to confirm this is a SERP page request, a failure message will be returned.

Query parameters

In this example the API will retrieve Google Scholar search for keyword newton in Push-Pull method:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google", "url": "https://scholar.google.com/scholar?hl=en&q=newton&btnG=&as_sdt=1%2C5&as_sdtp="}'

Here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google", "url": "https://scholar.google.com/scholar?hl=en&q=newton&btnG=&as_sdt=1%2C5&as_sdtp="}'

And via SuperAPI:

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 "https://scholar.google.com/scholar?hl=en&q=newton&btnG=&as_sdt=1%2C5&as_sdtp="

Search

google_search source is designed to retrieve Google Search results (SERP).

Query parameters

API makes request to google.nl to retrieve search results pages from number 11 to number 20 for keyword adidas. The results will be displayed in French, since results_language parameter is also passed through via context. The API will post a JSON request to your.callback.url containing the URL to download the raw HTML output once the data retrieval task is successfully finished. This is Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_search", "domain": "nl", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url", "context": [{"key": "results_language", "value": "fr"}]}}'

And here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_search", "domain": "nl", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url", "context": [{"key": "results_language", "value": "fr"}]}}'

Ads

google_ads source is optimized to retrieve Google Search results page (SERP) with paid ads. This source will return only 10 results per page, ensuring highest changes of paid results showing up. Other than that, it supports the same parameters as regular Search

Query parameters

API makes request to google.nl to retrieve search results for keyword adidas. The API will post a JSON request to your.callback.url containing the URL to download the raw HTML output once the data retrieval task is successfully finished. This is Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_ads", "domain": "nl", "query": "adidas", "callback_url": "https://your.callback.url"}'

And here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_ads", "domain": "nl", "query": "adidas"}'

Hotels

google_hotels data source is designed to retrieve Google Hotel search results.

Query parameters

Please note that with Google hotels you always need to send a keyword with ‘hotels' word inside, for example ‘hotels in Los Angeles', ‘hotels in Paris, France', etc. Both ‘hotel' and ‘hotels' work. Google also supports local languages, so you can send ‘Hotelli Helsingissä' for hotels in Helsinki or ‘viešbučiai Vilnius' for hotels in Vilnius.

In this example API will retrieve first 3 pages of hotel availability for 1 guest between 2019-10-01 and 2019-10-10 for hotels in Paris from google.com. This is Push-Pull method.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_hotels", "domain": "com", "pages": 3, "query": "hotels in Paris", "context": [{"key": "hotel_occupancy", "value": 1}, {"key": "hotel_dates", "value": "2019-10-01,2019-10-10"}]}'

This is in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_hotels", "domain": "com", "pages": 3, "query": "hotels in Paris", "context": [{"key": "hotel_occupancy", "value": 1}, {"key": "hotel_dates", "value": "2019-10-01,2019-10-10"}]}'

Travel: Hotels

google_travel_hotels data source is designed to retrieve Google Travel service's hotel search results.

Query parameters

Please note that with Google hotels you always need to send a keyword with ‘hotels' word inside, for example ‘hotels in Los Angeles', ‘hotels in Paris, France', etc. Both ‘hotel' and ‘hotels' work. Google also supports local languages, so you can send ‘Hotelli Helsingissä' for hotels in Helsinki or ‘viešbučiai Vilnius' for hotels in Vilnius.

In this example API will retrieve the 2nd page of results for of hotel availability for 2 guests between 2020-10-01 and 2020-10-10 for hotels in Paris from google.com. The results will be filtered to only show 2 and 4 star hotels. This is Push-Pull method.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_travel_hotels", "domain": "com", "start_page": 2, "query": "hotels in Paris", "callback_url": "https://your.callback.url", "context": [{"key": "hotel_occupancy", "value": 2}, {"key": "hotel_dates", "value": "2020-10-01,2020-10-10"}, {"key": "hotel_classes", "value": [2,4]}]}'

This is in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_travel_hotels", "domain": "com", "start_page": 2, "query": "hotels in Paris", "context": [{"key": "hotel_occupancy", "value": 2}, {"key": "hotel_dates", "value": "2020-10-01,2020-10-10"}, {"key": "hotel_classes", "value": [2,4]}]}'

Shopping Search

google_shopping_search source is designed to retrieve Google Shopping search results.

POST https://data.oxylabs.io/v1/queries

Query parameters

The API will download first 4 pages of Google Shopping search for keyword adidas, sorted by descending price and minimum price of $20. This is how it's done in Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_shopping_search", "domain": "com", "query": "adidas", "pages": 4, "context": [{"key": "sort_by", "value": "pd"}, {"key": "min_price", "value": 20}]}'

Here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_shopping_search", "domain": "com", "query": "adidas", "pages": 4, "context": [{"key": "sort_by", "value": "pd"}, {"key": "min_price", "value": 20}]}'

Shopping Product

google_shopping_product source is designed to retrieve Google Shopping product page for specified product.

Query parameters

Here the API will download product page for product ID 5007040952399054528 from Google Shopping on google.com. It will also get first 4 pages with pricing information. This is how it looks in Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_shopping_product", "domain": "com", "query": "5007040952399054528"}'

The same in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_shopping_product", "domain": "com", "query": "5007040952399054528"}'

Shopping Product Pricing

google_shopping_pricing source is designed to retrieve Google Shopping product pricing page for specified product.

Query parameters

Here the API will download product pricing page for product ID 5007040952399054528 from Google Shopping on google.com. Here is an example in Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_shopping_pricing", "domain": "com", "query": "5007040952399054528"}'

The same in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_shopping_pricing", "domain": "com", "query": "5007040952399054528"}'

Images

google_images source is designed to retrieve Images search page for images that are similar to the one provided with query parameter, as well as websites containing those images.

Query parameters

In this example the API will download Image search page of similar images for image https://newsneakernews-wpengine.netdna-ssl.com/wp-content/uploads/2017/03/adidas-boost-march-25-2017.jpg from google.com. This is Push-Pull method:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_images", "domain": "com", "query": "https://newsneakernews-wpengine.netdna-ssl.com/wp-content/uploads/2017/03/adidas-boost-march-25-2017.jpg"}'

And this is the same request in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_images", "domain": "com", "query": "https://www.example.com/img/image.jpg"}'

Suggestions

google_suggest source is designed to retrieve Google keyword suggestions.

Query parameters

API makes request to Google Suggestions page to retrieve suggestions for keyword adidas. The API will post a JSON payload to your.callback.url containing the URL to download the result once the task is finished. Here is an example with Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_suggest", "query": "adidas", "callback_url": "https://your.callback.url"}'

The same request with Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_suggest", "query": "adidas"}'

Keyword Data

google_msv data source will retrieve Google keyword data for specified keywords, as well as suggested keywords (unless passing ideas=False in context). Keywords are passed in query parameter as a string. Keywords are separated by commas. Commas within a keyword are not supported, so a keyword “Water Bottle 5,0L” will actually be interpreted as 2 keywords: “Water Bottle 5” and “0L”. See output example on the right for more details.

Query parameters

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
-d '{"source": "google_msv", "query": "meilleur restaurant", "geo_location": "Paris,Ile-de-France,France", "context": [{"key": "language", "value": "french"},{"key": "currency", "value": "EUR"}, {"key": "ideas", "value": true}]}'

# OR if you don't want ideas:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
-d '{"source": "google_msv", "query": "meilleur restaurant", "geo_location": "Paris,Ile-de-France,France", "context": [{"key": "language", "value": "french"},{"key": "currency", "value": "EUR"}, {"key": "ideas", "value": false}]}'

Sample output (historicalSearchVolume entries and ideas entries cut):

{
    "results": [
    {
        "content":
        {
            "ideas": [
            {
                "cpc": 4.712038,
                "keyword": "meilleur restaurant a paris",
                "currency": "EUR",
                "competition": 0.3385383889238515,
                "averageSearchVolume": 1900,
                "historicalSearchVolume": [
                {
                    "date": "201803",
                    "searchVolume": 1600
                },
                {
                    "date": "201802",
                    "searchVolume": 1900
                }]
            }],
            "seeds": [
            {
                "cpc": 4.05351,
                "keyword": "meilleur restaurant",
                "currency": "EUR",
                "competition": 0.3385341239238515,
                "averageSearchVolume": 2900,
                "historicalSearchVolume": [
                {
                    "date": "201803",
                    "searchVolume": 3600
                },
                {
                    "date": "201802",
                    "searchVolume": 2900
                }]
            }]
        }
    }]
}

Parsed data

Google Web Search (SERP) page is the only one that is extensively supported in parsed data delivery. Below you can find which particular SERP page fields we parse. Structure data is available with Search (all the time) and Direct (as long as SERP page URL is submitted).

Google Web Search ("source": "google_search") supports CSV output. To access it, please include these parameters in your Google Web Search job {"source": "google_search", "parse": true, "parser_type": "v2"}. The result retrieval URL for a CSV job is structured like this: http://data.oxylabs.io/v1/queries/{job_id}/results/normalized?format=csv.

Search

Organic & Paid

"results": {
  "paid": [
    {
      "pos": 1,
      "url": "https://www.adidas.com/us",
      "desc": "New York · 10 locations nearby",
      "title": "adidas.com | adidas® Official Site | Official adidas® Online Store‎",
      "url_shown": "www.adidas.com/Official/Site",
      "pos_overall": 1
    }
  ],
  "organic": [
    {
      "pos": 1,
      "url": "https://www.adidas.com/us",
      "desc": "Welcome to adidas Shop for adidas shoes, clothing and view new collections for adidas Originals, running, football, training and much more.",
      "title": "adidas Official Website | adidas US",
      "url_shown": "https://www.adidas.com › ...",
      "pos_overall": 2
    },
    {
      "pos": 2,
      "url": "https://www.mena.adidas.com/",
      "desc": "Browse for adidas shoes, clothing and collections, adidas Originals, Running, Football, Training and more on the official adidas website.",
      "title": "adidas Official Website | adidas",
      "url_shown": "https://www.mena.adidas.com",
      "pos_overall": 6
    },
    {
      "pos": 3,
      "url": "https://www.adidas-group.com/",
      "desc": "adidas AG Supervisory Board announces candidates as shareholder ... adidas celebrates its 70th anniversary and the opening of the Arena building. August 9 ...",
      "title": "adidas - Home",
      "url_shown": "https://www.adidas-group.com",
      "pos_overall": 7
    },
    {
      "pos": 4,
      "url": "https://www.nycgo.com/shopping/the-adidas-store",
      "desc": "You don't so much shop in this flagship Adidas store as you experience it. With an interior modeled on a high school stadium, this four-story Midtown outlet—the  ...",
      "title": "The Adidas Store (Midtown) | NYCgo - NYCgo.com",
      "url_shown": "https://www.nycgo.com › shopping › the-adidas-store",
      "pos_overall": 8
    },
    {
      "pos": 5,
      "url": "https://www.yelp.com/search?find_desc=adidas+store&find_loc=Manhattan%2C+NY",
      "desc": "Reviews on Adidas Store in Manhattan, NY - Adidas, Adidas Originals New York SoHo, adidas Sport Performance, Upper 90 Soccer - Manhattan, Nike Soho, ...",
      "title": "Adidas Store Manhattan, NY - Last Updated August 2019 - Yelp",
      "url_shown": "https://www.yelp.com › search › find_desc=adidas+store",
      "pos_overall": 9
    },
    {
      "pos": 6,
      "url": "https://en.wikipedia.org/wiki/Adidas",
      "desc": "Adidas AG is a multinational corporation, founded and headquartered in Herzogenaurach, Germany, that designs and manufactures shoes, clothing and ...",
      "title": "Adidas - Wikipedia",
      "url_shown": "https://en.wikipedia.org › wiki › Adidas",
      "pos_overall": 10
    }
  ]

Product Listing Ads

"pla": [
  {
    "pos": 1,
    "url": "http://www.adidas.com/us/asweego-shoes/F37038.html?cm_mmc=AdieSEM_Feeds-_-GoogleProductAds-_-NA-_-F37038&cm_mmca1=US&cm_mmca2=NA&kpid=F37038&sourceid=543457011",
    "price": "$40.00",
    "title": "adidas Asweego Shoes Black 10.5 - Mens Running Shoes",
    "seller": "adidas",
    "source": ""
  },
  {
    "pos": 2,
    "url": "http://www.adidas.com/us/baseline-shoes/AW4299.html?cm_mmc=AdieSEM_Feeds-_-GoogleProductAds-_-NA-_-AW4299&cm_mmca1=US&cm_mmca2=NA&kpid=AW4299&sourceid=543457011",
    "price": "$50.00",
    "title": "adidas Baseline Shoes White 13K - Originals Shoes",
    "seller": "adidas",
    "source": ""
  },
  ...
  {
    "pos": 29,
    "url": "https://www.zappos.com/product/8466374/color/21766",
    "price": "$79.95",
    "title": "adidas Superstar W Originals Women's Classic Shoes White/Black/White : 9 B - Medium",
    "seller": "Zappos.com",
    "source": ""
  }
]

Top Stories

"top_stories": [
  {
    "url": "https://www.cnet.com/news/spacex-starhopper-prototype-takes-giant-leap-for-elon-musk/",
    "source": "Cnet",
    "headline": "SpaceX Starhopper rocket prototype takes giant leap for Elon Musk",
    "timeframe": "13 hours ago"
  },
  {
    "url": "https://electrek.co/2019/08/27/elon-musk-tesla-china-made-model-3-rumor/",
    "source": "Electrek",
    "headline": "Elon Musk is rumored to unveil first China-made Tesla Model 3 at event this \nweek",
    "timeframe": "16 hours ago"
  },
  {
    "url": "https://www.bloomberg.com/news/articles/2019-08-28/musk-to-join-china-ai-summit-despite-trump-ordering-firms-out",
    "source": "Bloomberg",
    "headline": "Elon Musk and Jack Ma Will Debate AI at China Summit",
    "timeframe": "4 hours ago"
  }
]

Featured Snippet

"featured_snippet": [
  {
    "url": "https://en.wikipedia.org/wiki/Contract_for_difference",
    "desc": "In finance, a contract for difference (CFD) is a contract between two parties, typically described as \"buyer\" and \"seller\", stipulating that the seller will pay to the buyer the difference between the current value of an asset and its value at contract time (if the difference is negative, then the buyer pays instead to ...",
    "title": "Contract for difference - Wikipedia",
    "url_shown": "https://en.wikipedia.org › wiki › Contract_for_difference",
    "pos_overall": 1
  }
]

Knowledge Base

"knowledge": {
  "title": "Adidas",
  "factoids": [
    {
      "title": "Stock price",
      "content": "ADDDF (OTCMKTS) $291.81 +2.74 (+0.95%)Aug 23, 4:00 PM EDT - Disclaimer"
    },
    {
      "title": "Founder",
      "content": "Adolf Dassler"
    },
    {
      "title": "Founded",
      "content": "August 18, 1949, Herzogenaurach, Germany"
    },
    {
      "title": "Headquarters",
      "content": "Herzogenaurach, Germany"
    },
    {
      "title": "Subsidiaries",
      "content": "Reebok, Five Ten Footwear, Runtastic, Ashworth, MORE"
    },
    {
      "title": "Website",
      "content": "https://www.adidas.com/us"
    }
  ],
  "subtitle": "Design company",
  "description": "DescriptionAdidas AG is a multinational corporation, founded and headquartered in Herzogenaurach, Germany, that designs and manufactures shoes, clothing and accessories. It is the largest sportswear manufacturer in Europe, and the second largest in the world, after Nike. Wikipedia"
}

Local Pack

"local_pack": [
  {
    "links": [
      {
        "href": "https://www.adidas.com/us?utm_source=gmb&utm_medium=organic&utm_campaign=US470198_local",
        "title": "Website"
      },
      {
        "href": "#",
        "title": "Directions"
      }
    ],
    "phone": "",
    "title": "adidas Originals Flagship Store",
    "rating": 0,
    "address": "Open ⋅ Closes 7PM",
    "subtitle": "(212) 966-0954",
    "pos_overall": 3,
    "rating_count": 0
  }
]

Twitter Feed

"twitter": [
  {
    "pos": 1,
    "url": "https://twitter.com/elonmusk",
    "title": "Elon Musk (@elonmusk) · Twitter",
    "tweets": [
      {
        "url": "https://twitter.com/elonmusk/status/1166081488648949760?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet",
        "content": "Starhopper flight currently tracking to 5pm Texas time for 150m / ~500ft hover test",
        "timeframe": "11 hours ago"
      },
      {
        "url": "https://twitter.com/elonmusk/status/1165377786338406400?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet",
        "content": "Looks like @SpaceX Starhopper flight may be as soon as Monday. FAA support is much appreciated!",
        "timeframe": "2 days ago"
      },
      {
        "url": "https://twitter.com/elonmusk/status/1165371975528640512?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet",
        "content": "If you’re a utility or public utilities commission, please consider using the Tesla Megapack. Better for the environment & usually lower cost than fossil fuel peaker plants! www.tesla.com/megapack",
        "timeframe": "2 days ago"
      }
    ],
    "pos_overall": 1
  }
]

Job Listings

"jobs": {
  "listings": [
    {
      "title": "SR SOFTWARE DEVELOPER",
      "source": "via LinkedIn",
      "employer": "Jobs @ TheJobNetwork",
      "location": "Tulsa, OK",
      "extra_details": [
        "1 day ago",
        "Full-time"
      ]
    },
    {
      "title": "Autonomous Vehicle Simulation Software Engineer",
      "source": "via Built In Colorado",
      "employer": "Azevtec",
      "location": "United States",
      "extra_details": [
        "17 hours ago",
        "Full-time"
      ]
    },
    {
      "title": "Senior Software Engineer - Oracle Transportation Management",
      "source": "via LinkedIn",
      "employer": "XPO Logistics, Inc.",
      "location": "United States",
      "extra_details": [
        "21 hours ago",
        "Full-time"
      ]
    }
  ],
  "location_header": "Near United States"
}

Carousel

"item_carousel": {
  "items": [
    {
      "title": "Chris Evans",
      "subtitle": "Captain America"
    },
    {
      "title": "Mark Ruffalo",
      "subtitle": "Hulk"
    },
    {
      "title": "Tom Holland",
      "subtitle": "Spider-Man"
    },
    {
      "title": "Stan Lee",
      "subtitle": "Old Man in TV Report, Bus Driver"
    },
    {
      "title": "Chris Pratt",
      "subtitle": "Star-Lord"
    }
  ],
  "title": "The Avengers/Cast"
}

Images

"images": [
  {
    "alt": "Image result for contemporary wall clock",
    "href": "/search?q=contemporary+wall+clock&safe=off&hl=en&gl=US&tbm=isch&source=iu&ictx=1&fir=Qspcw8WiAmXYzM%253A%252C-m-5575uWYilbM%252C_&vet=1&usg=AI4_-kTGLIU9LAzoCJxO8gp7kK322MV8Yg&sa=X&ved=2ahUKEwjFy8rSy7HkAhWkDrkGHck7A24Q9QEwAXoECAkQBg#imgrc=Qspcw8WiAmXYzM:",
    "source": "https://www.allmodern.com/decor-pillows/sb0/wall-clocks-c429917.html"
  },
  {
    "alt": "Image result for contemporary wall clock",
    "href": "/search?q=contemporary+wall+clock&safe=off&hl=en&gl=US&tbm=isch&source=iu&ictx=1&fir=G0pFK8TQ91ls6M%253A%252Cr5nLxZQfxnA3MM%252C_&vet=1&usg=AI4_-kStPZh1tpSdQ5vTAZUIXwW4zThzQg&sa=X&ved=2ahUKEwjFy8rSy7HkAhWkDrkGHck7A24Q9QEwAnoECAkQCQ#imgrc=G0pFK8TQ91ls6M:",
    "source": "https://www.wayfair.com/decor-pillows/cat/modern-wall-clocks-c1869680.html"
  },
  ...
  {
    "alt": "Image result for contemporary wall clock",
    "href": "/search?q=contemporary+wall+clock&safe=off&hl=en&gl=US&tbm=isch&source=iu&ictx=1&fir=o4ZXIngZyr9HAM%253A%252C-m-5575uWYilbM%252C_&vet=1&usg=AI4_-kTIJMWyTs07HFcVKHTfTd6otLL82w&sa=X&ved=2ahUKEwjFy8rSy7HkAhWkDrkGHck7A24Q9QEwCnoECAkQIQ#imgrc=o4ZXIngZyr9HAM:",
    "source": "https://www.allmodern.com/decor-pillows/sb0/wall-clocks-c429917.html"
  }
]

Related Questions

"related_questions": [
  {
    "pos": 1,
    "question": "What does Adidas stand for?"
  },
  {
    "pos": 2,
    "question": "Is Adidas German?"
  },
  {
    "pos": 3,
    "question": "Are Jordans Adidas?"
  },
  {
    "pos": 4,
    "question": "What shoe brands does adidas own?"
  }
]

Shopping Search

...
"organic": [
            {
              "pos": 1,
              "url": "/aclk?sa=l&ai=DChcSEwju8fmd84jpAhUPTxgKHQshDIcYABAHGgJsZQ&sig=AOD64_1BTHVcnNzI5775j9xNkILrCU2KYA&ctype=5&q=&ved=0ahUKEwjpr_Sd84jpAhVI2aYKHYn1CeMQvxMI4wQ&adurl=",
              "type": "grid",
              "price": 85,
              "title": "Adidas Women's Swift Run Casual Shoes in White ...",
              "merchant": {
                "url": "/aclk?sa=l&ai=DChcSEwju8fmd84jpAhUPTxgKHQshDIcYABAHGgJsZQ&sig=AOD64_1BTHVcnNzI5775j9xNkILrCU2KYA&ctype=5&q=&ved=0ahUKEwjpr_Sd84jpAhVI2aYKHYn1CeMQg-UECOoE&adurl=",
                "name": "Finish Line"
              },
              "price_str": "$85.00.",
              "pos_overall": 1
            },
            {
              "pos": 2,
              "url": "/shopping/product/4092922174439754197?uule=w+CAIQICIXQ29sb3JhZG8sIFVuaXRlZCBTdGF0ZXM&q=adidas&prds=epd:6096059639745774212,paur:ClkAsKraX5cxKGk1E_r15f66xbFqydL47KoF9cO04jau1Hw_EeaJnz0EV5mb_JEjRlE5_m7N_B5Vg-krR5766rvdESfkczSSBqkGVDV7A5Ts8BlTUCNfpUxgtxIZAFPVH73vXbe47J5qGlzkfYH83D9zVPSv8w,prmr:1&sa=X&ved=0ahUKEwjpr_Sd84jpAhVI2aYKHYn1CeMQvxMI7AQ",
              "type": "grid",
              "price": 139.97,
              "title": "adidas Mens Alphaboost Training Shoes White ...",
              "merchant": {
                "url": "/aclk?sa=l&ai=DChcSEwju8fmd84jpAhUPTxgKHQshDIcYABAEGgJsZQ&sig=AOD64_3S0xuLlA1GOzNxCvYQdpeTLZkRyQ&ctype=5&q=&ved=0ahUKEwjpr_Sd84jpAhVI2aYKHYn1CeMQg-UECPQE&adurl=",
                "name": "Baseball Savings.com"
              },
              "price_str": "$139.97.",
              "pos_overall": 2
            },
...

Shopping Product

...
{
  "type": "Bundle",
  "items": [
    {
      "value": "Console Only",
      "selected": true,
      "available": true,
      "product_id": "5007040952399054528"
    },
    {
      "value": "Splatoon 2 Bundle",
      "available": false,
      "product_id": "6767220879106424425"
    },
    {
      "value": "Super Mario Odyssey Edition",
      "available": false,
      "product_id": "11634753303078094444"
    }
  ]
}
...

Shopping Product Pricing

"content": {
  "url": "https://www.google.com/shopping/product/5007040952399054528/online",
  "title": "Nintendo Switch with Joy-Con - 32 GB - Gray/Black",
  "rating": 4.5,
  "pricing": [
    {
      "price": 319.99,
      "seller": "Electronic Express",
      "details": "Free shipping",
      "currency": "$",
      "price_tax": 0,
      "price_total": 319.99,
      "seller_link": "/aclk?sa=l&ai=DChcSEwi9t9HqoJ7mAhVCXw0KHdyPBEYYABABGgJxYg&sig=AOD64_2gaL_J1BQ5J5PR-JazDM86N23Nww&adurl=&ctype=5&q=",
      "price_shipping": 0
    },
    {
      "price": 334.99,
      "seller": "ShopZodys",
      "details": "Arrives Dec 9 – 13",
      "currency": "$",
      "price_tax": 27.69,
      "price_total": 412.67,
      "seller_link": "/aclk?sa=l&ai=DChcSEwi9t9HqoJ7mAhVCXw0KHdyPBEYYABADGgJxYg&sig=AOD64_1Rqy4wxKvZXAaoX9FNDBy379EAAA&adurl=&ctype=5&q=",
      "price_shipping": 49.99
    }

Parameter Values

User-Agent

Download full list of user_agent_type values in JSON here.

[
  {
    "user_agent_type": "desktop",
    "description": "Random desktop browser User-Agent"
  },
  {
    "user_agent_type": "desktop_firefox",
    "description": "Random User-Agent of one of the latest versions of desktop Firefox"
  },
  {
    "user_agent_type": "desktop_chrome",
    "description": "Random User-Agent of one of the latest versions of desktop Chrome"
  },
  {
    "user_agent_type": "desktop_opera",
    "description": "Random User-Agent of one of the latest versions of desktop Opera"
  },
  {
    "user_agent_type": "desktop_edge",
    "description": "Random User-Agent of one of the latest versions of desktop Edge"
  },
  {
    "user_agent_type": "desktop_safari",
    "description": "Random User-Agent of one of the latest versions of desktop Safari"
  },
  {
    "user_agent_type": "mobile",
    "description": "Random mobile browser User-Agent"
  },
  {
    "user_agent_type": "mobile_android",
    "description": "Random User-Agent of one of the latest versions of Android browser"
  },
  {
    "user_agent_type": "mobile_ios",
    "description": "Random User-Agent of one of the latest versions of iPhone browser"
  },
  {
    "user_agent_type": "tablet",
    "description": "Random tablet browser User-Agent"
  },
  {
    "user_agent_type": "tablet_android",
    "description": "Random User-Agent of one of the latest versions of Android tablet"
  },
  {
    "user_agent_type": "tablet_ios",
    "description": "Random User-Agent of one of the latest versions of iPad tablet"
  }
]

Locale

Download full list of locale values in JSON here.

[  
   {  
      "locale":{  
         "en-ai":{  
            "description":"Anguilla - English",
            "domain":"com.ai"
         },
         "es-pr":{  
            "description":"Puerto Rico - Spanish",
            "domain":"com.pr"
         },
         ...
         "en-by":{  
            "description":"Belarus - English",
            "domain":"by"
         },
         "en-in":{  
            "description":"India - English",
            "domain":"co.in"
         }
      }
   }
]

Results Language

Download full list of results_language values in JSON here.

[
 {
   "results_language": "af",
   "language": "Afrikaans"
 },
 {
   "results_language": "ar",
   "language": "Arabic"
 },
 ...
 {
   "results_language": "vi",
   "language": "Vietnamese"
 }
]

Geo_location

There are a few ways you can use the geo_location parameter to get correctly-localized Google results.

• Using Google’s Canonical Location Name. This is very straightforward. Just pass us one of the values found on the CSV download here. Example: “geo_location”: “New York,New York,United States”.
• Using a state name. Strip the first part of a Google's Canonical Location Name and pass a geo_location value in a “State,Country” format. Works with United States, Australia, India and other countries with federated states. Example: “geo_location”: “California,United States”.
• Using a country name. To get results localized for the geographical center point of a country, pass an official country name. Example: “geo_location”: “United Kingdom”.
• Using coordinates and radius. To get hyperlocal search results (especially useful for searches such as “restaurants near me”), you can pass latitude, longitude and radius values. The following example passes the coordinates of Space Needle in Seattle, WA: “geo_location”: “lat: 47.6205, lng: -122.3493, rad: 25000”.

If you pass a misspelled geo_location parameter, chances are, either us or Google will interpret and correct it for you. Nonetheless, we recommend using the parameter structures outlined above, combined with the locale and domain parameters, to get the most accurate results.

Account Status

Usage Statistics

You can find your usage statistics by querying the following endpoint:

GET https://data.oxylabs.io/v1/stats

By default the API will return all time usage statistics. Adding ?group_by=month will return monthly stats, while ?group_by=day will return daily numbers.

This query will return all time statistics. You can find your daily and monthly usage by adding either ?group_by=day or ?group_by=month

curl --user user:pass1 'https://data.oxylabs.io/v1/stats'

Sample output:

{
    "data": {
        "sources": [
            {
                "realtime_results_count": "90",
                "results_count": "10",
                "title": "google_hotels"
            },
            {
                "realtime_results_count": "19",
                "results_count": "87",
                "title": "google_search"
            }
        ]
    },
    "meta": {
        "group_by": null
    }
}

Limits

The following endpoint will give your monthly commitment information as well as how much has already been used:

GET https://data.oxylabs.io/v1/stats/limits

curl --user user:pass1 'https://data.oxylabs.io/v1/stats/limits'

Sample output:

{
    "monthly_requests_commitment": 4500000,
    "used_requests": 985000
}

Response Codes

Parsed data response codes:

Cloud storage upload response codes:

References

• https://docs.oxylabs.io/rtc/source/google/index.html
• https://docs.oxylabs.io/rtc/index.html

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Do you know how to use OxyLabs Real-time Crawler for Yandex? This is the most comprehensive introduction from OxyLabs official.

Quick Start

Real-Time Crawler is built for heavy-duty data retrieval operations. You can use Real-Time Crawler to access various Yandex pages. It enables effortless web data extraction from search engines without any delays or errors.

Real-Time Crawler for Yandex uses basic HTTP authentication that requires sending username and password.

This is by far the fastest way to start using Real-Time Crawler for Yandex. You will send a query adidas to yandex_search using Realtime integration method. Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "yandex_search", "domain": "com", "query": "adidas"}'

If you have any questions not covered by this documentation, please contact your account manager or our support staff at support@oxylabs.io.

Integration Methods

Real-Time Crawler for Yandex supports three integration methods which have their unique benefits:

• Push-Pull. Using this method it is now required to mainain an active connection with our endpoint to retrieve the data. Upon making a request, our system is able to automatically ping users server when the job is done (see Callback). This method saves computing resources and can be scaled easily.
• Realtime. The method requires user to maintain an active connection with our endpoint in order to get the results successfully when the job is completed. This method can be implemented into one service while Push-Pull method is a two step process.
• SuperAPI. This method is very similar to Realtime but instead posting data to our endpoint, user can use HTML Cralwer as a proxy. To retrieve the data, user must set up a proxy endpoint and make GET request to a desired URL. Additional parameters must be added using headers.

Our recommended data extraction method is Push-Pull.

Push-Pull

This is the most simple yet the most reliable and recommended data delivery method. In Push-Pull scenario you send us a query, we return you job id, and once the job is done you can use that id to retrieve content from /results endpoint. You can check job completion status yourself, or you can set up a simple listener that is able to accept POST queries.

This way we will send you a callback message once the job is ready to be retrieved. In this particular example the results will be automatically uploaded to your S3 bucket named YOUR_BUCKET_NAME.

Single Query

The following endpoint will handle single queries for one keyword or URL. The API will return a confirmation message containing job information, including job id. You can check job completion status using that id, or you can ask us to ping your callback endpoint once the scraping task is finished by adding callback_url in the query.

POST https://data.oxylabs.io/v1/queries

You need to post query parameters as data in JSON body.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "yandex_search", "domain": "com", "query": "adidas", "callback_url": "https://your.callback.url", "storage_type": "s3", "storage_url": "YOUR_BUCKET_NAME"}'

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "callback_url": "https://your.callback.url",
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "query": "adidas",
  "source": "yandex_search",
  "start_page": 1,
  "status": "pending",
  "storage_type": "s3",
  "storage_url": "YOUR_BUCKET_NAME/12345678900987654321.json",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:01",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Check Job Status

If your query had callback_url, we will send you a message containing link to content once the scraping task is done. However, if there was no callback_url in the query, you will need to check job status yourself. For that you need to use the URL in href under rel:self in the response message you received after submitting your query to our API. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321.

GET https://data.oxylabs.io/v1/queries/{id}

Querying this link will return job information, including its status. There are 3 possible status values:

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321'

The API will respond with query information in JSON format, by printing it in response body. Notice that job status has been changed to done. You can now retrieve content by querying http://data.oxylabs.io/v1/queries/12345678900987654321/results.

You can also see that the task has been updated_at 2019-10-01 00:00:15 – the query took 14 seconds to complete.

{
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "query": "adidas",
  "source": "yandex_search",
  "start_page": 1,
  "status": "done",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:15",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Retrieve Job Content

Once you know the job is ready to retrieved either by checking its status or receiving a callback from us, you can GET it using the URL in href under rel:results in either our initial response or in callback message. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321/results.

GET https://data.oxylabs.io/v1/queries/{id}/results

The results can be automatically retrieved without periodically checking job status by setting up Callback service. User needs to specfy the IP or domain of the server where the Callback service is running. When our system completes a job, it will send a message to the provided IP or domain and the Callback service will download the results as described in the Callback implementation example.

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321/results'

The API will return job content:

{
  "results": [
    {
      "content": "<!doctype html>
        CONTENT      
      ",
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "page": 1,
      "url": "https://www.yandex.com/search?q=adidas&hl=en&gl=US",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

Callback

A callback is a POST request we send to your machine, informing that the data extraction task is completed and providing URL to download scraped content. This means that you no longer need to check job status manually. Once the data is here, we will let you know, and all you need to do now is retrieve it.

# Please see code samples in Python and PHP.

Sample callback output

{  
   "created_at":"2019-10-01 00:00:01",
   "updated_at":"2019-10-01 00:00:15",
   "locale":null,
   "client_id":163,
   "user_agent_type":"desktop",
   "source":"yandex_search",
   "pages":1,
   "subdomain":"www",
   "status":"done",
   "start_page":1,
   "parse":0,
   "render":null,
   "priority":0,
   "ttl":0,
   "origin":"api",
   "persist":true,
   "id":"12345678900987654321",
   "callback_url":"http://your.callback.url/",
   "query":"adidas",
   "domain":"com",
   "limit":10,
   "geo_location":null,
   {...}
   "_links":[
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321",
         "method":"GET",
         "rel":"self"
      },
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321/results",
         "method":"GET",
         "rel":"results"
      }
   ],
}

Batch Query

Real-Time Crawler also supports executing multiple keywords, up to 1,000 keywords with each batch. The following endpoint will submit multiple keywords to the extraction queue.

POST https://data.oxylabs.io/v1/queries/batch

You need to post query parameters as data in JSON body.

The system will handle every keyword as a separate request. If you provided callback URL, you will get a separate call for each keyword. Otherwise, our initial response will contain job ids for all keywords. For example, if you sent 50 keywords, we will return 50 unique job ids.

Important! query is the only parameter that can have multiple values. All other parameters are the same for that batch query.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries/batch' -H 'Content-Type: application/json'
 -d '@keywords.json'

keywords.json content:

{  
   "query":[  
      "adidas",
      "nike",
      "reebok"
   ],
   "source": "yandex_search",
   "domain": "com",
   "callback_url": "https://your.callback.url"
}

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "queries": [
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678900987654321",
      {...}
      "query": "adidas",
      "source": "yandex_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678901234567890",
      {...}
      "query": "nike",
      "source": "yandex_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678901234567890/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "01234567899876543210",
      {...}
      "query": "reebok",
      "source": "yandex_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/01234567899876543210/results",
          "method": "GET"
        }
      ]
    }
  ]
}

Get Notifier IP Address List

You may want to whitelist the IPs sending you callback messages or get the list of these IPs for other purposes. This can be done by GETing this endpoint: https://data.oxylabs.io/v1/info/callbacker_ips.

curl --user user:pass1 'https://data.oxylabs.io/v1/info/callbacker_ips'

The API will return the list of IPs making callback requests to your system:

{
    "ips": [
        "x.x.x.x",
        "y.y.y.y"
    ]
}

Upload to Storage

By default RTC job results are stored in our databases. This means that you will need to query our results endpoint and retrieve content yourself. Custom storage feature allows you to store results in your own cloud storage. The advantage of this feature is that you don't have to make extra requests in order to fetch results – everything goes directly to your storage bucket.

We support Amazon S3 and Google Cloud Storage. If you would like to use a different type of storage, please contact your account manager to discuss the feature delivery timeline.

Amazon S3

To get your job results uploaded to your Amazon S3 bucket, please set up access permissions for our service. To do that, go to https://s3.console.aws.amazon.com/ > S3 > Storage > Bucket Name (if don't have one, create new) > Permissions > Bucket Policy

You can find bucket policy in this JSON or in code sample area on the right. Don't forget to change bucket name under YOUR_BUCKET_NAME. This policy allows us to write to your bucket, give access to uploaded files to you, and know bucket location.

Google Cloud Storage

To get your job results uploaded to your Google Cloud Storage bucket, please set up special permissions for our service. To do that, please create a custom role with the storage.objects.create permission and assign it to the Oxylabs service account email oxyserps-storage@oxyserps-storage.iam.gserviceaccount.com.

Usage

To use this feature, please specify two additional parameters in your requests. Learn more here.

The upload path looks like this: YOUR_BUCKET_NAME/job_ID.json. You will find job ID in response body that you receive from us after submitting a request. In this example job ID is 12345678900987654321.

{
    "Version": "2012-10-17",
    "Id": "Policy1577442634787",
    "Statement": [
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
        },
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
        }
    ]
}

Realtime

The data submission is the same as in Push-Pull method, but Realtime case we will return the content on open connection. You send us a query, the connection remains open, we retrieve the content and bring it to you. The endpoint that handles that is this:

POST https://realtime.oxylabs.io/v1/queries

There is a timeout limit of 150 seconds for open connections, therefore in rare cases of heavy load we may not be able to ensure the data gets to you.

You need to post query parameters as data in JSON body. Please see example for more details.

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "yandex_search", "domain": "com", "query": "adidas"}'

Example response body that will be returned on open connection:

{
  "results": [
    {
      "content": "
      CONTENT
      "
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "id": null,
      "page": 1,
      "url": "https://www.yandex.com/search?q=adidas&hl=en&gl=US",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

SuperAPI

If you ever used regular proxies for data scraping, integrating SuperAPI delivery method will be a breeze. All that needs to be done is to use our entry node as proxy, authorize with Real-Time Crawler credentials, and ignore certificates. In cURL it's -k or --insecure. Your data will reach you on open connection.

GET realtime.oxylabs.io:60000

SuperAPI only supports a handful of parameters since it only works with Direct data source where full URL is provided. These parameters should be sent as headers. This is a list of accepted parameters:

If you need help setting up SuperAPI, drop a line at support@oxylabs.io.

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 -H "X-OxySERPs-User-Agent-Type: desktop_chrome" 
-H "X-OxySERPs-Geo-Location: 15550" "https://yandex.com/search/?text=nike"

Content Type

Real-Time Crawler returns raw HTML.

Data Sources

There are two approaches on how to retrieve data from Yandex using Real-Time Crawler. You can give us full URL via Direct, or you can specify parameters via specifically built data source – Search.

Technically not a content type, but Real-Time Crawler is able to render JavaScript when scraping. This may be necessary in some Yandex pages. A checkmark under Render JS will indicate whether a particular data source can be scraped with JavaScript enabled.

If you are unsure which way to choose, drop us a line at support@oxylabs.io or contact your account manager.

Direct

yandex source is designed to retrieve content of direct URLs of various Yandex pages. This means that instead of sending multiple parameters, you can provide us with a direct URL to required Yandex page. We do not strip any parameters or alter your URLs in any other way.

Query parameters

In this example the API will retrieve Yandex search for keyword nike in Push-Pull method:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "yandex", "url": "https://yandex.com/search/?text=nike&"}'

Here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "yandex", "url": "https://yandex.com/search/?text=nike&"}'

And via SuperAPI:

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 "https://yandex.com/search/?text=nike&"

Search

yandex_search source is designed to retrieve Yandex Search results (SERP).

Query parameters

API makes request to yandex.com to retrieve search results pages from number 11 to number 20 for keyword adidas. The API will post a JSON request to your.callback.url containing the URL to download the raw HTML output once the data retrieval task is successfully finished. This is Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "yandex_search", "domain": "com", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url"}'

And here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "yandex_search", "domain": "com", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url"}'

Parameter Values

User-Agent

Download full list of user_agent_type values in JSON here.

[
  {
    "user_agent_type": "desktop",
    "description": "Random desktop browser User-Agent"
  },
  {
    "user_agent_type": "desktop_firefox",
    "description": "Random User-Agent of one of the latest versions of desktop Firefox"
  },
  {
    "user_agent_type": "desktop_chrome",
    "description": "Random User-Agent of one of the latest versions of desktop Chrome"
  },
  {
    "user_agent_type": "desktop_opera",
    "description": "Random User-Agent of one of the latest versions of desktop Opera"
  },
  {
    "user_agent_type": "desktop_edge",
    "description": "Random User-Agent of one of the latest versions of desktop Edge"
  },
  {
    "user_agent_type": "desktop_safari",
    "description": "Random User-Agent of one of the latest versions of desktop Safari"
  },
  {
    "user_agent_type": "mobile",
    "description": "Random mobile browser User-Agent"
  },
  {
    "user_agent_type": "mobile_android",
    "description": "Random User-Agent of one of the latest versions of Android browser"
  },
  {
    "user_agent_type": "mobile_ios",
    "description": "Random User-Agent of one of the latest versions of iPhone browser"
  },
  {
    "user_agent_type": "tablet",
    "description": "Random tablet browser User-Agent"
  },
  {
    "user_agent_type": "tablet_android",
    "description": "Random User-Agent of one of the latest versions of Android tablet"
  },
  {
    "user_agent_type": "tablet_ios",
    "description": "Random User-Agent of one of the latest versions of iPad tablet"
  }
]

Account Status

Usage Statistics

You can find your usage statistics by querying the following endpoint:

GET https://data.oxylabs.io/v1/stats

By default the API will return all time usage statistics. Adding ?group_by=month will return monthly stats, while ?group_by=day will return daily numbers.

This query will return all time statistics. You can find your daily and monthly usage by adding either ?group_by=day or ?group_by=month

curl --user user:pass1 'https://data.oxylabs.io/v1/stats'

Sample output:

{
    "data": {
        "sources": [
            {
                "realtime_results_count": "90",
                "results_count": "10",
                "title": "yandex"
            },
            {
                "realtime_results_count": "19",
                "results_count": "87",
                "title": "yandex_search"
            }
        ]
    },
    "meta": {
        "group_by": null
    }
}

Limits

The following endpoint will give your monthly commitment information as well as how much has already been used:

GET https://data.oxylabs.io/v1/stats/limits

curl --user user:pass1 'https://data.oxylabs.io/v1/stats/limits'

Sample output:

{
    "monthly_requests_commitment": 4500000,
    "used_requests": 985000
}

Response Codes

Cloud storage upload response codes:

References

• https://docs.oxylabs.io/rtc/source/yandex/index.html
• https://docs.oxylabs.io/rtc/index.html

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Do you know how to use OxyLabs Real-time Crawler for Bing? This is the most comprehensive introduction from OxyLabs official.

Quick Start

Real-Time Crawler is built for heavy-duty data retrieval operations. You can use Real-Time Crawler to access various Bing pages. It enables effortless web data extraction from search engines without any delays or errors.

Real-Time Crawler for Bing uses basic HTTP authentication that requires sending username and password.

This is by far the fastest way to start using Real-Time Crawler for Bing. You will send a query adidas to bing_search using Realtime integration method. Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "bing_search", "domain": "com", "query": "adidas"}'

If you have any questions not covered by this documentation, please contact your account manager or our support staff at support@oxylabs.io.

Integration Methods

Real-Time Crawler for Bing supports three integration methods which have their unique benefits:

• Push-Pull. Using this method it is now required to mainain an active connection with our endpoint to retrieve the data. Upon making a request, our system is able to automatically ping users server when the job is done (see Callback). This method saves computing resources and can be scaled easily.
• Realtime. The method requires user to maintain an active connection with our endpoint in order to get the results successfully when the job is completed. This method can be implemented into one service while Push-Pull method is a two step process.
• SuperAPI. This method is very similar to Realtime but instead posting data to our endpoint, user can use HTML Cralwer as a proxy. To retrieve the data, user must set up a proxy endpoint and make GET request to a desired URL. Additional parameters must be added using headers.

Our recommended data extraction method is Push-Pull.

Push-Pull

This is the most simple yet the most reliable and recommended data delivery method. In Push-Pull scenario you send us a query, we return you job id, and once the job is done you can use that id to retrieve content from /results endpoint. You can check job completion status yourself, or you can set up a simple listener that is able to accept POST queries.

This way we will send you a callback message once the job is ready to be retrieved. In this particular example the results will be automatically uploaded to your S3 bucket named YOUR_BUCKET_NAME.

Single Query

The following endpoint will handle single queries for one keyword or URL. The API will return a confirmation message containing job information, including job id. You can check job completion status using that id, or you can ask us to ping your callback endpoint once the scraping task is finished by adding callback_url in the query.

POST https://data.oxylabs.io/v1/queries

You need to post query parameters as data in JSON body.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "bing_search", "domain": "com", "query": "adidas", "callback_url": "https://your.callback.url", "storage_type": "s3", "storage_url": "YOUR_BUCKET_NAME"}'

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "callback_url": "https://your.callback.url",
  "client_id": 5,
  "context": [
    {
      "key": "safe_search",
      "value": null
    }
  ],
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "render": null,
  "query": "adidas",
  "source": "bing_search",
  "start_page": 1,
  "status": "pending",  
  "storage_type": "s3",
  "storage_url": "YOUR_BUCKET_NAME/12345678900987654321.json",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:01",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Check Job Status

If your query had callback_url, we will send you a message containing link to content once the scraping task is done. However, if there was no callback_url in the query, you will need to check job status yourself. For that you need to use the URL in href under rel:self in the response message you received after submitting your query to our API. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321.

GET https://data.oxylabs.io/v1/queries/{id}

Querying this link will return job information, including its status. There are 3 possible status values:

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321'

The API will respond with query information in JSON format, by printing it in response body. Notice that job status has been changed to done. You can now retrieve content by querying http://data.oxylabs.io/v1/queries/12345678900987654321/results.

You can also see that the task has been updated_at 2019-10-01 00:00:15 – the query took 14 seconds to complete.

{
  "client_id": 5,
  "context": [
    {
      "key": "safe_search",
      "value": null
    }
  ],
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "render": null,
  "query": "adidas",
  "source": "bing_search",
  "start_page": 1,
  "status": "done",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:15",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Retrieve Job Content

Once you know the job is ready to retrieved either by checking its status or receiving a callback from us, you can GET it using the URL in href under rel:results in either our initial response or in callback message. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321/results

GET https://data.oxylabs.io/v1/queries/{id}/results

The results can be automatically retrieved without periodically checking job status by setting up Callback service. User needs to specfy the IP or domain of the server where the Callback service is running. When our system completes a job, it will send a message to the provided IP or domain and the Callback service will download the results as described in the Callback implementation example.

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321/results'

The API will return job content:

{
  "results": [
    {
      "content": "<!doctype html>
        CONTENT      
      ",
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "page": 1,
      "url": "https://www.bing.com/search?q=adidas",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

Callback

A callback is a POST request we send to your machine, informing that the data extraction task is completed and providing URL to download scraped content. This means that you no longer need to check job status manually. Once the data is here, we will let you know, and all you need to do now is retrieve it.

# Please see code samples in Python and PHP.

Sample callback output

{  
   "created_at":"2019-10-01 00:00:01",
   "updated_at":"2019-10-01 00:00:15",
   "locale":null,
   "client_id":163,
   "user_agent_type":"desktop",
   "source":"bing_search",
   "pages":1,
   "subdomain":"www",
   "status":"done",
   "start_page":1,
   "render":null,
   "priority":0,
   "ttl":0,
   "origin":"api",
   "persist":true,
   "id":"12345678900987654321",
   "callback_url":"http://your.callback.url/",
   "query":"adidas",
   "domain":"com",
   "limit":10,
   "geo_location":null,
   {...}
   "_links":[
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321",
         "method":"GET",
         "rel":"self"
      },
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321/results",
         "method":"GET",
         "rel":"results"
      }
   ],
}

Batch Query

Real-Time Crawler also supports executing multiple keywords, up to 1,000 keywords with each batch. The following endpoint will submit multiple keywords to the extraction queue.

POST https://data.oxylabs.io/v1/queries/batch

You need to post query parameters as data in JSON body.

The system will handle every keyword as a separate request. If you provided callback URL, you will get a separate call for each keyword. Otherwise, our initial response will contain job ids for all keywords. For example, if you sent 50 keywords, we will return 50 unique job ids.

Important! query is the only parameter that can have multiple values. All other parameters are the same for that batch query.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries/batch' -H 'Content-Type: application/json'
 -d '@keywords.json'

keywords.json content:

{  
   "query":[  
      "adidas",
      "nike",
      "reebok"
   ],
   "source": "bing_search",
   "domain": "com",
   "callback_url": "https://your.callback.url"
}

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "queries": [
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678900987654321",
      {...}
      "query": "adidas",
      "source": "bing_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678901234567890",
      {...}
      "query": "nike",
      "source": "bing_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678901234567890/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "01234567899876543210",
      {...}
      "query": "reebok",
      "source": "bing_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/01234567899876543210/results",
          "method": "GET"
        }
      ]
    }
  ]
}

Get Notifier IP Address List

You may want to whitelist the IPs sending you callback messages or get the list of these IPs for other purposes. This can be done by GETing this endpoint: https://data.oxylabs.io/v1/info/callbacker_ips.

curl --user user:pass1 'https://data.oxylabs.io/v1/info/callbacker_ips'

The API will return the list of IPs making callback requests to your system:

{
    "ips": [
        "x.x.x.x",
        "y.y.y.y"
    ]
}

Upload to Storage

By default RTC job results are stored in our databases. This means that you will need to query our results endpoint and retrieve content yourself. Custom storage feature allows you to store results in your own cloud storage. The advantage of this feature is that you don't have to make extra requests in order to fetch results – everything goes directly to your storage bucket.

We support Amazon S3 and Google Cloud Storage. If you would like to use a different type of storage, please contact your account manager to discuss the feature delivery timeline.

Amazon S3

To get your job results uploaded to your Amazon S3 bucket, please set up access permissions for our service. To do that, go to https://s3.console.aws.amazon.com/ > S3 > Storage > Bucket Name (if don't have one, create new) > Permissions > Bucket Policy

You can find bucket policy in this JSON or in code sample area on the right. Don't forget to change bucket name under YOUR_BUCKET_NAME. This policy allows us to write to your bucket, give access to uploaded files to you, and know bucket location.

Google Cloud Storage

To get your job results uploaded to your Google Cloud Storage bucket, please set up special permissions for our service. To do that, please create a custom role with the storage.objects.create permission and assign it to the Oxylabs service account email oxyserps-storage@oxyserps-storage.iam.gserviceaccount.com.

Usage

To use this feature, please specify two additional parameters in your requests. Learn more here.

The upload path looks like this: YOUR_BUCKET_NAME/job_ID.json. You will find job ID in response body that you receive from us after submitting a request. In this example job ID is 12345678900987654321.

{
    "Version": "2012-10-17",
    "Id": "Policy1577442634787",
    "Statement": [
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
        },
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
        }
    ]
}

Realtime

The data submission is the same as in Push-Pull method, but Realtime case we will return the content on open connection. You send us a query, the connection remains open, we retrieve the content and bring it to you. The endpoint that handles that is this:

POST https://realtime.oxylabs.io/v1/queries

There is a timeout limit of 150 seconds for open connections, therefore in rare cases of heavy load we may not be able to ensure the data gets to you.

You need to post query parameters as data in JSON body. Please see example for more details.

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "bing_search", "domain": "com", "query": "adidas"}'

Example response body that will be returned on open connection:

{
  "results": [
    {
      "content": "
      CONTENT
      "
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "id": null,
      "page": 1,
      "url": "https://www.bing.com/search?q=adidas",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

SuperAPI

If you ever used regular proxies for data scraping, integrating SuperAPI delivery method will be a breeze. All that needs to be done is to use our entry node as proxy, authorize with Real-Time Crawler credentials, and ignore certificates. In cURL it's -k or --insecure. Your data will reach you on open connection.

GET realtime.oxylabs.io:60000

SuperAPI only supports a handful of parameters since it only works with Direct data source where full URL is provided. These parameters should be sent as headers. This is a list of accepted parameters:

If you need help setting up SuperAPI, drop a line at support@oxylabs.io.

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 -H "X-OxySERPs-User-Agent-Type: desktop_chrome" 
-H "X-OxySERPs-Geo-Location: 15550" "https://bing.com/search/?text=nike"

Content Type

Real-Time Crawler returns raw HTML.

Data Sources

There are two approaches on how to retrieve data from Bing using Real-Time Crawler. You can give us full URL via Direct, or you can specify parameters via specifically built data source – Search.

If you are unsure which way to choose, drop us a line at support@oxylabs.io or contact your account manager.

Direct

bing source is designed to retrieve content of direct URLs of various Bing pages. This means that instead of sending multiple parameters, you can provide us with a direct URL to required Bing page. We do not strip any parameters or alter your URLs in any other way.

Query parameters

In this example the API will retrieve Bing search for keyword nike in Push-Pull method:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "bing", "url": "https://bing.com/search/?text=nike&"}'

Here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "bing", "url": "https://bing.com/search/?text=nike&"}'

And via SuperAPI:

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 "https://bing.com/search/?text=nike&"

Search

bing_search source is designed to retrieve Bing Search results (SERP).

Query parameters

API makes request to bing.com to retrieve search results pages from number 11 to number 20 for keyword adidas. The API will post a JSON request to your.callback.url containing the URL to download the raw HTML output once the data retrieval task is successfully finished. This is Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "bing_search", "domain": "com", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url"}'

And here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "bing_search", "domain": "com", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url"}'

Parameter Values

User-Agent

Download full list of user_agent_type values in JSON here.

[
  {
    "user_agent_type": "desktop",
    "description": "Random desktop browser User-Agent"
  },
  {
    "user_agent_type": "desktop_firefox",
    "description": "Random User-Agent of one of the latest versions of desktop Firefox"
  },
  {
    "user_agent_type": "desktop_chrome",
    "description": "Random User-Agent of one of the latest versions of desktop Chrome"
  },
  {
    "user_agent_type": "desktop_opera",
    "description": "Random User-Agent of one of the latest versions of desktop Opera"
  },
  {
    "user_agent_type": "desktop_edge",
    "description": "Random User-Agent of one of the latest versions of desktop Edge"
  },
  {
    "user_agent_type": "desktop_safari",
    "description": "Random User-Agent of one of the latest versions of desktop Safari"
  },
  {
    "user_agent_type": "mobile",
    "description": "Random mobile browser User-Agent"
  },
  {
    "user_agent_type": "mobile_android",
    "description": "Random User-Agent of one of the latest versions of Android browser"
  },
  {
    "user_agent_type": "mobile_ios",
    "description": "Random User-Agent of one of the latest versions of iPhone browser"
  },
  {
    "user_agent_type": "tablet",
    "description": "Random tablet browser User-Agent"
  },
  {
    "user_agent_type": "tablet_android",
    "description": "Random User-Agent of one of the latest versions of Android tablet"
  },
  {
    "user_agent_type": "tablet_ios",
    "description": "Random User-Agent of one of the latest versions of iPad tablet"
  }
]

Account Status

Usage Statistics

You can find your usage statistics by querying the following endpoint:

GET https://data.oxylabs.io/v1/stats

By default the API will return all time usage statistics. Adding ?group_by=month will return monthly stats, while ?group_by=day will return daily numbers.

This query will return all time statistics. You can find your daily and monthly usage by adding either ?group_by=day or ?group_by=month

curl --user user:pass1 'https://data.oxylabs.io/v1/stats'

Sample output:

{
    "data": {
        "sources": [
            {
                "realtime_results_count": "90",
                "results_count": "10",
                "title": "bing"
            },
            {
                "realtime_results_count": "19",
                "results_count": "87",
                "title": "bing_search"
            }
        ]
    },
    "meta": {
        "group_by": null
    }
}

Limits

The following endpoint will give your monthly commitment information as well as how much has already been used:

GET https://data.oxylabs.io/v1/stats/limits

curl --user user:pass1 'https://data.oxylabs.io/v1/stats/limits'

Sample output:

{
    "monthly_requests_commitment": 4500000,
    "used_requests": 985000
}

Response Codes

Cloud storage upload response codes:

References

• https://docs.oxylabs.io/rtc/source/bing/index.html
• https://docs.oxylabs.io/rtc/index.html

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Do you know how to use OxyLabs Real-time Crawler for Baidu? This is the most comprehensive introduction from OxyLabs official.

Quick Start

Real-Time Crawler is built for heavy-duty data retrieval operations. You can use Real-Time Crawler to access various Baidu pages. It enables effortless web data extraction from search engines without any delays or errors.

Real-Time Crawler for Baidu uses basic HTTP authentication that requires sending username and password.

This is by far the fastest way to start using Real-Time Crawler for Baidu. You will send a query adidas to baidu_search using Realtime integration method. Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "baidu_search", "domain": "com", "query": "adidas"}'

If you have any questions not covered by this documentation, please contact your account manager or our support staff at support@oxylabs.io.

Integration Methods

Real-Time Crawler for Baidu supports three integration methods which have their unique benefits:

• Push-Pull. Using this method it is now required to mainain an active connection with our endpoint to retrieve the data. Upon making a request, our system is able to automatically ping users server when the job is done (see Callback). This method saves computing resources and can be scaled easily.
• Realtime. The method requires user to maintain an active connection with our endpoint in order to get the results successfully when the job is completed. This method can be implemented into one service while Push-Pull method is a two step process.
• SuperAPI. This method is very similar to Realtime but instead posting data to our endpoint, user can use HTML Cralwer as a proxy. To retrieve the data, user must set up a proxy endpoint and make GET request to a desired URL. Additional parameters must be added using headers.

Our recommended data extraction method is Push-Pull.

Push-Pull

This is the most simple yet the most reliable and recommended data delivery method. In Push-Pull scenario you send us a query, we return you job id, and once the job is done you can use that id to retrieve content from /results endpoint. You can check job completion status yourself, or you can set up a simple listener that is able to accept POST queries.

This way we will send you a callback message once the job is ready to be retrieved. In this particular example the results will be automatically uploaded to your S3 bucket named YOUR_BUCKET_NAME.

Single Query

The following endpoint will handle single queries for one keyword or URL. The API will return a confirmation message containing job information, including job id. You can check job completion status using that id, or you can ask us to ping your callback endpoint once the scraping task is finished by adding callback_url in the query.

POST https://data.oxylabs.io/v1/queries

You need to post query parameters as data in JSON body.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "baidu_search", "domain": "com", "query": "adidas", "callback_url": "https://your.callback.url", "storage_type": "s3", "storage_url": "YOUR_BUCKET_NAME"}'

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "callback_url": "https://your.callback.url",
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "render": null,
  "query": "adidas",
  "source": "baidu_search",
  "start_page": 1,
  "status": "pending",
  "storage_type": "s3",
  "storage_url": "YOUR_BUCKET_NAME/12345678900987654321.json",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:01",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Check Job Status

If your query had callback_url, we will send you a message containing link to content once the scraping task is done. However, if there was no callback_url in the query, you will need to check job status yourself. For that you need to use the URL in href under rel:self in the response message you received after submitting your query to our API. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321.

GET https://data.oxylabs.io/v1/queries/{id}

Querying this link will return job information, including its status. There are 3 possible status values:

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321'

The API will respond with query information in JSON format, by printing it in response body. Notice that job status has been changed to done. You can now retrieve content by querying http://data.oxylabs.io/v1/queries/12345678900987654321/results.

You can also see that the task has been updated_at 2019-10-01 00:00:15 – the query took 14 seconds to complete.

{
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "render": null,
  "query": "adidas",
  "source": "baidu_search",
  "start_page": 1,
  "status": "done",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:15",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Retrieve Job Content

Once you know the job is ready to retrieved either by checking its status or receiving a callback from us, you can GET it using the URL in href under rel:results in either our initial response or in callback message. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321/results.

GET https://data.oxylabs.io/v1/queries/{id}/results

The results can be automatically retrieved without periodically checking job status by setting up Callback service. User needs to specfy the IP or domain of the server where the Callback service is running. When our system completes a job, it will send a message to the provided IP or domain and the Callback service will download the results as described in the Callback implementation example.

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321/results'

The API will return job content:

{
  "results": [
    {
      "content": "<!doctype html>
        CONTENT      
      ",
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "page": 1,
      "url": "https://www.baidu.com/search?q=adidas&hl=en&gl=US",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

Callback

A callback is a POST request we send to your machine, informing that the data extraction task is completed and providing URL to download scraped content. This means that you no longer need to check job status manually. Once the data is here, we will let you know, and all you need to do now is retrieve it.

# Please see code samples in Python and PHP.

Sample callback output

{  
   "created_at":"2019-10-01 00:00:01",
   "updated_at":"2019-10-01 00:00:15",
   "locale":null,
   "client_id":163,
   "user_agent_type":"desktop",
   "source":"baidu_search",
   "pages":1,
   "subdomain":"www",
   "status":"done",
   "start_page":1,
   "render":null,
   "priority":0,
   "ttl":0,
   "origin":"api",
   "persist":true,
   "id":"12345678900987654321",
   "callback_url":"http://your.callback.url/",
   "query":"adidas",
   "domain":"com",
   "limit":10,
   "geo_location":null,
   {...}
   "_links":[
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321",
         "method":"GET",
         "rel":"self"
      },
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321/results",
         "method":"GET",
         "rel":"results"
      }
   ],
}

Batch Query

Real-Time Crawler also supports executing multiple keywords, up to 1,000 keywords with each batch. The following endpoint will submit multiple keywords to the extraction queue.

POST https://data.oxylabs.io/v1/queries/batch

You need to post query parameters as data in JSON body.

The system will handle every keyword as a separate request. If you provided callback URL, you will get a separate call for each keyword. Otherwise, our initial response will contain job ids for all keywords. For example, if you sent 50 keywords, we will return 50 unique job ids.

Important! query is the only parameter that can have multiple values. All other parameters are the same for that batch query.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries/batch' -H 'Content-Type: application/json'
 -d '@keywords.json'

keywords.json content:

{  
   "query":[  
      "adidas",
      "nike",
      "reebok"
   ],
   "source": "baidu_search",
   "domain": "com",
   "callback_url": "https://your.callback.url"
}

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "queries": [
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678900987654321",
      {...}
      "query": "adidas",
      "source": "baidu_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678901234567890",
      {...}
      "query": "nike",
      "source": "baidu_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678901234567890/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "01234567899876543210",
      {...}
      "query": "reebok",
      "source": "baidu_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/01234567899876543210/results",
          "method": "GET"
        }
      ]
    }
  ]
}

Get Notifier IP Address List

You may want to whitelist the IPs sending you callback messages or get the list of these IPs for other purposes. This can be done by GETing this endpoint: https://data.oxylabs.io/v1/info/callbacker_ips.

curl --user user:pass1 'https://data.oxylabs.io/v1/info/callbacker_ips'

The API will return the list of IPs making callback requests to your system:

{
    "ips": [
        "x.x.x.x",
        "y.y.y.y"
    ]
}

Upload to Storage

By default RTC job results are stored in our databases. This means that you will need to query our results endpoint and retrieve content yourself. Custom storage feature allows you to store results in your own cloud storage. The advantage of this feature is that you don't have to make extra requests in order to fetch results – everything goes directly to your storage bucket.

We support Amazon S3 and Google Cloud Storage. If you would like to use a different type of storage, please contact your account manager to discuss the feature delivery timeline.

Amazon S3

To get your job results uploaded to your Amazon S3 bucket, please set up access permissions for our service. To do that, go to https://s3.console.aws.amazon.com/ > S3 > Storage > Bucket Name (if don't have one, create new) > Permissions > Bucket Policy

You can find bucket policy in this JSON or in code sample area on the right. Don't forget to change bucket name under YOUR_BUCKET_NAME. This policy allows us to write to your bucket, give access to uploaded files to you, and know bucket location.

Google Cloud Storage

To get your job results uploaded to your Google Cloud Storage bucket, please set up special permissions for our service. To do that, please create a custom role with the storage.objects.create permission and assign it to the Oxylabs service account email oxyserps-storage@oxyserps-storage.iam.gserviceaccount.com.

Usage

To use this feature, please specify two additional parameters in your requests. Learn more here.

The upload path looks like this: YOUR_BUCKET_NAME/job_ID.json. You will find job ID in response body that you receive from us after submitting a request. In this example job ID is 12345678900987654321.

Realtime

The data submission is the same as in Push-Pull method, but Realtime case we will return the content on open connection. You send us a query, the connection remains open, we retrieve the content and bring it to you. The endpoint that handles that is this:

POST https://realtime.oxylabs.io/v1/queries

There is a timeout limit of 150 seconds for open connections, therefore in rare cases of heavy load we may not be able to ensure the data gets to you.

You need to post query parameters as data in JSON body. Please see example for more details.

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "baidu_search", "domain": "com", "query": "adidas"}'

Example response body that will be returned on open connection:

{
  "results": [
    {
      "content": "
      CONTENT
      "
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "id": null,
      "page": 1,
      "url": "https://www.baidu.com/search?q=adidas&hl=en&gl=US",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

SuperAPI

If you ever used regular proxies for data scraping, integrating SuperAPI delivery method will be a breeze. All that needs to be done is to use our entry node as proxy, authorize with Real-Time Crawler credentials, and ignore certificates. In cURL it's -k or --insecure. Your data will reach you on open connection.

GET realtime.oxylabs.io:60000

SuperAPI only supports a handful of parameters since it only works with Direct data source where full URL is provided. These parameters should be sent as headers. This is a list of accepted parameters:

If you need help setting up SuperAPI, drop a line at support@oxylabs.io.

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 -H "X-OxySERPs-User-Agent-Type: desktop_chrome" "https://www.baidu.com/search?q=adidas"

Data Sources

There are multiple approaches how to retrieve data from Baidu using Real-Time Crawler. You can give us full URL via Direct, or you can specify parameters via specifically built data sources, such as Search, Shopping Product or Images.

If you are unsure which way to choose, drop us a line at support@oxylabs.io or contact your account manager.

Direct

baidu source is designed to retrieve content of direct URLs of various Baidu pages. This means that instead of sending multiple parameters, you can provide us with a direct URL to required Baidu page. We do not strip any parameters or alter your URLs in any other way.

Query parameters

In this example the API will retrieve Baidu search for keyword adidas in Push-Pull method:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "baidu", "url": "http://www.baidu.com/s?ie=utf-8&f=8&rsv_bp=1&rsv_idx=1&ch=&tn=baidu&bar=&wd=adidas"}'

Here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "baidu", "url": "http://www.baidu.com/s?ie=utf-8&f=8&rsv_bp=1&rsv_idx=1&ch=&tn=baidu&bar=&wd=adidas"}'

And via SuperAPI:

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 "http://www.baidu.com/s?ie=utf-8&f=8&rsv_bp=1&rsv_idx=1&ch=&tn=baidu&bar=&wd=adidas"

Search

baidu_search source is designed to retrieve Baidu Search results (SERP) in HTML format.

Query parameters

Real-Time Crawler makes request to baidu.com to retrieve search results pages from number 11 to number 20 for keyword adidas. Real-Time Crawler will post the download URL to raw HTML page output to your.callback.url once the data retrieval task is successfully finished.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "baidu_search", "domain": "com", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url"}'

And here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "baidu_search", "domain": "com", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url"}'

Parameter Values

User-Agent

Download full list of user_agent_type values in JSON here.

[
  {
    "user_agent_type": "desktop",
    "description": "Random desktop browser User-Agent"
  },
  {
    "user_agent_type": "desktop_firefox",
    "description": "Random User-Agent of one of the latest versions of desktop Firefox"
  },
  {
    "user_agent_type": "desktop_chrome",
    "description": "Random User-Agent of one of the latest versions of desktop Chrome"
  },
  {
    "user_agent_type": "desktop_opera",
    "description": "Random User-Agent of one of the latest versions of desktop Opera"
  },
  {
    "user_agent_type": "desktop_edge",
    "description": "Random User-Agent of one of the latest versions of desktop Edge"
  },
  {
    "user_agent_type": "desktop_safari",
    "description": "Random User-Agent of one of the latest versions of desktop Safari"
  },
  {
    "user_agent_type": "mobile",
    "description": "Random mobile browser User-Agent"
  },
  {
    "user_agent_type": "mobile_android",
    "description": "Random User-Agent of one of the latest versions of Android browser"
  },
  {
    "user_agent_type": "mobile_ios",
    "description": "Random User-Agent of one of the latest versions of iPhone browser"
  },
  {
    "user_agent_type": "tablet",
    "description": "Random tablet browser User-Agent"
  },
  {
    "user_agent_type": "tablet_android",
    "description": "Random User-Agent of one of the latest versions of Android tablet"
  },
  {
    "user_agent_type": "tablet_ios",
    "description": "Random User-Agent of one of the latest versions of iPad tablet"
  }
]

Account Status

Usage Statistics

You can find your usage statistics by querying the following endpoint:

GET https://data.oxylabs.io/v1/stats

By default the API will return all time usage statistics. Adding ?group_by=month will return monthly stats, while ?group_by=day will return daily numbers.

This query will return all time statistics. You can find your daily and monthly usage by adding either ?group_by=day or ?group_by=month

curl --user user:pass1 'https://data.oxylabs.io/v1/stats'

Sample output:

{
    "data": {
        "sources": [
            {
                "realtime_results_count": "90",
                "results_count": "10",
                "title": "baidu"
            },
            {
                "realtime_results_count": "19",
                "results_count": "87",
                "title": "baidu_search"
            }
        ]
    },
    "meta": {
        "group_by": null
    }
}

Limits

The following endpoint will give your monthly commitment information as well as how much has already been used:

GET https://data.oxylabs.io/v1/stats/limits

curl --user user:pass1 'https://data.oxylabs.io/v1/stats/limits'

Sample output:

{
    "monthly_requests_commitment": 4500000,
    "used_requests": 985000
}

Response Codes

Cloud storage upload response codes:

References

• https://docs.oxylabs.io/rtc/source/baidu/index.html
• https://docs.oxylabs.io/rtc/index.html

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Do you know how to use OxyLabs Real-time Crawler for web pages? This is the most comprehensive introduction from OxyLabs official.

Quick Start

HTML Crawler API is built to help you in your heavy-duty data retrieval operations. You can use HTML Crawler API to access various public pages. It enables effortless web data extraction without any delays or errors.

HTML Crawler API uses basic HTTP authentication that requires sending username and password.

This is by far the fastest way to start using HTML Crawler API. You will make a request to https://ip.oxylabs.io using Realtime integration method. Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "universal", "url": "https://ip.oxylabs.io"}'

If you have any questions not covered by this documentation, please contact your account manager or our support staff at support@oxylabs.io.

Push-Pull

This is the most simple yet the most reliable and recommended data delivery method. In Push-Pull scenario you send us a query, we return you job id, and once the job is done you can use that id to retrieve content from /results endpoint. You can check job completion status yourself, or you can set up a simple listener that is able to accept POST queries.

This way, we will send you a callback message once the job is ready to be retrieved. In this particular example the results will be automatically uploaded to your S3 bucket named YOUR_BUCKET_NAME.

Single Query

The following endpoint will handle single queries for one keyword or URL. The API will return a confirmation message containing job information, including job id. You can check job completion status using that id, or you can ask us to ping your callback endpoint once the scraping task is finished by adding callback_url in the query.

POST https://data.oxylabs.io/v1/queries

You need to post query parameters as data in the JSON body.

curl --user user:pass1\
'https://data.oxylabs.io/v1/queries' \
-H "Content-Type: application/json" \
-d '{"source": "universal", "url": "https://stackoverflow.com/questions/tagged/python", "callback_url": "https://your.callback.url", "storage_type": "s3", "storage_url": "YOUR_BUCKET_NAME"}'

The API will respond with query information in JSON format, by printing it in the response body, similar to this:

{
  "callback_url": "https://your.callback.url",
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "url": "https://stackoverflow.com/questions/tagged/python",
  "source": "universal",
  "start_page": 1,
  "status": "pending",
  "storage_type": "s3",
  "storage_url": "YOUR_BUCKET_NAME/12345678900987654321.json",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:01",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Check Job Status

If your query had a callback_url, we will send you a message containing a link to the content once the scraping task is done. However, if there was no callback_url in the query, you will need to check the job status yourself. For that, you need to use the URL in href under rel:self in the response message you received after submitting your query to our API. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321.

GET https://data.oxylabs.io/v1/queries/{id}

Querying this link will return the job information, including its status. There are three possible status values:

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321'

The API will respond with query information in JSON format, by printing it in the response body. Notice that job status has been changed to done. You can now retrieve content by querying http://data.oxylabs.io/v1/queries/12345678900987654321/results.

You can also see that the task has been updated_at 2019-10-01 00:00:15 – the query took 14 seconds to complete.

{
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "url": "sofa",
  "source": "universal",
  "start_page": 1,
  "status": "done",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:15",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Retrieve Job Content

Once you know the job is ready to be retrieved by checking its status, you can GET it using the URL in href under rel:results in our initial response. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321/results.

GET https://data.oxylabs.io/v1/queries/{id}/results

The results can be automatically retrieved without periodically checking job status by setting up Callback service. User needs to specfy the IP or domain of the server where the Callback service is running. When our system completes a job, it will send a message to the provided IP or domain and the Callback service will download the results as described in the Callback implementation example.

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321/results'

The API will return job content:

{
  "results": [
    {
      "content": "<!doctype html>
        CONTENT      
      ",
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "page": 1,
      "url": "https://stackoverflow.com/questions/tagged/python",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

Callback

A callback is a POST request we send to your machine, informing that the data extraction task is completed and providing URL to download scraped content. This means that you no longer need to check job status manually. Once the data is here, we will let you know, and all you need to do now is retrieve it.

# Please see the code samples in Python and PHP.

Sample callback output

{  
   "created_at":"2019-10-01 00:00:01",
   "updated_at":"2019-10-01 00:00:15",
   "locale":null,
   "client_id":163,
   "user_agent_type":"desktop",
   "source":"universal",
   "pages":1,
   "subdomain":"www",
   "status":"done",
   "start_page":1,
   "parse":0,
   "render":null,
   "priority":0,
   "ttl":0,
   "origin":"api",
   "persist":true,
   "id":"12345678900987654321",
   "callback_url":"http://your.callback.url/",
   "url":"https://stackoverflow.com/questions/tagged/python",
   "domain":"de",
   "limit":10,
   "geo_location":null,
   {...}
   "_links":[
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321",
         "method":"GET",
         "rel":"self"
      },
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321/results",
         "method":"GET",
         "rel":"results"
      }
   ],
}

Batch Query

HTML Crawler API also supports executing multiple keywords, up to 1,000 keywords with each batch. The following endpoint will submit multiple keywords to the extraction queue.

POST https://data.oxylabs.io/v1/queries/batch

You need to post query parameters as data in the JSON body.

The system will handle every keyword as a separate request. If you provided a callback URL, you will get a separate call for each keyword. Otherwise, our initial response will contain job ids for all keywords. For example, if you sent 50 keywords, we will return 50 unique job ids.

Important! query is the only parameter that can have multiple values. All other parameters are the same for that batch query.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries/batch' -H 'Content-Type: application/json' \
 -d '@keywords.json'

keywords.json content:

{  
   "url":[  
      "https://stackoverflow.com/questions/tagged/python",
      "https://stackoverflow.com/questions/tagged/golang",
      "https://stackoverflow.com/questions/tagged/php"
   ],
   "source": "universal",
   "callback_url": "https://your.callback.url"
}

The API will respond with query information in JSON format, by printing it in the response body, similar to this:

{
  "queries": [
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678900987654321",
      {...}
      "url": "https://stackoverflow.com/questions/tagged/python",
      "source": "universal",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678901234567890",
      {...}
      "url": "https://stackoverflow.com/questions/tagged/golang",
      "source": "universal",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678901234567890/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "01234567899876543210",
      {...}
      "url": "https://stackoverflow.com/questions/tagged/php",
      "source": "universal",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/01234567899876543210/results",
          "method": "GET"
        }
      ]
    }
  ]
}

Get Notifier IP Address List

You may want to whitelist the IPs sending you callback messages or get the list of these IPs for other purposes. This can be done by GETing this endpoint: https://data.oxylabs.io/v1/info/callbacker_ips.

curl --user user:pass1 'https://data.oxylabs.io/v1/info/callbacker_ips'

The API will return the list of IPs making callback requests to your system:

{
    "ips": [
        "x.x.x.x",
        "y.y.y.y"
    ]
}

Upload to Storage

By default RTC job results are stored in our databases. This means that you will need to query our results endpoint and retrieve content yourself. Custom storage feature allows you to store results in your own cloud storage. The advantage of this feature is that you don't have to make extra requests in order to fetch results – everything goes directly to your storage bucket.

We support Amazon S3 and Google Cloud Storage. If you would like to use a different type of storage, please contact your account manager to discuss the feature delivery timeline.

Amazon S3

To get your job results uploaded to your Amazon S3 bucket, please set up access permissions for our service. To do that, go to https://s3.console.aws.amazon.com/ > S3 > Storage > Bucket Name (if don't have one, create new) > Permissions > Bucket Policy

You can find bucket policy in this JSON or in code sample area on the right. Don't forget to change bucket name under YOUR_BUCKET_NAME. This policy allows us to write to your bucket, give access to uploaded files to you, and know bucket location.

Google Cloud Storage

To get your job results uploaded to your Google Cloud Storage bucket, please set up special permissions for our service. To do that, please create a custom role with the storage.objects.create permission and assign it to the Oxylabs service account email oxyserps-storage@oxyserps-storage.iam.gserviceaccount.com.

Usage

To use this feature, please specify two additional parameters in your requests. Learn more here.

The upload path looks like this: YOUR_BUCKET_NAME/job_ID.json. You will find job ID in response body that you receive from us after submitting a request. In this example job ID is 12345678900987654321.

{
    "Version": "2012-10-17",
    "Id": "Policy1577442634787",
    "Statement": [
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
        },
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
        }
    ]
}

Realtime

The data submission is the same as in Push-Pull method, but with Realtime, we will return the content on open connection. You send us a query, the connection remains open, we retrieve the content, and bring it to you. The endpoint that handles that is this:

POST https://realtime.oxylabs.io/v1/queries

The timeout limit for open connections is 100 seconds. Therefore, in rare cases of heavy load, we may not be able to ensure the data gets to you.

You need to post query parameters as data in the JSON body. Please see an example for more details.

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" \
 -d '{"source": "universal", "url": "https://stackoverflow.com/questions/tagged/python"}'

Example response body that will be returned on open connection:

{
  "results": [
    {
      "content": "
      CONTENT
      "
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "id": null,
      "page": 1,
      "url": "https://stackoverflow.com/questions/tagged/python",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

SuperAPI

If you have ever used regular proxies for data scraping, integrating SuperAPI delivery method will be a breeze. You simply need to use our entry node as proxy, authorize with HTML Crawler API credentials, and ignore certificates. In cURL it's -k or --insecure. Your data will reach you on open connection.

GET realtime.oxylabs.io:60000

SuperAPI only supports a handful of parameters since it only works with Direct data source where a full URL is provided. These parameters should be sent as headers. This is a list of accepted parameters:

If you need help setting up SuperAPI, get in touch with us at support@oxylabs.io.

curl -k \
-x realtime.oxylabs.io:60000 \
-U user:pass1 \
-H "X-OxySERPs-User-Agent-Type: desktop_chrome" \
"https://stackoverflow.com/questions/tagged/python"

Content Type

HTML Crawler API returns raw HTML.

Download Images

It is possible to download images via HTML Crawler API. If you are doing that through SuperAPI, you can simply save the output to image extension. For example:

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 "https://example.com/image.jpg" >> image.jpg

If you are using Push-Pull or Realtime methods, you will need to add content_encoding parameter with a value of base64. Once you receive the results, you then need to decode encoded data from content into bytes and save it as an image file. Please find an example in Python on the right.

Data Sources

HTML Crawler API accepts URLs, along with additional parameters, such as User-Agent type, proxy location, and others. See this method, which we call Direct, described below.

HTML Crawler API is able to render JavaScript when scraping. This enables you to get more data from the web page and get screenshots.

If you are unsure about any part of the documentation, drop us a line at support@oxylabs.io or contact your account manager.

Direct

universal source is designed to retrieve the contents of any URL on the internet. POST-ing the parameters in JSON format to the following endpoint will submit the specified URL to the extraction queue.

Query parameters

In this example, the API will retrieve a universal product page in Push-Pull method. All available parameters are included (though not always necessary or compatible within the same request), to give you an idea on how to format your requests:

curl --user user:pass1 \
'https://data.oxylabs.io/v1/queries' \
-H "Content-Type: application/json" \
 -d '{"source":"universal","url":"https://stackoverflow.com/questions/tagged/python","user_agent_type":"mobile","context":[{"key":"headers","value":{"Accept-Language":"en-US","Content-Type":"application/octet-stream","Custom-Header":"custom header content"}},{"key":"cookies","value":[{"key":"NID","value":"1234567890"},{"key":"1P JAR","value":"0987654321"}]},{"key":"follow_redirects","value":true},{"key":"http_method","value":"post"},{"key":"content","value":"YmFzZTY0RW5jb2RlZFBPU1RCb2R5"},{"key":"successful_status_codes","value":[808,909]}]}

Here is the same example in Realtime:

curl --user user:pass1 \
'https://data.oxylabs.io/v1/queries' \
-H "Content-Type: application/json" \
-d '{"source": "universal", "url": "https://stackoverflow.com/questions/tagged/python", "user_agent_type": "mobile", "context": [{"key": "headers", "value": ["Accept-Language": "en-US", "Content-Type": "application/octet-stream", "Custom-Header": "custom header content"]}, {"key": "cookies", "value": [{"key": "NID", "value": "1234567890"}, {"key": "1P JAR", "value": "0987654321"}, {"key": "follow_redirects", "value": true}, {"key": "http_method", "value": "post"}, {"key": "content", "value": "base64EncodedPOSTBody"}, {"key": "successful_status_codes", "value": [303, 808, 909]}]}]}'

And via SuperAPI:

# A GET request could look something like this:
curl -k \
-x http://realtime.oxylabs.io:60000 \
-U user:pass1 \
"https://stackoverflow.com/questions/tagged/python" \
-H "X-OxySERPs-Session-Id: 1234567890abcdef" \
-H "X-OxySERPs-Geo-Location: India" \
-H "Accept-Language: en-US" \
-H "Content-Type: application/octet-stream" \
-H "Custom-Header: custom header content" \
-H "Cookie: NID=1234567890; 1P_JAR=0987654321" \
-H "X-Status-Code: 303, 808, 909"

# A POST request would have the same structure but contain a parameter specifying that it is a POST request:
curl -X POST \
-k \
-x http://realtime.oxylabs.io:60000 \
-U user:pass1 "https://stackoverflow.com/questions/tagged/python" \
-H "X-OxySERPs-Session-Id: 1234567890abcdef" \
-H "X-OxySERPs-Geo-Location: India" \
-H "Custom-Header: custom header content" \
-H "Cookie: NID=1234567890; 1P_JAR=0987654321" \
-H "X-Status-Code: 303, 808, 909"

Parameter Values

Geo_Location

Full list of supported geo locations can be found in CSV format here.

"United Arab Emirates",
"Albania",
"Armenia",
"Angola",
"Argentina",
"Australia",
...
"Uruguay",
"Uzbekistan",
"Venezuela Bolivarian Republic of",
"Viet Nam",
"South Africa",
"Zimbabwe"

HTTP_Method

Universal Crawler supports two HTTP(S) methods: GET (default) and POST.

"GET",
"POST"

Render

Universal Crawler can render Javascript and return either a rendered HTML document or a PNG screenshot of the web page.

"html",
"png"

User_Agent_Type

Download full list of user_agent_type values in JSON here.

[
  {
    "user_agent_type": "desktop",
    "description": "Random desktop browser User-Agent"
  },
  {
    "user_agent_type": "desktop_firefox",
    "description": "Random User-Agent of one of the latest versions of desktop Firefox"
  },
  {
    "user_agent_type": "desktop_chrome",
    "description": "Random User-Agent of one of the latest versions of desktop Chrome"
  },
  {
    "user_agent_type": "desktop_opera",
    "description": "Random User-Agent of one of the latest versions of desktop Opera"
  },
  {
    "user_agent_type": "desktop_edge",
    "description": "Random User-Agent of one of the latest versions of desktop Edge"
  },
  {
    "user_agent_type": "desktop_safari",
    "description": "Random User-Agent of one of the latest versions of desktop Safari"
  },
  {
    "user_agent_type": "mobile",
    "description": "Random mobile browser User-Agent"
  },
  {
    "user_agent_type": "mobile_android",
    "description": "Random User-Agent of one of the latest versions of Android browser"
  },
  {
    "user_agent_type": "mobile_ios",
    "description": "Random User-Agent of one of the latest versions of iPhone browser"
  },
  {
    "user_agent_type": "tablet",
    "description": "Random tablet browser User-Agent"
  },
  {
    "user_agent_type": "tablet_android",
    "description": "Random User-Agent of one of the latest versions of Android tablet"
  },
  {
    "user_agent_type": "tablet_ios",
    "description": "Random User-Agent of one of the latest versions of iPad tablet"
  }
]

Account Status

Usage Statistics

You can find your usage statistics by querying the following endpoint:

GET https://data.oxylabs.io/v1/stats

By default, the API will return all-time usage statistics. Adding ?group_by=month will return monthly stats, while ?group_by=day will return daily numbers.

This query will return all-time statistics. You can find your daily and monthly usage by adding either ?group_by=day or ?group_by=month

curl --user user:pass1 'https://data.oxylabs.io/v1/stats'

Sample output:

{
    "data": {
        "sources": [
            {
                "realtime_results_count": "90",
                "results_count": "10",
                "title": "universal"
            }
        ]
    },
    "meta": {
        "group_by": null
    }
}

Limits

The following endpoint will give your monthly commitment information as well as how much of it has already been used:

GET https://data.oxylabs.io/v1/stats/limits

curl --user user:pass1 'https://data.oxylabs.io/v1/stats/limits'

Sample output:

{
    "monthly_requests_commitment": 4500000,
    "used_requests": 985000
}

Response Codes

References

• https://docs.oxylabs.io/rtc/source/html-crawler-api/index.html
• https://docs.oxylabs.io/rtc/index.html

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Do you know how to use Oxylabs E-Commerce Scraper API? This is the most comprehensive introduction from OxyLabs official.

Quick Start

Scraper API is built to help you in your heavy-duty data retrieval operations. You can use Scraper API to access various public pages. It enables effortless web data extraction without any delays or errors.

Scraper API uses basic HTTP authentication that requires sending username and password.

This is by far the fastest way to start using Scraper API. You will make a request to https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html using Realtime integration method from United States geo-location and retrieve already parsed data in JSON. If you wish to get page HTML instead of parsed data, simply remove parse and parser_type parameters. Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

Integration Methods

Scraper API supports three integration methods which have their unique benefits:

• Push-Pull. Using this method it is now required to mainain an active connection with our endpoint to retrieve the data. Upon making a request, our system is able to automatically ping users server when the job is done (see Callback). This method saves computing resources and can be scaled easily.
• Realtime. The method requires user to maintain an active connection with our endpoint in order to get the results successfully when the job is completed. This method can be implemented into one service while Push-Pull method is a two step process.
• SuperAPI. This method is very similar to Realtime but instead posting data to our endpoint, user can use HTML Cralwer as a proxy. To retrieve the data, user must set up a proxy endpoint and make GET request to a desired URL. Additional parameters must be added using headers.

Our recommended data extraction method is Push-Pull.

Push-Pull

This is the most simple yet the most reliable and recommended data delivery method. In Push-Pull scenario you send us a query, we return you job id, and once the job is done you can use that id to retrieve content from /results endpoint. You can check job completion status yourself, or you can set up a simple listener that is able to accept POST queries. This way, we will send you a callback message once the job is ready to be retrieved. In this particular example the results will be automatically uploaded to your S3 bucket named YOUR_BUCKET_NAME.

Single Query

The following endpoint will handle single queries for one keyword or URL. The API will return a confirmation message containing job information, including job id. You can check job completion status using that id, or you can ask us to ping your callback endpoint once the scraping task is finished by adding callback_url in the query.

POST https://data.oxylabs.io/v1/queries

You need to post query parameters as data in the JSON body.

curl --user user:pass1\
'https://data.oxylabs.io/v1/queries' \
-H "Content-Type: application/json" \
-d '{"source": "universal_ecommerce", "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "callback_url": "https://your.callback.url", "storage_type": "s3", "storage_url": "YOUR_BUCKET_NAME"}'

import requests
from pprint import pprint


# Structure payload.
payload = {
    'source': 'universal_ecommerce',
    'url': 'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html',
    'callback_url': 'https://your.callback.url',
    'storage_type': 's3',
    'storage_url': 'YOUR_BUCKET_NAME'
}

# Get response.
response = requests.request(
    'POST',
    'https://data.oxylabs.io/v1/queries',
    auth=('user', 'pass1'),
    json=payload,
)

# Print prettified response to stdout.
pprint(response.json())

<?php

$params = array(
    'source' => 'universal_ecommerce',
    'url' => 'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html',
    'callback_url' => 'https://your.callback.url',
    'storage_type' => 's3',
    'storage_url' => 'YOUR_BUCKET_NAME'
);

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://data.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");

$headers = array();
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close ($ch);
?>

HTTP method is currently not supported

The API will respond with query information in JSON format, by printing it in the response body, similar to this:

{
  "callback_url": "https://your.callback.url",
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "source": "universal_ecommerce",
  "start_page": 1,
  "status": "pending",
  "storage_type": "s3",
  "storage_url": "YOUR_BUCKET_NAME/12345678900987654321.json",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:01",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Check Job Status

If your query had a callback_url, we will send you a message containing a link to the content once the scraping task is done. However, if there was no callback_url in the query, you will need to check the job status yourself. For that, you need to use the URL in href under rel:self in the response message you received after submitting your query to our API. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321.

GET https://data.oxylabs.io/v1/queries/{id}

Querying this link will return the job information, including its status. There are three possible status values:

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321'

import requests
from pprint import pprint

# Get a response from the stats endpoint.
response = requests.request(
    method='GET',
    url='http://data.oxylabs.io/v1/queries/12345678900987654321',
    auth=('user', 'pass1'),
)

# Print prettified JSON response to stdout.
pprint(response.json())

<?php

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "http://data.oxylabs.io/v1/queries/12345678900987654321");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close ($ch);
?>

HTTP method is currently not supported

The API will respond with query information in JSON format, by printing it in the response body. Notice that job status has been changed to done. You can now retrieve content by querying http://data.oxylabs.io/v1/queries/12345678900987654321/results.

You can also see that the task has been updated_at 2019-10-01 00:00:15 – the query took 14 seconds to complete.

{
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "url": "sofa",
  "source": "universal_ecommerce",
  "start_page": 1,
  "status": "done",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:15",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Retrieve Job Content

Once you know the job is ready to be retrieved by checking its status, you can GET it using the URL in href under rel:results in our initial response. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321/results.

GET https://data.oxylabs.io/v1/queries/{id}/results

The results can be automatically retrieved without periodically checking job status by setting up Callback service. User needs to specfy the IP or domain of the server where the Callback service is running. When our system completes a job, it will send a message to the provided IP or domain and the Callback service will download the results as described in the Callback implementation example.

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321/results'

import requests
from pprint import pprint

# Get response from the stats endpoint.
response = requests.request(
    method='GET',
    url='http://data.oxylabs.io/v1/queries/12345678900987654321/results',
    auth=('user', 'pass1'),
)

# Print the prettified JSON response to stdout.
pprint(response.json())

<?php

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "http://data.oxylabs.io/v1/queries/12345678900987654321/results");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close ($ch);
?>

HTTP method is currently not supported

The API will return job content:

{
  "results": [
    {
      "content": "<!doctype html><html>
        CONTENT      
      </html>",
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "page": 1,
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

Callback

A callback is a POST request we send to your machine, informing that the data extraction task is completed and providing URL to download scraped content. This means that you no longer need to check job status manually. Once the data is here, we will let you know, and all you need to do now is retrieve it.

# Please see the code samples in Python and PHP.

# This is a simple Sanic web server with a route listening for callbacks on localhost:8080.
# It will print job results to stdout.
import requests
from pprint import pprint
from sanic import Sanic, response


AUTH_TUPLE = ('user', 'pass1')

app = Sanic()


# Define /job_listener endpoint that accepts POST requests.
@app.route('/job_listener', methods=['POST'])
async def job_listener(request):
    try:
        res = request.json
        links = res.get('_links', [])
        for link in links:
            if link['rel'] == 'results':
                # Sanic is async, but requests are synchronous, to fully take
                # advantage of Sanic, use aiohttp.
                res_response = requests.request(
                    method='GET',
                    url=link['href'],
                    auth=AUTH_TUPLE,
                )
                pprint(res_response.json())
                break
    except Exception as e:
        print("Listener exception: {}".format(e))
    return response.json(status=200, body={'status': 'ok'})


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

<?php
$stdout = fopen('php://stdout', 'w');

if (isset($_POST)) {
    $result = array_merge($_POST, (array) json_decode(file_get_contents('php://input')));

    $ch = curl_init();

    curl_setopt($ch, CURLOPT_URL, "https://data.oxylabs.io/v1/queries/".$result['id'].'/results');
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
    curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");

    $result = curl_exec($ch);
    fwrite($stdout, $result);

    if (curl_errno($ch)) {
        echo 'Error:' . curl_error($ch);
    }
    curl_close ($ch);
}
?>

HTTP method is currently not supported

Sample callback output

{  
   "created_at":"2019-10-01 00:00:01",
   "updated_at":"2019-10-01 00:00:15",
   "locale":null,
   "client_id":163,
   "user_agent_type":"desktop",
   "source":"universal_ecommerce",
   "pages":1,
   "subdomain":"www",
   "status":"done",
   "start_page":1,
   "parse":0,
   "render":null,
   "priority":0,
   "ttl":0,
   "origin":"api",
   "persist":true,
   "id":"12345678900987654321",
   "callback_url":"http://your.callback.url/",
   "url":"https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
   "domain":"de",
   "limit":10,
   "geo_location":null,
   {...}
   "_links":[
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321",
         "method":"GET",
         "rel":"self"
      },
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321/results",
         "method":"GET",
         "rel":"results"
      }
   ],
}

Batch Query

Scraper API also accepts multiple keywords per query, up to 1,000 keywords with each batch. The following endpoint will submit multiple keywords to the extraction queue.

POST https://data.oxylabs.io/v1/queries/batch

You need to post query parameters as data in the JSON body.

The system will handle every keyword as a separate request. If you provided a callback URL, you will get a separate call for each keyword. Otherwise, our initial response will contain job ids for all keywords. For example, if you sent 50 keywords, we will return 50 unique job ids.

Important! query is the only parameter that can have multiple values. All other parameters are the same for that batch query.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries/batch' -H 'Content-Type: application/json' \
 -d '@keywords.json'

import requests
import json
from pprint import pprint


# Get payload from file.
with open('keywords.json', 'r') as f:
    payload = json.loads(f.read())

response = requests.request(
    'POST',
    'https://data.oxylabs.io/v1/queries/batch',
    auth=('user', 'pass1'),
    json=payload,
)

# Print prettified response.
pprint(response.json())

<?php

$paramsFile = file_get_contents(realpath("keywords.json"));
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://data.oxylabs.io/v1/queries/batch");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $paramsFile);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");

$headers = array();
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close ($ch);
?>

HTTP method is currently not supported

keywords.json content:

{  
   "url":[  
      "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
      "https://books.toscrape.com/catalogue/soumission_998/index.html"
   ],
   "source": "universal_ecommerce",
   "callback_url": "https://your.callback.url"
}

The API will respond with query information in JSON format, by printing it in the response body, similar to this:

{
  "queries": [
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678900987654321",
      {...}
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "source": "universal_ecommerce",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678901234567890",
      {...}
      "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
      "source": "universal_ecommerce",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678901234567890/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "01234567899876543210",
      {...}
      "url": "https://books.toscrape.com/catalogue/soumission_998/index.html",
      "source": "universal_ecommerce",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/01234567899876543210/results",
          "method": "GET"
        }
      ]
    }
  ]
}

Get Notifier IP Address List

You may want to whitelist the IPs sending you callback messages or get the list of these IPs for other purposes. This can be done by GETing this endpoint: https://data.oxylabs.io/v1/info/callbacker_ips.

curl --user user:pass1 'https://data.oxylabs.io/v1/info/callbacker_ips'

import requests
from pprint import pprint

# Get response from the callback IPs endpoint.
response = requests.request(
    method='GET',
    url='https://data.oxylabs.io/v1/info/callbacker_ips',
    auth=('user', 'pass1'),
)

# Print the prettified JSON response to stdout.
pprint(response.json())

<?php

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://data.oxylabs.io/v1/info/callbacker_ips");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close ($ch);
?>

HTTP method is currently not supported

The API will return the list of IPs making callback requests to your system:

{
    "ips": [
        "x.x.x.x",
        "y.y.y.y"
    ]
}

Upload to Storage

By default, Ecommerce Spraper API's job results are stored in our databases. This means that you will need to query our results endpoint and retrieve the content yourself. Custom storage feature allows you to store results in your own cloud storage. The advantages of this feature are that you do not need extra requests to fetch results – everything goes directly to your storage bucket.

At the moment, we only support Amazon S3. If you want to use a different type of storage, contact your account manager to discuss the timeline.

In order to upload job results to your Amazon S3 bucket, you need to set up special permissions. To do that, go to https://s3.console.aws.amazon.com/ > S3 > Storage > Bucket Name (if don't have one, create new) > Permissions > Bucket Policy

You can find the bucket policy in this JSON or in the code sample area on the right. Do not forget to change the bucket name under YOUR_BUCKET_NAME. This policy allows us to write to your bucket, upload files for you, and know the bucket location.

To use this feature and you will need to specify two additional parameters in your requests. Learn more here.

The upload path looks like this: YOUR_BUCKET_NAME/job_ID.json. You will find the job ID in the response body that you will receive from us after submitting a request. In this example job ID is 12345678900987654321.

{
    "Version": "2012-10-17",
    "Id": "Policy1577442634787",
    "Statement": [
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
        },
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
        }
    ]
}

Realtime

The data submission is the same as in Push-Pull method, but with Realtime, we will return the content on open connection. You send us a query, the connection remains open, we retrieve the content, and bring it to you. The endpoint that handles that is this:

POST https://realtime.oxylabs.io/v1/queries

The timeout limit for open connections is 100 seconds. Therefore, in rare cases of heavy load, we may not be able to ensure the data gets to you.

You need to post query parameters as data in the JSON body. Please see an example for more details.

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" \
 -d '{"source": "universal_ecommerce", "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"}'

import requests
from pprint import pprint


# Structure payload.
payload = {
    'source': 'universal_ecommerce',
    'url': 'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html',
}

# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('user', 'pass1'),
    json=payload,
)

# Instead of response with job status and results url, this will return the
# JSON response with results.
pprint(response.json())

<?php

$params = array(
    'source' => 'universal_ecommerce',
    'query' => 'sofa',
);

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");

$headers = array();
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close ($ch);
?>

# URL has to be encoded to escape `&` and `=` characters. It is not necessary in this example.

https://realtime.oxylabs.io/v1/queries?source=universal_ecommerce&url=https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html&access_token=12345abcde

Example response body that will be returned on open connection:

{
  "results": [
    {
      "content": "<html>
      CONTENT
      </html>"
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "id": null,
      "page": 1,
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

SuperAPI

If you have ever used regular proxies for data scraping, integrating SuperAPI delivery method will be a breeze. You simply need to use our entry node as proxy, authorize with Scraper API credentials, and ignore certificates. In cURL it's -k or --insecure. Your data will reach you on open connection.

GET realtime.oxylabs.io:60000

SuperAPI only supports a handful of parameters since it only works with Direct data source where a full URL is provided. These parameters should be sent as headers. This is a list of accepted parameters:

If you need help setting up SuperAPI, get in touch with us at support@oxylabs.io.

curl -k \
-x realtime.oxylabs.io:60000 \
-U user:pass1 \
-H "X-OxySERPs-User-Agent-Type: desktop_chrome" \
"https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"

import requests
from pprint import pprint

# Define proxy dict. Do not forget to put your real user and pass here as well.
proxies = {
  'http': 'http://user:pass1@realtime.oxylabs.io:60000',
  'https': 'https://user:pass1@realtime.oxylabs.io:60000',
}

response = requests.request(
    'GET',
    'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html',
    auth=('user', 'pass1'),
    verify=False,  # Or accept our certificate.
    proxies=proxies,
)

# Print result page to stdout
pprint(response.text)

# Save returned HTML to result.html file
with open('result.html', 'w') as f:
    f.write(response.text)

<?php
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXY, 'realtime.oxylabs.io:60000');
curl_setopt($ch, CURLOPT_PROXYUSERPWD, "user" . ":" . "pass1");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close ($ch);
?>