Topic Related: #IncognitoMode #OnlinePrivacy #VPN #BrowsingHistory #InternetSecurity

• 🕵️ Incognito mode provides some privacy but isn't completely secure or private.
• 🌐 Different browsers have different names for incognito mode, but they all have the same purpose.
• 📛 Incognito mode doesn't save browsing history, autofill credentials, or cookies.
• 🚫 Your identity, location, IP address, and internet traffic are not hidden in incognito mode.
• 💻 Incognito mode can be useful for using multiple accounts, shopping for gifts, and booking travel without storing cookies.
• 🔐 For true privacy and anonymity, use a VPN to encrypt data and mask IP addresses.
• ⚠️ Web browsers often provide warnings about the limitations of incognito mode.

You must have heard people recommending the use of incognito mode when you are using a public or shared computer. For the longest time, people have enjoyed a sense of security when they browse in incognito mode. But unfortunately, people think they can browse any website they want in this mode, and the passwords will not be saved, or the website history will not be recorded.

Unfortunately, there’s more to incognito mode than what you have been led to believe. Also, it might not be as private and secure as you think. Therefore, you must learn how to delete incognito mode history, so the next person using your computer doesn’t get their hands on the sites you visited. Continue scrolling to learn more about the incognito mode.

What is the Meaning of Incognito Mode?

Every web browser has a different name for this type of web browsing. For example, Google Chrome calls it Incognito, but Safari and Firefox call it Private Browsing. But they all mean the same thing.

Essentially, incognito mode means you are browsing privately, and it does not save your browsing history. If you log into online websites, the credentials are not saved for autofill purposes. Also, the cookies are not saved.

However, this mode is not completely private because one can view your incognito mode browsing history using DNS cache or third-party programs.

Why is the Incognito Mode not as Private as you Think?

Browsing in incognito mode is safer than browsing in normal mode. However, it is not entirely private because it cannot do the following:

• Your identity is not hidden from websites when you create an account or sign in.
• Your location is not hidden in incognito mode. Websites can find your location based on your GPS or IP address.
• Your Internet traffic is not hidden from third parties like the government, your ISP, or the network admin at your university or company. These entities can view the log and find out which websites you have visited.
• Your IP address will not be masked; the websites you visit can see this. They can use this information to track you.
• Incognito mode does not secure your web traffic from hackers and other cyber attacks.
• It is possible for tech-savvy individuals to quickly find your incognito mode browsing history if you haven’t deleted it.

What private browsing blocks is recording your traffic. Therefore, your browser cannot record your traffic. But this mode doesn’t hide your IP address or encrypt your traffic. As a result, snoopers can easily see and collect your information by tracking your IP address.

Also, web browsers are upfront about this. For example, whenever you open a new incognito window on Chrome, your browser will warn you that your activities might be visible to the websites, school authorities or employers, your ISP, and the government. Microsoft Edge and Firefox offer similar disclaimers as well. But if you are using Safari, you will not get such warnings or disclaimers.

Therefore, do not get your guard down because you are browsing in incognito mode. Your data isn’t safe. If you are sharing your computer with someone else, they might not be able to see what you are doing.

When is it alright to use Incognito Mode?

Since the incognito mode is not as private as you think, you may wonder why you use this mode at all. Here are some instances when using incognito mode makes sense.

• If you need to use two accounts on the same platform simultaneously, you can log into one of the accounts in normal mode and the other in incognito mode. In this mode, you can log into the same platforms without logging out on the normal browser tabs. This function is handy if you want to separate your work and personal profiles.
• You can use incognito mode when shopping for gifts. No site data and cookies mean there will be no targeted advertising.
• When booking hotel rooms or flight tickets, you can use incognito mode. As this mode doesn’t store cookies, the hotel or airline website will not know that you have checked for the same dates before. This will prevent them from hiking their prices, and you will get better deals.

If you truly want to browse privately, use premium VPNs. VPNs encrypt the data sent and received between your device and the server. Also, they help to mask IP addresses and locations. So you can truly surf the web anonymously if you use VPN.

Conclusion

Incognito mode doesn’t stop the government, your ISP, your employer, or the websites you visit from tracking your activities. But they won’t be able to tell whether you have visited the website in normal or incognito mode. However, for true anonymity and privacy, use VPN when browsing the web.

Want to learn more about Blazing SEO's dedicated proxies? This article discusses their advantages and how they differ from other types of proxies, so let's reading!

About BlazingSEO's Dedicated Proxies

What are Dedicated Proxies?

Dedicated Proxies are proxies that are only used by one single person. Which is why they are called “Dedicated”. They are dedicated to the one person using them.

Advantages:

• A dedicated proxy is a proxy that is used only by you.
• It hides your IP(or MAC) address while adding security and protecting your privacy.
• Dedicated proxies deliver fully anonymous surfing on the internet. It is important to remember that you leave a digital footprint in the form of your IP address when using the web. This means that almost anyone can see you online, including hackers and spammers. A dedicated proxy server can almost guarantee your IP address will be invisible.
• Higher speed since no one else is working with the same proxy.
• Ability to bypass filters and firewalls since a proxy server can circumvent these types of blocked websites.
• Ability to unblock numerous applications and bypass numerous internet restrictions (i.e. unblock YouTube, Facebook, Twitter at work or school where the “IT administrator” has restricted your access to these and similar sites).
• Protect privacy for minimal worry about your personal information being exposed or compromised when you make purchases or surf on the web.
• Deliver a higher level of security on the internet since a dedicated proxy provides a unique internet address dedicated to only you.
• Avoid and block spam successfully.
• Get around geographic restrictions (Restricted content based on country).

These proxies are best used for account use because the proxy never changes (Unless you would like the proxies to change).

Dedicated can also be used for games as well. People can create a game account through the proxy and play the game using the proxy.

Dedicated is best because only one person is using it. This means faster speeds, and better reliability site wise for the proxy. To compare, if the proxy was shared (Semi-Dedicated/Rotating), the other person using the proxy might've already created an account for a certain site. As result, the other users on the proxy will not be able to use that same proxy on the same site and create another account.

You might ask why you wouldn't be able to use dedicated proxies for sneaker sites. Please read this article the reasoning behind this.

Typical Uses of Dedicated Proxies:

• Social Media
• Games
• Account Creation/Use
• General Browsing
• SEO

Locations:

• USA
• Brazil
• Germany
• UK
• India
• Canada
• Japan

What are Semi-dedicated Proxies?

Semi-Dedicated Proxies are proxies that are used by up to 3 people (Shared Proxy). This is why they are called “Semi-Dedicated”. They are shared by up to 2 other people. However, this allows the proxy the ability to be more affordable. You may not need a proxy entirely dedicated to only you, and this would be a good option if so.

Advantages:

• Inexpensive
• Sharing the proxy server allows you to share in the cost of that server
• They are useful for the user who requires several proxies but uses each proxy only for a few hours
• Proxies can be helpful with some of SEO software that need harvest data from internet (like keyword ranking or page rank).
• They can support both an HTTP and SOCKS service
• Proxies can also be used with web browsers and other apps that support proxy use
• Get around geographic restrictions (Restricted content based on country)

These proxies are best used for general browsing and they are not recommended for account creation/management on social media sites.

Authorization Types:

• IP Authorization – HTTP (3128), SOCKS (1080)
• Username/Password – (4444)

Locations:

• USA
• Brazil
• Germany

What is the Difference Between Dedicated Proxies and Sneaker Proxies?

What is the difference between Blazing SEO USA Dedicated proxies and their sneaker proxies? A lot — if you're a sneaker buyer. Dedicated proxies DO allow you to choose your “location” after purchasing, however, what most people do not realize is that the location is the GEOLOCATION, not the HOSTED LOCATION of the proxy.

The geolocation is an arbitrary location to appear as if you're from somewhere you're not, but your proxies will be located all around the USA in various datacenters. Sneaker proxies allow you to choose the HOSTED LOCATION after purchasing: either LA or NY. This allows you to select the city that is CLOSEST to your server/computer.

Having a proxy server closest to your server is the ideal situation for the fastest speeds possible. We do recommend you to use their sneaker proxies that are specifically purposed and optimized for shoe sites instead.

Dedicated Proxies Vs Rotating Proxies

BlazingSEO also provide the Rotating proxies, To learn the difference between dedicated Proxies and Rotating-proxies, I would suggest you learn this part at first,

What are Rotating Proxies?

Rotating proxies are proxies that are shared, but the IPs change every set number of minutes to another proxy IP from a larger pool of IPs. This is why they are called “Rotating”. They rotate IPs after a set number of minutes (10-120 minutes) to a different IP. These IPs are shared between other users through which can mean slower speeds and potential website blocks.

Blazing SEO rotating proxies are proxies that switch it's public IP address within an interval of 10 minutes between the switch. Once you have successfully purchased the rotating proxy package, you will be given a single IP address with different port numbers like the following

108.46.12.64:1081

108.46.12.64:1082

108.46.12.64:1083

108.46.12.64:1084

108.46.12.64:1085

The proxy IP has different port numbers and will show a different public IP address. When you load the proxy onto Firefox for testing, and when you browse to whatismyipaddress.com, you will not see 108.46.12.64 as your public IP, but rather something like this 106.39.12.61.

If you purchased 10 rotating proxy ports, that will grant you access to a pool of IPs that has 200 unique IPs. If you purchased 100 rotating proxy ports, that will grant you access to 2000 unique IPs in your pool. The pool of IP will remain the same throughout the entire billing cycle.

If you let your rotating proxy subscription expire at the end of the billing cycle, and re-purchase after that, this should grant you access to another pool of IP – do take note that the IPs are allocated randomly, thus we can't guarantee that all the IPs will be different from your previous pool.

Locations

• USA
• Brazil
• Germany

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Here is a guide to IPv4 proxies API Settings, We're using the sample of BlazingSEO's IPv4 proxies.

API Key: mZQjSDSDSD-SAMPALE

Welcome to the Blazing proxy API page! The following guidelines provide the information you need to get started using your API to automate proxy management.

You will need your unique API key (mZQjSDSDSD-SAMPALE) any time you use the API. Treat this like you would any password, because it is unique to you and allows access to your proxy product details. It is worth noting that it does not enable access to any of your personal or billing information.

The following sections provide instructions to setup and use your API to view and manage your inventory:

Setup

The API will only work if you have set up authorization using either IP Auth or User/Password, so before using it be sure that your IPs are authorized. You can find instructions to do so by clicking here.

Every request to the Customer Blazing Proxies API should contain an email and API key. If a user does not pass the email and API key or it is invalid – the API will respond with status 401 (Unauthorized).

Authorization type

Get your current authorization type:

https://blazingseollc.com/proxy/dashboard/api/auth-type/privateproxyreviews@gmail.com/mZQjUfW3

Change authorization type:

https://blazingseollc.com/proxy/dashboard/api/auth-type/privateproxyreviews@gmail.com/mZQjUfW3/authType

Change proxy authorization configuration with allowed types:

• {authType} – authorization type to set, possible values – [‘PW', ‘IP']
• {ipAuthType} (query parameter) – can be specified if authType = IP, possible values – [‘HTTP', ‘SOCKS']

View Whitelisted Authorized Addresses

Returns IPs that are authorized on your account to access proxies. It could take 10-30 minutes after an authorization request for the newly authorized IPs to show in this listing.

https://blazingseollc.com/proxy/dashboard/api/export/ips/privateproxyreviews@gmail.com/mZQjUfW3

List All Proxy IPs in Your Account

Every time you call the API URL you will be given the newest list of authorized proxies in your account.

Export Proxy IPs to a CSV file

https://blazingseollc.com/proxy/dashboard/api/export/4/all/privateproxyreviews@gmail.com/mZQjUfW3/list.csv

• additionalValues – You can specify this parameter to display additional values in the list (region,country,category)

https://blazingseollc.com/proxy/dashboard/api/export/4/all/privateproxyreviews@gmail.com/mZQjUfW3/list.csv?additionalValues=region,country

Get Proxy IPs by Country or Category

https://blazingseollc.com/proxy/dashboard/api/export/4/COUNTRY-CATEGORY/privateproxyreviews@gmail.com/mZQjUfW3/list.csv

Get list of Proxy IPs of certain package

• {COUNTRY} – package country in country code (us/de/br/fr/ca etc)
• {CATEGORY} – package category (static/semi-3/rotate/sneaker/block/shopify/static-res)

Adding and Removing IPs

Add new IP

https://blazingseollc.com/proxy/dashboard/api/ips/add/privateproxyreviews@gmail.com/mZQjUfW3/IPTOADD

• {IPTOADD} – IP address which should be enabled for proxy authorization (Note that it may take 10-30 minutes after the submission before your IP is ready for use).
• {IPTOADD1,IPTOADD2,…,IPTOADD20} – You can add multiple IP addresses (separated by commas) for proxy authorization.

Delete / Remove existing IP

https://blazingseollc.com/proxy/dashboard/api/ips/delete/privateproxyreviews@gmail.com/mZQjUfW3/IPTODELETE

• {IPTODELETE} – previously added proxy authorization IP which you now want to remove from your account. Once this is submitted, the IP will no longer be authorized after 10-30 minutes.

Replacing Proxy IPs

Get available replacements

https://blazingseollc.com/proxy/dashboard/api/available-replacements/privateproxyreviews@gmail.com/mZQjUfW3

Replace a Single Proxy IP

Replaces single Proxy IP, and returns # of available replacements for Proxy IPs in the same package type.

https://blazingseollc.com/proxy/dashboard/api/replace-ip/privateproxyreviews@gmail.com/mZQjUfW3/IP_Single

• {IP} – Proxy IP to replace (will subsequently reduce the replacements amount)

Replace multiple Proxy IPs

Replaces multiple IPs at the same time. The list of ips must be comma-separated and provided as query parameter ‘replace';
This also returns a list of successfully flagged (IPs to be replaced by cron job) IPs, and # of available replacements (similar to ‘Get Available Replacements' API method).

https://blazingseollc.com/proxy/dashboard/api/replace-multiple-ips/privateproxyreviews@gmail.com/mZQjUfW3?replace=IP1,IP2,IPn

Third-party Tool Setup: GSA SER

To setup GSA SER to automatically grab your proxies at a certain interval, follow the instructions below. This is useful for semi-dedicated and dedicated users who request proxies to be changed every 30 days, or if a proxy dies by chance — GSA will automatically grab the new proxies for you 🙂

1. 1. Open up Proxy Settings tab
   2. Click on Add/Edit Proxy
   3. Right click the list on the left and select ‘Check None’
   4. Click the Add button
   5. Add your API URL (above) and select the option to ‘Add as private proxy type’
   6. Go back to the main Proxy settings page and click on the Options tab
   7. Set ‘Automatically search for new proxies every’ — to 120 minutes
   8. Done!

Rotation Options

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Want to know which locations the BlazingSEO proxy supports? This article will give you a detailed introduction.

What Locations Do You Offer for Proxies? (Country – City)

The following countries and cities are available through our Blazing SEO proxies as of May 2021.  We continue to expand our locations.  If you have a specific request that you do not see on this list, you can email our address and speak to a member of our sales team.

• United States (17):  Los Angeles, Chicago, San Francisco, Dallas, New York City, Secaucus, Denver, Atlanta, San Jose, Omaha, Des Moines, Salem, Seattle, Buffalo, Anchorage, Miami, Phoenix.
• Argentina, Buenos Aires
• Australia (2):  Sydney, Melbourne
• Belgium, Brussels
• Brazil, Sao Paulo
• Canada, Toronto
• Colombia, Bogota
• Egypt, Cairo
• France (3): Paris, Marseille, Lyon
• Germany (2):  Frankfurt, Berlin
• India, Mumbai
• Indonesia, Jakarta
• Italy, Rome
• Japan, Tokyo
• Mexico, Mexico City
• Netherlands, Amsterdam
• Pakistan, Islamabad
• Philippines, Manila
• Poland, Warsaw
• Saudi Arabia, Riyadh
• Seychelles, Victoria
• South Africa, Cape Town
• South Korea, Seoul
• Spain, Madrid
• Taiwan, Taipei
• Thailand, Bangkok
• United Kingdom, London
• Vietnam, Hanoi

If you put ANY of your IP addresses you have from us at the top right of this PAGE you will find that your IP address is in fact in the correct location. Since ARIN owns IP addresses, they govern this location and it is up to third parties, such as MaxMind, to honor this governance – which unfortunately is not the case.

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

How much do you know about Blazing SEO's sneaker prxoies? Before you purchase, please read this detailed manuals!

What Are Sneaker Proxies?

A sneaker proxy is a dedicated proxy that allows for customers who are looking to purchase a limited shoe to be able to do so through the sneaker sites.

Typically users will purchase as many sneaker proxies as their budget allows, a VPS, and a bot. However, these 3 components are not necessarily required. In theory, they could try using their home computer and try going to the shoe site normally purchase. Their chances might not be as great though, because they would only be using one IP (their home IP).

For the best chance at success, users need multiple sneaker proxies, and a sneaker server, which we provide. The server is in the same datacenter as the website server and proxies, allowing for the absolute fastest connection to the footsites. Giving the best chance to cop sneakers. Also, the more proxies you have, the less chance that the footsite will block all the proxies, and in a result, you have a much greater chance of getting the shoes you want.

Advantages:

• The proxy is only used by you (not shared).
• Can be used with shoe servers for fastest possible speeds
• In same datacenters as shoe site servers.
• Higher speed since no one else is working with the same proxy.
• Ability to bypass filters and firewalls since a private proxy server can circumvent these types of blocked websites.
• Protect privacy for minimal worry about your personal information being exposed or compromised when you make purchases or surf on the web.
• Deliver a higher level of security on the internet since a private proxy provides a unique internet address dedicated to only you.

These proxies are best used for account use because the proxy never changes (Unless you would like the proxies to change).

Locations:

USA

What Is the Difference Between Dedicated Proxies and Sneaker Proxies?

What is the difference between our USA Dedicated proxies and our sneaker proxies? A lot — if you're a sneaker buyer.

Dedicated proxies DO allow you to choose your “location” after purchasing, however, what most people do not realize is that the location is the GEOLOCATION, not the HOSTED LOCATION of the proxy. The geolocation is an arbitrary location to appear as if you're from somewhere you're not, but your proxies will be located all around the USA in various datacenters.

Sneaker proxies allow you to choose the HOSTED LOCATION after purchasing: either LA or NY. This allows you to select the city that is CLOSEST to your server/computer. Having a proxy server closest to your server is the ideal situation for the fastest speeds possible.

• https://portal.blazingseollc.com/en/support/solutions/articles/64000256618-what-are-sneaker-proxies
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256675-what-is-the-difference-between-dedicated-proxies-and-sneaker-proxies
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256676-how-many-tasks-can-i-run-on-a-blazing-server
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256677-what-are-the-proxy-speeds-for-sneaker-purchases
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256678-how-do-i-prevent-my-ip-from-being-banned-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256635-why-is-my-aio-proxy-tester-showing-that-a-connection-timed-out
• https://portal.blazingseollc.com/en/support/solutions/articles/64000263274-which-proxies-work-for-shoe-or-foot-sites-

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

If you have decided to purchase the Blaze SEO proxy, please check out the purchasing process and user manual.

BlazingSEO Purchase

Are There Any Discount Codes Available?

We have the code BHW for 5% recurring on our proxies and 5OFF for a 5 % recurring discount on our servers.

What do I do If I Do Not Receive A Verification Code?

If you are not receiving your verification code after signing up or after signing in from a new device, please contact our support team by email.

Why Are My Proxies Are Not Assigning?

There may not be some locations available at this time. We highly recommend assigning your proxies to the “Mixed” location preference in your dashboard here. Once the proxies are assigned to the respective “Mixed” location setting then the proxies will be provided to you. If for some reason they are not provided to you after selecting the “Mixed” location after 10 minutes please let us know right away.

Why Are My Proxies Not Showing in My Dashboard or Are Blank?

If you have purchased proxies, and find that there are blank areas where proxies should be in your proxy dashboard, then this means that you have selected a location that we may not have in stock currently.

In this case, you have a few options:

1. You may wait up to 48+ hours, and we may receive more stock in, but it is not guaranteed.
2. Recommended –  Change your proxy locations to Mixed under the Location Preferences tab in the proxy dashboard here.

• You should receive the proxy in about 10 minutes after changing the location to mixed, and saving.
• Note: Setting to Mixed location will provide better subnet diversity, which is the best thing to have when connecting to websites. You will spread across more ISPs, ASNs, and A+B+C class subnets.  

How Do I Purchase More Replacements?

We currently offer one free proxy replacement, per proxy, per 30 days. So if one of your proxies is not working to a site you using the proxies on, then you may use one of your replacements.

If you have used all your replacements, then you will receive more than 30 days from the day that you used the replacements. So if you used the replacements 15 days into your 30 day billing period, then you will receive the new replacements 15 days into your next 30 day billing period.

If you would not like to wait for this, you are able to purchase additional replacements. Currently, you will need to speak with our support team and they will be able to assist you in completing this process at the link.

We will be implementing a client friendly way of doing this soon.

BlazingSEO Account

What do I do After I Complete the Purchase Process?

After you have completed the purchase, to access your proxies, follow the instructions below:

1. Please login your account here. You may also follow the below video guide on how to access your client area too

2.  After logging in, you may access your proxies here. You may also use the Shortcuts link found on the bottom left corner of, “Log In To Proxy Dashboard“, to login to your proxy dashboard as well.

If this is your first time logging in, you will need to assign your proxy's location in, and after saving your assigned proxy's locations, your proxies will appear on your proxy dashboard.

Make sure you authorize your computer's IP address prior to usage, so that your proxies will grant usage access to the IP that you authorize. You may follow this video guide on how to do just that: How to authorize your IP address for proxies:

Can You Tell Me How to Login to My Dashboard and Manage My Account?

This article highlights Login Areas, and Account Management locations for your account.

Login Option 1 – Control Panel 

Proxy Dashboard Login (found under Shortcuts>”Log In to Proxy Dashboard”)

• To Manage Your Services, 
• Pay / Download Invoices, 
• Change Account Password
• Update Billing Information / Address

Please login using your registered email address and the password you've set during registration. If you have forgotten your password, please use “Forgot Password?” to reset your password.

Login Option 2 – Proxy Dashboard 

To View and Administer Your Proxies, Authorize Your IP Addresses, Set Authorization Type (IP Authorization or Username/Password Authorization),

• Upgrade A Proxy Subscription
• Purchase A New Proxy Subscription
• Replace Proxies, Proxy Location Settings 
• Proxy Settings

Please login using your registered email address and the password you've set during registration.

How do I Reset or Change My Password?

If you would like to change your account password, you may do so at this link.

How do I Remove a Cancellation Request?

If you have accidentally requested cancellation for your respective subscription you can remove it through your client area. Please follow the steps below on how to remove your cancellation request.

1. Login to your client area

2. Then proceed to your active service(s) that you request a cancellation for

3. From there you will select “Continue billing”

4. Next select “Ok” after this window shows

BlazingSEO Refund & Cancel

Does Blazing SEO Process Refunds?

Blazing SEO does not guarantee refunds for lack of usage or dissatisfaction with the Services. The refund policies can also be referred to in their TOS and AUP.

Trial Refunds

We offer a 2 day trial for our products so that you are able to test our proxies for your specific use-case prior to purchasing. If you find that the proxies do not work for you purpose, you can contact and request a refund.

Once you pass the two-day trial period you can continue to use them and they will automatically renew monthly until you cancel. If you believe you are entitled to a refund due to a special circumstance, you can submit a refund request but we do not guarantee refunds to any individual.

3/6/12 month Refunds

If you have purchased a 3/6/12 month service with us, then please contact our support for details on refunds.

How Do I Cancel a Proxy Trial?

If you would like to cancel your proxy trial service, and not renew, you will need to request a refund and your service will be cancelled at that time. We do offer two day trials and we encourage you to use those two days to fully test the proxies for your use case.  Refunds can be processed if you notify us within the first two days of sign up.  If you decide that particular type of proxy may not meet your needs but there is another type you'd like to purchase, our support team can work with you to determine the best type of proxy to recommend based on your use case.  If you have questions, please email.

How Do I Cancel My Service?

Video on how to cancel:

Text instructions:

1. Access your Client Area.

2. You will then select the active service you want to cancel.

3. Select ‘Request Cancellation'.

4. Enter a reason for the cancellation, then click the final Cancellation button.

This cancellation request will ensure that your service is cancelled at the end of the 30 day period of the service, and that you will not be charged again for the service.

*Please double check and make sure all of your payment subscription are cancelled through PayPal. You may cancel your recurring subscription payment profile in your PayPal account by following this guide here.

Please make sure your subscription is properly cancelled in PayPal, in order to prevent PayPal from automatically sending payment on your behalf for renewal.

• https://portal.blazingseollc.com/en/support/solutions/articles/64000256658-are-there-any-discount-codes-available
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256674-what-do-i-do-if-i-do-not-receive-a-verification-code
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256634-why-are-my-proxies-are-not-assigning
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256616-why-are-my-proxies-not-showing-in-my-dashboard-or-are-blank
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256619-how-do-i-purchase-more-replacements-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256665-what-do-i-do-after-i-complete-the-purchase-process
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256605-can-you-tell-me-how-to-login-to-my-dashboard-and-manage-my-account
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256664-how-do-i-reset-or-change-my-password
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256662-how-do-i-remove-a-cancellation-request-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256660-does-blazing-seo-process-refunds
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256670-how-do-i-cancel-a-proxy-trial
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256659-how-do-i-cancel-my-service-

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Can you believeBlazing SEO provides IPv6 Proxies? Through our article you will know if it's worth having their IPv6 Proxies.

What Are IPv6 Proxies?

IPv6 Proxies are proxies that are similar to our Dedicated proxies in the sense that they are only used by one single person. However, the Dedicated proxies are in IPv4 format, and the IPv6 are in a different format (explained below).

IPv4 has been the standard IP address used for most device's public IP addresses and proxies. All of our proxies are in IPv4 format. Example – 192.117.63.126 However, IPv4 addresses are actually running out entirely across the internet as more and more devices are introduced into the market and IPv6 will be the next step once the IPv4 runs out.

IPv6 looks VERY different from IPv4. You'll know its an IPv6 just by glancing. You can see that IPv6 is much longer and more complicated. As well as being separated by a colon “:”. However, IPv4 and IPv6 work in a very similar way when being used as a proxy.
Example – 2001:0db8:85a3:0000:0000:8a2e:0370:7334

IPv6 support is available on many major sites, and because IPv6 is very new, many of the proxies can be cleaner than our IPv4, making it a better option.

Advantages:

• A private proxy is a proxy that is used only by you.
• It hides your IP(or MAC) address while adding security and protecting your privacy.
• Private proxies deliver fully anonymous surfing on the internet. It is important to remember that you leave a digital footprint in the form of your IP address when using the web. This means that almost anyone can see you online, including hackers and spammers. A private proxy server can almost guarantee your IP address will be invisible.
• Higher speed since no one else is working with the same proxy.
• Ability to bypass filters and firewalls since a private proxy server can circumvent these types of blocked websites.
• Ability to unblock numerous applications and bypass numerous internet restrictions (i.e. unblock YouTube, Facebook, Twitter at work or school where the “IT administrator” has restricted your access to these and similar sites).
• Cleaner than IPv4
• Protect privacy for minimal worry about your personal information being exposed or compromised when you make purchases or surf on the web.
• Deliver a higher level of security on internet since a private proxy provides a unique internet address dedicated to only you.
• Avoid and block spam successfully.
• Get around geographic restrictions (Restricted content based on country).

These proxies are best used for account use because the proxy never changes (Unless you would like the proxies to change).

IPv6 is best because only one person is using it. This means faster speeds, and better reliability site wise for the proxy. To compare, if the proxy was shared (Semi-Dedicated/Rotating), the other person using the proxy might've already created an account for a certain site. As result, the other users on the proxy will not be able to use that same proxy on the same site and create another account.

Typical Uses of IPv6 Proxies:

• Social Media
• Games
• Account Creation/Use
• General Browsing
• SEO

Locations:

• USA

• https://portal.blazingseollc.com/en/support/solutions/articles/64000256608-what-are-ipv6-proxies
• https://portal.blazingseollc.com/en/support/solutions/articles/64000262778-what-sites-will-ipv6-ips-work-for-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000262779-how-many-locations-do-you-offer-for-ipv6-proxies-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000262776-how-does-your-authorization-scheme-work-for-ipv6-proxies-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000262777-what-are-your-bandwidth-limits-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000262780-do-you-have-rotating-ipv6-proxies-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000262781-do-you-offer-replacements-on-ipv6-proxies-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000262782-where-can-i-find-additional-information-about-ipv6-proxies
• https://portal.blazingseollc.com/en/support/solutions/articles/64000262787-what-additional-information-can-you-provide-about-blazing-seo-s-ipv6-proxy-offering-

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

This is the using manuals of BlazingSEO Proxies that you must not miss！Includind how to use their dashboard, test the proxy on the browser, and so on…

How to Use BlazingSEO Proxies

How do I Use Proxies?

You may use proxies on a web browser or through a software that supports proxy usage.

How to Use Proxy on Firefox:

How to Use Proxy on Chrome: 

How do I Authorize My Proxies?

It is important to understand that authorization for the proxies is required to prevent non-authorized users from being able to use the proxies. It keeps the proxies secure to you. That is why the proxies will not work if you do not authorize correctly.

There are a few different types of authorization options we provide depending on your use case. Please find the authorization type you need below for instructions.

IP Authorization: 

• IP Authorization uses the port 3128 with the proxies.

1. Login to Proxy Dashboard

2. To find your PC/Device's IP, open a web browser

3. Go to Proxy Dashboard, and select ‘Via IP', ‘HTTP', then enter your IP into the blank box in the screenshot on ‘2.'. Then click ‘Add IP'.

4. The proxies will not work for 30 minutes until your IP is fully authorized in the system. The IP will be added to the system after 30 minutes, and then you will be able to use the proxies.

5. Complete! You may then use the proxies as needed.

• Video on how to use proxies in browser:

Username/Password Authorization: 

• Username/Password Authorization uses the port 4444 with the proxies.

1. Login to Proxy Dashboard

2. Go to Proxy Dashboard, and select ‘Username/Password'

3. The proxies will not work for 30 minutes until the system changes the authorization type. The authorization type will be adjusted in the system after 30 minutes, and then you will be able to use the proxies.

4. Complete! You may then use the proxies as needed.

• Video on how to use proxies in browser:

SOCKS Authorization:

• SOCKS Authorization uses the port 1080 with the proxies. 

1. Login to Proxy Dashboard

2. To find your PC/Device's IP, open a web browser

3. Go to Proxy Dashboard, and select ‘Via IP', SOCKS', then enter your IP into the blank box in the screenshot on ‘3.'. Then click ‘Add IP':

How do I Use and Test My Proxies in a Browser?

Before testing proxies in a browser, you'll want to make sure they are authorized in your dashboard. You may either use IP authorization method or username/password authorization method for this purpose.

Testing with a Chrome Browser

1. Open Chrome, and click on the ellipsis (3 dots) in the upper right corner of your browser and choose ‘Settings'

2. Scroll to bottom of Settings page, and select ‘Advanced'

3. Scroll to bottom of Advanced section, and select ‘Open your computer's proxy settings'

4. Select the ‘Use a proxy server' option, then fill in your IP address and Port as shown

5. To confirm that your proxy is loaded into the browser, go to google and type ‘my ip'. Your browser should now show the proxy IP address instead of your regular one – nice work!

6. If your proxy IP shows up in google, you can then try loading any website just as you normally would when browsing from your computer. Voila!

• To remove proxy, just reverse the steps above.

Testing with a Firefox Browser

1. Open Firefox, and click on the hamburger icon (aka 3-bar menu) in the upper right corner and select ‘Options'

2. Scroll to bottom of Options page, and select ‘Settings'

3. Select the ‘Manual proxy configuration bubble' and set the values in the HTTP field – be sure to check the box as shown so it applies to FTP and HTTPS as well!

4. To confirm that your proxy is loaded into the browser, go to google and type ‘my ip'. Your browser should now show the proxy IP address instead of your regular one – nice work!

5. If your proxy IP shows up in google, you can then try loading any website just as you normally would when browsing from your computer. Voila!

• To remove proxy, simply reverse the steps above.

How can I Get a Diverse Subnet for My Proxies?

Our system is automatically set to provide you with a diverse subnet for your proxy’s IP. If you still find the proxy IPs to be lacking in diversity, you may proceed to replace the proxy’s IP via your proxy DASHBOARD accordingly. Kindly login to your proxy dashboard and browse to the “Proxy Replacements” tab and replace the IP that you find unsuitable. You can also follow this video guide HERE on how to replace your proxies.

DISCLAIMER: Each replacement will use a single proxy replacement credit. If you purchased 10 proxies, you are entitled to 10 proxy replacements per billing cycle. The replacement credit will be replenished on the next billing cycle. Kindly use the replacements cautiously, as we are unable to replenish the credit manually.

How do I Test Proxies?

To test the proxies, you may simply follow this video guide:

How to Use Proxy Dashboard

How do I Replace Proxies?

At Blazing SEO, we know that sometimes, you need new proxies. Therefore we offer free replacements to all of our customers for all of our proxies. We offer one free proxy replacement per month, per proxy in your package. For example, if you have 20 proxies, you are entitled to 20 free replacements each month. Your replacement balance is replenished 30 days from the date of replacement respectively.

If you need to replace your proxies, please follow the steps below:

1. Log in to the Blazing SEO proxy dashboard.

2. On the dashboard home screen, select “IPv4 Proxies” to drop down the menu. Then select the “Proxy Replacements” option.

3. Once you are on the proxy replacements page, you have two options to replace your proxies:

• Option 1 (Individual): You can select individual proxies from your list. Simply select the “Replace” button to replace that proxy with a new proxy.

• Option 2 (Multiple): You can manually enter the IPs you want to replace in the box shown below. This allows you to replace multiple proxies at the same time. Input the IPs you want to replace and then press “Submit”.

4. Once you have processed the request, our system will provide you with a new IP in your dashboard. Please note that this replacement process can take up to 10 minutes before the new proxy is delivered.

How do I Cancel Proxies?

Before cancelling your proxies, please refer to this article for information on troubleshooting proxies. We are happy to work with you on any issues you are facing with your proxies whether it is issued with functionality or perhaps financial costs.

If you are certain you would like to cancel your proxies, there are two methods available for you to do so. Please see the methods listed below:

Cancelling Through The Proxy Dashboard:

1. Login to your proxy dashboard.

2. Select “Upgrade Your Plan” under “Manage Plan”:

3. Click on the red Cancel button:

4. A warning will pop up. You will then enter your cancellation reason, then click ‘Yes, please cancel my proxies':

Cancelling Through The Client Area:

1. The client area can be accessed through the link here.

2. You will then click ‘Services':

3. You will then find the active service you are looking to cancel:

4. You can then click ‘Request Cancellation' here:

5. On the next page, you will enter your cancellation reason, then click ‘Request Cancellation':

How do I Upgrade My Proxies?

If you would like to upgrade your proxies, please follow the steps below:

1. Login to your Proxy dashboard

2. Select “Upgrade Your Plan” under “Manage Plan”:

3. If you wish to upgrade to a 15 proxies package, from 10, please enter the “New Amount” as 15 for the package that you want to upgrade, click on the upgrade button to proceed:

4. An upgrade invoice will be generated. You would need to pay the invoice to activate the upgraded proxies:

5. Once the invoice is paid, the proxies will automatically be added to your account and you will be able to select locations!

How do I Downgrade My Proxies?

If you would like to downgrade your proxies, please follow the steps below:

1. Login to your Proxy dashboard

2. Select “Upgrade Your Plan” under “Manage Plan”

3. To reduce the number of proxies in your account, enter the new number you want going forward.  For example, if you currently have 20 proxies and wish to downgrade to a new total of 15 in your proxy package, enter the number '15' in the field shown below. Then click on ‘Downgrade‘ to proceed.

4. Now you can optionally specify which IPs you would like to keep, if some are working better than others

• Enter the list of IPs as directed. This should include the IP only, and should not include the port number.
• NOTE: If you leave this field blank, then IPs will be randomly selected for removal from your account.

5. Click ‘Downgrade‘ and you are all set!  A credit will be added to your account to prorate for the remainder of the billing period.

Please note that proxies will be downgraded at the end of the billing cycle and can be used till then.

Why Are My Proxies Not Working on Some Sites?

Does your proxy work on some sites, but does not work on others?

Did your proxy work in the past, but now it does not work?

If you answered ‘Yes' to either of the above questions, then it is likely that your proxies are banned (or ‘blocked') on the website you are using them on. In this part, we will talk about why this happens, as well as what you can do about it.

If you answered ‘No' to both of the above questions, then please visit our article about why your proxies might not be providing connectivity to any sites. You can read about it here. 

Why did I Get Banned?

There are a number of reasons as to why a proxy can get banned, some of which are predictable and obvious, while other reasons are only theoretical guesses. Here are a few reasons:

1. You are using your proxies too fast

One of the main reasons for getting banned on a website is that you have your software configured to run too fast at a particular website. Websites will not tolerate significant, parallel, connections as it may cause harm to their services. Imagine a scenario where a website allowed a single IP address to connect to it at [1 million times] per second – wow! It's clear that they would place a ban, or block, on this IP address so that it cannot connect to their website anymore.

What is the appropriate speed that you can connect to the site you are using the proxies on? Unfortunately, that's a hard question to answer, but what we recommend is that people run their own experiments to find that rate:

1. Set your software to run at a slow speed that will not cause a ban (ex: 1 request per hour)
2. If you do not receive a ban at that rate, then increase it higher (ex: 2 requests per hour)
3. Repeat this until you receive a ban, and then set your rate at the previous rate before

 2. You are running at a slow rate and some of your proxies work, but others do not

The next question we frequently receive is customers asking why some of their proxies work, and why some of their proxies are banned. This is an unfortunate side effect of how IPv4 address schema works and why it impacts proxies.

There are two terms used frequently in the proxy business:

1. Subnets (A, B, C subnets – read here for more info)
2. ISPs (or ASNs). Example:  Verizon is an ISP;  Sprious is an ISP; etc

Why do subnets and ISPs (Internet Service Providers) matter in the terms of proxies? It's because it makes it easier for the website you are using the proxy on to see that you are sharing the same ISP or subnet as other proxy customers of ours that are also using the proxies for the same use case.

Given that the proxy industry is driven largely by certain niches (Scraping, social media marketing, and others), it's very common that many customers are using proxies for the same use case as yours. Therefore, when this happens, your proxy speeds can negatively impact each other because you share the same subnet or ISP as another customer using the same use case.

What can you do about this? Please read on!

What Can I do About My Proxies Being Banned?

In this section we will cover various options you have to be more successful with your proxies with Blazing SEO:

1. From day one, Blazing SEO made it a top priority that our customers could have the freedom to automatically replace their proxies if any of them were not working for any reason.

2. If you run out of replacements, please contact us and our support team will assist you with restoring the replacements.

3. Slow your proxy speed down:  As discussed in the previous section, sometimes it just requires you to slow down your rates a little in order to achieve maximum throughput.

4. Contact us – seriously!

• If pricing is a problem at the rates you are running your proxies now, please contact us and we can discuss custom pricing that will make you the most profitable.
• There are a number of strategies we can offer you (at no additional cost) that will give your account better proxies. This includes only giving you IPs from certain ISPs/ASNs and only IPs from certain subnets that appear to work better for your use case.
• The reason that certain IPs on certain ISPs or subnets work better than others is that some ISPs that we partner with are on “blacklists” for certain use cases. By configuring your account to receive IPs from ISPs/subnets that work for you, you will inherently avoid getting IPs from the “blacklisted” ISPs/subnets

1. How can I switch between proxies in my browser?

We recommend using the FoxyProxy Add-on for Chrome or Firefox to easily switch between proxies in your browser.

2. What happens if I use a dynamic changing personal IP? 

If you have an IP address that is dynamic (changing), then you will be unable to authorize your proxies via IP authorization in the proxy dashboard. You will need to instead use Username/Password authorization.
However, it is not currently possible to use Username/Password with our Rotating Proxies, only IP authorization. So if you have a dynamic IP, then rotating proxies will not work for you.

3. How do I know if I set up my proxies correctly?

Many users report that their proxies are “not working”, but in actuality, it is a server or user error that is the cause of it.  If you see the proxy IP at whatismyipaddress.com then you know you have set it up correctly.

4. When should I replace a proxy in my account?

In rare occasions, the proxy may go offline due to unforeseen circumstances. If you urgently need to use all the available proxies in your account, you may replace the proxy that isn’t working through your proxy DASHBOARD and you can also follow this video guide for reference below. Also, if you find a particular IP address to be unsuitable to your usage, you may also replace the respective IP via your proxy dashboard.

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

We know BlazingSEO provides SOCKS protocol, but do you know how to use it correctly? This article will be a complete guide for you.

Do Your Proxies Work with SOCKS Protocol?

Our dedicated and semi-dedicated proxies support SOCKS. You will need to change the proxy’s port to 1080 for SOCKS connectivity. You may also find the following video guides on how to use SOCKS protocol with blazing proxies in your browsers(Chrome, Firefox).

You can only use IP authorization along with SOCKS. Unfortunately, username/password authorization method is not available for SOCKS for our proxies.

How do I Use SOCKS Protocol?

Dedicated and semi-dedicated proxies support SOCKS protocol.

For some websites and use-cases, you may need to switch from HTTP/HTTPS and instead use SOCKS protocol. We offer SOCKS protocol to assist with these specific use-cases. When using SOCKS, you are only able to use IP authorization as we currently do not offer username/password for SOCKS protocol.

1. Login to your proxy dashboard.

2. On the right side of the proxy dashboard home page,  select “Via IP” under Authorization Type and then select SOCKS.

3. If you have not already authorized your IP, please proceed to authorize your IP.  f you do not know how to authorize your IP you may simply follow this video guide here.

You may also find the following video guides on how to use SOCKS protocol with Blazing SEO proxies in your browser.

Chrome

Firefox

• https://portal.blazingseollc.com/en/support/solutions/articles/64000263275-do-your-proxies-work-with-socks-protocol
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256630-how-do-i-use-socks-protocol-

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Do you know what Proxy Pilot does and how it works? By learning this article, you will master the above knowledge and know-how to troubleshoot Proxy Pilot.

What Is Proxy Pilot?

Description

Proxy Pilot is a software that completely manages your proxy lists intelligently. This means you can build your system to be more flexible by using this “microservice” that handles a significant workload that is otherwise put into your main code base.

Features

The list of current features are:

• Specific cooldowns between each proxy attempt
• Specific cooldown after a ban is detected
• Ban detection (the certificate you install on your server allows us to decrypt your traffic as a man-in-the-middle attack and read the resulting HTML for ban messages), specific per each site
• Optimal proxy usage (round-robin at first, and then once various cooldown timers are started, it will use the proxies that are most cooled-down)
• Automatically downloads your proxy list from a proxy API endpoint
• Geo-targeting, if you use multiple countries in your proxy list
• Advanced statistics powered by ELK (see below)

Powered by ELK

The ELK stack is one that has grown massively in popularity over the past few years… and Proxy Pilot is powered by it! See the amazing statistics that you will have at your fingertips through Kibana’s amazing visualizations:

const fs = require('fs');
const path = require('path');

/* Note: please replace the ./public/ca.pem with a local path to the ca.pem file on your server and simply remove lines with X-ProxyPilot-Region if you don’t need to use geo-targeting feature*/

// const cert = fs.readFileSync(path.resolve(__dirname, './public/ca.pem'));
const request = require('request');
request(
    {
        url: 'https://www.amazon.com/dp/B07HNW68ZC/',
        proxy: 'http://PROXY_LOGIN:PROXY_PASS@PROXY_IP:PROXY_PORT',
        // ca: cert,
        // tunnel: true,
        followAllRedirects: true,
        timeout: 60000,
        method: "GET",
        rejectUnauthorized: false,
        gzip: true,
        // proxyHeaderWhiteList: ['X-Sprious-Region'],
        headers: {
            // 'X-ProxyPilot-Region': 'GB',
            'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36',
        }
    },
    (err, response, body) => {
        console.log(err, body);
    }
);

const puppeteer = require('puppeteer');
const proxyChain = require('proxy-chain');

(async () => {
    const anonymizeProxy = await proxyChain.anonymizeProxy('http://PROXY_LOGIN:PROXY_PASS@PROXY_IP:PROXY_PORT');

    const browser = await puppeteer.launch({
        args: ['--no-sandbox', '--disable-setuid-sandbox', '--proxy-server=' + anonymizeProxy, '--ignore-certificate-errors'],
    });
    const page = await browser.newPage();
    await page.goto('https://www.amazon.com/p/dp/B08GL2XTV6');

    let pageTitle = await page.title();
    let element = await page.$("#priceblock_ourprice");
    let price = await (await element.getProperty('textContent')).jsonValue();;

    console.log(price + ' - ' + pageTitle);
    // should be '$199.00 - LOVESHACKFANCY Women's Antonella Dress, Brilliant Blue, 4 at Amazon Women’s Clothing store'

    await browser.close();
    await proxyChain.closeAnonymizedProxy(anonymizeProxy, true)
})();

curl -s 'https://www.amazon.com/dp/B07HNW68ZC/' \
     -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36' \
     -k --compressed \
     -x 'PROXY_LOGIN:PROXY_PASS@PROXY_IP:PROXY_PORT'

import requests

url = "https://www.amazon.com/dp/B07HNW68ZC/"
proxies = {
    "https": f"http://PROXY_LOGIN:PROXY_PASS@PROXY_IP:PROXY_PORT/",
    "http": f"http://PROXY_LOGIN:PROXY_PASS@PROXY_IP:PROXY_PORT/"
}
headers = {
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36',
    'content-encoding': 'gzip'
}

# r = requests.get(url, headers=headers, proxies=proxies, verify='./public/ca.pem'')
r = requests.get(url, headers=headers, proxies=proxies, verify=False)

print(f"Response Body: {r.text}\n"
    "Request Headers:"
    f"{r.request.headers}\n\n"
    f"Response Time: {r.elapsed.total_seconds()}\n"
    f"Response Code: {r.status_code}\n"
    f"Response Headers: {r.headers}\n\n"
    f"Response Cookies: {r.cookies.items()}\n\n"
    f"Requesting {url}\n"
)

• https://portal.blazingseollc.com/en/support/solutions/articles/64000264080-what-is-proxy-pilot-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000264081-proxy-pilot-setup-instructions
• https://portal.blazingseollc.com/en/support/solutions/articles/64000264085-using-python-requests-with-proxy-pilot
• https://portal.blazingseollc.com/en/support/solutions/articles/64000264083-using-node-js-requests-with-proxy-pilot
• https://portal.blazingseollc.com/en/support/solutions/articles/64000264084-using-node-js-puppeteer-with-proxy-pilot
• https://portal.blazingseollc.com/en/support/solutions/articles/64000264086-using-curl-with-proxy-pilot
• https://portal.blazingseollc.com/en/support/solutions/articles/64000264087-using-firefox-with-proxy-pilot
• https://portal.blazingseollc.com/en/support/solutions/articles/64000264203-general-troubleshooting-for-proxy-pilot

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Are you having trouble using BlazingSEO? This article will tell you how to troubleshoot BlazingSEO Proxies.

Why Are My Proxies Not Working?

This article addresses:

• Proxies not showing up in your proxy dashboard
• Dead Proxies

NOTE: There is a difference between the proxies actually being completely dead (not loading any websites at all), and the proxies being blocked by a particular website, but not ‘dead’.

Steps to Diagnose Proxy Issues:

1. Is your PC/Device IP authorized in the proxy dashboard?

This can be checked following this article: How do I authorize my proxies?

Additionally, you go to the browser and access whatismyip.com to verify what your IP address is.  (Taking a screen capture of this so that you can send that in with a support ticket if you proceed to the other steps but still have issues, would be helpful.)

Once you confirm that your IPs are authorized and you’ve waited 30 minutes for authorization to complete, then continue to step 2.

2. Are all proxies loaded into your account dashboard?

If yes, then proceed to step 3

If no, it is possible that we are temporarily out of stock for the location(s) you selected. This does not happen often, but it can occur if we recently received an unexpectedly large order. We typically have new stock within 24-48 hours, so if that time frame is acceptable for you then hold tight! We’ll add them to your account as soon as they are replenished.

However, if your needs are more urgent, then you should change your locations to ‘mixed’ in the Location Preferences tab in the proxy dashboard. Setting to Mixed location will provide better subnet diversity, which is the best thing when connecting to websites. You will spread across more ISPs, ASNs, and A+B+C class subnets.

Your proxies will all load within 5-10 minutes, and you’ll be back in business in less time than it takes to run out for a Big Mac and fries.

3. Do your proxies access other websites, but not your preferred website?

Great news!  Your proxies are not dead. Unfortunately, this means that you may have a more challenging problem for your use case: The website you are trying to access with your proxies is banning or blocking them for some reason. Please read this article on what to do when proxies do now work on all websites.

We also offer one free replacement of IPs every 30 days, which you can learn more about in the knowledge base article about replacing proxies.

4. I’ve replaced my proxies, but they still aren’t working on my preferred website!

Unfortunately, if a website is blocking the proxies from accessing their site because of their site security, yet the proxy still works on other sites, then there is little we can do with that particular proxy.

But wait – you have more options!  You can try different proxy locations, various types of proxies (e.g. switch from semi-dedicated to dedicated, give IPv6 a whirl, or go big and get a premium proxy).

• https://portal.blazingseollc.com/en/support/solutions/articles/64000256628-why-are-my-proxies-not-working-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256603-how-can-i-check-to-see-if-a-proxy-is-blocked-from-a-certain-website
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256611-what-are-virgin-24-c-block-ips
• https://portal.blazingseollc.com/en/support/solutions/articles/64000262802-can-my-isp-see-that-i-m-using-a-proxy-or-block-it-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000262806-why-am-i-continuously-prompted-for-username-password
• https://portal.blazingseollc.com/en/support/solutions/articles/64000263276-what-should-i-do-if-my-proxy-is-banned-

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

What do you know about Blazing SEO residential proxies? Do you know how to set up them? Be sure to read this article before you buy!

Does Blazing SEO Offer Residential Proxies?

Residential proxies are currently in open beta testing. At present, we offer premium Datacenter proxies, and we’ll have ethically supported residential proxies very soon.

It’s important to us to deliver the highest quality residential proxies with thoughtful ethical practices. Unfortunately, many residential proxy providers are providing IPs that are unethically sold and resold without explicit consent from the source. Not only is this shady business, but it also puts the user’s IP at risk when using these services.

That’s why we strive to hold our products and services to a higher standard–it’s more fair, it’s legal, and everyone benefits from ethical residential proxy practices.

Residential Proxies Setup Guide

Welcome to the Blazing SEO Rotating Residential beta program! To start using your rotating residential proxies, just follow the instructions for your programming language of choice to install our proxy controller software:

• Node.js (Requests)
• Node.js with Puppeteer
• Curl
• Python (Requests)
• Firefox Browser

Need instructions for a language that’s not listed above? Reach out to your account manager. For details, see Proxy Pilot.

What Am I Installing?

Proxy Pilot is Blazing SEO’s brand new, 100% free proxy management application which comes with many benefits for the residential proxy user such as yourself.

To use our proxies you need to add a line of code to your software to “ignore” a certificate warning (recommended option), or install our secure certificate on your server. This certificate allows us to intercept your traffic and decrypt it, which emulates a man-in-the-middle-attack with your connection.

This certificate then stores your data insights in the powerful ELK stack, fully accessible by you and your team. You’ll be able to see data like the list below, and much more:

• How many connections were sent to domain.com?
• How many non-200 HTTP response codes?
• How much bandwidth have I consumed?
• How many total connections in the past 7 days?
• How much bandwidth on a specific domain?
• Look at the full header sent to the target website to understand why you may have received a non-200 HTTP response (a ban, in many cases)

Why do I Need This?

We want you to have the best possible experience with your Blazing SEO proxies, and the Proxy Pilot certificate will make your scraping project much more successful than it would be otherwise.

There are two primary reasons you must work with this certificate.

1. Improve your scraping with advanced statistics

Proxy Pilot lets you access amazing statistics at your fingertips, with detail far beyond most of our users’ in-house solutions. Without such data, we believe customers are just “spraying” connections at our competitors’ pools without really knowing what’s going on or what they’re being charged for. With our solution, you will know everything!

Here are actual screenshots of the ELK dashboard you will be provided with:

And in this ten-minute video, our VP of Technology shows the wide array of benefits you get from installing our simple certificate.

2. Protect our users and our proxy pool

Most proxy providers cannot technologically prohibit a customer from opening a tunnel and sending 100k requests to a single device.

Blazing SEO works a little differently. We know that setting high ethical standards doesn’t just protect the pool of users who are entrusting us with their IP addresses – it also protects YOU from getting banned/blocked from a site you want to scrape.

By giving our software the insights to know about each connection being sent, we are able to ethically spread out connections to our residential devices.

• https://portal.blazingseollc.com/en/support/solutions/articles/64000256604-does-blazing-seo-offer-residential-proxies-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000264222-residential-proxies-setup-guide
• https://portal.blazingseollc.com/en/support/solutions/articles/64000263384-residential-proxy-faq
• https://portal.blazingseollc.com/en/support/solutions/articles/64000264987-static-residential-ip-asns

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Do you want to learn what's the blazing proxies and type of proxies that offered? We take time research on Their customer support manuals, learn form it here,

What are Proxies?

A proxy is an IP that forwards your internet request/data, without the websites that you are connected to, finding out your actual IP address. Proxies can be used for a wide variety of use cases which include anonymous browsing, social media, scraping, etc.

What is Your Proxy Anonymity?

A proxy is an IP that forwards your internet request/data, without the websites that you are connected to, finding out your actual IP address. Proxies can be used for a wide variety of use cases which include anonymous browsing, social media, scraping, etc.

How do Proxies Work and How is Ping Measured in MS?

Understanding the Proxy's Network Path

A proxy's network path is defined as your data that you send when you visit a website.

In order for you to ‘download' the website content to view on your computer, your request is sent to the website's server and then back to your home computer.

Unfortunately, your computer does not have a direct ethernet cable from your home computer to the website’s server, thus, network hops are required between many ISPs (POPs) to reach the end target.

With that in mind, your proxy's network path looks something like this:

Based on the image above, the total time taken from sending a request to download data back to your computer when using a proxy = 620 ms.

Now, if you remove the proxy from the equation, or move your computer closer to the proxy server, you reduce Blue's network latency (in theory).

Thus, customers who are experiencing slow speeds can many times be attested to the fact that they are connecting to one of our nationwide proxy servers (LA, Buffalo, etc), from a computer/server that is very far away (thousands of miles), which is connecting to a website that is additionally thousands of miles away — and THUS, create a latency that is noticeable.

Can My ISP See that I'm Using a Proxy or Block It?

We enforce high level proxy anonymity protocol to ensure that your IP does not leak out. Your ISP will not recognize our proxy IP usage.

What Port do Blazing SEO Proxies Use?

For IP authorization, Blazing SEO proxies utilize port 3128. For username/password authorization, their proxies utilize port 4444. And for SOCKS connectivity, our proxies utilize port 1080.

What Type of Proxies do BlazingSEO Provider?

Now BlazingSEO offer,

• Residential Proxies
• Dedicated IP Proxies (DC Proxies)
• Rotating Proxies
• IPv6 Proxies
• Proxy Pilot
• Mobile Proxies (developing)

• https://portal.blazingseollc.com/en/support/solutions/articles/64000262799-what-are-proxies-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256610-what-is-your-proxy-anonymity-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000256636-how-do-proxies-work-and-how-is-ping-measured-in-ms-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000262802-can-my-isp-see-that-i-m-using-a-proxy-or-block-it-
• https://portal.blazingseollc.com/en/support/solutions/articles/64000263273-what-port-do-your-proxies-use-

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Do you know how to use OxyLabs Real-time Crawler for web pages? This is the most comprehensive introduction from OxyLabs official.

Quick Start

HTML Crawler API is built to help you in your heavy-duty data retrieval operations. You can use HTML Crawler API to access various public pages. It enables effortless web data extraction without any delays or errors.

HTML Crawler API uses basic HTTP authentication that requires sending username and password.

This is by far the fastest way to start using HTML Crawler API. You will make a request to https://ip.oxylabs.io using Realtime integration method. Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "universal", "url": "https://ip.oxylabs.io"}'

If you have any questions not covered by this documentation, please contact your account manager or our support staff at support@oxylabs.io.

Push-Pull

This is the most simple yet the most reliable and recommended data delivery method. In Push-Pull scenario you send us a query, we return you job id, and once the job is done you can use that id to retrieve content from /results endpoint. You can check job completion status yourself, or you can set up a simple listener that is able to accept POST queries.

This way, we will send you a callback message once the job is ready to be retrieved. In this particular example the results will be automatically uploaded to your S3 bucket named YOUR_BUCKET_NAME.

Single Query

The following endpoint will handle single queries for one keyword or URL. The API will return a confirmation message containing job information, including job id. You can check job completion status using that id, or you can ask us to ping your callback endpoint once the scraping task is finished by adding callback_url in the query.

POST https://data.oxylabs.io/v1/queries

You need to post query parameters as data in the JSON body.

curl --user user:pass1\
'https://data.oxylabs.io/v1/queries' \
-H "Content-Type: application/json" \
-d '{"source": "universal", "url": "https://stackoverflow.com/questions/tagged/python", "callback_url": "https://your.callback.url", "storage_type": "s3", "storage_url": "YOUR_BUCKET_NAME"}'

The API will respond with query information in JSON format, by printing it in the response body, similar to this:

{
  "callback_url": "https://your.callback.url",
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "url": "https://stackoverflow.com/questions/tagged/python",
  "source": "universal",
  "start_page": 1,
  "status": "pending",
  "storage_type": "s3",
  "storage_url": "YOUR_BUCKET_NAME/12345678900987654321.json",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:01",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Check Job Status

If your query had a callback_url, we will send you a message containing a link to the content once the scraping task is done. However, if there was no callback_url in the query, you will need to check the job status yourself. For that, you need to use the URL in href under rel:self in the response message you received after submitting your query to our API. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321.

GET https://data.oxylabs.io/v1/queries/{id}

Querying this link will return the job information, including its status. There are three possible status values:

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321'

The API will respond with query information in JSON format, by printing it in the response body. Notice that job status has been changed to done. You can now retrieve content by querying http://data.oxylabs.io/v1/queries/12345678900987654321/results.

You can also see that the task has been updated_at 2019-10-01 00:00:15 – the query took 14 seconds to complete.

{
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "url": "sofa",
  "source": "universal",
  "start_page": 1,
  "status": "done",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:15",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Retrieve Job Content

Once you know the job is ready to be retrieved by checking its status, you can GET it using the URL in href under rel:results in our initial response. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321/results.

GET https://data.oxylabs.io/v1/queries/{id}/results

The results can be automatically retrieved without periodically checking job status by setting up Callback service. User needs to specfy the IP or domain of the server where the Callback service is running. When our system completes a job, it will send a message to the provided IP or domain and the Callback service will download the results as described in the Callback implementation example.

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321/results'

The API will return job content:

{
  "results": [
    {
      "content": "<!doctype html>
        CONTENT      
      ",
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "page": 1,
      "url": "https://stackoverflow.com/questions/tagged/python",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

Callback

A callback is a POST request we send to your machine, informing that the data extraction task is completed and providing URL to download scraped content. This means that you no longer need to check job status manually. Once the data is here, we will let you know, and all you need to do now is retrieve it.

# Please see the code samples in Python and PHP.

Sample callback output

{  
   "created_at":"2019-10-01 00:00:01",
   "updated_at":"2019-10-01 00:00:15",
   "locale":null,
   "client_id":163,
   "user_agent_type":"desktop",
   "source":"universal",
   "pages":1,
   "subdomain":"www",
   "status":"done",
   "start_page":1,
   "parse":0,
   "render":null,
   "priority":0,
   "ttl":0,
   "origin":"api",
   "persist":true,
   "id":"12345678900987654321",
   "callback_url":"http://your.callback.url/",
   "url":"https://stackoverflow.com/questions/tagged/python",
   "domain":"de",
   "limit":10,
   "geo_location":null,
   {...}
   "_links":[
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321",
         "method":"GET",
         "rel":"self"
      },
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321/results",
         "method":"GET",
         "rel":"results"
      }
   ],
}

Batch Query

HTML Crawler API also supports executing multiple keywords, up to 1,000 keywords with each batch. The following endpoint will submit multiple keywords to the extraction queue.

POST https://data.oxylabs.io/v1/queries/batch

You need to post query parameters as data in the JSON body.

The system will handle every keyword as a separate request. If you provided a callback URL, you will get a separate call for each keyword. Otherwise, our initial response will contain job ids for all keywords. For example, if you sent 50 keywords, we will return 50 unique job ids.

Important! query is the only parameter that can have multiple values. All other parameters are the same for that batch query.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries/batch' -H 'Content-Type: application/json' \
 -d '@keywords.json'

keywords.json content:

{  
   "url":[  
      "https://stackoverflow.com/questions/tagged/python",
      "https://stackoverflow.com/questions/tagged/golang",
      "https://stackoverflow.com/questions/tagged/php"
   ],
   "source": "universal",
   "callback_url": "https://your.callback.url"
}

The API will respond with query information in JSON format, by printing it in the response body, similar to this:

{
  "queries": [
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678900987654321",
      {...}
      "url": "https://stackoverflow.com/questions/tagged/python",
      "source": "universal",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678901234567890",
      {...}
      "url": "https://stackoverflow.com/questions/tagged/golang",
      "source": "universal",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678901234567890/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "01234567899876543210",
      {...}
      "url": "https://stackoverflow.com/questions/tagged/php",
      "source": "universal",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/01234567899876543210/results",
          "method": "GET"
        }
      ]
    }
  ]
}

Get Notifier IP Address List

You may want to whitelist the IPs sending you callback messages or get the list of these IPs for other purposes. This can be done by GETing this endpoint: https://data.oxylabs.io/v1/info/callbacker_ips.

curl --user user:pass1 'https://data.oxylabs.io/v1/info/callbacker_ips'

The API will return the list of IPs making callback requests to your system:

{
    "ips": [
        "x.x.x.x",
        "y.y.y.y"
    ]
}

Upload to Storage

By default RTC job results are stored in our databases. This means that you will need to query our results endpoint and retrieve content yourself. Custom storage feature allows you to store results in your own cloud storage. The advantage of this feature is that you don't have to make extra requests in order to fetch results – everything goes directly to your storage bucket.

We support Amazon S3 and Google Cloud Storage. If you would like to use a different type of storage, please contact your account manager to discuss the feature delivery timeline.

Amazon S3

To get your job results uploaded to your Amazon S3 bucket, please set up access permissions for our service. To do that, go to https://s3.console.aws.amazon.com/ > S3 > Storage > Bucket Name (if don't have one, create new) > Permissions > Bucket Policy

You can find bucket policy in this JSON or in code sample area on the right. Don't forget to change bucket name under YOUR_BUCKET_NAME. This policy allows us to write to your bucket, give access to uploaded files to you, and know bucket location.

Google Cloud Storage

To get your job results uploaded to your Google Cloud Storage bucket, please set up special permissions for our service. To do that, please create a custom role with the storage.objects.create permission and assign it to the Oxylabs service account email oxyserps-storage@oxyserps-storage.iam.gserviceaccount.com.

Usage

To use this feature, please specify two additional parameters in your requests. Learn more here.

The upload path looks like this: YOUR_BUCKET_NAME/job_ID.json. You will find job ID in response body that you receive from us after submitting a request. In this example job ID is 12345678900987654321.

{
    "Version": "2012-10-17",
    "Id": "Policy1577442634787",
    "Statement": [
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
        },
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
        }
    ]
}

Realtime

The data submission is the same as in Push-Pull method, but with Realtime, we will return the content on open connection. You send us a query, the connection remains open, we retrieve the content, and bring it to you. The endpoint that handles that is this:

POST https://realtime.oxylabs.io/v1/queries

The timeout limit for open connections is 100 seconds. Therefore, in rare cases of heavy load, we may not be able to ensure the data gets to you.

You need to post query parameters as data in the JSON body. Please see an example for more details.

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" \
 -d '{"source": "universal", "url": "https://stackoverflow.com/questions/tagged/python"}'

Example response body that will be returned on open connection:

{
  "results": [
    {
      "content": "
      CONTENT
      "
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "id": null,
      "page": 1,
      "url": "https://stackoverflow.com/questions/tagged/python",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

SuperAPI

If you have ever used regular proxies for data scraping, integrating SuperAPI delivery method will be a breeze. You simply need to use our entry node as proxy, authorize with HTML Crawler API credentials, and ignore certificates. In cURL it's -k or --insecure. Your data will reach you on open connection.

GET realtime.oxylabs.io:60000

SuperAPI only supports a handful of parameters since it only works with Direct data source where a full URL is provided. These parameters should be sent as headers. This is a list of accepted parameters:

If you need help setting up SuperAPI, get in touch with us at support@oxylabs.io.

curl -k \
-x realtime.oxylabs.io:60000 \
-U user:pass1 \
-H "X-OxySERPs-User-Agent-Type: desktop_chrome" \
"https://stackoverflow.com/questions/tagged/python"

Content Type

HTML Crawler API returns raw HTML.

Download Images

It is possible to download images via HTML Crawler API. If you are doing that through SuperAPI, you can simply save the output to image extension. For example:

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 "https://example.com/image.jpg" >> image.jpg

If you are using Push-Pull or Realtime methods, you will need to add content_encoding parameter with a value of base64. Once you receive the results, you then need to decode encoded data from content into bytes and save it as an image file. Please find an example in Python on the right.

Data Sources

HTML Crawler API accepts URLs, along with additional parameters, such as User-Agent type, proxy location, and others. See this method, which we call Direct, described below.

HTML Crawler API is able to render JavaScript when scraping. This enables you to get more data from the web page and get screenshots.

If you are unsure about any part of the documentation, drop us a line at support@oxylabs.io or contact your account manager.

Direct

universal source is designed to retrieve the contents of any URL on the internet. POST-ing the parameters in JSON format to the following endpoint will submit the specified URL to the extraction queue.

Query parameters

In this example, the API will retrieve a universal product page in Push-Pull method. All available parameters are included (though not always necessary or compatible within the same request), to give you an idea on how to format your requests:

curl --user user:pass1 \
'https://data.oxylabs.io/v1/queries' \
-H "Content-Type: application/json" \
 -d '{"source":"universal","url":"https://stackoverflow.com/questions/tagged/python","user_agent_type":"mobile","context":[{"key":"headers","value":{"Accept-Language":"en-US","Content-Type":"application/octet-stream","Custom-Header":"custom header content"}},{"key":"cookies","value":[{"key":"NID","value":"1234567890"},{"key":"1P JAR","value":"0987654321"}]},{"key":"follow_redirects","value":true},{"key":"http_method","value":"post"},{"key":"content","value":"YmFzZTY0RW5jb2RlZFBPU1RCb2R5"},{"key":"successful_status_codes","value":[808,909]}]}

Here is the same example in Realtime:

curl --user user:pass1 \
'https://data.oxylabs.io/v1/queries' \
-H "Content-Type: application/json" \
-d '{"source": "universal", "url": "https://stackoverflow.com/questions/tagged/python", "user_agent_type": "mobile", "context": [{"key": "headers", "value": ["Accept-Language": "en-US", "Content-Type": "application/octet-stream", "Custom-Header": "custom header content"]}, {"key": "cookies", "value": [{"key": "NID", "value": "1234567890"}, {"key": "1P JAR", "value": "0987654321"}, {"key": "follow_redirects", "value": true}, {"key": "http_method", "value": "post"}, {"key": "content", "value": "base64EncodedPOSTBody"}, {"key": "successful_status_codes", "value": [303, 808, 909]}]}]}'

And via SuperAPI:

# A GET request could look something like this:
curl -k \
-x http://realtime.oxylabs.io:60000 \
-U user:pass1 \
"https://stackoverflow.com/questions/tagged/python" \
-H "X-OxySERPs-Session-Id: 1234567890abcdef" \
-H "X-OxySERPs-Geo-Location: India" \
-H "Accept-Language: en-US" \
-H "Content-Type: application/octet-stream" \
-H "Custom-Header: custom header content" \
-H "Cookie: NID=1234567890; 1P_JAR=0987654321" \
-H "X-Status-Code: 303, 808, 909"

# A POST request would have the same structure but contain a parameter specifying that it is a POST request:
curl -X POST \
-k \
-x http://realtime.oxylabs.io:60000 \
-U user:pass1 "https://stackoverflow.com/questions/tagged/python" \
-H "X-OxySERPs-Session-Id: 1234567890abcdef" \
-H "X-OxySERPs-Geo-Location: India" \
-H "Custom-Header: custom header content" \
-H "Cookie: NID=1234567890; 1P_JAR=0987654321" \
-H "X-Status-Code: 303, 808, 909"

Parameter Values

Geo_Location

Full list of supported geo locations can be found in CSV format here.

"United Arab Emirates",
"Albania",
"Armenia",
"Angola",
"Argentina",
"Australia",
...
"Uruguay",
"Uzbekistan",
"Venezuela Bolivarian Republic of",
"Viet Nam",
"South Africa",
"Zimbabwe"

HTTP_Method

Universal Crawler supports two HTTP(S) methods: GET (default) and POST.

"GET",
"POST"

Render

Universal Crawler can render Javascript and return either a rendered HTML document or a PNG screenshot of the web page.

"html",
"png"

User_Agent_Type

Download full list of user_agent_type values in JSON here.

[
  {
    "user_agent_type": "desktop",
    "description": "Random desktop browser User-Agent"
  },
  {
    "user_agent_type": "desktop_firefox",
    "description": "Random User-Agent of one of the latest versions of desktop Firefox"
  },
  {
    "user_agent_type": "desktop_chrome",
    "description": "Random User-Agent of one of the latest versions of desktop Chrome"
  },
  {
    "user_agent_type": "desktop_opera",
    "description": "Random User-Agent of one of the latest versions of desktop Opera"
  },
  {
    "user_agent_type": "desktop_edge",
    "description": "Random User-Agent of one of the latest versions of desktop Edge"
  },
  {
    "user_agent_type": "desktop_safari",
    "description": "Random User-Agent of one of the latest versions of desktop Safari"
  },
  {
    "user_agent_type": "mobile",
    "description": "Random mobile browser User-Agent"
  },
  {
    "user_agent_type": "mobile_android",
    "description": "Random User-Agent of one of the latest versions of Android browser"
  },
  {
    "user_agent_type": "mobile_ios",
    "description": "Random User-Agent of one of the latest versions of iPhone browser"
  },
  {
    "user_agent_type": "tablet",
    "description": "Random tablet browser User-Agent"
  },
  {
    "user_agent_type": "tablet_android",
    "description": "Random User-Agent of one of the latest versions of Android tablet"
  },
  {
    "user_agent_type": "tablet_ios",
    "description": "Random User-Agent of one of the latest versions of iPad tablet"
  }
]

Account Status

Usage Statistics

You can find your usage statistics by querying the following endpoint:

GET https://data.oxylabs.io/v1/stats

By default, the API will return all-time usage statistics. Adding ?group_by=month will return monthly stats, while ?group_by=day will return daily numbers.

This query will return all-time statistics. You can find your daily and monthly usage by adding either ?group_by=day or ?group_by=month

curl --user user:pass1 'https://data.oxylabs.io/v1/stats'

Sample output:

{
    "data": {
        "sources": [
            {
                "realtime_results_count": "90",
                "results_count": "10",
                "title": "universal"
            }
        ]
    },
    "meta": {
        "group_by": null
    }
}

Limits

The following endpoint will give your monthly commitment information as well as how much of it has already been used:

GET https://data.oxylabs.io/v1/stats/limits

curl --user user:pass1 'https://data.oxylabs.io/v1/stats/limits'

Sample output:

{
    "monthly_requests_commitment": 4500000,
    "used_requests": 985000
}

Response Codes

References

• https://docs.oxylabs.io/rtc/source/html-crawler-api/index.html
• https://docs.oxylabs.io/rtc/index.html

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Do you know how to use OxyLabs Real-time Crawler for Baidu? This is the most comprehensive introduction from OxyLabs official.

Quick Start

Real-Time Crawler is built for heavy-duty data retrieval operations. You can use Real-Time Crawler to access various Baidu pages. It enables effortless web data extraction from search engines without any delays or errors.

Real-Time Crawler for Baidu uses basic HTTP authentication that requires sending username and password.

This is by far the fastest way to start using Real-Time Crawler for Baidu. You will send a query adidas to baidu_search using Realtime integration method. Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "baidu_search", "domain": "com", "query": "adidas"}'

If you have any questions not covered by this documentation, please contact your account manager or our support staff at support@oxylabs.io.

Integration Methods

Real-Time Crawler for Baidu supports three integration methods which have their unique benefits:

• Push-Pull. Using this method it is now required to mainain an active connection with our endpoint to retrieve the data. Upon making a request, our system is able to automatically ping users server when the job is done (see Callback). This method saves computing resources and can be scaled easily.
• Realtime. The method requires user to maintain an active connection with our endpoint in order to get the results successfully when the job is completed. This method can be implemented into one service while Push-Pull method is a two step process.
• SuperAPI. This method is very similar to Realtime but instead posting data to our endpoint, user can use HTML Cralwer as a proxy. To retrieve the data, user must set up a proxy endpoint and make GET request to a desired URL. Additional parameters must be added using headers.

Our recommended data extraction method is Push-Pull.

Push-Pull

This is the most simple yet the most reliable and recommended data delivery method. In Push-Pull scenario you send us a query, we return you job id, and once the job is done you can use that id to retrieve content from /results endpoint. You can check job completion status yourself, or you can set up a simple listener that is able to accept POST queries.

This way we will send you a callback message once the job is ready to be retrieved. In this particular example the results will be automatically uploaded to your S3 bucket named YOUR_BUCKET_NAME.

Single Query

The following endpoint will handle single queries for one keyword or URL. The API will return a confirmation message containing job information, including job id. You can check job completion status using that id, or you can ask us to ping your callback endpoint once the scraping task is finished by adding callback_url in the query.

POST https://data.oxylabs.io/v1/queries

You need to post query parameters as data in JSON body.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "baidu_search", "domain": "com", "query": "adidas", "callback_url": "https://your.callback.url", "storage_type": "s3", "storage_url": "YOUR_BUCKET_NAME"}'

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "callback_url": "https://your.callback.url",
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "render": null,
  "query": "adidas",
  "source": "baidu_search",
  "start_page": 1,
  "status": "pending",
  "storage_type": "s3",
  "storage_url": "YOUR_BUCKET_NAME/12345678900987654321.json",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:01",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Check Job Status

If your query had callback_url, we will send you a message containing link to content once the scraping task is done. However, if there was no callback_url in the query, you will need to check job status yourself. For that you need to use the URL in href under rel:self in the response message you received after submitting your query to our API. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321.

GET https://data.oxylabs.io/v1/queries/{id}

Querying this link will return job information, including its status. There are 3 possible status values:

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321'

The API will respond with query information in JSON format, by printing it in response body. Notice that job status has been changed to done. You can now retrieve content by querying http://data.oxylabs.io/v1/queries/12345678900987654321/results.

You can also see that the task has been updated_at 2019-10-01 00:00:15 – the query took 14 seconds to complete.

{
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "render": null,
  "query": "adidas",
  "source": "baidu_search",
  "start_page": 1,
  "status": "done",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:15",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Retrieve Job Content

Once you know the job is ready to retrieved either by checking its status or receiving a callback from us, you can GET it using the URL in href under rel:results in either our initial response or in callback message. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321/results.

GET https://data.oxylabs.io/v1/queries/{id}/results

The results can be automatically retrieved without periodically checking job status by setting up Callback service. User needs to specfy the IP or domain of the server where the Callback service is running. When our system completes a job, it will send a message to the provided IP or domain and the Callback service will download the results as described in the Callback implementation example.

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321/results'

The API will return job content:

{
  "results": [
    {
      "content": "<!doctype html>
        CONTENT      
      ",
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "page": 1,
      "url": "https://www.baidu.com/search?q=adidas&hl=en&gl=US",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

Callback

A callback is a POST request we send to your machine, informing that the data extraction task is completed and providing URL to download scraped content. This means that you no longer need to check job status manually. Once the data is here, we will let you know, and all you need to do now is retrieve it.

# Please see code samples in Python and PHP.

Sample callback output

{  
   "created_at":"2019-10-01 00:00:01",
   "updated_at":"2019-10-01 00:00:15",
   "locale":null,
   "client_id":163,
   "user_agent_type":"desktop",
   "source":"baidu_search",
   "pages":1,
   "subdomain":"www",
   "status":"done",
   "start_page":1,
   "render":null,
   "priority":0,
   "ttl":0,
   "origin":"api",
   "persist":true,
   "id":"12345678900987654321",
   "callback_url":"http://your.callback.url/",
   "query":"adidas",
   "domain":"com",
   "limit":10,
   "geo_location":null,
   {...}
   "_links":[
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321",
         "method":"GET",
         "rel":"self"
      },
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321/results",
         "method":"GET",
         "rel":"results"
      }
   ],
}

Batch Query

Real-Time Crawler also supports executing multiple keywords, up to 1,000 keywords with each batch. The following endpoint will submit multiple keywords to the extraction queue.

POST https://data.oxylabs.io/v1/queries/batch

You need to post query parameters as data in JSON body.

The system will handle every keyword as a separate request. If you provided callback URL, you will get a separate call for each keyword. Otherwise, our initial response will contain job ids for all keywords. For example, if you sent 50 keywords, we will return 50 unique job ids.

Important! query is the only parameter that can have multiple values. All other parameters are the same for that batch query.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries/batch' -H 'Content-Type: application/json'
 -d '@keywords.json'

keywords.json content:

{  
   "query":[  
      "adidas",
      "nike",
      "reebok"
   ],
   "source": "baidu_search",
   "domain": "com",
   "callback_url": "https://your.callback.url"
}

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "queries": [
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678900987654321",
      {...}
      "query": "adidas",
      "source": "baidu_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678901234567890",
      {...}
      "query": "nike",
      "source": "baidu_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678901234567890/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "01234567899876543210",
      {...}
      "query": "reebok",
      "source": "baidu_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/01234567899876543210/results",
          "method": "GET"
        }
      ]
    }
  ]
}

Get Notifier IP Address List

You may want to whitelist the IPs sending you callback messages or get the list of these IPs for other purposes. This can be done by GETing this endpoint: https://data.oxylabs.io/v1/info/callbacker_ips.

curl --user user:pass1 'https://data.oxylabs.io/v1/info/callbacker_ips'

The API will return the list of IPs making callback requests to your system:

{
    "ips": [
        "x.x.x.x",
        "y.y.y.y"
    ]
}

Upload to Storage

By default RTC job results are stored in our databases. This means that you will need to query our results endpoint and retrieve content yourself. Custom storage feature allows you to store results in your own cloud storage. The advantage of this feature is that you don't have to make extra requests in order to fetch results – everything goes directly to your storage bucket.

We support Amazon S3 and Google Cloud Storage. If you would like to use a different type of storage, please contact your account manager to discuss the feature delivery timeline.

Amazon S3

To get your job results uploaded to your Amazon S3 bucket, please set up access permissions for our service. To do that, go to https://s3.console.aws.amazon.com/ > S3 > Storage > Bucket Name (if don't have one, create new) > Permissions > Bucket Policy

You can find bucket policy in this JSON or in code sample area on the right. Don't forget to change bucket name under YOUR_BUCKET_NAME. This policy allows us to write to your bucket, give access to uploaded files to you, and know bucket location.

Google Cloud Storage

To get your job results uploaded to your Google Cloud Storage bucket, please set up special permissions for our service. To do that, please create a custom role with the storage.objects.create permission and assign it to the Oxylabs service account email oxyserps-storage@oxyserps-storage.iam.gserviceaccount.com.

Usage

To use this feature, please specify two additional parameters in your requests. Learn more here.

The upload path looks like this: YOUR_BUCKET_NAME/job_ID.json. You will find job ID in response body that you receive from us after submitting a request. In this example job ID is 12345678900987654321.

Realtime

The data submission is the same as in Push-Pull method, but Realtime case we will return the content on open connection. You send us a query, the connection remains open, we retrieve the content and bring it to you. The endpoint that handles that is this:

POST https://realtime.oxylabs.io/v1/queries

There is a timeout limit of 150 seconds for open connections, therefore in rare cases of heavy load we may not be able to ensure the data gets to you.

You need to post query parameters as data in JSON body. Please see example for more details.

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "baidu_search", "domain": "com", "query": "adidas"}'

Example response body that will be returned on open connection:

{
  "results": [
    {
      "content": "
      CONTENT
      "
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "id": null,
      "page": 1,
      "url": "https://www.baidu.com/search?q=adidas&hl=en&gl=US",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

SuperAPI

If you ever used regular proxies for data scraping, integrating SuperAPI delivery method will be a breeze. All that needs to be done is to use our entry node as proxy, authorize with Real-Time Crawler credentials, and ignore certificates. In cURL it's -k or --insecure. Your data will reach you on open connection.

GET realtime.oxylabs.io:60000

SuperAPI only supports a handful of parameters since it only works with Direct data source where full URL is provided. These parameters should be sent as headers. This is a list of accepted parameters:

If you need help setting up SuperAPI, drop a line at support@oxylabs.io.

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 -H "X-OxySERPs-User-Agent-Type: desktop_chrome" "https://www.baidu.com/search?q=adidas"

Data Sources

There are multiple approaches how to retrieve data from Baidu using Real-Time Crawler. You can give us full URL via Direct, or you can specify parameters via specifically built data sources, such as Search, Shopping Product or Images.

If you are unsure which way to choose, drop us a line at support@oxylabs.io or contact your account manager.

Direct

baidu source is designed to retrieve content of direct URLs of various Baidu pages. This means that instead of sending multiple parameters, you can provide us with a direct URL to required Baidu page. We do not strip any parameters or alter your URLs in any other way.

Query parameters

In this example the API will retrieve Baidu search for keyword adidas in Push-Pull method:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "baidu", "url": "http://www.baidu.com/s?ie=utf-8&f=8&rsv_bp=1&rsv_idx=1&ch=&tn=baidu&bar=&wd=adidas"}'

Here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "baidu", "url": "http://www.baidu.com/s?ie=utf-8&f=8&rsv_bp=1&rsv_idx=1&ch=&tn=baidu&bar=&wd=adidas"}'

And via SuperAPI:

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 "http://www.baidu.com/s?ie=utf-8&f=8&rsv_bp=1&rsv_idx=1&ch=&tn=baidu&bar=&wd=adidas"

Search

baidu_search source is designed to retrieve Baidu Search results (SERP) in HTML format.

Query parameters

Real-Time Crawler makes request to baidu.com to retrieve search results pages from number 11 to number 20 for keyword adidas. Real-Time Crawler will post the download URL to raw HTML page output to your.callback.url once the data retrieval task is successfully finished.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "baidu_search", "domain": "com", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url"}'

And here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "baidu_search", "domain": "com", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url"}'

Parameter Values

User-Agent

Download full list of user_agent_type values in JSON here.

[
  {
    "user_agent_type": "desktop",
    "description": "Random desktop browser User-Agent"
  },
  {
    "user_agent_type": "desktop_firefox",
    "description": "Random User-Agent of one of the latest versions of desktop Firefox"
  },
  {
    "user_agent_type": "desktop_chrome",
    "description": "Random User-Agent of one of the latest versions of desktop Chrome"
  },
  {
    "user_agent_type": "desktop_opera",
    "description": "Random User-Agent of one of the latest versions of desktop Opera"
  },
  {
    "user_agent_type": "desktop_edge",
    "description": "Random User-Agent of one of the latest versions of desktop Edge"
  },
  {
    "user_agent_type": "desktop_safari",
    "description": "Random User-Agent of one of the latest versions of desktop Safari"
  },
  {
    "user_agent_type": "mobile",
    "description": "Random mobile browser User-Agent"
  },
  {
    "user_agent_type": "mobile_android",
    "description": "Random User-Agent of one of the latest versions of Android browser"
  },
  {
    "user_agent_type": "mobile_ios",
    "description": "Random User-Agent of one of the latest versions of iPhone browser"
  },
  {
    "user_agent_type": "tablet",
    "description": "Random tablet browser User-Agent"
  },
  {
    "user_agent_type": "tablet_android",
    "description": "Random User-Agent of one of the latest versions of Android tablet"
  },
  {
    "user_agent_type": "tablet_ios",
    "description": "Random User-Agent of one of the latest versions of iPad tablet"
  }
]

Account Status

Usage Statistics

You can find your usage statistics by querying the following endpoint:

GET https://data.oxylabs.io/v1/stats

By default the API will return all time usage statistics. Adding ?group_by=month will return monthly stats, while ?group_by=day will return daily numbers.

This query will return all time statistics. You can find your daily and monthly usage by adding either ?group_by=day or ?group_by=month

curl --user user:pass1 'https://data.oxylabs.io/v1/stats'

Sample output:

{
    "data": {
        "sources": [
            {
                "realtime_results_count": "90",
                "results_count": "10",
                "title": "baidu"
            },
            {
                "realtime_results_count": "19",
                "results_count": "87",
                "title": "baidu_search"
            }
        ]
    },
    "meta": {
        "group_by": null
    }
}

Limits

The following endpoint will give your monthly commitment information as well as how much has already been used:

GET https://data.oxylabs.io/v1/stats/limits

curl --user user:pass1 'https://data.oxylabs.io/v1/stats/limits'

Sample output:

{
    "monthly_requests_commitment": 4500000,
    "used_requests": 985000
}

Response Codes

Cloud storage upload response codes:

References

• https://docs.oxylabs.io/rtc/source/baidu/index.html
• https://docs.oxylabs.io/rtc/index.html

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Do you know how to use OxyLabs Real-time Crawler for Bing? This is the most comprehensive introduction from OxyLabs official.

Quick Start

Real-Time Crawler is built for heavy-duty data retrieval operations. You can use Real-Time Crawler to access various Bing pages. It enables effortless web data extraction from search engines without any delays or errors.

Real-Time Crawler for Bing uses basic HTTP authentication that requires sending username and password.

This is by far the fastest way to start using Real-Time Crawler for Bing. You will send a query adidas to bing_search using Realtime integration method. Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "bing_search", "domain": "com", "query": "adidas"}'

If you have any questions not covered by this documentation, please contact your account manager or our support staff at support@oxylabs.io.

Integration Methods

Real-Time Crawler for Bing supports three integration methods which have their unique benefits:

• Push-Pull. Using this method it is now required to mainain an active connection with our endpoint to retrieve the data. Upon making a request, our system is able to automatically ping users server when the job is done (see Callback). This method saves computing resources and can be scaled easily.
• Realtime. The method requires user to maintain an active connection with our endpoint in order to get the results successfully when the job is completed. This method can be implemented into one service while Push-Pull method is a two step process.
• SuperAPI. This method is very similar to Realtime but instead posting data to our endpoint, user can use HTML Cralwer as a proxy. To retrieve the data, user must set up a proxy endpoint and make GET request to a desired URL. Additional parameters must be added using headers.

Our recommended data extraction method is Push-Pull.

Push-Pull

This is the most simple yet the most reliable and recommended data delivery method. In Push-Pull scenario you send us a query, we return you job id, and once the job is done you can use that id to retrieve content from /results endpoint. You can check job completion status yourself, or you can set up a simple listener that is able to accept POST queries.

This way we will send you a callback message once the job is ready to be retrieved. In this particular example the results will be automatically uploaded to your S3 bucket named YOUR_BUCKET_NAME.

Single Query

The following endpoint will handle single queries for one keyword or URL. The API will return a confirmation message containing job information, including job id. You can check job completion status using that id, or you can ask us to ping your callback endpoint once the scraping task is finished by adding callback_url in the query.

POST https://data.oxylabs.io/v1/queries

You need to post query parameters as data in JSON body.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "bing_search", "domain": "com", "query": "adidas", "callback_url": "https://your.callback.url", "storage_type": "s3", "storage_url": "YOUR_BUCKET_NAME"}'

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "callback_url": "https://your.callback.url",
  "client_id": 5,
  "context": [
    {
      "key": "safe_search",
      "value": null
    }
  ],
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "render": null,
  "query": "adidas",
  "source": "bing_search",
  "start_page": 1,
  "status": "pending",  
  "storage_type": "s3",
  "storage_url": "YOUR_BUCKET_NAME/12345678900987654321.json",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:01",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Check Job Status

If your query had callback_url, we will send you a message containing link to content once the scraping task is done. However, if there was no callback_url in the query, you will need to check job status yourself. For that you need to use the URL in href under rel:self in the response message you received after submitting your query to our API. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321.

GET https://data.oxylabs.io/v1/queries/{id}

Querying this link will return job information, including its status. There are 3 possible status values:

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321'

The API will respond with query information in JSON format, by printing it in response body. Notice that job status has been changed to done. You can now retrieve content by querying http://data.oxylabs.io/v1/queries/12345678900987654321/results.

You can also see that the task has been updated_at 2019-10-01 00:00:15 – the query took 14 seconds to complete.

{
  "client_id": 5,
  "context": [
    {
      "key": "safe_search",
      "value": null
    }
  ],
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "render": null,
  "query": "adidas",
  "source": "bing_search",
  "start_page": 1,
  "status": "done",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:15",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Retrieve Job Content

Once you know the job is ready to retrieved either by checking its status or receiving a callback from us, you can GET it using the URL in href under rel:results in either our initial response or in callback message. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321/results

GET https://data.oxylabs.io/v1/queries/{id}/results

The results can be automatically retrieved without periodically checking job status by setting up Callback service. User needs to specfy the IP or domain of the server where the Callback service is running. When our system completes a job, it will send a message to the provided IP or domain and the Callback service will download the results as described in the Callback implementation example.

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321/results'

The API will return job content:

{
  "results": [
    {
      "content": "<!doctype html>
        CONTENT      
      ",
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "page": 1,
      "url": "https://www.bing.com/search?q=adidas",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

Callback

A callback is a POST request we send to your machine, informing that the data extraction task is completed and providing URL to download scraped content. This means that you no longer need to check job status manually. Once the data is here, we will let you know, and all you need to do now is retrieve it.

# Please see code samples in Python and PHP.

Sample callback output

{  
   "created_at":"2019-10-01 00:00:01",
   "updated_at":"2019-10-01 00:00:15",
   "locale":null,
   "client_id":163,
   "user_agent_type":"desktop",
   "source":"bing_search",
   "pages":1,
   "subdomain":"www",
   "status":"done",
   "start_page":1,
   "render":null,
   "priority":0,
   "ttl":0,
   "origin":"api",
   "persist":true,
   "id":"12345678900987654321",
   "callback_url":"http://your.callback.url/",
   "query":"adidas",
   "domain":"com",
   "limit":10,
   "geo_location":null,
   {...}
   "_links":[
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321",
         "method":"GET",
         "rel":"self"
      },
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321/results",
         "method":"GET",
         "rel":"results"
      }
   ],
}

Batch Query

Real-Time Crawler also supports executing multiple keywords, up to 1,000 keywords with each batch. The following endpoint will submit multiple keywords to the extraction queue.

POST https://data.oxylabs.io/v1/queries/batch

You need to post query parameters as data in JSON body.

The system will handle every keyword as a separate request. If you provided callback URL, you will get a separate call for each keyword. Otherwise, our initial response will contain job ids for all keywords. For example, if you sent 50 keywords, we will return 50 unique job ids.

Important! query is the only parameter that can have multiple values. All other parameters are the same for that batch query.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries/batch' -H 'Content-Type: application/json'
 -d '@keywords.json'

keywords.json content:

{  
   "query":[  
      "adidas",
      "nike",
      "reebok"
   ],
   "source": "bing_search",
   "domain": "com",
   "callback_url": "https://your.callback.url"
}

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "queries": [
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678900987654321",
      {...}
      "query": "adidas",
      "source": "bing_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678901234567890",
      {...}
      "query": "nike",
      "source": "bing_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678901234567890/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "01234567899876543210",
      {...}
      "query": "reebok",
      "source": "bing_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/01234567899876543210/results",
          "method": "GET"
        }
      ]
    }
  ]
}

Get Notifier IP Address List

You may want to whitelist the IPs sending you callback messages or get the list of these IPs for other purposes. This can be done by GETing this endpoint: https://data.oxylabs.io/v1/info/callbacker_ips.

curl --user user:pass1 'https://data.oxylabs.io/v1/info/callbacker_ips'

The API will return the list of IPs making callback requests to your system:

{
    "ips": [
        "x.x.x.x",
        "y.y.y.y"
    ]
}

Upload to Storage

By default RTC job results are stored in our databases. This means that you will need to query our results endpoint and retrieve content yourself. Custom storage feature allows you to store results in your own cloud storage. The advantage of this feature is that you don't have to make extra requests in order to fetch results – everything goes directly to your storage bucket.

We support Amazon S3 and Google Cloud Storage. If you would like to use a different type of storage, please contact your account manager to discuss the feature delivery timeline.

Amazon S3

To get your job results uploaded to your Amazon S3 bucket, please set up access permissions for our service. To do that, go to https://s3.console.aws.amazon.com/ > S3 > Storage > Bucket Name (if don't have one, create new) > Permissions > Bucket Policy

You can find bucket policy in this JSON or in code sample area on the right. Don't forget to change bucket name under YOUR_BUCKET_NAME. This policy allows us to write to your bucket, give access to uploaded files to you, and know bucket location.

Google Cloud Storage

To get your job results uploaded to your Google Cloud Storage bucket, please set up special permissions for our service. To do that, please create a custom role with the storage.objects.create permission and assign it to the Oxylabs service account email oxyserps-storage@oxyserps-storage.iam.gserviceaccount.com.

Usage

To use this feature, please specify two additional parameters in your requests. Learn more here.

The upload path looks like this: YOUR_BUCKET_NAME/job_ID.json. You will find job ID in response body that you receive from us after submitting a request. In this example job ID is 12345678900987654321.

{
    "Version": "2012-10-17",
    "Id": "Policy1577442634787",
    "Statement": [
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
        },
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
        }
    ]
}

Realtime

The data submission is the same as in Push-Pull method, but Realtime case we will return the content on open connection. You send us a query, the connection remains open, we retrieve the content and bring it to you. The endpoint that handles that is this:

POST https://realtime.oxylabs.io/v1/queries

There is a timeout limit of 150 seconds for open connections, therefore in rare cases of heavy load we may not be able to ensure the data gets to you.

You need to post query parameters as data in JSON body. Please see example for more details.

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "bing_search", "domain": "com", "query": "adidas"}'

Example response body that will be returned on open connection:

{
  "results": [
    {
      "content": "
      CONTENT
      "
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "id": null,
      "page": 1,
      "url": "https://www.bing.com/search?q=adidas",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

SuperAPI

If you ever used regular proxies for data scraping, integrating SuperAPI delivery method will be a breeze. All that needs to be done is to use our entry node as proxy, authorize with Real-Time Crawler credentials, and ignore certificates. In cURL it's -k or --insecure. Your data will reach you on open connection.

GET realtime.oxylabs.io:60000

SuperAPI only supports a handful of parameters since it only works with Direct data source where full URL is provided. These parameters should be sent as headers. This is a list of accepted parameters:

If you need help setting up SuperAPI, drop a line at support@oxylabs.io.

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 -H "X-OxySERPs-User-Agent-Type: desktop_chrome" 
-H "X-OxySERPs-Geo-Location: 15550" "https://bing.com/search/?text=nike"

Content Type

Real-Time Crawler returns raw HTML.

Data Sources

There are two approaches on how to retrieve data from Bing using Real-Time Crawler. You can give us full URL via Direct, or you can specify parameters via specifically built data source – Search.

If you are unsure which way to choose, drop us a line at support@oxylabs.io or contact your account manager.

Direct

bing source is designed to retrieve content of direct URLs of various Bing pages. This means that instead of sending multiple parameters, you can provide us with a direct URL to required Bing page. We do not strip any parameters or alter your URLs in any other way.

Query parameters

In this example the API will retrieve Bing search for keyword nike in Push-Pull method:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "bing", "url": "https://bing.com/search/?text=nike&"}'

Here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "bing", "url": "https://bing.com/search/?text=nike&"}'

And via SuperAPI:

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 "https://bing.com/search/?text=nike&"

Search

bing_search source is designed to retrieve Bing Search results (SERP).

Query parameters

API makes request to bing.com to retrieve search results pages from number 11 to number 20 for keyword adidas. The API will post a JSON request to your.callback.url containing the URL to download the raw HTML output once the data retrieval task is successfully finished. This is Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "bing_search", "domain": "com", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url"}'

And here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "bing_search", "domain": "com", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url"}'

Parameter Values

User-Agent

Download full list of user_agent_type values in JSON here.

[
  {
    "user_agent_type": "desktop",
    "description": "Random desktop browser User-Agent"
  },
  {
    "user_agent_type": "desktop_firefox",
    "description": "Random User-Agent of one of the latest versions of desktop Firefox"
  },
  {
    "user_agent_type": "desktop_chrome",
    "description": "Random User-Agent of one of the latest versions of desktop Chrome"
  },
  {
    "user_agent_type": "desktop_opera",
    "description": "Random User-Agent of one of the latest versions of desktop Opera"
  },
  {
    "user_agent_type": "desktop_edge",
    "description": "Random User-Agent of one of the latest versions of desktop Edge"
  },
  {
    "user_agent_type": "desktop_safari",
    "description": "Random User-Agent of one of the latest versions of desktop Safari"
  },
  {
    "user_agent_type": "mobile",
    "description": "Random mobile browser User-Agent"
  },
  {
    "user_agent_type": "mobile_android",
    "description": "Random User-Agent of one of the latest versions of Android browser"
  },
  {
    "user_agent_type": "mobile_ios",
    "description": "Random User-Agent of one of the latest versions of iPhone browser"
  },
  {
    "user_agent_type": "tablet",
    "description": "Random tablet browser User-Agent"
  },
  {
    "user_agent_type": "tablet_android",
    "description": "Random User-Agent of one of the latest versions of Android tablet"
  },
  {
    "user_agent_type": "tablet_ios",
    "description": "Random User-Agent of one of the latest versions of iPad tablet"
  }
]

Account Status

Usage Statistics

You can find your usage statistics by querying the following endpoint:

GET https://data.oxylabs.io/v1/stats

By default the API will return all time usage statistics. Adding ?group_by=month will return monthly stats, while ?group_by=day will return daily numbers.

This query will return all time statistics. You can find your daily and monthly usage by adding either ?group_by=day or ?group_by=month

curl --user user:pass1 'https://data.oxylabs.io/v1/stats'

Sample output:

{
    "data": {
        "sources": [
            {
                "realtime_results_count": "90",
                "results_count": "10",
                "title": "bing"
            },
            {
                "realtime_results_count": "19",
                "results_count": "87",
                "title": "bing_search"
            }
        ]
    },
    "meta": {
        "group_by": null
    }
}

Limits

The following endpoint will give your monthly commitment information as well as how much has already been used:

GET https://data.oxylabs.io/v1/stats/limits

curl --user user:pass1 'https://data.oxylabs.io/v1/stats/limits'

Sample output:

{
    "monthly_requests_commitment": 4500000,
    "used_requests": 985000
}

Response Codes

Cloud storage upload response codes:

References

• https://docs.oxylabs.io/rtc/source/bing/index.html
• https://docs.oxylabs.io/rtc/index.html

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Do you know how to use OxyLabs Real-time Crawler for Yandex? This is the most comprehensive introduction from OxyLabs official.

Quick Start

Real-Time Crawler is built for heavy-duty data retrieval operations. You can use Real-Time Crawler to access various Yandex pages. It enables effortless web data extraction from search engines without any delays or errors.

Real-Time Crawler for Yandex uses basic HTTP authentication that requires sending username and password.

This is by far the fastest way to start using Real-Time Crawler for Yandex. You will send a query adidas to yandex_search using Realtime integration method. Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "yandex_search", "domain": "com", "query": "adidas"}'

If you have any questions not covered by this documentation, please contact your account manager or our support staff at support@oxylabs.io.

Integration Methods

Real-Time Crawler for Yandex supports three integration methods which have their unique benefits:

• Push-Pull. Using this method it is now required to mainain an active connection with our endpoint to retrieve the data. Upon making a request, our system is able to automatically ping users server when the job is done (see Callback). This method saves computing resources and can be scaled easily.
• Realtime. The method requires user to maintain an active connection with our endpoint in order to get the results successfully when the job is completed. This method can be implemented into one service while Push-Pull method is a two step process.
• SuperAPI. This method is very similar to Realtime but instead posting data to our endpoint, user can use HTML Cralwer as a proxy. To retrieve the data, user must set up a proxy endpoint and make GET request to a desired URL. Additional parameters must be added using headers.

Our recommended data extraction method is Push-Pull.

Push-Pull

This is the most simple yet the most reliable and recommended data delivery method. In Push-Pull scenario you send us a query, we return you job id, and once the job is done you can use that id to retrieve content from /results endpoint. You can check job completion status yourself, or you can set up a simple listener that is able to accept POST queries.

This way we will send you a callback message once the job is ready to be retrieved. In this particular example the results will be automatically uploaded to your S3 bucket named YOUR_BUCKET_NAME.

Single Query

The following endpoint will handle single queries for one keyword or URL. The API will return a confirmation message containing job information, including job id. You can check job completion status using that id, or you can ask us to ping your callback endpoint once the scraping task is finished by adding callback_url in the query.

POST https://data.oxylabs.io/v1/queries

You need to post query parameters as data in JSON body.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "yandex_search", "domain": "com", "query": "adidas", "callback_url": "https://your.callback.url", "storage_type": "s3", "storage_url": "YOUR_BUCKET_NAME"}'

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "callback_url": "https://your.callback.url",
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "query": "adidas",
  "source": "yandex_search",
  "start_page": 1,
  "status": "pending",
  "storage_type": "s3",
  "storage_url": "YOUR_BUCKET_NAME/12345678900987654321.json",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:01",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Check Job Status

If your query had callback_url, we will send you a message containing link to content once the scraping task is done. However, if there was no callback_url in the query, you will need to check job status yourself. For that you need to use the URL in href under rel:self in the response message you received after submitting your query to our API. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321.

GET https://data.oxylabs.io/v1/queries/{id}

Querying this link will return job information, including its status. There are 3 possible status values:

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321'

The API will respond with query information in JSON format, by printing it in response body. Notice that job status has been changed to done. You can now retrieve content by querying http://data.oxylabs.io/v1/queries/12345678900987654321/results.

You can also see that the task has been updated_at 2019-10-01 00:00:15 – the query took 14 seconds to complete.

{
  "client_id": 5,
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "query": "adidas",
  "source": "yandex_search",
  "start_page": 1,
  "status": "done",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:15",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Retrieve Job Content

Once you know the job is ready to retrieved either by checking its status or receiving a callback from us, you can GET it using the URL in href under rel:results in either our initial response or in callback message. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321/results.

GET https://data.oxylabs.io/v1/queries/{id}/results

The results can be automatically retrieved without periodically checking job status by setting up Callback service. User needs to specfy the IP or domain of the server where the Callback service is running. When our system completes a job, it will send a message to the provided IP or domain and the Callback service will download the results as described in the Callback implementation example.

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321/results'

The API will return job content:

{
  "results": [
    {
      "content": "<!doctype html>
        CONTENT      
      ",
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "page": 1,
      "url": "https://www.yandex.com/search?q=adidas&hl=en&gl=US",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

Callback

A callback is a POST request we send to your machine, informing that the data extraction task is completed and providing URL to download scraped content. This means that you no longer need to check job status manually. Once the data is here, we will let you know, and all you need to do now is retrieve it.

# Please see code samples in Python and PHP.

Sample callback output

{  
   "created_at":"2019-10-01 00:00:01",
   "updated_at":"2019-10-01 00:00:15",
   "locale":null,
   "client_id":163,
   "user_agent_type":"desktop",
   "source":"yandex_search",
   "pages":1,
   "subdomain":"www",
   "status":"done",
   "start_page":1,
   "parse":0,
   "render":null,
   "priority":0,
   "ttl":0,
   "origin":"api",
   "persist":true,
   "id":"12345678900987654321",
   "callback_url":"http://your.callback.url/",
   "query":"adidas",
   "domain":"com",
   "limit":10,
   "geo_location":null,
   {...}
   "_links":[
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321",
         "method":"GET",
         "rel":"self"
      },
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321/results",
         "method":"GET",
         "rel":"results"
      }
   ],
}

Batch Query

Real-Time Crawler also supports executing multiple keywords, up to 1,000 keywords with each batch. The following endpoint will submit multiple keywords to the extraction queue.

POST https://data.oxylabs.io/v1/queries/batch

You need to post query parameters as data in JSON body.

The system will handle every keyword as a separate request. If you provided callback URL, you will get a separate call for each keyword. Otherwise, our initial response will contain job ids for all keywords. For example, if you sent 50 keywords, we will return 50 unique job ids.

Important! query is the only parameter that can have multiple values. All other parameters are the same for that batch query.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries/batch' -H 'Content-Type: application/json'
 -d '@keywords.json'

keywords.json content:

{  
   "query":[  
      "adidas",
      "nike",
      "reebok"
   ],
   "source": "yandex_search",
   "domain": "com",
   "callback_url": "https://your.callback.url"
}

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "queries": [
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678900987654321",
      {...}
      "query": "adidas",
      "source": "yandex_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678901234567890",
      {...}
      "query": "nike",
      "source": "yandex_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678901234567890/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "01234567899876543210",
      {...}
      "query": "reebok",
      "source": "yandex_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/01234567899876543210/results",
          "method": "GET"
        }
      ]
    }
  ]
}

Get Notifier IP Address List

You may want to whitelist the IPs sending you callback messages or get the list of these IPs for other purposes. This can be done by GETing this endpoint: https://data.oxylabs.io/v1/info/callbacker_ips.

curl --user user:pass1 'https://data.oxylabs.io/v1/info/callbacker_ips'

The API will return the list of IPs making callback requests to your system:

{
    "ips": [
        "x.x.x.x",
        "y.y.y.y"
    ]
}

Upload to Storage

By default RTC job results are stored in our databases. This means that you will need to query our results endpoint and retrieve content yourself. Custom storage feature allows you to store results in your own cloud storage. The advantage of this feature is that you don't have to make extra requests in order to fetch results – everything goes directly to your storage bucket.

We support Amazon S3 and Google Cloud Storage. If you would like to use a different type of storage, please contact your account manager to discuss the feature delivery timeline.

Amazon S3

To get your job results uploaded to your Amazon S3 bucket, please set up access permissions for our service. To do that, go to https://s3.console.aws.amazon.com/ > S3 > Storage > Bucket Name (if don't have one, create new) > Permissions > Bucket Policy

You can find bucket policy in this JSON or in code sample area on the right. Don't forget to change bucket name under YOUR_BUCKET_NAME. This policy allows us to write to your bucket, give access to uploaded files to you, and know bucket location.

Google Cloud Storage

To get your job results uploaded to your Google Cloud Storage bucket, please set up special permissions for our service. To do that, please create a custom role with the storage.objects.create permission and assign it to the Oxylabs service account email oxyserps-storage@oxyserps-storage.iam.gserviceaccount.com.

Usage

To use this feature, please specify two additional parameters in your requests. Learn more here.

The upload path looks like this: YOUR_BUCKET_NAME/job_ID.json. You will find job ID in response body that you receive from us after submitting a request. In this example job ID is 12345678900987654321.

{
    "Version": "2012-10-17",
    "Id": "Policy1577442634787",
    "Statement": [
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
        },
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
        }
    ]
}

Realtime

The data submission is the same as in Push-Pull method, but Realtime case we will return the content on open connection. You send us a query, the connection remains open, we retrieve the content and bring it to you. The endpoint that handles that is this:

POST https://realtime.oxylabs.io/v1/queries

There is a timeout limit of 150 seconds for open connections, therefore in rare cases of heavy load we may not be able to ensure the data gets to you.

You need to post query parameters as data in JSON body. Please see example for more details.

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "yandex_search", "domain": "com", "query": "adidas"}'

Example response body that will be returned on open connection:

{
  "results": [
    {
      "content": "
      CONTENT
      "
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "id": null,
      "page": 1,
      "url": "https://www.yandex.com/search?q=adidas&hl=en&gl=US",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

SuperAPI

If you ever used regular proxies for data scraping, integrating SuperAPI delivery method will be a breeze. All that needs to be done is to use our entry node as proxy, authorize with Real-Time Crawler credentials, and ignore certificates. In cURL it's -k or --insecure. Your data will reach you on open connection.

GET realtime.oxylabs.io:60000

SuperAPI only supports a handful of parameters since it only works with Direct data source where full URL is provided. These parameters should be sent as headers. This is a list of accepted parameters:

If you need help setting up SuperAPI, drop a line at support@oxylabs.io.

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 -H "X-OxySERPs-User-Agent-Type: desktop_chrome" 
-H "X-OxySERPs-Geo-Location: 15550" "https://yandex.com/search/?text=nike"

Content Type

Real-Time Crawler returns raw HTML.

Data Sources

There are two approaches on how to retrieve data from Yandex using Real-Time Crawler. You can give us full URL via Direct, or you can specify parameters via specifically built data source – Search.

Technically not a content type, but Real-Time Crawler is able to render JavaScript when scraping. This may be necessary in some Yandex pages. A checkmark under Render JS will indicate whether a particular data source can be scraped with JavaScript enabled.

If you are unsure which way to choose, drop us a line at support@oxylabs.io or contact your account manager.

Direct

yandex source is designed to retrieve content of direct URLs of various Yandex pages. This means that instead of sending multiple parameters, you can provide us with a direct URL to required Yandex page. We do not strip any parameters or alter your URLs in any other way.

Query parameters

In this example the API will retrieve Yandex search for keyword nike in Push-Pull method:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "yandex", "url": "https://yandex.com/search/?text=nike&"}'

Here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "yandex", "url": "https://yandex.com/search/?text=nike&"}'

And via SuperAPI:

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 "https://yandex.com/search/?text=nike&"

Search

yandex_search source is designed to retrieve Yandex Search results (SERP).

Query parameters

API makes request to yandex.com to retrieve search results pages from number 11 to number 20 for keyword adidas. The API will post a JSON request to your.callback.url containing the URL to download the raw HTML output once the data retrieval task is successfully finished. This is Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "yandex_search", "domain": "com", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url"}'

And here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "yandex_search", "domain": "com", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url"}'

Parameter Values

User-Agent

Download full list of user_agent_type values in JSON here.

[
  {
    "user_agent_type": "desktop",
    "description": "Random desktop browser User-Agent"
  },
  {
    "user_agent_type": "desktop_firefox",
    "description": "Random User-Agent of one of the latest versions of desktop Firefox"
  },
  {
    "user_agent_type": "desktop_chrome",
    "description": "Random User-Agent of one of the latest versions of desktop Chrome"
  },
  {
    "user_agent_type": "desktop_opera",
    "description": "Random User-Agent of one of the latest versions of desktop Opera"
  },
  {
    "user_agent_type": "desktop_edge",
    "description": "Random User-Agent of one of the latest versions of desktop Edge"
  },
  {
    "user_agent_type": "desktop_safari",
    "description": "Random User-Agent of one of the latest versions of desktop Safari"
  },
  {
    "user_agent_type": "mobile",
    "description": "Random mobile browser User-Agent"
  },
  {
    "user_agent_type": "mobile_android",
    "description": "Random User-Agent of one of the latest versions of Android browser"
  },
  {
    "user_agent_type": "mobile_ios",
    "description": "Random User-Agent of one of the latest versions of iPhone browser"
  },
  {
    "user_agent_type": "tablet",
    "description": "Random tablet browser User-Agent"
  },
  {
    "user_agent_type": "tablet_android",
    "description": "Random User-Agent of one of the latest versions of Android tablet"
  },
  {
    "user_agent_type": "tablet_ios",
    "description": "Random User-Agent of one of the latest versions of iPad tablet"
  }
]

Account Status

Usage Statistics

You can find your usage statistics by querying the following endpoint:

GET https://data.oxylabs.io/v1/stats

By default the API will return all time usage statistics. Adding ?group_by=month will return monthly stats, while ?group_by=day will return daily numbers.

This query will return all time statistics. You can find your daily and monthly usage by adding either ?group_by=day or ?group_by=month

curl --user user:pass1 'https://data.oxylabs.io/v1/stats'

Sample output:

{
    "data": {
        "sources": [
            {
                "realtime_results_count": "90",
                "results_count": "10",
                "title": "yandex"
            },
            {
                "realtime_results_count": "19",
                "results_count": "87",
                "title": "yandex_search"
            }
        ]
    },
    "meta": {
        "group_by": null
    }
}

Limits

The following endpoint will give your monthly commitment information as well as how much has already been used:

GET https://data.oxylabs.io/v1/stats/limits

curl --user user:pass1 'https://data.oxylabs.io/v1/stats/limits'

Sample output:

{
    "monthly_requests_commitment": 4500000,
    "used_requests": 985000
}

Response Codes

Cloud storage upload response codes:

References

• https://docs.oxylabs.io/rtc/source/yandex/index.html
• https://docs.oxylabs.io/rtc/index.html

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Do you know how to use OxyLabs Real-time Crawler for Google? This is the most comprehensive introduction from OxyLabs official.

Quick Start

Real-Time Crawler is built for heavy-duty data retrieval operations. You can use Real-Time Crawler to access various Google pages, including regular search, hotel availability and Google Shopping. It enables effortless web data extraction from search engines without any delays or errors.

Real-Time Crawler for Google uses basic HTTP authentication that requires sending username and password.

This is by far the fastest way to start using Real-Time Crawler for Google. You will send a query adidas to google_search using Realtime integration method. Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "google_search", "domain": "com", "query": "adidas"}'

If you have any questions not covered by this documentation, please contact your account manager or our support staff at support@oxylabs.io.

Postman

Download and import this Postman collection to try out all of the Google crawler functions and data delivery methods documented on this page.

Integration Methods

Real-Time Crawler for Google supports three integration methods which have their unique benefits:

• Push-Pull. Using this method it is now required to mainain an active connection with our endpoint to retrieve the data. Upon making a request, our system is able to automatically ping users server when the job is done (see Callback). This method saves computing resources and can be scaled easily.
• Realtime. The method requires user to maintain an active connection with our endpoint in order to get the results successfully when the job is completed. This method can be implemented into one service while Push-Pull method is a two step process.
• SuperAPI. This method is very similar to Realtime but instead posting data to our endpoint, user can use HTML Cralwer as a proxy. To retrieve the data, user must set up a proxy endpoint and make GET request to a desired URL. Additional parameters must be added using headers.

Our recommended data extraction method is Push-Pull.

Push-Pull

This is the most simple yet the most reliable and recommended data delivery method. In Push-Pull scenario you send us a query, we return you job id, and once the job is done you can use that id to retrieve content from /results endpoint. You can check job completion status yourself, or you can set up a simple listener that is able to accept POST queries. This way we will send you a callback message once the job is ready to be retrieved. In this particular example the results will be automatically uploaded to your S3 bucket named YOUR_BUCKET_NAME.

You can also try and see how Push-Pull method works via Postman. Download this file to get started.

Single Query

The following endpoint will handle single queries for one keyword or URL. The API will return a confirmation message containing job information, including job id. You can check job completion status using that id, or you can ask us to ping your callback endpoint once the scraping task is finished by adding callback_url in the query.

POST https://data.oxylabs.io/v1/queries

You need to post query parameters as data in JSON body.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "google_search", "domain": "com", "query": "adidas", "callback_url": "https://your.callback.url", "storage_type": "s3", "storage_url": "YOUR_BUCKET_NAME"}'

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "callback_url": "https://your.callback.url",
  "client_id": 5,
  "context": [
    {
      "key": "results_language",
      "value": null
    },
    {
      "key": "safe_search",
      "value": null
    },
    {
      "key": "tbm",
      "value": null
    },
    {
      "key": "cr",
      "value": null
    },
    {
      "key": "filter",
      "value": null
    }
  ],
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "query": "adidas",
  "source": "google_search",
  "start_page": 1,
  "status": "pending",
  "storage_type": "s3",
  "storage_url": "YOUR_BUCKET_NAME/12345678900987654321.json",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:01",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Check Job Status

If your query had callback_url, we will send you a message containing link to content once the scraping task is done. However, if there was no callback_url in the query, you will need to check job status yourself. For that you need to use the URL in href under rel:self in the response message you received after submitting your query to our API. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321.

GET https://data.oxylabs.io/v1/queries/{id}

Querying this link will return job information, including its status. There are 3 possible status values:

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321'

The API will respond with query information in JSON format, by printing it in response body. Notice that job status has been changed to done. You can now retrieve content by querying http://data.oxylabs.io/v1/queries/12345678900987654321/results.

You can also see that the task has been updated_at 2019-10-01 00:00:15 – the query took 14 seconds to complete.

{
  "client_id": 5,
  "context": [
    {
      "key": "results_language",
      "value": null
    },
    {
      "key": "safe_search",
      "value": null
    },
    {
      "key": "tbm",
      "value": null
    },
    {
      "key": "cr",
      "value": null
    },
    {
      "key": "filter",
      "value": null
    }
  ],
  "created_at": "2019-10-01 00:00:01",
  "domain": "com",
  "geo_location": null,
  "id": "12345678900987654321",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "render": null,
  "query": "adidas",
  "source": "google_search",
  "start_page": 1,
  "status": "done",
  "subdomain": "www",
  "updated_at": "2019-10-01 00:00:15",
  "user_agent_type": "desktop",
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
      "method": "GET"
    }
  ]
}

Retrieve Job Content

Once you know the job is ready to retrieved either by checking its status or receiving a callback from us, you can GET it using the URL in href under rel:results in either our initial response or in callback message. It should look similar to this: http://data.oxylabs.io/v1/queries/12345678900987654321/results.

GET https://data.oxylabs.io/v1/queries/{id}/results

The results can be automatically retrieved without periodically checking job status by setting up Callback service. User needs to specfy the IP or domain of the server where the Callback service is running. When our system completes a job, it will send a message to the provided IP or domain and the Callback service will download the results as described in the Callback implementation example.

curl --user user:pass1 'http://data.oxylabs.io/v1/queries/12345678900987654321/results'

The API will return job content:

{
  "results": [
    {
      "content": "<!doctype html>
        CONTENT      
      ",
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "page": 1,
      "url": "https://www.google.com/search?q=adidas&hl=en&gl=US",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

Callback

A callback is a POST request we send to your machine, informing that the data extraction task is completed and providing URL to download scraped content. This means that you no longer need to check job status manually. Once the data is here, we will let you know, and all you need to do now is retrieve it.

# Please see code samples in Python and PHP.

Sample callback output

{  
   "created_at":"2019-10-01 00:00:01",
   "updated_at":"2019-10-01 00:00:15",
   "locale":null,
   "client_id":163,
   "user_agent_type":"desktop",
   "source":"google_search",
   "pages":1,
   "subdomain":"www",
   "status":"done",
   "start_page":1,
   "parse":0,
   "render":null,
   "priority":0,
   "ttl":0,
   "origin":"api",
   "persist":true,
   "id":"12345678900987654321",
   "callback_url":"http://your.callback.url/",
   "query":"adidas",
   "domain":"com",
   "limit":10,
   "geo_location":null,
   {...}
   "_links":[
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321",
         "method":"GET",
         "rel":"self"
      },
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321/results",
         "method":"GET",
         "rel":"results"
      }
   ],
}

Batch Query

Real-Time Crawler also supports executing multiple keywords, up to 1,000 keywords with each batch. The following endpoint will submit multiple keywords to the extraction queue.

POST https://data.oxylabs.io/v1/queries/batch

You need to post query parameters as data in JSON body.

The system will handle every keyword as a separate request. If you provided callback URL, you will get a separate call for each keyword. Otherwise, our initial response will contain job ids for all keywords. For example, if you sent 50 keywords, we will return 50 unique job ids.

Important! query is the only parameter that can have multiple values. All other parameters are the same for that batch query.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries/batch' -H 'Content-Type: application/json'
 -d '@keywords.json'

keywords.json content:

{  
   "query":[  
      "adidas",
      "nike",
      "reebok"
   ],
   "source": "google_search",
   "domain": "com",
   "callback_url": "https://your.callback.url"
}

The API will respond with query information in JSON format, by printing it in response body, similar to this:

{
  "queries": [
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678900987654321",
      {...}
      "query": "adidas",
      "source": "google_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678901234567890",
      {...}
      "query": "nike",
      "source": "google_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678901234567890/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "01234567899876543210",
      {...}
      "query": "reebok",
      "source": "google_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/01234567899876543210/results",
          "method": "GET"
        }
      ]
    }
  ]
}

Get Notifier IP Address List

You may want to whitelist the IPs sending you callback messages or get the list of these IPs for other purposes. This can be done by GETing this endpoint: https://data.oxylabs.io/v1/info/callbacker_ips.

curl --user user:pass1 'https://data.oxylabs.io/v1/info/callbacker_ips'

The API will return the list of IPs making callback requests to your system:

{
    "ips": [
        "x.x.x.x",
        "y.y.y.y"
    ]
}

Upload to Storage

By default RTC job results are stored in our databases. This means that you will need to query our results endpoint and retrieve content yourself. Custom storage feature allows you to store results in your own cloud storage. The advantage of this feature is that you don't have to make extra requests in order to fetch results – everything goes directly to your storage bucket.

We support Amazon S3 and Google Cloud Storage. If you would like to use a different type of storage, please contact your account manager to discuss the feature delivery timeline.

Amazon S3

To get your job results uploaded to your Amazon S3 bucket, please set up access permissions for our service. To do that, go to https://s3.console.aws.amazon.com/ > S3 > Storage > Bucket Name (if don't have one, create new) > Permissions > Bucket Policy

You can find bucket policy in this JSON or in code sample area on the right. Don't forget to change bucket name under YOUR_BUCKET_NAME. This policy allows us to write to your bucket, give access to uploaded files to you, and know bucket location.

Google Cloud Storage

To get your job results uploaded to your Google Cloud Storage bucket, please set up special permissions for our service. To do that, please create a custom role with the storage.objects.create permission and assign it to the Oxylabs service account email oxyserps-storage@oxyserps-storage.iam.gserviceaccount.com.

Usage

To use this feature, please specify two additional parameters in your requests. Learn more here.

The upload path looks like this: YOUR_BUCKET_NAME/job_ID.json. You will find job ID in response body that you receive from us after submitting a request. In this example job ID is 12345678900987654321.

{
    "Version": "2012-10-17",
    "Id": "Policy1577442634787",
    "Statement": [
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
        },
        {
            "Sid": "Stmt1577442633719",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
        }
    ]
}

Realtime

The data submission is the same as in Push-Pull method, but Realtime case we will return the content on open connection. You send us a query, the connection remains open, we retrieve the content and bring it to you. The endpoint that handles that is this:

POST https://realtime.oxylabs.io/v1/queries

There is a timeout limit of 150 seconds for open connections, therefore in rare cases of heavy load we may not be able to ensure the data gets to you.

You need to post query parameters as data in JSON body. Please see example for more details.

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" 
 -d '{"source": "google_search", "domain": "com", "query": "adidas"}'

Example response body that will be returned on open connection:

{
  "results": [
    {
      "content": "
      CONTENT
      "
      "created_at": "2019-10-01 00:00:01",
      "updated_at": "2019-10-01 00:00:15",
      "id": null,
      "page": 1,
      "url": "https://www.google.com/search?q=adidas&hl=en&gl=US",
      "job_id": "12345678900987654321",
      "status_code": 200
    }
  ]
}

SuperAPI

If you ever used regular proxies for data scraping, integrating SuperAPI delivery method will be a breeze. All that needs to be done is to use our entry node as proxy, authorize with Real-Time Crawler credentials, and ignore certificates. In cURL it's -k or --insecure. Your data will reach you on open connection.

GET realtime.oxylabs.io:60000

SuperAPI only supports a handful of parameters since it only works with a Direct data source where full URL is provided. These parameters should be sent as headers. This is a list of accepted parameters:

If you need help setting up SuperAPI, drop a line at support@oxylabs.io.

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 -H "X-OxySERPs-User-Agent-Type: desktop_chrome" -H "X-OxySERPs-Geo-Location: New York,New York,United States" "https://www.google.com/search?q=adidas"

Content Type

Real-Time Crawler can return either raw HTML, or structured (parsed) JSON. Bear in mind that not all data sources can be returned structured. An icon under each data source in this documentation will indicate whether we are able to parse it, or we can only return raw HTML.

Please see Parsed Data to see which fields we return with each Data Source.

Data Sources

There are multiple approaches how to retrieve data from Google using Real-Time Crawler. You can give us full URL via Direct, or you can specify parameters via specifically built data sources, such as Search, Shopping Product or Images.

Technically not a content type, but Real-Time Crawler is able to render JavaScript when scraping. This is necessary in some Google pages, such as Flights and Patents. A checkmark under Render JS will indicate whether a particular data source can be scraped with JavaScript enabled.

If you are unsure which way to choose, drop us a line at support@oxylabs.io or contact your account manager.

Direct

google source is designed to retrieve content of direct URLs of various Google pages. This means that instead of sending multiple parameters, you can provide us with a direct URL to required Google page. We do not strip any parameters or alter your URLs in any other way.

This data source also supports parsed data (Parsed JSON), as long as the URL submitted is for Google Search (SERP page). If we are unable to confirm this is a SERP page request, a failure message will be returned.

Query parameters

In this example the API will retrieve Google Scholar search for keyword newton in Push-Pull method:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google", "url": "https://scholar.google.com/scholar?hl=en&q=newton&btnG=&as_sdt=1%2C5&as_sdtp="}'

Here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google", "url": "https://scholar.google.com/scholar?hl=en&q=newton&btnG=&as_sdt=1%2C5&as_sdtp="}'

And via SuperAPI:

curl -k -x realtime.oxylabs.io:60000 -U user:pass1 "https://scholar.google.com/scholar?hl=en&q=newton&btnG=&as_sdt=1%2C5&as_sdtp="

Search

google_search source is designed to retrieve Google Search results (SERP).

Query parameters

API makes request to google.nl to retrieve search results pages from number 11 to number 20 for keyword adidas. The results will be displayed in French, since results_language parameter is also passed through via context. The API will post a JSON request to your.callback.url containing the URL to download the raw HTML output once the data retrieval task is successfully finished. This is Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_search", "domain": "nl", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url", "context": [{"key": "results_language", "value": "fr"}]}}'

And here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_search", "domain": "nl", "query": "adidas", "start_page": 11, "pages": 10, "callback_url": "https://your.callback.url", "context": [{"key": "results_language", "value": "fr"}]}}'

Ads

google_ads source is optimized to retrieve Google Search results page (SERP) with paid ads. This source will return only 10 results per page, ensuring highest changes of paid results showing up. Other than that, it supports the same parameters as regular Search

Query parameters

API makes request to google.nl to retrieve search results for keyword adidas. The API will post a JSON request to your.callback.url containing the URL to download the raw HTML output once the data retrieval task is successfully finished. This is Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_ads", "domain": "nl", "query": "adidas", "callback_url": "https://your.callback.url"}'

And here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_ads", "domain": "nl", "query": "adidas"}'

Hotels

google_hotels data source is designed to retrieve Google Hotel search results.

Query parameters

Please note that with Google hotels you always need to send a keyword with ‘hotels' word inside, for example ‘hotels in Los Angeles', ‘hotels in Paris, France', etc. Both ‘hotel' and ‘hotels' work. Google also supports local languages, so you can send ‘Hotelli Helsingissä' for hotels in Helsinki or ‘viešbučiai Vilnius' for hotels in Vilnius.

In this example API will retrieve first 3 pages of hotel availability for 1 guest between 2019-10-01 and 2019-10-10 for hotels in Paris from google.com. This is Push-Pull method.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_hotels", "domain": "com", "pages": 3, "query": "hotels in Paris", "context": [{"key": "hotel_occupancy", "value": 1}, {"key": "hotel_dates", "value": "2019-10-01,2019-10-10"}]}'

This is in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_hotels", "domain": "com", "pages": 3, "query": "hotels in Paris", "context": [{"key": "hotel_occupancy", "value": 1}, {"key": "hotel_dates", "value": "2019-10-01,2019-10-10"}]}'

Travel: Hotels

google_travel_hotels data source is designed to retrieve Google Travel service's hotel search results.

Query parameters

Please note that with Google hotels you always need to send a keyword with ‘hotels' word inside, for example ‘hotels in Los Angeles', ‘hotels in Paris, France', etc. Both ‘hotel' and ‘hotels' work. Google also supports local languages, so you can send ‘Hotelli Helsingissä' for hotels in Helsinki or ‘viešbučiai Vilnius' for hotels in Vilnius.

In this example API will retrieve the 2nd page of results for of hotel availability for 2 guests between 2020-10-01 and 2020-10-10 for hotels in Paris from google.com. The results will be filtered to only show 2 and 4 star hotels. This is Push-Pull method.

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_travel_hotels", "domain": "com", "start_page": 2, "query": "hotels in Paris", "callback_url": "https://your.callback.url", "context": [{"key": "hotel_occupancy", "value": 2}, {"key": "hotel_dates", "value": "2020-10-01,2020-10-10"}, {"key": "hotel_classes", "value": [2,4]}]}'

This is in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_travel_hotels", "domain": "com", "start_page": 2, "query": "hotels in Paris", "context": [{"key": "hotel_occupancy", "value": 2}, {"key": "hotel_dates", "value": "2020-10-01,2020-10-10"}, {"key": "hotel_classes", "value": [2,4]}]}'

Shopping Search

google_shopping_search source is designed to retrieve Google Shopping search results.

POST https://data.oxylabs.io/v1/queries

Query parameters

The API will download first 4 pages of Google Shopping search for keyword adidas, sorted by descending price and minimum price of $20. This is how it's done in Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_shopping_search", "domain": "com", "query": "adidas", "pages": 4, "context": [{"key": "sort_by", "value": "pd"}, {"key": "min_price", "value": 20}]}'

Here is the same example in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_shopping_search", "domain": "com", "query": "adidas", "pages": 4, "context": [{"key": "sort_by", "value": "pd"}, {"key": "min_price", "value": 20}]}'

Shopping Product

google_shopping_product source is designed to retrieve Google Shopping product page for specified product.

Query parameters

Here the API will download product page for product ID 5007040952399054528 from Google Shopping on google.com. It will also get first 4 pages with pricing information. This is how it looks in Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_shopping_product", "domain": "com", "query": "5007040952399054528"}'

The same in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_shopping_product", "domain": "com", "query": "5007040952399054528"}'

Shopping Product Pricing

google_shopping_pricing source is designed to retrieve Google Shopping product pricing page for specified product.

Query parameters

Here the API will download product pricing page for product ID 5007040952399054528 from Google Shopping on google.com. Here is an example in Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_shopping_pricing", "domain": "com", "query": "5007040952399054528"}'

The same in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_shopping_pricing", "domain": "com", "query": "5007040952399054528"}'

Images

google_images source is designed to retrieve Images search page for images that are similar to the one provided with query parameter, as well as websites containing those images.

Query parameters

In this example the API will download Image search page of similar images for image https://newsneakernews-wpengine.netdna-ssl.com/wp-content/uploads/2017/03/adidas-boost-march-25-2017.jpg from google.com. This is Push-Pull method:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_images", "domain": "com", "query": "https://newsneakernews-wpengine.netdna-ssl.com/wp-content/uploads/2017/03/adidas-boost-march-25-2017.jpg"}'

And this is the same request in Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_images", "domain": "com", "query": "https://www.example.com/img/image.jpg"}'

Suggestions

google_suggest source is designed to retrieve Google keyword suggestions.

Query parameters

API makes request to Google Suggestions page to retrieve suggestions for keyword adidas. The API will post a JSON payload to your.callback.url containing the URL to download the result once the task is finished. Here is an example with Push-Pull:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_suggest", "query": "adidas", "callback_url": "https://your.callback.url"}'

The same request with Realtime:

curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
 -d '{"source": "google_suggest", "query": "adidas"}'

Keyword Data

google_msv data source will retrieve Google keyword data for specified keywords, as well as suggested keywords (unless passing ideas=False in context). Keywords are passed in query parameter as a string. Keywords are separated by commas. Commas within a keyword are not supported, so a keyword “Water Bottle 5,0L” will actually be interpreted as 2 keywords: “Water Bottle 5” and “0L”. See output example on the right for more details.

Query parameters

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
-d '{"source": "google_msv", "query": "meilleur restaurant", "geo_location": "Paris,Ile-de-France,France", "context": [{"key": "language", "value": "french"},{"key": "currency", "value": "EUR"}, {"key": "ideas", "value": true}]}'

# OR if you don't want ideas:

curl --user user:pass1 'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json"
-d '{"source": "google_msv", "query": "meilleur restaurant", "geo_location": "Paris,Ile-de-France,France", "context": [{"key": "language", "value": "french"},{"key": "currency", "value": "EUR"}, {"key": "ideas", "value": false}]}'

Sample output (historicalSearchVolume entries and ideas entries cut):

{
    "results": [
    {
        "content":
        {
            "ideas": [
            {
                "cpc": 4.712038,
                "keyword": "meilleur restaurant a paris",
                "currency": "EUR",
                "competition": 0.3385383889238515,
                "averageSearchVolume": 1900,
                "historicalSearchVolume": [
                {
                    "date": "201803",
                    "searchVolume": 1600
                },
                {
                    "date": "201802",
                    "searchVolume": 1900
                }]
            }],
            "seeds": [
            {
                "cpc": 4.05351,
                "keyword": "meilleur restaurant",
                "currency": "EUR",
                "competition": 0.3385341239238515,
                "averageSearchVolume": 2900,
                "historicalSearchVolume": [
                {
                    "date": "201803",
                    "searchVolume": 3600
                },
                {
                    "date": "201802",
                    "searchVolume": 2900
                }]
            }]
        }
    }]
}

Parsed data

Google Web Search (SERP) page is the only one that is extensively supported in parsed data delivery. Below you can find which particular SERP page fields we parse. Structure data is available with Search (all the time) and Direct (as long as SERP page URL is submitted).

Google Web Search ("source": "google_search") supports CSV output. To access it, please include these parameters in your Google Web Search job {"source": "google_search", "parse": true, "parser_type": "v2"}. The result retrieval URL for a CSV job is structured like this: http://data.oxylabs.io/v1/queries/{job_id}/results/normalized?format=csv.

Search

Organic & Paid

"results": {
  "paid": [
    {
      "pos": 1,
      "url": "https://www.adidas.com/us",
      "desc": "New York · 10 locations nearby",
      "title": "adidas.com | adidas® Official Site | Official adidas® Online Store‎",
      "url_shown": "www.adidas.com/Official/Site",
      "pos_overall": 1
    }
  ],
  "organic": [
    {
      "pos": 1,
      "url": "https://www.adidas.com/us",
      "desc": "Welcome to adidas Shop for adidas shoes, clothing and view new collections for adidas Originals, running, football, training and much more.",
      "title": "adidas Official Website | adidas US",
      "url_shown": "https://www.adidas.com › ...",
      "pos_overall": 2
    },
    {
      "pos": 2,
      "url": "https://www.mena.adidas.com/",
      "desc": "Browse for adidas shoes, clothing and collections, adidas Originals, Running, Football, Training and more on the official adidas website.",
      "title": "adidas Official Website | adidas",
      "url_shown": "https://www.mena.adidas.com",
      "pos_overall": 6
    },
    {
      "pos": 3,
      "url": "https://www.adidas-group.com/",
      "desc": "adidas AG Supervisory Board announces candidates as shareholder ... adidas celebrates its 70th anniversary and the opening of the Arena building. August 9 ...",
      "title": "adidas - Home",
      "url_shown": "https://www.adidas-group.com",
      "pos_overall": 7
    },
    {
      "pos": 4,
      "url": "https://www.nycgo.com/shopping/the-adidas-store",
      "desc": "You don't so much shop in this flagship Adidas store as you experience it. With an interior modeled on a high school stadium, this four-story Midtown outlet—the  ...",
      "title": "The Adidas Store (Midtown) | NYCgo - NYCgo.com",
      "url_shown": "https://www.nycgo.com › shopping › the-adidas-store",
      "pos_overall": 8
    },
    {
      "pos": 5,
      "url": "https://www.yelp.com/search?find_desc=adidas+store&find_loc=Manhattan%2C+NY",
      "desc": "Reviews on Adidas Store in Manhattan, NY - Adidas, Adidas Originals New York SoHo, adidas Sport Performance, Upper 90 Soccer - Manhattan, Nike Soho, ...",
      "title": "Adidas Store Manhattan, NY - Last Updated August 2019 - Yelp",
      "url_shown": "https://www.yelp.com › search › find_desc=adidas+store",
      "pos_overall": 9
    },
    {
      "pos": 6,
      "url": "https://en.wikipedia.org/wiki/Adidas",
      "desc": "Adidas AG is a multinational corporation, founded and headquartered in Herzogenaurach, Germany, that designs and manufactures shoes, clothing and ...",
      "title": "Adidas - Wikipedia",
      "url_shown": "https://en.wikipedia.org › wiki › Adidas",
      "pos_overall": 10
    }
  ]

Product Listing Ads

"pla": [
  {
    "pos": 1,
    "url": "http://www.adidas.com/us/asweego-shoes/F37038.html?cm_mmc=AdieSEM_Feeds-_-GoogleProductAds-_-NA-_-F37038&cm_mmca1=US&cm_mmca2=NA&kpid=F37038&sourceid=543457011",
    "price": "$40.00",
    "title": "adidas Asweego Shoes Black 10.5 - Mens Running Shoes",
    "seller": "adidas",
    "source": ""
  },
  {
    "pos": 2,
    "url": "http://www.adidas.com/us/baseline-shoes/AW4299.html?cm_mmc=AdieSEM_Feeds-_-GoogleProductAds-_-NA-_-AW4299&cm_mmca1=US&cm_mmca2=NA&kpid=AW4299&sourceid=543457011",
    "price": "$50.00",
    "title": "adidas Baseline Shoes White 13K - Originals Shoes",
    "seller": "adidas",
    "source": ""
  },
  ...
  {
    "pos": 29,
    "url": "https://www.zappos.com/product/8466374/color/21766",
    "price": "$79.95",
    "title": "adidas Superstar W Originals Women's Classic Shoes White/Black/White : 9 B - Medium",
    "seller": "Zappos.com",
    "source": ""
  }
]

Top Stories

"top_stories": [
  {
    "url": "https://www.cnet.com/news/spacex-starhopper-prototype-takes-giant-leap-for-elon-musk/",
    "source": "Cnet",
    "headline": "SpaceX Starhopper rocket prototype takes giant leap for Elon Musk",
    "timeframe": "13 hours ago"
  },
  {
    "url": "https://electrek.co/2019/08/27/elon-musk-tesla-china-made-model-3-rumor/",
    "source": "Electrek",
    "headline": "Elon Musk is rumored to unveil first China-made Tesla Model 3 at event this \nweek",
    "timeframe": "16 hours ago"
  },
  {
    "url": "https://www.bloomberg.com/news/articles/2019-08-28/musk-to-join-china-ai-summit-despite-trump-ordering-firms-out",
    "source": "Bloomberg",
    "headline": "Elon Musk and Jack Ma Will Debate AI at China Summit",
    "timeframe": "4 hours ago"
  }
]

Featured Snippet

"featured_snippet": [
  {
    "url": "https://en.wikipedia.org/wiki/Contract_for_difference",
    "desc": "In finance, a contract for difference (CFD) is a contract between two parties, typically described as \"buyer\" and \"seller\", stipulating that the seller will pay to the buyer the difference between the current value of an asset and its value at contract time (if the difference is negative, then the buyer pays instead to ...",
    "title": "Contract for difference - Wikipedia",
    "url_shown": "https://en.wikipedia.org › wiki › Contract_for_difference",
    "pos_overall": 1
  }
]

Knowledge Base

"knowledge": {
  "title": "Adidas",
  "factoids": [
    {
      "title": "Stock price",
      "content": "ADDDF (OTCMKTS) $291.81 +2.74 (+0.95%)Aug 23, 4:00 PM EDT - Disclaimer"
    },
    {
      "title": "Founder",
      "content": "Adolf Dassler"
    },
    {
      "title": "Founded",
      "content": "August 18, 1949, Herzogenaurach, Germany"
    },
    {
      "title": "Headquarters",
      "content": "Herzogenaurach, Germany"
    },
    {
      "title": "Subsidiaries",
      "content": "Reebok, Five Ten Footwear, Runtastic, Ashworth, MORE"
    },
    {
      "title": "Website",
      "content": "https://www.adidas.com/us"
    }
  ],
  "subtitle": "Design company",
  "description": "DescriptionAdidas AG is a multinational corporation, founded and headquartered in Herzogenaurach, Germany, that designs and manufactures shoes, clothing and accessories. It is the largest sportswear manufacturer in Europe, and the second largest in the world, after Nike. Wikipedia"
}

Local Pack

"local_pack": [
  {
    "links": [
      {
        "href": "https://www.adidas.com/us?utm_source=gmb&utm_medium=organic&utm_campaign=US470198_local",
        "title": "Website"
      },
      {
        "href": "#",
        "title": "Directions"
      }
    ],
    "phone": "",
    "title": "adidas Originals Flagship Store",
    "rating": 0,
    "address": "Open ⋅ Closes 7PM",
    "subtitle": "(212) 966-0954",
    "pos_overall": 3,
    "rating_count": 0
  }
]

Twitter Feed

"twitter": [
  {
    "pos": 1,
    "url": "https://twitter.com/elonmusk",
    "title": "Elon Musk (@elonmusk) · Twitter",
    "tweets": [
      {
        "url": "https://twitter.com/elonmusk/status/1166081488648949760?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet",
        "content": "Starhopper flight currently tracking to 5pm Texas time for 150m / ~500ft hover test",
        "timeframe": "11 hours ago"
      },
      {
        "url": "https://twitter.com/elonmusk/status/1165377786338406400?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet",
        "content": "Looks like @SpaceX Starhopper flight may be as soon as Monday. FAA support is much appreciated!",
        "timeframe": "2 days ago"
      },
      {
        "url": "https://twitter.com/elonmusk/status/1165371975528640512?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet",
        "content": "If you’re a utility or public utilities commission, please consider using the Tesla Megapack. Better for the environment & usually lower cost than fossil fuel peaker plants! www.tesla.com/megapack",
        "timeframe": "2 days ago"
      }
    ],
    "pos_overall": 1
  }
]

Job Listings

"jobs": {
  "listings": [
    {
      "title": "SR SOFTWARE DEVELOPER",
      "source": "via LinkedIn",
      "employer": "Jobs @ TheJobNetwork",
      "location": "Tulsa, OK",
      "extra_details": [
        "1 day ago",
        "Full-time"
      ]
    },
    {
      "title": "Autonomous Vehicle Simulation Software Engineer",
      "source": "via Built In Colorado",
      "employer": "Azevtec",
      "location": "United States",
      "extra_details": [
        "17 hours ago",
        "Full-time"
      ]
    },
    {
      "title": "Senior Software Engineer - Oracle Transportation Management",
      "source": "via LinkedIn",
      "employer": "XPO Logistics, Inc.",
      "location": "United States",
      "extra_details": [
        "21 hours ago",
        "Full-time"
      ]
    }
  ],
  "location_header": "Near United States"
}

Carousel

"item_carousel": {
  "items": [
    {
      "title": "Chris Evans",
      "subtitle": "Captain America"
    },
    {
      "title": "Mark Ruffalo",
      "subtitle": "Hulk"
    },
    {
      "title": "Tom Holland",
      "subtitle": "Spider-Man"
    },
    {
      "title": "Stan Lee",
      "subtitle": "Old Man in TV Report, Bus Driver"
    },
    {
      "title": "Chris Pratt",
      "subtitle": "Star-Lord"
    }
  ],
  "title": "The Avengers/Cast"
}

Images

"images": [
  {
    "alt": "Image result for contemporary wall clock",
    "href": "/search?q=contemporary+wall+clock&safe=off&hl=en&gl=US&tbm=isch&source=iu&ictx=1&fir=Qspcw8WiAmXYzM%253A%252C-m-5575uWYilbM%252C_&vet=1&usg=AI4_-kTGLIU9LAzoCJxO8gp7kK322MV8Yg&sa=X&ved=2ahUKEwjFy8rSy7HkAhWkDrkGHck7A24Q9QEwAXoECAkQBg#imgrc=Qspcw8WiAmXYzM:",
    "source": "https://www.allmodern.com/decor-pillows/sb0/wall-clocks-c429917.html"
  },
  {
    "alt": "Image result for contemporary wall clock",
    "href": "/search?q=contemporary+wall+clock&safe=off&hl=en&gl=US&tbm=isch&source=iu&ictx=1&fir=G0pFK8TQ91ls6M%253A%252Cr5nLxZQfxnA3MM%252C_&vet=1&usg=AI4_-kStPZh1tpSdQ5vTAZUIXwW4zThzQg&sa=X&ved=2ahUKEwjFy8rSy7HkAhWkDrkGHck7A24Q9QEwAnoECAkQCQ#imgrc=G0pFK8TQ91ls6M:",
    "source": "https://www.wayfair.com/decor-pillows/cat/modern-wall-clocks-c1869680.html"
  },
  ...
  {
    "alt": "Image result for contemporary wall clock",
    "href": "/search?q=contemporary+wall+clock&safe=off&hl=en&gl=US&tbm=isch&source=iu&ictx=1&fir=o4ZXIngZyr9HAM%253A%252C-m-5575uWYilbM%252C_&vet=1&usg=AI4_-kTIJMWyTs07HFcVKHTfTd6otLL82w&sa=X&ved=2ahUKEwjFy8rSy7HkAhWkDrkGHck7A24Q9QEwCnoECAkQIQ#imgrc=o4ZXIngZyr9HAM:",
    "source": "https://www.allmodern.com/decor-pillows/sb0/wall-clocks-c429917.html"
  }
]

Related Questions

"related_questions": [
  {
    "pos": 1,
    "question": "What does Adidas stand for?"
  },
  {
    "pos": 2,
    "question": "Is Adidas German?"
  },
  {
    "pos": 3,
    "question": "Are Jordans Adidas?"
  },
  {
    "pos": 4,
    "question": "What shoe brands does adidas own?"
  }
]

Shopping Search

...
"organic": [
            {
              "pos": 1,
              "url": "/aclk?sa=l&ai=DChcSEwju8fmd84jpAhUPTxgKHQshDIcYABAHGgJsZQ&sig=AOD64_1BTHVcnNzI5775j9xNkILrCU2KYA&ctype=5&q=&ved=0ahUKEwjpr_Sd84jpAhVI2aYKHYn1CeMQvxMI4wQ&adurl=",
              "type": "grid",
              "price": 85,
              "title": "Adidas Women's Swift Run Casual Shoes in White ...",
              "merchant": {
                "url": "/aclk?sa=l&ai=DChcSEwju8fmd84jpAhUPTxgKHQshDIcYABAHGgJsZQ&sig=AOD64_1BTHVcnNzI5775j9xNkILrCU2KYA&ctype=5&q=&ved=0ahUKEwjpr_Sd84jpAhVI2aYKHYn1CeMQg-UECOoE&adurl=",
                "name": "Finish Line"
              },
              "price_str": "$85.00.",
              "pos_overall": 1
            },
            {
              "pos": 2,
              "url": "/shopping/product/4092922174439754197?uule=w+CAIQICIXQ29sb3JhZG8sIFVuaXRlZCBTdGF0ZXM&q=adidas&prds=epd:6096059639745774212,paur:ClkAsKraX5cxKGk1E_r15f66xbFqydL47KoF9cO04jau1Hw_EeaJnz0EV5mb_JEjRlE5_m7N_B5Vg-krR5766rvdESfkczSSBqkGVDV7A5Ts8BlTUCNfpUxgtxIZAFPVH73vXbe47J5qGlzkfYH83D9zVPSv8w,prmr:1&sa=X&ved=0ahUKEwjpr_Sd84jpAhVI2aYKHYn1CeMQvxMI7AQ",
              "type": "grid",
              "price": 139.97,
              "title": "adidas Mens Alphaboost Training Shoes White ...",
              "merchant": {
                "url": "/aclk?sa=l&ai=DChcSEwju8fmd84jpAhUPTxgKHQshDIcYABAEGgJsZQ&sig=AOD64_3S0xuLlA1GOzNxCvYQdpeTLZkRyQ&ctype=5&q=&ved=0ahUKEwjpr_Sd84jpAhVI2aYKHYn1CeMQg-UECPQE&adurl=",
                "name": "Baseball Savings.com"
              },
              "price_str": "$139.97.",
              "pos_overall": 2
            },
...

Shopping Product

...
{
  "type": "Bundle",
  "items": [
    {
      "value": "Console Only",
      "selected": true,
      "available": true,
      "product_id": "5007040952399054528"
    },
    {
      "value": "Splatoon 2 Bundle",
      "available": false,
      "product_id": "6767220879106424425"
    },
    {
      "value": "Super Mario Odyssey Edition",
      "available": false,
      "product_id": "11634753303078094444"
    }
  ]
}
...

Shopping Product Pricing

"content": {
  "url": "https://www.google.com/shopping/product/5007040952399054528/online",
  "title": "Nintendo Switch with Joy-Con - 32 GB - Gray/Black",
  "rating": 4.5,
  "pricing": [
    {
      "price": 319.99,
      "seller": "Electronic Express",
      "details": "Free shipping",
      "currency": "$",
      "price_tax": 0,
      "price_total": 319.99,
      "seller_link": "/aclk?sa=l&ai=DChcSEwi9t9HqoJ7mAhVCXw0KHdyPBEYYABABGgJxYg&sig=AOD64_2gaL_J1BQ5J5PR-JazDM86N23Nww&adurl=&ctype=5&q=",
      "price_shipping": 0
    },
    {
      "price": 334.99,
      "seller": "ShopZodys",
      "details": "Arrives Dec 9 – 13",
      "currency": "$",
      "price_tax": 27.69,
      "price_total": 412.67,
      "seller_link": "/aclk?sa=l&ai=DChcSEwi9t9HqoJ7mAhVCXw0KHdyPBEYYABADGgJxYg&sig=AOD64_1Rqy4wxKvZXAaoX9FNDBy379EAAA&adurl=&ctype=5&q=",
      "price_shipping": 49.99
    }

Parameter Values

User-Agent

Download full list of user_agent_type values in JSON here.

[
  {
    "user_agent_type": "desktop",
    "description": "Random desktop browser User-Agent"
  },
  {
    "user_agent_type": "desktop_firefox",
    "description": "Random User-Agent of one of the latest versions of desktop Firefox"
  },
  {
    "user_agent_type": "desktop_chrome",
    "description": "Random User-Agent of one of the latest versions of desktop Chrome"
  },
  {
    "user_agent_type": "desktop_opera",
    "description": "Random User-Agent of one of the latest versions of desktop Opera"
  },
  {
    "user_agent_type": "desktop_edge",
    "description": "Random User-Agent of one of the latest versions of desktop Edge"
  },
  {
    "user_agent_type": "desktop_safari",
    "description": "Random User-Agent of one of the latest versions of desktop Safari"
  },
  {
    "user_agent_type": "mobile",
    "description": "Random mobile browser User-Agent"
  },
  {
    "user_agent_type": "mobile_android",
    "description": "Random User-Agent of one of the latest versions of Android browser"
  },
  {
    "user_agent_type": "mobile_ios",
    "description": "Random User-Agent of one of the latest versions of iPhone browser"
  },
  {
    "user_agent_type": "tablet",
    "description": "Random tablet browser User-Agent"
  },
  {
    "user_agent_type": "tablet_android",
    "description": "Random User-Agent of one of the latest versions of Android tablet"
  },
  {
    "user_agent_type": "tablet_ios",
    "description": "Random User-Agent of one of the latest versions of iPad tablet"
  }
]

Locale

Download full list of locale values in JSON here.

[  
   {  
      "locale":{  
         "en-ai":{  
            "description":"Anguilla - English",
            "domain":"com.ai"
         },
         "es-pr":{  
            "description":"Puerto Rico - Spanish",
            "domain":"com.pr"
         },
         ...
         "en-by":{  
            "description":"Belarus - English",
            "domain":"by"
         },
         "en-in":{  
            "description":"India - English",
            "domain":"co.in"
         }
      }
   }
]

Results Language

Download full list of results_language values in JSON here.

[
 {
   "results_language": "af",
   "language": "Afrikaans"
 },
 {
   "results_language": "ar",
   "language": "Arabic"
 },
 ...
 {
   "results_language": "vi",
   "language": "Vietnamese"
 }
]

Geo_location

There are a few ways you can use the geo_location parameter to get correctly-localized Google results.

• Using Google’s Canonical Location Name. This is very straightforward. Just pass us one of the values found on the CSV download here. Example: “geo_location”: “New York,New York,United States”.
• Using a state name. Strip the first part of a Google's Canonical Location Name and pass a geo_location value in a “State,Country” format. Works with United States, Australia, India and other countries with federated states. Example: “geo_location”: “California,United States”.
• Using a country name. To get results localized for the geographical center point of a country, pass an official country name. Example: “geo_location”: “United Kingdom”.
• Using coordinates and radius. To get hyperlocal search results (especially useful for searches such as “restaurants near me”), you can pass latitude, longitude and radius values. The following example passes the coordinates of Space Needle in Seattle, WA: “geo_location”: “lat: 47.6205, lng: -122.3493, rad: 25000”.

If you pass a misspelled geo_location parameter, chances are, either us or Google will interpret and correct it for you. Nonetheless, we recommend using the parameter structures outlined above, combined with the locale and domain parameters, to get the most accurate results.

Account Status

Usage Statistics

You can find your usage statistics by querying the following endpoint:

GET https://data.oxylabs.io/v1/stats

By default the API will return all time usage statistics. Adding ?group_by=month will return monthly stats, while ?group_by=day will return daily numbers.

This query will return all time statistics. You can find your daily and monthly usage by adding either ?group_by=day or ?group_by=month

curl --user user:pass1 'https://data.oxylabs.io/v1/stats'

Sample output:

{
    "data": {
        "sources": [
            {
                "realtime_results_count": "90",
                "results_count": "10",
                "title": "google_hotels"
            },
            {
                "realtime_results_count": "19",
                "results_count": "87",
                "title": "google_search"
            }
        ]
    },
    "meta": {
        "group_by": null
    }
}

Limits

The following endpoint will give your monthly commitment information as well as how much has already been used:

GET https://data.oxylabs.io/v1/stats/limits

curl --user user:pass1 'https://data.oxylabs.io/v1/stats/limits'

Sample output:

{
    "monthly_requests_commitment": 4500000,
    "used_requests": 985000
}

Response Codes

Parsed data response codes:

Cloud storage upload response codes:

References

• https://docs.oxylabs.io/rtc/source/google/index.html
• https://docs.oxylabs.io/rtc/index.html

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Did you heard of the Next-Gen Residential Proxies by Oxylabs? This article will show you how to use it in detail.

Quick Start

Next-Gen Residential Proxies are built for heavy-duty data retrieval operations. They enable effortless web data extraction without any delays or errors. The product is as customizable as a regular proxy, but at the same time it guarantees a much higher success rate. Custom headers and IP stickiness are both supported, alongside reusable cookies and POST requests.

If you have ever used regular proxies for data scraping, integrating Next-Gen Residential Proxies will be a breeze. The only difference is that we require to accept our certificate, or ignore it altogether with -k or --insecure cURL flags (or an equivalent expression in the language of your choice).

To make a request using Next-Gen Residential Proxies, we need to use ngrp.oxylabs.io:60000 endpoint. Below is an example in cURL. You can find code samples in other languages here or full code examples on our GitHub.

curl -k -x ngrp.oxylabs.io:60000 -U "USERNAME:PASSWORD" http://ip.oxylabs.io

If you have any questions not covered by this documentation, please contact your account manager or our support staff at support@oxylabs.io.

Making requests

GET ngrp.oxylabs.io:60000

The easiest way to start is to send us a simple query without any custom options. We will add all standard headers on our end, pick the fastest proxy and deliver you the response body.

To utilize desired functionalities of Next-Gen Residential Proxies such as setting up proxy geolocation or reusing the same IP via session control, we need to send additional headers with the request.

Here's the full list of supported functionalities and headers:

Query parameters

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io"

Session

If you want to use the same proxy to make multiple requests, you can do that by adding the X-Oxylabs-Session-Id header with a randomly-generated string for the session ID. We will assign a proxy to this ID and keep it for up to 10 minutes. After that a new proxy will be assigned to that particular session ID.

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "X-Oxylabs-Session-Id: 123randomString"

Geo-Location

Some websites will not serve content if accessed from unsupported geo-locations. You can specify in your request which country you want to access the target from. Just add the X-Oxylabs-Geo-Location header with the value set to a country name, for example, Germany for Germany or United States for the United States.

Next-Gen Residential Proxies also support city-level targeting. If you want to add a city, simply add the city name before the country, e.g,: Munich, Germany.

The full list of supported geo-location parameter values can be found here.

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "X-Oxylabs-Geo-Location: Munich,Germany"

Headers

If you know more than we do about a particular target, you can add your own headers to your request. It can be both standard headers, such as User-Agent or Accept-Language, and something completely custom and target-specific.

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "Your-Custom-Header: interesting header content" -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36" -H "Accept-Language: en-US"

Cookies

The system also allows setting custom cookies to target website. With your initial request we will return all response headers and cookies. You can modify them on your end and send them back to our system with the next request. It may be a good idea to reuse the cookies if you use the same IP address to make a few consecutive requests (see Session for more)

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "Cookie: NID=1234567890; 1P_JAR=0987654321"

Custom Status Code

By default we assume that the request is successful as long as it returns a 2xx or a 4xx status code. However, sometimes websites return the required content together with a non-standard HTTPS status code. If one of your targets does that, you can indicate which status codes are acceptable and actually valuable for you. Simply add X-Oxylabs-Status-Code header with all HTTP response codes that actually work for you. Please note that 2xx and 4xx will still be automatically marked as successful.

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "X-Oxylabs-Status-Code: 500,501,502,503" 

POST Requests

Next-Gen Residential Proxies don't only support sending GET requests, but also let you POST to a web endpoint of your choice. This means that you can send data to a target website, which can then make the website return a different result.

curl -X POST -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -d "@/path/to/file.json"

JavaScript rendering

Enable JavaScript rendering. Use when the target requires JavaScript to load content. There are two available values for this parameter: html (get raw output) and png (get a Base64-encoded screenshot).

curl -k -v -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "X-Oxylabs-Render: html"

Adaptive parsing

Adaptive parsing is capable of parsing the most important fields from any e-commerce product page. To enable adaptive parsing for e-commerce product pages it is required to send two additional headers: X-Oxylabs-Parser-Type: ecommerce_product to select the parser type and X-Oxylabs-Parse: 1 to parse the retrieved page.

Fields that can be parsed by Next-Gen Residential Proxies Adaptive Parsing for e-commerce product pages:

• Title
• Regular price
• Offer price
• Currency
• Product description
• Product IDs
• Image URLs
• URLs from the page

curl -v -k -x ngrp.oxylabs.io:60000 -U user:pass1 "https://ip.oxylabs.io" -H "X-Oxylabs-Parser-Type: ecommerce_product" -H "X-Oxylabs-Parse: 1"

Sample output:

{
    "results": [
        {
            "content": {
                "url": "https://ip.oxylabs.io/product/product_example.html",
                "body": {
                    "price": 11.99,
                    "title": "Example product title",
                    "currency": "$",
                    "old_price": 15.99,
                    "description": null,
                    "image_links": [
                        "https://ip.oxylabs.io/img/product_example.png"
                    ],
                    "ids_from_url": [],
                    "simple_links": [
                        {
                            "link": "https://ip.oxylabs.io/product/another_example.html",
                            "description": "Another product example"
                        }
                    ],
                    "ids_from_html": [
                        {
                            "Product number": "14158288"
                        }
                    ],
                    "price_range_lower": null,
                    "price_range_upper": null
                },
                "meta": {
                    "title": "Example product meta title",
                    "keywords": [],
                    "description": "Example product description"
                },
                "parse_status_code": 12000
            },
            "created_at": "2020-01-01 10:00:00",
            "updated_at": "2020-01-01 10:00:07",
            "id": 29964797,
            "page": 1,
            "url": "https://ip.oxylabs.io/product/product_example.html",
            "job_id": "6699272813062145025",
            "status_code": 200
        }
    ]
}

Usage Statistics

You can find your usage statistics by visiting our Dashboard or by querying the following endpoint:

GET https://data.oxylabs.io/v1/stats

By default the API will return all time usage statistics. Adding ?group_by=month will return monthly stats, while ?group_by=day will return daily numbers. Traffic is being displayed in bytes.

his query will return all time statistics. You can find your daily and monthly usage by adding either ?group_by=day or ?group_by=month. Traffic is being displayed in bytes.

curl --user user:pass1 'https://data.oxylabs.io/v1/stats'

Sample output:

{
    "meta": {
        "group_by": null,
        "date_from": null,
        "date_to": null,
        "source": null
    },
    "data": {
        "sources": [
            {
                "results_count_all": "1482",
                "results_count": "0",
                "realtime_results_count": "0",
                "super_api_results_count": "12777",
                "render": "0",
                "geo_location": "0",
                "average_response_time": 2.18,
                "request_traffic": "6629", 
                "response_traffic": "17850",
                "title": "universal"
            }
        ]
    }
}

Sample Response

This is what the response looks like. To see headers and cookies, Verbose mode must be enabled.

HTTP/1.1 200 OK
X-Job-Id: 1234567890123456
X-Session-Id: 123randomString
cf-cache-status: DYNAMIC
cf-ray: 55c2ab837eddcba8-VIE
content-encoding: gzip
content-length: 72657
content-type: text/html
date: Tue, 1 Jan 2020 00:00:01 GMT,expect-ct: max-age=604800,report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
last-modified: Tue, 1 Dec 2020 00:00:00 GMT
server: cloudflare
status: 200
strict-transport-security: max-age=31536000
X-DNS-Prefetch-Control: off
Set-Cookie: NID=1234567890; expires=Wed, 29-Jul-2020 10:56:21 GMT
Set-Cookie: 1P_JAR=0987654321; expires=Wed, 29-Jul-2020 10:56:21 GMT
x-cache: MISS from localhost
x-cache-lookup: MISS from localhost:3129content here

References

https://docs.oxylabs.io/next-gen-residential-proxies/index.html#quick-start

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Want to use OxyLabs' residential proxies? Then you can't miss this article. It is the most comprehensive OxyLabs residential proxies user guide.

Quick Start

This is by far the fastest way to start using Residential Proxies. You will make a request to https://ipinfo.io using a residential proxy from a random location through our Backconnect Entry. Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME:PASSWORD" https://ipinfo.io

Let's make a request using a proxy from France. All you need to do is add cc-FR to your username string. Below is an example in cURL, and here you can find code samples in other languages.

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-cc-FR:PASSWORD" https://ipinfo.io

Here is one more example of a request that goes through a proxy from London, United Kingdom. Adding cc-GB-city-london to username string enables you to do just that. Click here to get more information on how to master city-level proxy targeting.

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-cc-GB-city-london:PASSWORD" https://ipinfo.io

What if you want to keep the same proxy for more than one request? Session control enables that. Just add sessid-abc12345 to your username string, where abc12345 can be any random string, and as long as you keep sending requests with this string (session ID), we will try our best to give you the same proxy IP.

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-sessid-randomString123:PASSWORD" https://ipinfo.io

Check out our integration tutorials with the most popular 3rd party tools, such as Multilogin, ShadowRocket, or SwitchyOmega.

If you have any additional questions, please contact your account manager or our support team at support@oxylabs.io.

To find out your usage statistics, generated traffic, request count and more, please login to dashboard.oxylabs.io. You can also add proxy users (subusers) and track their statistics separately.

Backconnect Entry Node

pr.oxylabs.io:7777

The single backconnect proxy enables to choose a specific country or city proxy via additional parameters in the username. This approach also supports session control. Below is a sample credentials structure:

customer-USERNAME-cc-US-city-CITY-sessid-abcde12345:PASSWORD

If you add more than one location parameter in the username, the system will ignore all but one. Therefore it's important to understand which parameters get prioritized. The hierarchy is as follows:

1. city
2. state
3. cc

For example, if your query has these parameters customer-username-cc-US-city-paris-st-us_california, you will get a proxy from Paris, US.

Entry Node for China

https://a81298871.com:8000 - Hong Kong

https://cnt9t1is.com:8000 - Beijing

For better connectivity from China, we have configured entry nodes in Hong Kong and Beijing regions. Note that it is an HTTPS proxy node. All username parameters are identical to the regular entry node.

Please note that some 3rd party tools currently do not work with Hong Kong and Beijing entry nodes. 3rd party tools that work with said entry nodes are:

1. 1. 1. Oxylabs Proxy Manager
      2. SwitchyOmega
      3. Shadowrocket
      4. Postern

We are currently working on fixing the rest of the integrations on which HTTPS protocol will be supported.

If you are connecting to our residential network from China and still have issues with provided entry nodes, please get in touch at support@oxylabs.io and we will figure out a solution together.

Basic Query

Basic query only requires to pass username and password. No other parameters are needed. Such query will result in the request being made from a random IP address (proxy). Every new request will use a different proxy.

In this example a query to ipinfo.io is performed from a random IP:

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME:PASSWORD" https://ipinfo.io

Select Country

Adding cc flag to the authorization header enables to specify which country IP to use to process the request. The value of this parameter is a case insensitive country code in 2-letter 3166-1 alpha-2 format. For example, DE for Germany, GB for the United Kingdom, TH for Thailand proxy. See examples for more details.

In this example a query to ipinfo.io is performed from a random IP address from Germany:

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-cc-DE:PASSWORD" https://ipinfo.io

Select State

To get proxy from a specified US state, add st flag to the authorization string. For example, us_california, us_illinois. Full list of supported cities in TXT format can be downloaded here.

In this example a query to ipinfo.io is performed from a random IP address from California, USA:

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-st-us_california:PASSWORD" https://ipinfo.io

Select City

To narrow down to a city-level targeting, a city parameter needs to be added. For example, cc-DE-city-munich means that a proxy from Munich, Germany, will handle the query. We support every city in the world, but we do not guarantee we will have proxies there. Most popular cities are well covered, and will have many proxies to choose from.

Here are a few examples of valid combinations of cc and city parameters:
cc-US-city-los_angeles
cc-IT-city-rome
cc-TH-city-bangkok
cc-JP-city-tokyo
cc-AU-city-sydney
cc-ES-city-barcelona

In this example a query to ipinfo.io is performed from a random IP address from Munich, Germany:

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-cc-DE-city-munich:PASSWORD" https://ipinfo.io

Session Control

Adding sessid parameter with session ID string, for example sessid-abc12345, enables session control. This means that the proxy will not change with the following requests. The example below shows how session control works:

The system keeps the same IP associated with a particular session ID as long as there is some activity (requests). After 60 seconds of inactivity, the IP is automatically changed to a different one.

For example: * Your initial query with sessid-random123 got assigned proxy IP address 1.1.1.1 * As long as you keep sending new requests with the same session ID and that IP address is online and available, the system will route your queries through 1.1.1.1 * If you stop sending requests or the IP address is no longer online, the system will assign a new proxy IP after 60 seconds. * This means that your next query with sessid-random123 will be routed through a different IP address.

Standard session time is 10 minutes, but it can be increased. Contact your account manager to learn more.

German IP will be chosen for the first request, and then the same IP will be kept with new queries (session control):

curl -x pr.oxylabs.io:7777 -U "customer-USERNAME-cc-DE-sessid-abcde12345:PASSWORD" https://ipinfo.io

Country Specific Entry Nodes

Random Proxy Entry Nodes

Oxylabs residential network has coverage in almost every country in the world. Country-specific random proxy entry point will return new IP with every new request.

You do not need to pass any additional parameters, just customer-username:password. If you have whitelisted IPs, you do not need to pass login credentials. Country-specific ports do not support city-level results.

Please see the dashboard for entry point information or download a full list of random entry points in XLS here.

In this example a query to ipinfo.io is performed from a random Turkey IP:

curl -x tr-pr.oxylabs.io:30000 -U "customer-USERNAME:PASSWORD" https://ipinfo.io

pr.oxylabs.io:7777
us-pr.oxylabs.io:10000
ca-pr.oxylabs.io:30000
gb-pr.oxylabs.io:20000
de-pr.oxylabs.io:30000
fr-pr.oxylabs.io:40000
es-pr.oxylabs.io:10000
it-pr.oxylabs.io:20000
se-pr.oxylabs.io:30000
gr-pr.oxylabs.io:40000
pt-pr.oxylabs.io:10000
nl-pr.oxylabs.io:20000
be-pr.oxylabs.io:30000
ru-pr.oxylabs.io:40000
ua-pr.oxylabs.io:10000
pl-pr.oxylabs.io:20000
il-pr.oxylabs.io:20000
tr-pr.oxylabs.io:30000
au-pr.oxylabs.io:40000
my-pr.oxylabs.io:10000
th-pr.oxylabs.io:20000
kr-pr.oxylabs.io:30000
jp-pr.oxylabs.io:40000
ph-pr.oxylabs.io:10000
sg-pr.oxylabs.io:20000
cn-pr.oxylabs.io:30000
hk-pr.oxylabs.io:40000
tw-pr.oxylabs.io:10000
in-pr.oxylabs.io:20000
pk-pr.oxylabs.io:30000
ir-pr.oxylabs.io:40000
id-pr.oxylabs.io:10000
az-pr.oxylabs.io:20000
kz-pr.oxylabs.io:30000
ae-pr.oxylabs.io:40000
mx-pr.oxylabs.io:10000
br-pr.oxylabs.io:20000
ar-pr.oxylabs.io:30000
cl-pr.oxylabs.io:40000
pe-pr.oxylabs.io:10000
ec-pr.oxylabs.io:20000
co-pr.oxylabs.io:30000
za-pr.oxylabs.io:40000
eg-pr.oxylabs.io:10000
sa-pr.oxylabs.io:44000
dk-pr.oxylabs.io:19000
ao-pr.oxylabs.io:40000
cm-pr.oxylabs.io:41000
cf-pr.oxylabs.io:42000
td-pr.oxylabs.io:43000
bj-pr.oxylabs.io:44000
et-pr.oxylabs.io:45000
dj-pr.oxylabs.io:46000
gm-pr.oxylabs.io:47000
gh-pr.oxylabs.io:48000
ci-pr.oxylabs.io:49000
ke-pr.oxylabs.io:10000
lr-pr.oxylabs.io:11000
mg-pr.oxylabs.io:12000
ml-pr.oxylabs.io:13000
mr-pr.oxylabs.io:14000
mu-pr.oxylabs.io:15000
ma-pr.oxylabs.io:16000
mz-pr.oxylabs.io:17000
ng-pr.oxylabs.io:18000
sn-pr.oxylabs.io:19000
sc-pr.oxylabs.io:20000
zw-pr.oxylabs.io:21000
ss-pr.oxylabs.io:22000
sd-pr.oxylabs.io:23000
tg-pr.oxylabs.io:24000
tn-pr.oxylabs.io:25000
ug-pr.oxylabs.io:26000
zm-pr.oxylabs.io:27000
af-pr.oxylabs.io:28000
bh-pr.oxylabs.io:29000
bd-pr.oxylabs.io:30000
am-pr.oxylabs.io:31000
bt-pr.oxylabs.io:32000
mm-pr.oxylabs.io:33000
kh-pr.oxylabs.io:34000
ge-pr.oxylabs.io:36000
iq-pr.oxylabs.io:37000
jo-pr.oxylabs.io:38000
lb-pr.oxylabs.io:39000
mv-pr.oxylabs.io:40000
mn-pr.oxylabs.io:41000
om-pr.oxylabs.io:42000
qa-pr.oxylabs.io:43000
sa-pr.oxylabs.io:44000
vn-pr.oxylabs.io:45000
tm-pr.oxylabs.io:46000
uz-pr.oxylabs.io:47000
ye-pr.oxylabs.io:48000
al-pr.oxylabs.io:49000
ad-pr.oxylabs.io:10000
at-pr.oxylabs.io:11000
ba-pr.oxylabs.io:13000
bg-pr.oxylabs.io:14000
by-pr.oxylabs.io:15000
hr-pr.oxylabs.io:16000
cy-pr.oxylabs.io:35000
cz-pr.oxylabs.io:18000
dk-pr.oxylabs.io:19000
ee-pr.oxylabs.io:20000
fi-pr.oxylabs.io:21000
hu-pr.oxylabs.io:23000
is-pr.oxylabs.io:24000
ie-pr.oxylabs.io:25000
lv-pr.oxylabs.io:26000
li-pr.oxylabs.io:27000
lt-pr.oxylabs.io:28000
lu-pr.oxylabs.io:29000
mt-pr.oxylabs.io:30000
mc-pr.oxylabs.io:31000
md-pr.oxylabs.io:32000
me-pr.oxylabs.io:33000
no-pr.oxylabs.io:34000
ro-pr.oxylabs.io:35000
rs-pr.oxylabs.io:36000
sk-pr.oxylabs.io:37000
si-pr.oxylabs.io:38000
ch-pr.oxylabs.io:39000
mk-pr.oxylabs.io:40000
bs-pr.oxylabs.io:41000
bz-pr.oxylabs.io:42000
vg-pr.oxylabs.io:43000
cr-pr.oxylabs.io:44000
cu-pr.oxylabs.io:45000
dm-pr.oxylabs.io:46000
ht-pr.oxylabs.io:47000
hn-pr.oxylabs.io:48000
jm-pr.oxylabs.io:49000
aw-pr.oxylabs.io:10000
pa-pr.oxylabs.io:11000
pr-pr.oxylabs.io:12000
tt-pr.oxylabs.io:13000
fj-pr.oxylabs.io:14000
nz-pr.oxylabs.io:15000
bo-pr.oxylabs.io:16000
py-pr.oxylabs.io:17000
uy-pr.oxylabs.io:18000
ve-pr.oxylabs.io:19000
kw-pr.oxylabs.io:20000

Sticky Proxy Entry Nodes

Country-specific sticky proxy entry point will return the same IP with every new request while you will use the same port. IP stickiness works for up to 10 minutes. After that, the IP is replaced with a new one.

You do not need to pass any additional parameters, just customer-username:password. If you have whitelisted IPs, you do not need to pass login credentials. Country-specific ports do not support city-level results.

Please see the dashboard for entry point information or download a full list of sticky entry points in XLS here.

In this example, a query to ipinfo.io is performed from Turkey IP, and the same IP will stay for up to 10 minutes with each request. We have chosen port 30001, however, Turkey sticky entry points cover ports from 30001 to 39999.

curl -x tr-pr.oxylabs.io:30001 -U "customer-USERNAME:PASSWORD" https://ipinfo.io

pr.oxylabs.io:10000 - 49999
us-pr.oxylabs.io:10001 - 19999
ca-pr.oxylabs.io:30001 - 39999
gb-pr.oxylabs.io:20001 - 29999
de-pr.oxylabs.io:30001 - 39999
fr-pr.oxylabs.io:40001 - 49999
es-pr.oxylabs.io:10001 - 19999
it-pr.oxylabs.io:20001 - 29999
se-pr.oxylabs.io:30001 - 39999
gr-pr.oxylabs.io:40001 - 49999
pt-pr.oxylabs.io:10001 - 19999
nl-pr.oxylabs.io:20001 - 29999
be-pr.oxylabs.io:30001 - 39999
ru-pr.oxylabs.io:40001 - 49999
ua-pr.oxylabs.io:10001 - 19999
pl-pr.oxylabs.io:20001 - 29999
il-pr.oxylabs.io:20001 - 29999
tr-pr.oxylabs.io:30001 - 39999
au-pr.oxylabs.io:40001 - 49999
my-pr.oxylabs.io:10001 - 19999
th-pr.oxylabs.io:20001 - 29999
kr-pr.oxylabs.io:30001 - 39999
jp-pr.oxylabs.io:40001 - 49999
ph-pr.oxylabs.io:10001 - 19999
sg-pr.oxylabs.io:20001 - 29999
cn-pr.oxylabs.io:30001 - 39999
hk-pr.oxylabs.io:40001 - 49999
tw-pr.oxylabs.io:10001 - 19999
in-pr.oxylabs.io:20001 - 29999
pk-pr.oxylabs.io:30001 - 39999
ir-pr.oxylabs.io:40001 - 49999
id-pr.oxylabs.io:10001 - 19999
az-pr.oxylabs.io:20001 - 29999
kz-pr.oxylabs.io:30001 - 39999
ae-pr.oxylabs.io:40001 - 49999
mx-pr.oxylabs.io:10001 - 19999
br-pr.oxylabs.io:20001 - 29999
ar-pr.oxylabs.io:30001 - 39999
cl-pr.oxylabs.io:40001 - 49999
pe-pr.oxylabs.io:10001 - 19999
ec-pr.oxylabs.io:20001 - 29999
co-pr.oxylabs.io:30001 - 39999
za-pr.oxylabs.io:40001 - 49999
eg-pr.oxylabs.io:10001 - 19999
sa-pr.oxylabs.io:44001 - 44999
dk-pr.oxylabs.io:19001 - 19999

Additional Information

Country, State, City Mapping

We use MaxMind GeoIP2 database to assign our exit nodes cc, st and city values. This database is updated weekly.

Some other public IP databases may show different locations. In cases where location mismatch is crucial, we encourage to double-check if the location of the assigned proxy is the same on the IP database used by the target website.

Response Codes

These are the most common response codes that you can get while using our residential proxy pool. If you receive any other response code, get in touch with your dedicated account manager for more information.

Backconnect Entry IPs

If you want to use your own backconnect entry node DNS name, you can do that by pointing your subdomain/domain to our entry node. To do that, you need to add a DNS CNAME record that has pr.oxylabs.io as the target. This is how such setup looks on Cloudflare DNS manager for entry subdomain:

For more information, please contact support@oxylabs.io or your account manager.

Restricted URLs

To avoid abuse and unclear activities, some websites are restricted on our residential proxy network. The list includes, but is not limited to websites such as all Google domains including Play, linkedin.com, sonyentertainmentnetwork.com, all Apple domains, including iTunes, netflix.com and others. For more information please contact our support staff at support@oxylabs.io.

Public API

To manage proxy users via Public API, please refer to this documentation in OpenAPI format. The API allows to create, delete, and modify proxy users, set their limits and see current usage, disable and enable them.

3rd Party Integrations

Please note that you must use an HTTPS protocol when accessing China entry nodes for 3rd party tools to work correctly.

Oxylabs Proxy Manager

Oxylabs Proxy Manager is a proxy management extension for Chrome. It is not only free but also works with any proxy provider of your choice. Enable your proxy without having to jump through settings and menu options. The extension has an additional feature specifically designed to use with Oxylabs Residential Proxies, which enables to easily change Oxylabs Residential Proxy session without editing your username. Download the extension from Chrome Web Store here

• Click the extension icon to open the application
• If you haven't added any proxies yet, you will see Add new proxy button. Click it to continue.

• Add proxy profile. Provide a name for the profile under Enter name. Enter pr.oxylabs.io as Proxy server IP and 7777 as Port. Fill in your Username and Password. Do not forget to add customer- before your username. Finally, click Save changes.

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Proxy server IP and 10001 under Port, you will receive US exit node with sticky session.

If you have whitelisted your IP, you can skip this step.

• Open the extension and click Connect. That's it, you are now using proxies.

• Chrome is notorious for caching proxy credentials, and on many proxy management extensions sticky sessions via username string does not work. If you are using sessions with main entry node (pr.oxylabs.io:7777), we created a special feature that allows rotating session much more easily. First, enter your username with sessid parameter, like this:

• Then, open the extenion window. Connect to the proxy and you will see a new button – Refresh session. Click it to change session ID and get a new proxy.

SwitchyOmega

SwitchyOmega is a powerful and reliable proxy manager that works with many popular browsers. First you need to get the plugin. For Chrome, click here, for Firefox click here. Once you have it installed, you should:

• Open Switchy Omega options.

• Click New profile….

• Give this new profile a name, choose Proxy Profile type and click Create

• Change Protocol to HTTP. Under Server enter our backconnect entry node pr.oxylabs.io. Under Port add 7777

• You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Server and 10001 under Port, you will receive US exit node with sticky session.
• Next, click the lock button on the right. Fill in your Username and Password. Do not forget to add customer- before your given username. Finally, click Save changes.

If you have whitelisted your IP, please skip this step.

• Click Apply changes

• For the final step, click on SwitchyOmega icon and choose your newly configured proxy. You are now ready to go!

Shadowrocket

If you want to use our Residential Proxies with Shadowrocket, you can easily do that by following the instructions below.

• Open Shadowrocket app.
• Click Add Server.

• In the following window click on Type.

• Choose HTTP and go back to the previous screen.

• Enter proxy and authentication information. Under Host type in pr.oxylabs.io. Port is 7777 to use our backconnect entry. Do not forget your crendentials under User and Password. Do not forget to add customer- before your given username. Once you are done, go back to the previous screen by pressing Back arrow.

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Host and 10001 under Port, you will receive US exit node with a sticky session.

• You need to turn proxy on. Do that by activating a toggle under Not Connected

• If that is your first time using Shadowrocket, you will get an iOS prompt to add Shadowrocket to VPN configurations. Click Add. You will need to enter your iPhone passcode to confirm.

• That is it. A toggle at the top of the app should now be activated.

• You can test if the proxy is working by visiting icanhazip.com.

Proxifier

If you want to use our Residential Proxies with Proxfier, you can easily do that by following the instructions below.

• Open Proxifier app.
• Add proxies. Click on Proxies (1). Click Add(2).

• In the following window enter required information. Address(1), which is pr.oxylabs.io, Port(2) which is 7777, select HTTPS(3) protocol. In authentication box click Enable(4) and enter provided Username(5) and Password(6). Do not forget to add customer- at the beginning of the username. If you have whitelisted your devices, do not click on Enable, your authenticiation will be done without username/password. If you wish, you can enable Appear as Web Browser option. Click OK.
• You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Address(1)* and **10001 under Port(2), you will receive US exit node with sticky session.

• You will be prompted with a window asking if you want Proxifier to use this proxy by default? This means that this setup will be user for all of your browsers. It is up to you to decide. If you select Yes, you are done setting up. If you wish to use proxies on a particular browser, follow the next steps.
• Click on Rules(1) and then click Add…(2).

• Click +(1) button to add applications that you want to use proxy settings. Select the browser with which you want to use these proxy settings and click on it. Finally, you have to change Action(2). Click on it and select your created proxy.

• This is it, all of your requests, while using browser which you defined in your rules of Proxifier, will be done from proxies.

Multilogin

Our proxies also work with Multilogin app. Setting up is easy, just follow the instructions below.

• Open Multilogin app.
• Add proxies. Click on Create new under Browser profile.
• In the following window enter Browser profile name (for example Oxylabs). Next, click Edit proxy settings.
• Choose HTTP proxy as Connection type. Now fill in the required details: under New address and Port insert pr.oxylabs.io and 7777. Under Login enter your username, and Password. Do not forget to add customer- at the beginning of the username. If you have whitelisted your devices, you do not need to provide the login credentials.
• You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under New address and 10001 under Port, you will receive US exit node with a sticky session.
• Click Check proxy. If everything is fine, you should get a response indicating External IP.
• Confirm changes by clicking Create profile.
• This is it, you are good to go.

Postern

Postern is a very popular proxy/VPN management application on Android. You can get it on Google Play Store. Here is a step-by-step guide how to configure our Residential Proxies on Postern:

• Open the app.
• Click Add Proxy.

• In the following window enter required proxy information. Server Name is Oxylabs, Server Address is pr.oxylabs.io. Under Server Port enter 7777. Click Server Type and choose HTTPS/HTTP CONNECT. Next, enter your login credentials under Username and Password. For this to work, you must type customer- before you type in your username, e.g., customer-username.Finally, click Save.

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Server Address and 10001 under Server Port, you will receive the US exit node with sticky session.

• Now, open the app menu and go to Rules.

• Click Add Rule

• Under Match Method select Match All. Choose Proxy/Tunnel as the Rule. Proxy/Proxy Group should automatically fill up with Oxylabs – pr.oxylabs.io:7777″. Click Save.

• Now you need to turn on the VPN. Open the app menu and click on VPN Off to activate the connection. That is it, you are now using proxies.

GoLogin

GoLogin is a multi-login browser management application for Windows, Linux and Mac OS. Get it on its official website here. Here is a step-by-step guide how to configure our Residential Proxies on GoLogin for Windows:

• Open the app.
• Click Create Your First Profile.

• In the following window enter the required proxy information. Under Profile Name write Oxylabs. Next, choose Connection Type as HTTP Proxy. Under Proxy Host and Port enter pr.oxylabs.io and 7777. Next, enter your login credentials under Username and Password. Finally, click Create Profile.

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io and 10001 under Proxy Host and Port, you will receive the US exit node with a sticky session. To fully utilize all available Residential Proxies features such as sessions and country, state, city geolocation parameters, please reffer to the corresponding sections in the documentation.

• All set, now simply click Run and wait for the browser to load.

• • If everything is fine, a website http://myip.gologin.app will open and show the IP information.

Kameleo

Kameleo is a multi-login browser management application for Windows and Android. Get it on its official website here. Here is a step-by-step guide how to configure our Residential Proxies on Kameleo for Windows:

• Open the app.
• Click APPLICATION SETTINGS.

• In the following window enter the required proxy information under Global proxy settings. For Upstream Proxy Type choose Http. Next, fill in Proxy IP or hostname with pr.oxylabs.io. Under Proxy Port enter 7777. Finally, enter your login credentials under Username and Password. Close the window to save settings.

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Proxy IP or hostname and 10001 under Proxy Port, you will receive the US exit node with a sticky session.

• That is it, you are ready to go. Either create a NEW PROFILE or click QUICK PROFILE to launch the browser which now has proxy enabled.

ClonBrowser

ClonBrowser is another multi-login browser management application for Windows and Mac. Get it on its official website here. Here is a step-by-step guide how to configure our Residential Proxies on ClonBrowser for Windows:

• Open the app.
• Click Create a New browser profile.

• Click Setting next to Proxy settings.

• In the following window choose Http as the Connection type. Under IP or host enter pr.oxylabs.io. Fill in 7777 for Port. Finally, enter your login credentials under Username in customer-username format and Password.

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under Proxy IP or hostname and 10001 under Proxy Port, you will receive the US exit node with a sticky session.

You can check if the proxy works by clicking Connection test. If everything is fine, click Create profile.

• That is all, you can now launch the browser by clicking Start under Option.

AdsPower

AdsPower is a multi-login browser management application for Windows and Mac. You get it on its official website here. Here is the step-by-step guide how to configure our Residential Proxies on AdsPower for Windows:

• Open the app.
• Click Upload..

• Under Proxy Type choose http. Now fill in the form bellow. Enter pr.oxylabs.io for Proxy Host and 7777 for Proxy Port. Enter your login credentials customer-username under Proxy User and your password for Password.

You can also use country specific entries. For example, if you put us-pr.oxylabs.io under Proxy Host and 10001 under *Proxy Port, you will receive US exit node with sticky session.

You can check if the proxy works by clicking Check Proxy. If everything is fine, click Create profile.

• That's all, you can now launch the browser by clicking Open under Account Management.

VMLogin

Virtual Multi Login, or VMLogin, is a tool for creating virtual browser profiles that help control browser fingerprinting, setup business workflows, and develop web automation, among other things. To use VMLogin with Oxylabs Residential Proxies, download the latest version of VMLogin here.

• Launch VMLogin and create a new browser profile. You can either click Get random profile (1), or select your settings.
• Click on Setting proxy server (2)

• Create a new Residential Proxy user.

• Click on Setting proxy server and choose HTTP Proxy as the Connection type. Under IP or host enter pr.oxylabs.io. Fill in 7777 for Port. Finally, enter your login credentials under Username (in customer-username format) and Password. You can check if the proxy works by clicking Test Proxy. If everything is fine, save profile and launch the browser.

• You can also use country specific entries. For example, if you put us-pr.oxylabs.io under Proxy IP address and 10001 under Proxy Port, you will receive a US exit node with a sticky session.

• This is all, you are ready to use Residential Proxies with VMLogin.

Undetectable

Firstly, you will need to get Undetectable. To do that, simply go to undetectable.io, sign up, and download the browser.

Then, open the application and click “+” to create a new profile.

In the following window, under New Profile, enter the new profile name. Choose the settings (OS, browser, configurations, and screen) of how you want your browser to be recognized by websites.

Also, you can select the main and additional languages, while geo-location will be determined by your proxy.

Go to the Proxy tab and select the “New proxy” option in the box.

You can choose either SOCKS5 or HTTP (select HTTP proxy as a connection type). Now fill in proxy and authentication details: type pr.oxylabs.io for your Host and use 7777 for Port. Enter your credentials in the Login and Password spaces. Do not forget to add customer- at the beginning of your username. If you have whitelisted your devices, you do not need to provide login credentials.

You can also use country-specific entries. For example, if you put us- pr.oxylabs.io under Host and 10001 under Port, you will receive US exit node with a sticky session. Please check out our complete list of country-secific entry nodes.

Click the Check button. If everything is fine, you should get a response indicating External IP. Press the Save Proxy button to confirm the changes.

Now save the profile by pressing the Save button, and you will see it appear on the left side menu. Edit your profiles easily by simply clicking on them.

That's it! You have successfully set up your proxies with Undetectable.io.

Puppeteer

Before getting started with Puppeteer, you’ll need to install some basic tools: Node.js and a code editor of your choice. After that, create a Node.js project and install the required packages. You may find a detailed guide on how to install and run Puppeteer on our blog post.

Once everything is set up, we can move on to the next part – Oxylabs Residential Proxies integration with Puppeteer.

• Within Puppeteer, fill in the value, for example:

pr.oxylabs.io:7777

• Under ‘page.authenticate’ input your Oxylabs proxy sub-user username in the ‘username’ value and your password.
• The example of a code looks like this:

const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch({
    headless: false,
    args: ['--proxy-server=pr.oxylabs.io:7777]  
});
  const page = await browser.newPage();
    await page.authenticate({
        username: 'USERNAME',
        password: 'PASSWORD'
    });
    await page.goto('https://ip.oxylabs.io');
    await page.screenshot({path: 'example.png'});
    await browser.close();
})();

You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under ‘host’ and 10001 under ‘port’, you will receive a US exit node with a sticky session. Please check out our documentation for a complete list of country-specific entry nodes.

And that’s it! You’ve successfully integrated Oxylabs Residential proxies with Puppeteer.

References,

https://docs.oxylabs.io/residential/index.html

Disclaimer: This part of the content is mainly from the merchant. If the merchant does not want it to be displayed on my website, please contact us to delete your content.

Want to use OxyLabs' datacenter proxies? Then you can't miss this article. It is the most comprehensive OxyLabs datacenter proxies user guide.

Quick Start

Before we begin, please make sure you have received a username, password, and all URLs linking to your IP list from a Sales Representative, Dedicated Account Manager, or Support Team.

There are two ways you can get to your IP list. You can either access the IP list directly from your browser by opening the URL you received (e.g.: https://proxy.oxylabs.io/all). You will then be asked to enter your login credentials (HTTP Basic Auth).

Or you can retrieve the list by making a GET request to the URL you received (for example: https://proxy.oxylabs.io/all). Don't forget to replace USERNAME and PASSWORD with your proxy user credentials.

curl -u "USERNAME:PASSWORD" https://proxy.oxylabs.io/all

curl -x 1.2.3.4:60000 -U "USERNAME:PASSWORD" http://ip.oxylabs.io

Proxy Lists

Basic

You should receive a URL to your proxy list from your Dedicated Account Manager or our Support and it should look something like this:

https://proxy.oxylabs.io/key/a27c5343ce4c4637a85a9297ddf1cace

Simply open this URL in your browser (or any other prefered software). You will then be asked to enter your login credentials (HTTP Basic Auth):

Once you do that, the list of proxies will be shown in a plain text.

If you have multiple lists and wish to get all IPs assigned to your account, you can use this URL:

https://proxy.oxylabs.io/all

You can also include ?showCountry or ?showCity at the end of your URL to get IPs with their corresponding locations.

RESTful

Showing All Available Proxy Lists

With this endpoint, you can retrieve proxy lists that belong to your user:

GET https://api.oxylabs.io/v1/proxies/lists

curl -U username:password "https://api.oxylabs.io/v1/proxies/lists"

Sample output:

[
{
"uuid": "123e4567-e89b-12d3-a456-426655440000",
"key": "12345678",
"updated_timestamp": "2016-09-28T13:03:50+0300",
"ips_count": 2000,
"href": "http://api.oxylabs.io/v1/proxies/lists/123e4567-e89b-12d3-a456-426655440000"
}
]

Retrieving a Specific Proxy List

With this endpoint, you can retrieve a specific proxy list:

GET https://api.oxylabs.io/v1/proxies/lists/{uuid}

Request parameters:

curl -u username:password "https://api.oxylabs.io/v1/proxies/lists/{uuid}"

Sample output:

[
{
"ip": "127.0.0.1",
"port": "60000",
"country": "US"
},
{ 
"ip": "127.0.0.2",
"port": "60000",
"country": "GB"
}
]

Making Requests

By default, Oxylabs proxies and Proxy Rotator use a basic HTTP authentication that requires you to provide a username and password. You can get your credentials by contacting your Dedicated Account Manager or our Support Team at support@oxylabs.io.

We also support authentication based on whitelisted IP addresses. See Whitelisting IPs.

Depending on your authentication method, you might need to change the proxy port:

Proxy

If you want to use Datacenter Proxies via user/pass authentification method:

curl -x 1.2.3.4:60000 -U user1:pass1 http://ip.oxylabs.io

If you want to use Datacenter Proxies via Whitelisted IP authentification method:

curl -x 1.2.3.4:65432 http://ip.oxylabs.io

Whitelisting IPs

Dashboard

We also support authentication based on an IP address. If you are using proxies directly (without proxy Proxy Rotator), you can whitelist your IPs by yourself. Just log into https://stats.oxylabs.io and on the sidebar at the bottom you will see a section named “Add Whitelist”. After clicking on the link you will be navigated to self-whitelist page which looks like this:

Here you can manage your whitelisted IPs that you can use to access proxies. You can whitelist individual IPs, as well as up to 24 subnets. If you would like to whitelist a larger subnet, please contact your Dedicated Account Manager or Support Team at support@oxylabs.io.

If you are using proxy Proxy Rotator or SOCKS proxies, you cannot whitelist your IP address by yourself. Instead, you will have to give us a list of IP address you want to use to connect to proxies. We will whitelist them (add them to firewall exception rules). After doing so, you will be able to access your proxies without login credentials.

Bear in mind, that if you are using whitelisted IPs to access your proxies, you will need to use port 65432. For proxy Proxy Rotator, the port remains 60000.

RESTful

You can also whitelist IPs by sending queries directly to our endpoints. Please note there is a 5-minute cooldown for confirming any changes to your whitelisted IP list.

Getting Whitelisted IPs List

GET https://api.oxylabs.io/v1/whitelisted_ips

curl -U username:password "https://api.oxylabs.io/v1/whitelisted_ips"

Sample output:

{
"data": [
{
"id": 9765,
"address": "127.0.0.1"
},
{
"id": 9766,
"address": "127.0.0.2"
}
]
}

Adding a Whitelisted IP

POST https://api.oxylabs.io/v1/whitelisted_ips

curl -X POST -u username:password "https://api.oxylabs.io/v1/whitelisted_ips"
--header "Content-Type: application/json" --data '{"address": "127.0.0.1"}'

{
"error": {
"message": "This is not a valid IP address.",
"code": 422
}
}

Sample output:

{
"id": 9767,
"address": "127.0.0.1"
}

Removing a Whitelisted IP

DELETE https://api.oxylabs.io/v1/whitelisted_ips/{id}

curl -X DELETE -u username:password "https://api.oxylabs.io/v1/whitelisted_ips/{id}"

Saving Changes (5min Cooldown)

POST https://api.oxylabs.io/v1/whitelisted_ips/upload_to_servers

curl -X POST -u username:password "https://api.oxylabs.io/v1/whitelisted_ips/upload_to_servers"

Datacenter Proxy API

Replacing subnet IPs

POST https://api.oxylabs.io/v1/proxy-list/rest/proxy-lists/{PROXY_LIST_UUID}/by-uuid/network-modification-requests/replace

For you to replace IPs from a specific subnet in your existing proxy list while using Proxy Management API, you will need to make a POST request to the endpoint above with specific subnets you want to replace. To make it work correctly, you must insert a proxy list UUID that can be queried from the API (as described here).

curl -X POST -H "Content-Type: application/json" -d '{"subnets": ["x.x.x.0/24","y.y.y.0/24"]}' https://api.oxylabs.io/v1/proxy-list/rest/proxy-lists/{PROXY_LIST_UUID}/by-uuid/subnet-replacement-requests -u user:pass

Sample output:

{
    "id": "1", 
    "status": "processing", 
    "message": None, 
    "type": "replace", 
    "created_at": "2021-01-01T00:00:00+00:00", 
    "link_to_list_plain": "https://proxy.oxylabs.io/key/4f34fe5862603138c1bc5ce28b181aef", 
    "link_to_list_json": "https://api.oxylabs.io/v1/proxies/lists/8f11274c-a900-11eb-b910-901b0ec4424b"
}

Checking replacement status

GET https://api.oxylabs.io/v1/proxy-list/rest/subnet-replacement-requests/{INITIATED_REPLACEMENT_PROCESS_ID}

To confirm that the subnet replacement was a success, you must make a GET request to the endpoint above while inserting Initiated replacement process ID from the replacement process initiated before.

curl -u user:pass https://api.oxylabs.io/v1/proxy-list/rest/subnet-replacement-requests/{INITIATED_REPLACEMENT_PROCESS_ID}"