Documentation

And see it for yourself !

IMPORTANT : If you are testing on a retail site, do not paste the URL of the homepage or a page with multiple products, you have to paste a single product page URL 😉

Getting started

API endpoints :

POST  http://api.scraping-bot.io/scrape/raw-html  ➡️ Read documentation here

POST  http://api.scraping-bot.io/scrape/retail  ➡️  Read documentation here

POST  http://api.scraping-bot.io/scrape/real-estate  ➡️  Read documentation here

 


 

Hey ! Just getting started with ScrapingBot API ? No problem, we will introduce you to this magical world here you can find scraping code samples!

  • In case a request fails we will return a 200 code with a JSON populated with an error, you may retry the request Make sure to catch these errors! They will occur for hard to scrape websites.

  • If you exceed your plan concurrent connection limit, the API will respond with a 402 status code, this can be solved by slowing down your request rate

  • If you exceed your plan api calls, the API will respond with a 402 status code, this can be solved by upgrading your plan.

  • There is no overage allowed on the free plan, if you exceed 100 requests per month on the free plan, you will receive a 402 error.

  • Note that requests on Google count for 10 request credits and same if you use Premium Proxies.

Each request will return a string with raw JSON

  • The error field is null by default and is filled with a string if some error occurred

  • The data field contains by default all fields listed below

  • siteHtml is always null you can get it with advanced options example here 
{
    "error": null,
    "data": {
      "title": "Apple iPhone XR 64GB Red Unlocked A2105 GSM SEALED BOX- 1 Year Apple Warranty",
      "description": "Apple iPhone XR. 1 YEAR APPLE CARE WARRANTY.",
      "image": "https://www.scraping-bot.io/iphone_example_ebay_files/s-l500.png",
      "price": 689,
      "shippingFees": 18,
      "currency": "GBP",
      "isInStock": true,
      "EAN13": "0190198770660",
      "ASIN": null,
      "ISBN": null,
      "color": "White",
      "brand": "Apple",
      "category": {
        "name": "Mobile & Smart Phones",
        "url": "https://www.ebay.co.uk/b/Mobile-Smart-Phones-/9355"
      },
      "categories": [
        {
          "name": "Mobile Phones & Communication",
          "url": "https://www.ebay.co.uk/b/Mobile-Phones-Communication-/15032"
        },
        {
          "name": "Mobile & Smart Phones",
          "url": "https://www.ebay.co.uk/b/Mobile-Smart-Phones-/9355"
        }
      ],
      "siteURL": "https://www.ebay.co.uk/itm/Apple-iPhone-XR-64GB-Red-Unlocked-A2105-GSM-SEALED-BOX-1-Year-Apple-Warranty-/123902112947",
      "siteHtml": null,
      "productHasVariations": null,
      "error": null,
      "statusCode": null,
      "isFinished": null,
      "isDead": null,
      "htmlLength": 128016
    }
  }

Basic usage Raw HTML

You are using ScrapingBot for a basic usage ? You’ll find more information just here :
endpoint : POST  http://api.scraping-bot.io/scrape/raw-html
You only need to give us the website URL you want to scrape and you will get the fully rendered HTML of that page avoiding captcha and blocking.

If you need javascript rendering look at advanced options right here

var request = require('request');

var username = "yourUsername",
    apiKey = "yourApiKey",
    url = "https://www.scraping-bot.io/rawHtmlPage.html",
    auth = "Basic " + Buffer.from(username + ":" + apiKey).toString("base64");

request(
    {
        method: 'POST',
        url: 'http://api.scraping-bot.io/scrape/raw-html',
        json: {
            url: url
        },
        headers: {
            Accept: 'application/json',
            Authorization : auth
        },
    },
    function(error, response, body) {
        console.log(body);
    }
);

#!/bin/bash
url='https://www.scraping-bot.io/rawHtmlPage.html'
username='yourUsername'
api_key='yourApiKey'
auth=$(echo -ne "$username:$api_key" | base64);

curl -X POST \
  http://api.scraping-bot.io/scrape/raw-html \
  -H "Authorization: Basic $auth" \
  -H "Content-Type: application/json" \
  -d "{\"url\":\"$url\"}"

<?php
$userName="yourUsername";
$apiKey="yourApiKey";

$auth = base64_encode($userName.":".$apiKey);

$curl = curl_init();

curl_setopt_array($curl, array(
  CURLOPT_URL => "http://api.scraping-bot.io/scrape/raw-html",
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
  CURLOPT_CUSTOMREQUEST => "POST",
  CURLOPT_POSTFIELDS => "{\"url\":\"https://www.scraping-bot.io/rawHtmlPage.html\"}",
  CURLOPT_HTTPHEADER => array(
    "Authorization: Basic ".$auth,
    "Content-Type: application/json"
  ),
));

$response = curl_exec($curl);
$err = curl_error($curl);

curl_close($curl);

if ($err) {
  echo "cURL Error #:" . $err;
} else {
  $info = curl_getinfo($curl);
  if($info["http_code"]>399){
    echo "HTTP Error #:" . $response;
  }else{
    echo $response;
  }
}

import requests
import json

url='https://www.scraping-bot.io/rawHtmlPage.html'
username = 'yourUsername'
apiKey = 'yourApiKey'

apiUrl = "http://api.scraping-bot.io/scrape/raw-html"

payload = json.dumps({"url":url})
headers = {
    'Content-Type': "application/json"
}

response = requests.request("POST", apiUrl, data=payload, auth=(username,apiKey), headers=headers)

print(response.text)

require 'uri'
require 'net/http'
require "base64"

url='https://www.scraping-bot.io/rawHtmlPage.html'
username='yourUsername'
api_key='yourApiKey'

auth=Base64.encode64(username+":"+api_key)

apiUrl = URI("http://api.scraping-bot.io/scrape/raw-html")

http = Net::HTTP.new(apiUrl.host, apiUrl.port)

request = Net::HTTP::Post.new(apiUrl)
request["Content-Type"] = 'application/json'
request["Authorization"] = 'Basic '+auth
request.body = "{\"url\":\""+url+"\"}"

response = http.request(request)
puts response.read_body

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Base64;

public class RawJava {
    public static void main(String[] args) {
        try {
            String username = "yourUsername";
            String apiKey = "yourApiKey";
            String originalInput = username + ":" + apiKey;
            String encodedString = "Basic " + Base64.getEncoder().encodeToString(originalInput.getBytes());
            URL url = new URL("http://api.scraping-bot.io/scrape/raw-html");
            HttpURLConnection con = (HttpURLConnection) url.openConnection();
            con.setRequestMethod("POST");
            con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
            con.setRequestProperty("Authorization", encodedString);

            String param = "{\"url\":\"https://www.scraping-bot.io/rawHtmlPage.html\"}";

            con.setDoOutput(true);
            OutputStream out = con.getOutputStream();
            out.write(param.getBytes());
            out.flush();
            out.close();

            int status = con.getResponseCode();
            System.out.println(status);
            BufferedReader in = new BufferedReader(
                    new InputStreamReader(con.getInputStream()));
            String inputLine;
            StringBuilder content = new StringBuilder();
            while ((inputLine = in.readLine()) != null) {
                content.append(inputLine);
            }
            String jsonResponse = content.toString();
            System.out.println(jsonResponse);
            in.close();
            con.disconnect();

        } catch (Exception e) {
            System.out.println("An error occured while scraping:" + e);
        }
    }
}

Each request will return raw HTML content of the page

Result ⤵
<html>
  <head>
  </head>
  <body>
    <h1>Here is my title</h1>
    <p>
        This is the content of my paragraph
    </p>
  </body>
</html>

Advanced options

You choose the best usage of ScrapingBot ? I knew that you were the best. Here you can find details about the options you can choose.

Options : 

  • useChrome : Boolean, set this option to true if you want to use headless chrome that is able to render Javascript and get full result WARNING this option consumes two api calls

  • premiumProxy : Boolean, set this option to true to use Premium proxy pool (better for Amazon, Rakuten, Google etc…)WARNING this option consumes 10 api calls and 20 calls if combined with js rendering

var request = require('request');

var username = "yourUsername",
    apiKey = "yourApiKey",
    url = "https://www.scraping-bot.io/rawHtmlPage.html",
    apiEndPoint = "http://api.scraping-bot.io/scrape/raw-html",
    auth = "Basic " + new Buffer(username + ":" + apiKey).toString("base64");

request(
    {
        method: 'POST',
        url: apiEndPoint,
        json: {
            url: url,
            options: {
                useChrome:false, //set to 'true' if you want to use headless chrome for javascript rendering
                premiumProxy:false, //set to 'true' if you want to use premium proxies Unblock Amazon,Google,Rakuten
            }
        },
        headers: {
            Accept: 'application/json',
            Authorization : auth
        },
    },
    function(error, response, body) {
        console.log(body);
    }
);

#!/bin/bash
url='https://www.scraping-bot.io/rawHtmlPage.html'
username='yourUsername'
api_key='yourApiKey'
auth=$(echo -ne "$username:$api_key" | base64);
#parameters
useChrome='false' #set to 'true' if you want to use headless chrome for javascript rendering
premiumProxy='false' #set to 'true' if you want to use premium proxies Unblock Amazon,Google,Rakuten

apiEndPoint='http://api.scraping-bot.io/scrape/raw-html'

curl -X POST \
  $apiEndPoint \
  -H "Authorization: Basic $auth" \
  -H "Content-Type: application/json" \
  -d "{\"url\":\"$url\",\"options\":{\"useChrome\":$useChrome,\"premiumProxy\":$premiumProxy}}"

<?php
$userName="yourUsername";
$apiKey="yourApiKey";

$auth = base64_encode($userName.":".$apiKey);

$postParams = array(
    "url" => "https://www.scraping-bot.io/rawHtmlPage.html",
    'options' => array(
      "useChrome" => false, //set to 'true' if you want to use headless chrome for javascript rendering
      "premiumProxy" => false  //set to 'true' if you want to use premium proxies Unblock Amazon,Google,Rakuten
    )
);

$apiEndPoint = "http://api.scraping-bot.io/scrape/raw-html";

$json = json_encode($postParams);


$curl = curl_init();

curl_setopt_array($curl, array(
  CURLOPT_URL => $apiEndPoint,
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
  CURLOPT_CUSTOMREQUEST => "POST",
  CURLOPT_POSTFIELDS => $json,
  CURLOPT_HTTPHEADER => array(
    "Authorization: Basic ".$auth,
    "Content-Type: application/json"
  ),
));

$response = curl_exec($curl);
$err = curl_error($curl);

curl_close($curl);

if ($err) {
  echo "cURL Error #:" . $err;
} else {
  echo $response;
}

import requests
import json

url='https://www.scraping-bot.io/rawHtmlPage.html'
username = 'yourUsername'
apiKey = 'yourApiKey'

apiEndPoint = "http://api.scraping-bot.io/scrape/raw-html"

options = {
    "useChrome": False,#set to True if you want to use headless chrome for javascript rendering
    "premiumProxy": False # set to True if you want to use premium proxies Unblock Amazon,Google,Rakuten
}

payload = json.dumps({"url":url,"options":options})
headers = {
    'Content-Type': "application/json"
}

response = requests.request("POST", apiEndPoint, data=payload, auth=(username,apiKey), headers=headers)

print(response.text)

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Base64;

public class AdvancedRawJava {
    public static void main(String[] args) {
        try {
            String username = "yourUsername";
            String apiKey = "yourApiKey";
            String originalInput = username + ":" + apiKey;
            String encodedString = "Basic " + Base64.getEncoder().encodeToString(originalInput.getBytes());

            String apiEndPoint = "http://api.scraping-bot.io/scrape/raw-html";
            URL url = new URL(apiEndPoint);
            HttpURLConnection con = (HttpURLConnection) url.openConnection();
            con.setRequestMethod("POST");
            con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
            con.setRequestProperty("Authorization", encodedString);

            String useChrome = "false";//set to "true" if you want to use headless chrome for javascript rendering
            String premiumProxy = "false";//set to "true" if you want to use premium proxies Unblock Amazon,Google,Rakuten
            String urlToScrape = "https://www.scraping-bot.io/rawHtmlPage.html";

            String param = "{\"url\":\""+urlToScrape+"\","+
                                "\"options\":{"+
                                    "\"useChrome\":"+useChrome+","+
                                    "\"premiumProxy\":"+premiumProxy+
                                    "}"+
                            "}";

            con.setDoOutput(true);
            OutputStream out = con.getOutputStream();
            out.write(param.getBytes());
            out.flush();
            out.close();

            int status = con.getResponseCode();
            System.out.println(status);
            BufferedReader in = new BufferedReader(
                    new InputStreamReader(con.getInputStream()));
            String inputLine;
            StringBuilder content = new StringBuilder();
            while ((inputLine = in.readLine()) != null) {
                content.append(inputLine);
            }
            String jsonResponse = content.toString();
            System.out.println(jsonResponse);
            in.close();
            con.disconnect();

        } catch (Exception e) {
            System.out.println("An error occured while scraping:" + e);
        }
    }
}

Retail API

You want to scrape Retail websites and don’t want to waste time on finding, price tags title, brand, color and many other properties full list of props here.

Stop wasting your precious time and use our Retail API , give us the product page you want to scrape and we will give you all the data already extracted

Endpoint : POST  http://api.scraping-bot.io/scrape/retail


Optional parameters : 

  • useChrome : Boolean, set this option to true if you want to use headless chrome that is able to render Javascript and get full result

  • premiumProxy : Boolean, set this option to true to use Premium proxy pool (better for Amazon, Rakuten, Google etc…)

var request = require('request');

var username = "yourUsername",
    apiKey = "yourApiKey",
    url = "https://www.scraping-bot.io/example-ebay.html",
    apiEndPoint = "http://api.scraping-bot.io/scrape/retail",
    auth = "Basic " + new Buffer(username + ":" + apiKey).toString("base64");

request(
    {
        method: 'POST',
        url: apiEndPoint,
        json: {
            url: url,
            options: {
                useChrome:false, //set to 'true' if you want to use headless chrome for javascript rendering
                premiumProxy:false, //set to 'true' if you want to use premium proxies Unblock Amazon,Google,Rakuten
            }
        },
        headers: {
            Accept: 'application/json',
            Authorization : auth
        },
    },
    function(error, response, body) {
        console.log(body);
    }
);

#!/bin/bash
url='https://www.scraping-bot.io/example-ebay.html'
username='yourUsername'
api_key='yourApiKey'
auth=$(echo -ne "$username:$api_key" | base64);
#parameters
useChrome='false' #set to 'true' if you want to use headless chrome for javascript rendering
premiumProxy='false' #set to 'true' if you want to use premium proxies Unblock Amazon,Google,Rakuten

apiEndPoint='http://api.scraping-bot.io/scrape/retail'

curl -X POST \
  $apiEndPoint \
  -H "Authorization: Basic $auth" \
  -H "Content-Type: application/json" \
  -d "{\"url\":\"$url\",\"options\":{\"useChrome\":$useChrome,\"premiumProxy\":$premiumProxy}}"

<?php
$userName="yourUsername";
$apiKey="yourApiKey";

$auth = base64_encode($userName.":".$apiKey);

$postParams = array(
    "url" => "https://www.scraping-bot.io/example-ebay.html",
    'options' => array(
      "useChrome" => false, //set to 'true' if you want to use headless chrome for javascript rendering
      "premiumProxy" => false  //set to 'true' if you want to use premium proxies Unblock Amazon,Google,Rakuten
    )
);

$apiEndPoint = "http://api.scraping-bot.io/scrape/retail";

$json = json_encode($postParams);


$curl = curl_init();

curl_setopt_array($curl, array(
  CURLOPT_URL => $apiEndPoint,
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
  CURLOPT_CUSTOMREQUEST => "POST",
  CURLOPT_POSTFIELDS => $json,
  CURLOPT_HTTPHEADER => array(
    "Authorization: Basic ".$auth,
    "Content-Type: application/json"
  ),
));

$response = curl_exec($curl);
$err = curl_error($curl);

curl_close($curl);

if ($err) {
  echo "cURL Error #:" . $err;
} else {
  echo $response;
}

import requests
import json

url='https://www.scraping-bot.io/example-ebay.html'
username = 'yourUsername'
apiKey = 'yourApiKey'

apiEndPoint = "http://api.scraping-bot.io/scrape/retail"

options = {
    "useChrome": False,#set to True if you want to use headless chrome for javascript rendering
    "premiumProxy": False # set to True if you want to use premium proxies Unblock Amazon,Google,Rakuten
}

payload = json.dumps({"url":url,"options":options})
headers = {
    'Content-Type': "application/json"
}

response = requests.request("POST", apiEndPoint, data=payload, auth=(username,apiKey), headers=headers)

print(response.text)

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Base64;

public class AdvancedRawJava {
    public static void main(String[] args) {
        try {
            String username = "yourUsername";
            String apiKey = "yourApiKey";
            String originalInput = username + ":" + apiKey;
            String encodedString = "Basic " + Base64.getEncoder().encodeToString(originalInput.getBytes());

            String apiEndPoint = "http://api.scraping-bot.io/scrape/retail";
            URL url = new URL(apiEndPoint);
            HttpURLConnection con = (HttpURLConnection) url.openConnection();
            con.setRequestMethod("POST");
            con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
            con.setRequestProperty("Authorization", encodedString);

            String useChrome = "false";//set to "true" if you want to use headless chrome for javascript rendering
            String premiumProxy = "false";//set to "true" if you want to use premium proxies Unblock Amazon,Google,Rakuten
            String urlToScrape = "https://www.scraping-bot.io/example-ebay.html";

            String param = "{\"url\":\""+urlToScrape+"\","+
                                "\"options\":{"+
                                    "\"useChrome\":"+useChrome+","+
                                    "\"premiumProxy\":"+premiumProxy+
                                    "}"+
                            "}";

            con.setDoOutput(true);
            OutputStream out = con.getOutputStream();
            out.write(param.getBytes());
            out.flush();
            out.close();

            int status = con.getResponseCode();
            System.out.println(status);
            BufferedReader in = new BufferedReader(
                    new InputStreamReader(con.getInputStream()));
            String inputLine;
            StringBuilder content = new StringBuilder();
            while ((inputLine = in.readLine()) != null) {
                content.append(inputLine);
            }
            String jsonResponse = content.toString();
            System.out.println(jsonResponse);
            in.close();
            con.disconnect();

        } catch (Exception e) {
            System.out.println("An error occured while scraping:" + e);
        }
    }
}

Each request will return a string with raw JSON

  • The error field is null by default and is filled with a string if some error occurred

  • The data field contains by default all fields listed below
Result example ⤵
{
    "error": null,
    "data": {
      "title": "Apple iPhone XR 64GB Red Unlocked A2105 GSM SEALED BOX- 1 Year Apple Warranty",
      "description": "Apple iPhone XR. 1 YEAR APPLE CARE WARRANTY.",
      "image": "https://www.scraping-bot.io/iphone_example_ebay_files/s-l500.png",
      "price": 689,
      "shippingFees": 18,
      "currency": "GBP",
      "isInStock": true,
      "EAN13": "0190198770660",
      "ASIN": null,
      "ISBN": null,
      "color": "White",
      "brand": "Apple",
      "category": {
        "name": "Mobile & Smart Phones",
        "url": "https://www.ebay.co.uk/b/Mobile-Smart-Phones-/9355"
      },
      "categories": [
        {
          "name": "Mobile Phones & Communication",
          "url": "https://www.ebay.co.uk/b/Mobile-Phones-Communication-/15032"
        },
        {
          "name": "Mobile & Smart Phones",
          "url": "https://www.ebay.co.uk/b/Mobile-Smart-Phones-/9355"
        }
      ],
      "siteURL": "https://www.ebay.co.uk/itm/Apple-iPhone-XR-64GB-Red-Unlocked-A2105-GSM-SEALED-BOX-1-Year-Apple-Warranty-/123902112947",
      "siteHtml": null,
      "productHasVariations": null,
      "error": null,
      "statusCode": null,
      "isFinished": null,
      "isDead": null,
      "htmlLength": 128016
    }
  }

Real Estate API

You want to scrape Real estate websites and don’t want to waste time on finding, price tags title, number of rooms, surfaceArea and many other properties full list of props here.

Stop wasting your precious time and use our Real estate API , give us the product page you want to scrape and we will give you all the data already extracted

Endpoint : POST  http://api.scraping-bot.io/scrape/real-estate

Optional parameters : 

  • useChrome : Boolean, set this option to true if you want to use headless chrome that is able to render Javascript and get full result

  • premiumProxy : Boolean, set this option to true to use Premium proxy pool (better for Amazon, Rakuten, Google etc…)

var request = require('request');

var username = "yourUsername",
    apiKey = "yourApiKey",
    url = "https://www.scraping-bot.io/realEstate.html",
    apiEndPoint = "http://api.scraping-bot.io/scrape/real-estate",
    auth = "Basic " + new Buffer(username + ":" + apiKey).toString("base64");

request(
    {
        method: 'POST',
        url: apiEndPoint,
        json: {
            url: url,
            options: {
                useChrome:false, //set to 'true' if you want to use headless chrome for javascript rendering
                premiumProxy:false, //set to 'true' if you want to use premium proxies Unblock Amazon,Google,Rakuten
            }
        },
        headers: {
            Accept: 'application/json',
            Authorization : auth
        },
    },
    function(error, response, body) {
        console.log(body);
    }
);

#!/bin/bash
url='https://www.scraping-bot.io/realEstate.html'
username='yourUsername'
api_key='yourApiKey'
auth=$(echo -ne "$username:$api_key" | base64);
#parameters
useChrome='false' #set to 'true' if you want to use headless chrome for javascript rendering
premiumProxy='false' #set to 'true' if you want to use premium proxies Unblock Amazon,Google,Rakuten

apiEndPoint='http://api.scraping-bot.io/scrape/real-estate'

curl -X POST \
  $apiEndPoint \
  -H "Authorization: Basic $auth" \
  -H "Content-Type: application/json" \
  -d "{\"url\":\"$url\",\"options\":{\"useChrome\":$useChrome,\"premiumProxy\":$premiumProxy}}"

<?php
$userName="yourUsername";
$apiKey="yourApiKey";

$auth = base64_encode($userName.":".$apiKey);

$postParams = array(
    "url" => "https://www.scraping-bot.io/realEstate.html",
    'options' => array(
      "useChrome" => false, //set to 'true' if you want to use headless chrome for javascript rendering
      "premiumProxy" => false  //set to 'true' if you want to use premium proxies Unblock Amazon,Google,Rakuten
    )
);

$apiEndPoint = "http://api.scraping-bot.io/scrape/real-estate";

$json = json_encode($postParams);


$curl = curl_init();

curl_setopt_array($curl, array(
  CURLOPT_URL => $apiEndPoint,
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
  CURLOPT_CUSTOMREQUEST => "POST",
  CURLOPT_POSTFIELDS => $json,
  CURLOPT_HTTPHEADER => array(
    "Authorization: Basic ".$auth,
    "Content-Type: application/json"
  ),
));

$response = curl_exec($curl);
$err = curl_error($curl);

curl_close($curl);

if ($err) {
  echo "cURL Error #:" . $err;
} else {
  echo $response;
}

import requests
import json

url='https://www.scraping-bot.io/realEstate.html'
username = 'yourUsername'
apiKey = 'yourApiKey'

apiEndPoint = "http://api.scraping-bot.io/scrape/real-estate"

options = {
    "useChrome": False,#set to True if you want to use headless chrome for javascript rendering
    "premiumProxy": False # set to True if you want to use premium proxies Unblock Amazon,Google,Rakuten
}

payload = json.dumps({"url":url,"options":options})
headers = {
    'Content-Type': "application/json"
}

response = requests.request("POST", apiEndPoint, data=payload, auth=(username,apiKey), headers=headers)

print(response.text)

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Base64;

public class AdvancedRawJava {
    public static void main(String[] args) {
        try {
            String username = "yourUsername";
            String apiKey = "yourApiKey";
            String originalInput = username + ":" + apiKey;
            String encodedString = "Basic " + Base64.getEncoder().encodeToString(originalInput.getBytes());

            String apiEndPoint = "http://api.scraping-bot.io/scrape/real-estate";
            URL url = new URL(apiEndPoint);
            HttpURLConnection con = (HttpURLConnection) url.openConnection();
            con.setRequestMethod("POST");
            con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
            con.setRequestProperty("Authorization", encodedString);

            String useChrome = "false";//set to "true" if you want to use headless chrome for javascript rendering
            String premiumProxy = "false";//set to "true" if you want to use premium proxies Unblock Amazon,Google,Rakuten
            String urlToScrape = "https://www.scraping-bot.io/realEstate.html";

            String param = "{\"url\":\""+urlToScrape+"\","+
                                "\"options\":{"+
                                    "\"useChrome\":"+useChrome+","+
                                    "\"premiumProxy\":"+premiumProxy+
                                    "}"+
                            "}";

            con.setDoOutput(true);
            OutputStream out = con.getOutputStream();
            out.write(param.getBytes());
            out.flush();
            out.close();

            int status = con.getResponseCode();
            System.out.println(status);
            BufferedReader in = new BufferedReader(
                    new InputStreamReader(con.getInputStream()));
            String inputLine;
            StringBuilder content = new StringBuilder();
            while ((inputLine = in.readLine()) != null) {
                content.append(inputLine);
            }
            String jsonResponse = content.toString();
            System.out.println(jsonResponse);
            in.close();
            con.disconnect();

        } catch (Exception e) {
            System.out.println("An error occured while scraping:" + e);
        }
    }
}

Each request will return a string with raw JSON

  • The error field is null by default and is filled with a string if some error occurred

  • The data field contains by default all fields listed below
Result example ⤵
{
  "error": null,
  "data": {
    "title": "Location Studio Montpellier - 415€/mois - appartement F1/T1/1 pièce 18m²",
    "description": "34000 : Appartement disponible le 22 / 01 / 2020 Situé à MONTPELLIER (34000) rue des amaryllis, proche arceaux, du tramway ligne 3 et de la ligne de bus n°10, cet appartement est un T1 de 18,35 m² comprenant une pièce de vie entièrement repeinte avec une kitchenette neuve, ...",
    "surfaceArea": 18,
    "surfaceAreaUnit": "sqm",
    "price": null,
    "currency": "EUR",
    "numberOfRooms": 1,
    "numberOfBedrooms": 0,
    "publishingDate": null,
    "monthlyRent": 415,
    "weeklyRent": null,
    "marketedBy": {
      "name": "GUY HOQUET BPMI",
      "address": "1 Place Pierre Renaudel  34080 Montpellier",
      "phoneNumber": "0411280090"
    },
    "energyClass": "E:284",
    "greenhouseGazClass": null,
    "siteURL": "https://www.seloger.com/annonces/locations/appartement/montpellier-34/les-cevennes/154966057.htm",
    "siteHtml": null,
    "error": null,
    "statusCode": null,
    "htmlLength": 181005,
    "captchaFound": false,
    "isHtmlPage": true,
    "host": "www.seloger.com",
    "codeinsee": "340172"
  }
}

You want to contact the geniuses that created this amazing Scraping Bot API ?

Please fill this form and make your dreams come true !

reCAPTCHA