How to scrape Twitter?

Twitter Data Scraper API
Posted by on | Featured
  • Twitter Data Scraper API

Scrape and collect data from Twitter sounds like a good idea. With more than 200 million users in the world, Twitter is a big social network.
Very popular with users for getting quickly informed, following trends or giving their opinion, Twitter has a lot of interesting marketing data for brands.

Collect this data to know more your audience, or to be informed of something you need.

Now, ScrapingBot has a Twitter scraper to automatically scrape and collect public data from Twitter profiles and Twitter search results (keyword or hashtag).
Retrieve the data you want in JSON, without any blocking.

What data can you collect with our Twitter Scraper?

Twitter profiles 

Profile name, profile image, profile background image, number of tweets, account verification, profile bio, number followed, number of followers, latest posts data (text, time, ID, replies, retweets, likes, views, media)

Twitter search results

From a keyword or a hashtag: Username, tweet URL, created time, ID, other hashtags, user mentions, source, geo coordinates, reply count, quote count, conversation ID, retweeted, possibly sensitive, language, media data (links, expanded URL, type, width and height)

Try for free

How to start scraping Twitter?

ScrapingBot is here to help you scrape Twitter profiles and hashtags as easily and efficiently as possible.

Start scraping Twitter in 3 steps:

1. Create an account

Want to start scraping Twitter right now? No problem, ScrapingBot offers FREE access with 100 credits per month to get started. No payment information required, and no any engagement.
If you already have an account, just log in.

Try for free

2. Configure your scraping

Once your account has been created, go to Documentation, to the "Data Scraper API" section to be able to start scraping what you want.

Setting up the endpoint is a two-step process with two API calls.
It's a little longer than usual because social networks have many protections to avoid robots and get scraped. These two calls make it possible to recover the data without being blocked.

Step 1 : Get your Response ID

First, you must make a first POST request to get the ResponseID which will be used later.
To do this, you must enter the following parameters in the body, then run a first API call:

  • scraper
    ("twitterProfile" to scrape Twitter profiles, or "twitterSearch" to scrape Twitter search results)
    • If you chose twitterProfile as a scraper:
      • url
        Add the URL of the Twitter profile you want to scrape
    • If you chose twitterSearch as a scraper:
      • search
        Enter the keyword or hashtag you want to retrieve search results data from

Here is the Request Endpoint:
http://api.scraping-bot.io/scrape/data-scraper

Step 2 : Configure your second request

Now that you have your responseId, you will be able to configure a GET request to ask for the response.

Here is the Response Endpoint:
http://api.scraping-bot.io/scrape/data-scraper-response?responseId=xxxx&scraper=name

Complete the Response Endpoint with these parameters:

  • responseId
    Insert here the Response ID that you had with the POST API call
  • scraper
    You must fill in the same as that used previously for the POST API call ("twitterProfile" or "twitterSearch")

When it's ready, execute the API call. The result you will get will tell you if the scraping is ready to use or not.
If it's not, you will get this pending message:

{status: "pending", message: "Scraping is not finished for this request, try again in a few"}

In this case, you have to try again until it's ready.

Try for free

3. Let's scraping!

Your web scraping setup is now ready to use!
You can start collecting data from Twitter Profiles and Search results.


NB: If you need more help, a sample code to execute the call to this API is available in the documentation right here ("Data Scraper API" section).

Tags: , ,