X Twitter Scraper API: Extract Profiles, Tweets and Search Results with ScrapingBot

ScrapingBot's X Twitter scraper API gives developers programmatic access to public X (formerly Twitter) data — profiles, tweets, and search results by keyword or hashtag — in structured JSON format, without dealing with rate limits, authentication walls, or bot detection. This guide covers everything you need to integrate the API into your pipeline: endpoints, parameters, response structure, and production best practices.

1. Why use an X Twitter scraper API?

An X Twitter scraper API lets you collect public data from X (formerly Twitter) programmatically — without managing proxies, authentication, or bot detection. With over 500 million monthly active users generating hundreds of millions of posts per day, X remains one of the most valuable real-time data sources on the internet. Over time, it has consequently become a critical platform for brand monitoring, trend detection, and social intelligence pipelines. According to X's official API documentation, access to this data at scale has never been more in demand — yet the official pricing has never been more restrictive.

As a result, access to X data at scale is highly valuable for a wide range of technical applications:

• Brand monitoring — track mentions, sentiment, and engagement around your brand or competitors in real time
• Trend detection — monitor emerging topics, viral hashtags, and breaking news as they develop
• Audience research — analyze follower profiles, interests, and posting patterns for a given account
• Competitive intelligence — benchmark competitor accounts by follower growth, engagement rate, and content strategy
• NLP and ML datasets — collect large-scale tweet corpora for sentiment analysis, classification, or language model fine-tuning

X's official API has become extremely restrictive since 2023 — the free tier limits access to just 1,500 tweets per month, while paid tiers start at $100/month. Therefore, the X Twitter scraper API from ScrapingBot is the most practical solution for developers who need reliable, cost-effective access to public X data.

2. What data can the X Twitter scraper API extract?

ScrapingBot's X Twitter scraper API supports two types of data extraction. Below is the full field reference for each:

X Twitter Profile data

X Twitter Search Results data

Search by keyword or hashtag to retrieve matching tweets with full metadata:

3. Technical challenges of scraping X Twitter

X is one of the most aggressively protected platforms against automated data collection. Before integrating any scraping solution, it is important to understand what makes it technically difficult:

Platform-level protections

• Authentication wall — since 2023, X requires login to view most content, making unauthenticated scraping extremely unreliable.
• Aggressive bot detection — behavioral fingerprinting, TLS fingerprinting, and IP reputation scoring block scrapers within seconds.
• JavaScript rendering — all tweet data loads dynamically; therefore, plain HTTP requests return empty shells.
• Strict rate limiting — even authenticated requests are throttled heavily, with temporary bans triggered after a small number of requests.

Maintenance and cost barriers

• Frequent API changes — X regularly modifies its internal GraphQL endpoints, which means custom scrapers break without warning.
• Restrictive official API pricing — the free tier allows only 1,500 tweets/month, making it unusable for most production use cases.

Furthermore, building and maintaining a custom X scraper that handles all of these challenges requires significant ongoing engineering effort. Consequently, ScrapingBot's X Twitter scraper API abstracts all of this complexity so you can focus on using the data.

4. How the X Twitter scraper API works

Two-step asynchronous pattern

ScrapingBot's X Twitter scraper API uses a two-step asynchronous pattern — specifically designed to handle X's protections without getting blocked:

This pattern is necessary because X scraping requires time to bypass protections reliably. As a result, the API returns a job ID immediately and processes the request asynchronously in the background.

Scraper values reference

5. Step-by-step: integrate the X Twitter scraper API

Step 1 — Create your ScrapingBot account

To get started, ScrapingBot offers free access with 100 credits per month — no payment information required. Already have an account? Simply log in and retrieve your API credentials from the dashboard.

Step 2 — Submit the scraping job (POST request)

Send a POST request to the Data Scraper endpoint. The scraper parameter determines whether you are scraping a profile or a search query:

POST https://api.scraping-bot.io/scrape/data-scraper

Authorization: Basic {base64(username:api_key)}
Content-Type: application/json

// Scraping a Twitter/X profile:
{
  "scraper": "twitterProfile",
  "url": "https://x.com/username"
}

// Scraping search results by keyword or hashtag:
{
  "scraper": "twitterSearch",
  "search": "#AI"
}

The API immediately returns a responseId:

{
  "responseId": "abc123xyz789"
}

Step 3 — Poll for the result (GET request)

Next, use the responseId to retrieve the scraped data:

GET https://api.scraping-bot.io/scrape/data-scraper-response?responseId=abc123xyz789&scraper=twitterProfile

Authorization: Basic {base64(username:api_key)}

Should the job still be processing, you will receive a pending response:

{
  "status": "pending",
  "message": "Scraping is not finished for this request, try again in a few"
}

In that case, retry after 2–3 seconds. Once complete, the API returns the full structured JSON data.

Step 4 — Parse the JSON response

Once the result is ready, the API returns a fully structured JSON object. Below is an example of the profile response structure:

{
  "profileName": "Elon Musk",
  "verified": true,
  "bio": "Owner of X",
  "followers": 185000000,
  "following": 890,
  "tweetsCount": 42000,
  "posts": [
    {
      "id": "1234567890",
      "text": "The bird is freed.",
      "createdTime": "2022-10-27T00:00:00Z",
      "likes": 980000,
      "retweets": 210000,
      "replies": 75000,
      "views": 45000000,
      "media": []
    }
  ]
}

6. Key use cases for the X Twitter scraper API

Brand monitoring and sentiment analysis

Use the search endpoint to collect all public mentions of your brand, product, or campaign hashtag in real time. By combining tweet text with engagement metrics, you can furthermore build a sentiment scoring pipeline that alerts your team when negative mentions spike — before they go viral. In addition, the structured JSON response makes it straightforward to pipe results into any downstream analytics tool.

Trend detection and news intelligence

Track emerging hashtags and keywords to identify trends as they develop — hours before they appear in traditional media. Additionally, monitoring keyword velocity (how fast tweet volume grows) gives you a reliable early signal for breaking news or viral content. As a result, your team can respond faster than competitors who rely solely on traditional news feeds.

Competitive benchmarking

Scrape competitor profiles to track follower growth, posting frequency, and engagement rates over time. As a result, your content and social media teams can make data-driven decisions about posting strategy and content format. Moreover, combining profile data across multiple competitors gives you a clear industry benchmark rather than a point-in-time snapshot.

NLP and ML dataset collection

Collect large-scale tweet corpora filtered by keyword, language, or hashtag for sentiment analysis or topic classification. Moreover, the rich metadata returned by the search endpoint — language detection, geo-coordinates, media type — makes it straightforward to build well-structured training datasets. Consequently, teams can move from raw data collection to model training far more quickly than with manual approaches.

7. Going further

Scaling your X data pipeline

Once your integration is working, you can scale it by batching requests across multiple profiles or search queries in parallel. ScrapingBot handles concurrency and IP rotation automatically — so your pipeline can therefore process hundreds of profiles per hour without manual infrastructure management.

Combining with other data sources

Beyond X, ScrapingBot's unified X Twitter scraper API also supports Instagram, Threads, LinkedIn, and Facebook with the same authentication pattern.

Furthermore, this makes it straightforward to build multi-source social listening pipelines — combining X real-time data with LinkedIn professional data and Instagram visual content, all through a single API key.