At the era of big data, web scraping is a life saver. To save even more time, you can couple ScrapingBot to a web crawling bot. What is a web crawler? A crawler, or spider, is an internet bot indexing … Read More
Scrape an entire HTML page without getting blocked
When you want to collect and analyze data, let it be for price comparison, statistics or to see a general evolution, scraping is a great and essential time saver.
However, many websites do not appreciate to be heavily scraped, some of them don’t allow it at all, especially in the retail industry sector.
There are some generic rules and tricks to respect/follow if you do not want to be blocked from scraping a website, temporarily or permanently. Thanks to our API you will be able scrape the content of a page without getting blocked.
The Raw HTML API collects the data while preserving its structure.
This module is especially made for scraping HTML from Google results for example.
Web scraping is particularly interesting for retail listings. This is where a small price difference can make a huge difference to your sales volume. You can use web scraping wether you’re selling the exact same product, or an alternative. Some … Read More