Scrape the Web with scrapestack (Sponsored)

Celý článek

I first grew to love Firefox not as a web developer but as user, and what drew me to this amazing new browser was its add-on ecosystem. The add-on I used the most? Web scrapers. Piracy had just hit mainstream and I also need imagery and documentation to create my first websites. Scrapers were the most valuable add-ons ever!

These days writing scrapers, even as a seasoned software engineer, is a nightmare. Storage, CAPTCHA, DDOS/Proxies…to protect our sites we’ve killed the generic scraper on a large scale level. These days you need a service like scrapestack, a world class website scraper that can use a variety of strategies to get you the content you want without the pitfalls and walls in between.

Quick Hits

• Free to start!
• Can bypass CAPTCHAs to deliver the content you desire
• Simple API with that lets you define your own proxy
• 99.9% uptime with 1+billion requests served per month
• From the creators of currencylayer, ipstack, mailboxlayer, and more rock solid APIs

Start by signing up for free — you’ll immediately be provided an API token to use, as well as get detailed API instructions for usage.

Basic Usage

The most basic usage includes sending an API key and a URL:

https://api.scrapestack.com/scrape?access_key=MY_API_KEY&url=https://davidwalsh.name]

The following address scrapes the source code of the provided url parameter, allowing you to download store it or simply mirror the content at that address.

Websites are super dynamic these days so you can even include the JavaScript resources for a given page:

https://api.scrapestack.com/scrape?access_key=MY_API_KEY&url=https://davidwalsh.name&render_js=1

You can also send custom header information with your scrape request:

curl --header "X-SomeHeader: SomeValue" \
"https://api.scrapestack.com/scrape?access_key=MY_API_KEY&url=https://davidwalsh.name"

As well as choose the location from which the request should originate:

https://api.scrapestack.com/scrape?access_key=MY_API_KEY&url=https://davidwalsh.name& proxy_location=uk

And of course you can choose a request type:

curl -d 'key=value' \
-X POST \
"https://api.scrapestack.com/scrape?access_key=MY_API_KEY&url=https://davidwalsh.name"

Scraping seems easy until you get hit with CAPTCHAs, IP limits, region restrictions, DDoS prevention utilities, and more. scrapestack helps to avoid those problems and provide you the contents you want without needing to be an expert at everything else!

The post Scrape the Web with scrapestack (Sponsored) appeared first on David Walsh Blog.