Scrapinghub's company blog covers web scraping with Scrapy, Portia, and Splash, plus data projects built on scraped content.
Commenters see it as useful for Scrapy and Portia guidance, though tied to the company's own tools, and point to ScrapyRT and AutoExtract posts.
Disclaimer, I wrote the article.
> I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do.
It was not my intention to give that implication. The main implication behind CityBikes is that public services should already provide this information since, well, it is a public service. On the same line, a private company providing a public service should already do so. See motives [1].
> I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down my results, I would not contest, since data is what drives the Internet ecosystem.
Same as CityBikes is doing. If we receive a cease and desist, we remove their service from our API. As for Foursquare, I do not see Foursquare as a public service. Your taxdollars at work, and all that.
I tried to keep the article balanced but maybe it wasn't clear. There are many transportation companies willing and happy to be scraped, or looking forward to provide their information for people to reuse [2].
[1]: https://blog.scrapinghub.com/2016/03/30/web-scraping-to-create-open-data/#benefits
[2]: http://nabsa.net/current-members/
All sorts of careers, for example:
- developers who want to develop some data-based product (a travel agency website, who finds the best deals from airline companies);
- lawyers can use it to structure the data from Judgments and Laws (so that they are able to query the data for things like: which judges have interpreted this law in their judgments) (more on this: http://blog.scrapinghub.com/2016/01/13/vizlegal-rise-of-machine-readable-laws-and-court-judgments/)
- (data-)journalists who work on investigative data-based articles (they use it to gather the data to build visualizations, infographics, and also to support their arguments).
- real state agencies can use it to grab the prices of their competitors, or to get a map of what people are selling, what are the areas where there is more demand.
- large companies that want to track their online reputation can scrape forums, blogs, etc, for further analysis.
- online retailers that want to keep their prices balanced with their competitors can scrape the competitors websites collecting prices from them.
More on Quora: https://www.quora.com/What-are-examples-of-how-real-businesses-use-web-scraping
Haven't tried this[0] yet, but Scrapy should be able to handle JavaScript sites with the JavaScript rendering service Splash[1]. scrapy-splash[2] is the plugin to integrate Scrapy and Splash.
[0] https://blog.scrapinghub.com/2015/03/02/handling-javascript-in-scrapy-with-splash/
[1] https://splash.readthedocs.io/en/stable/index.html
[2] https://github.com/scrapy-plugins/scrapy-splash
There is ScrapyRT: http://blog.scrapinghub.com/2015/01/22/introducing-scrapyrt-an-api-for-scrapy-spiders/
In the project I work on we do have the usual periodic crawls and use ScrapyRT to let the frontend trigger realtime scrapes of specific items, all of this using the same spider code.
Edit: Worth nothing that we trigger the realtime scrapes via AMQP.
Not sure if you're interested, but we (Scrapinghub) do offer a Kimono to Portia migration https://blog.scrapinghub.com/2016/02/25/migrate-your-kimono-projects-to-portia/
Otherwise, I'd recommend you check out Portia (open source). We're in the middle of releasing the beta 2.0 version.
ScrapingHub was something I was interested in - as an example of opensource project which the founders used as a base to build a company.
Does it help to use their model for you - https://blog.scrapinghub.com/2015/03/16/history-of-scrapinghub