Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

321–330 of 402 posts

Re: Web Scraping in 2016

#321

Post by LunaSea »

stupidcar wrote: post #154
ralusek wrote: post #140

Earlier quoted context omitted.

Threading in node is very easy, just use clusters. Alternatively, take any of the CPU intensive activity, like parsing the HTML and formatting as JSON, and just put that on an AWS lambda. You can invoke as many lambdas from your application as you want in parallel and you're not going to be bottlenecked by your CPU :)

Clustering in Node creates isolated child processes, not threads. I needed to have shared queues, in-memory caches, and hashes to coordinate workers and avoid them doing duplicate work. I'm did consider using clustering and having some master process coordinate everything, and using some shared-memory caching library. But it would not be "easy" to set up, especially compared to something like Java where you get threa…

Threading for a crawler is just a dirty way of not handling distribution. When you will need more than one server your threads won't save you. It has nothing to do with Node.js and thread support.

Re: Web Scraping in 2016

#322

Post by lamby »

Whilst they mean well, I find this a fundamentally deceptive — the arduous parts of "real world" scraping simply aren't in the parsing and extraction of data from the target page, the typical focus of these "scrape the web with X" articles.

The difficulties are invariably in "post-processing"; working around incomplete data on the page, handling errors gracefully and retrying in some (but not all) situations, keeping on top of layout/URL/data changes to the target site, not hitting your target site too often, logging into the target site if necessary and rotating credentials and IP addresses, respecting robots.txt, target site being utterly braindead, keeping users meaningfully informed of scraping progress if they are waiting of it, target site adding and removing data resulting in a null-leaning database schema, sane parallelisation in the presence of prioritisation of important requests, difficulties in monitoring a scraping system due to its implicitly non-deterministic nature, and general problems associated with long-running background processes in web stacks.

Et cetera.

In other words, extracting the right text on the page is the easiest and trivial part by far, with little practical difference between an admittedly cute jQuery-esque parsing library or even just using a blunt regular expression.

It would be quixotic to simply retort that sites should provide "proper" APIs but I would love to see more attempts at solutions that go beyond the superficial.

Re: Web Scraping in 2016

#323

Post by LunaSea »

nreece wrote: post #230

At Feedity ( https://feedity.com ), we "index" webpages to generate custom feeds. Over the years, we've designed our system to use a mix of technologies like .NET (C#) and node.js, and implemented a bunch of tweaks and optimizations for seamless & scalable access to public content.

Any tips and tricks you are able to share about the technologies you guys developed? It would be especially interesting to see what you use for text extraction from HTML.

Re: Web Scraping in 2016

#324

Post by Twisell »

What bother me the most is that recently I wanted to extract and archive of all the threads I participated in from an Internet forum. The webmaster told me that the BBS he use don't provide such a function and that I just had to download each thread manually... (300+ thread in my case).

He then say that it don't bother him if I scrape theses thread. And I'm currently figuring out how to manage his site's cookie protected search feature, so that my painstaking effort (I'm not a dev, more a DB guy) could be reproducible more easily by other users of this service.

But this shouldn't appen in the first place because all post of this service are stored in a cleanly organized MySQl DB. Yet as no method is provided the only way to get back structured data is by scrapping (as the webmaster told me that no, he won't run custom SQL because "he don't want to mess his DB").

So even if all the data is publicly available through the internet forum only a geek can download a personal archive... or google because google scrape and store everything.

Re: Web Scraping in 2016

#325

Post by CalRobert »

Twisell wrote: post #324

What bother me the most is that recently I wanted to extract and archive of all the threads I participated in from an Internet forum. The webmaster told me that the BBS he use don't provide such a function and that I just had to download each thread manually... (300+ thread in my case). He then say that it don't bother him if I scrape theses thread. And I'm currently figuring out how to manage his site's cookie prote…

It's overkill for most things, but I have found that on occasion the best way to scrape stuff behind annoying frontends is with Selenium. pysaunter is a useful library that's one layer of abstraction higher, if you're familiar with Python.

Re: Web Scraping in 2016

#326

Post by CalRobert »

CalRobert wrote: post #325
Twisell wrote: post #324

What bother me the most is that recently I wanted to extract and archive of all the threads I participated in from an Internet forum. The webmaster told me that the BBS he use don't provide such a function and that I just had to download each thread manually... (300+ thread in my case). He then say that it don't bother him if I scrape theses thread. And I'm currently figuring out how to manage his site's cookie prote…

It's overkill for most things, but I have found that on occasion the best way to scrape stuff behind annoying frontends is with Selenium. pysaunter is a useful library that's one layer of abstraction higher, if you're familiar with Python.

Well I see now that I'm really late to the party with that comment.

Re: Web Scraping in 2016

#327

Post by stupidcar »

LunaSea wrote: post #321
stupidcar wrote: post #154

Earlier quoted context omitted.

Clustering in Node creates isolated child processes, not threads. I needed to have shared queues, in-memory caches, and hashes to coordinate workers and avoid them doing duplicate work. I'm did consider using clustering and having some master process coordinate everything, and using some shared-memory caching library. But it would not be "easy" to set up, especially compared to something like Java where you get threa…

Threading for a crawler is just a dirty way of not handling distribution. When you will need more than one server your threads won't save you. It has nothing to do with Node.js and thread support.

I wasn't creating a new search engine, I was doing a one-off scraping job in my spare time. Creating a fully distributed solution would have been total overkill. But threading could and would have helped.

Honestly, stupidly hostile and ignorant comments like this are the absolute worst thing about Hacker News.

Re: Web Scraping in 2016

#328

Post by prashnts »

pault wrote: post #294
esac wrote: post #279

Earlier quoted context omitted.

do you have something? i was going to write the very same extension (but distributed so i could add it to my pc and my friends) but never did that

This is the boilerplate I used last time: http://extensionizr.com

Didn't know about `extensionizr`, Looks super cool. Thanks!

Re: Web Scraping in 2016

#329

Post by vidarh »

lerpa wrote: post #311
louprado wrote: post #304

Earlier quoted context omitted.

"As the user of large websites I don't care". Are you sure ? Do you want your OK Cupid or LinkedIn profile to be crossposted on another website without your knowledge.

If you don't want your data public, then don't make it public in the first place. That's a good rule of thumb.

Putting it behind a signup page with terms that don't allow sharing is not "making it public".

And while in the US that may "just" be treated as unauthorized access, in the EU, if you make the data public it's also a violation of the Data Protection Directive, putting you at risk of prosecution in every EU country from which you have included data.

You may be right from a risk minimisation perspective. But for a lot of data the risk in the case of exposure is low enough that it is a totally valid risk management strategy to assume that legal protections will be a sufficient deterrent to prevent enough of the most blatant abuses.

Re: Web Scraping in 2016

#330

Post by jacquesm »

Bootvis wrote: post #319
nerdponx wrote: post #310

Earlier quoted context omitted.

The difference is that Google didn't agree to not scrape your data. You, as per their TOS, agreed not to scrape theirs, as part of the condition of using their service.

Which TOS? I might have accepted terms when I created a Google Account but in no way do I agree to a TOS by visiting a URL.

Try using google from a fresh install, they´ll force you to accept their TOS.
Post reply on HN