Really cool. I have some scraping projects I'd like to do in Go, so this is a great starting point.
goquery is really neat, does a lot of the lifting for you!
Show HN: BBC Good Food Scraper in Go
11–20 of 26 posts
Re: Show HN: BBC Good Food Scraper in Go
#12It seems with all these website scrapers, we have forgotten about the Internet Archive. http://archive.org/web/
And the fantastic ArchiveBot.. http://www.archiveteam.org/index.php?title=ArchiveBot
Re: Show HN: BBC Good Food Scraper in Go
#13It seems with all these website scrapers, we have forgotten about the Internet Archive. http://archive.org/web/
As awesome as the Internet Archive's Wayback Machine is, it is still under central control. Worse, as they (reasonably) abide to robots.txt rules, the BBC could easily block access to pages they removed at their end in the archive as well. If you care about something, you need to fully "own" it.
Re: Show HN: BBC Good Food Scraper in Go
#14I also wrote a scraper for BBC Food, and implemented https://github.com/NYTimes/ingredient-phrase-tagger into it quite successfully, I guess I should push it up to Github.
Re: Show HN: BBC Good Food Scraper in Go
#15Be careful on your conversion from Sodium -> salt (salt is 40% Sodium)
e.g. http://healthyeating.sfgate.com/difference-between-salt-sodi...
edit: oh actually the BBC website has it labeled as "Salt", but the HTML ID is "sodiumContent". Weird. Worth a comment then :P
Re: Show HN: BBC Good Food Scraper in Go
#16Be careful on your conversion from Sodium -> salt (salt is 40% Sodium) e.g. http://healthyeating.sfgate.com/difference-between-salt-sodi... edit: oh actually the BBC website has it labeled as "Salt", but the HTML ID is "sodiumContent". Weird. Worth a comment then :P
Yup, I thought that when I was implementing the nutrition stuff. My guess is that the original implementer went "salt content, well that's really a sodium property"
Re: Show HN: BBC Good Food Scraper in Go
#17BBC Good Food is not going anywhere. So you can stop the scraping... It's BBC Food (different name, different website) the one who is going to close (even the recipes will remain online, btw)
Re: Show HN: BBC Good Food Scraper in Go
#18BBC Good Food is not going anywhere. So you can stop the scraping... It's BBC Food (different name, different website) the one who is going to close (even the recipes will remain online, btw)
"article implied that the bbc good food website would be taken down. This turned out to be false but by the time I realised that I'd already written this, so here you are."
Re: Show HN: BBC Good Food Scraper in Go
#19Earlier quoted context omitted.
Agree, used it to build this as a proof of concept: https://emailprofile.herokuapp.com/ Many sites don't consider user enumeration a bug/threat, but theoretically given enough sites, one could build a profile around a specific email address.
Have you open-sources this package somewhere? As an amateur web security researcher this project looks really interesting.
I haven't yet. I'm a bit torn on whether or not to release it given the potential privacy implications. With more sites and a classification algorithm, one could say an email is a "Gender, Race, Age Range, Job Industry, Interests, etc."
What makes this tool work is most sites (as an UX feature) will tell you if an account/email already exists. Whether that be an API call or a notice saying "Your password is incorrect", you'll be able to get the data you need. It was a learning experience for me to use Go to wrap each site check in its own goroutine to leverage concurrency. Quite nice.
Re: Show HN: BBC Good Food Scraper in Go
#20BBC Good Food is not going anywhere. So you can stop the scraping... It's BBC Food (different name, different website) the one who is going to close (even the recipes will remain online, btw)
I'm left-liberal British and am therefore expected to support the BBC I guess, but why in heaven this publicly-funded organisation has one recipe site, let alone two, baffles me.