Show HN: BBC Good Food Scraper in Go

#1

Post by sleepychu » Thu, Jun 23, 2016, 12:12 AM UTC

Show HN: BBC Good Food Scraper in Go
github.com

Re: Show HN: BBC Good Food Scraper in Go

#2

Post by krat0sprakhar » Thu, Jun 23, 2016, 2:11 AM UTC

Goquery is pretty great - used it last semester to write the crawler for a side-project of mine. https://github.com/prakhar1989/bekanjoos

Re: Show HN: BBC Good Food Scraper in Go

#3

Post by lxchase » Thu, Jun 23, 2016, 3:37 AM UTC

krat0sprakhar wrote: post #2

Goquery is pretty great - used it last semester to write the crawler for a side-project of mine. https://github.com/prakhar1989/bekanjoos

Agree, used it to build this as a proof of concept: https://emailprofile.herokuapp.com/

Many sites don't consider user enumeration a bug/threat, but theoretically given enough sites, one could build a profile around a specific email address.

Re: Show HN: BBC Good Food Scraper in Go

#4

Post by imwally » Thu, Jun 23, 2016, 3:54 AM UTC

krat0sprakhar wrote: post #2

Goquery is pretty great - used it last semester to write the crawler for a side-project of mine. https://github.com/prakhar1989/bekanjoos

It really is a great package and works extremely well in the context of scraping. It's what I opted to use for an API server [1] I wrote that handles the RFC Instant Answer [2] running on DuckDuckGo.

[1] https://github.com/imwally/rfcsearch [2] https://duck.co/ia/view/request_for_comments

Re: Show HN: BBC Good Food Scraper in Go

#5

Post by callmeed » Thu, Jun 23, 2016, 4:21 AM UTC

Really cool. I have some scraping projects I'd like to do in Go, so this is a great starting point.

Re: Show HN: BBC Good Food Scraper in Go

#6

Post by imjacobclark » Thu, Jun 23, 2016, 7:47 AM UTC

It seems with all these website scrapers, we have forgotten about the Internet Archive. http://archive.org/web/

Re: Show HN: BBC Good Food Scraper in Go

#7

Post by iNerdier » Thu, Jun 23, 2016, 7:56 AM UTC

Is it not http://www.bbc.co.uk/food/ that's being merged into bbc good food and therefore gaining ads? Surely that should be the target for this scraper...

Re: Show HN: BBC Good Food Scraper in Go

#8

Post by LunaSea » Thu, Jun 23, 2016, 7:59 AM UTC

lxchase wrote: post #3

krat0sprakhar wrote: post #2

Goquery is pretty great - used it last semester to write the crawler for a side-project of mine. https://github.com/prakhar1989/bekanjoos

Agree, used it to build this as a proof of concept: https://emailprofile.herokuapp.com/ Many sites don't consider user enumeration a bug/threat, but theoretically given enough sites, one could build a profile around a specific email address.

Have you open-sources this package somewhere?

As an amateur web security researcher this project looks really interesting.

Re: Show HN: BBC Good Food Scraper in Go

#9

Post by Jaruzel » Thu, Jun 23, 2016, 8:00 AM UTC

iNerdier wrote: post #7

Is it not http://www.bbc.co.uk/food/ that's being merged into bbc good food and therefore gaining ads? Surely that should be the target for this scraper...

It is, but haven't a 1000+ people already scraped it? (myself included!)

Re: Show HN: BBC Good Food Scraper in Go

#10

Post by sambeau » Thu, Jun 23, 2016, 8:01 AM UTC

Jaruzel wrote: post #9

iNerdier wrote: post #7

Is it not http://www.bbc.co.uk/food/ that's being merged into bbc good food and therefore gaining ads? Surely that should be the target for this scraper...

It is, but haven't a 1000+ people already scraped it? (myself included!)

Me too!