Show HN: BBC Good Food Scraper in Go • Hacker News Board

Re: Show HN: BBC Good Food Scraper in Go

#21

Post by open-source-ux » Thu, Jun 23, 2016, 3:02 PM UTC

Doctor_Fegg wrote: post #20

ryck wrote: post #17

BBC Good Food is not going anywhere. So you can stop the scraping... It's BBC Food (different name, different website) the one who is going to close (even the recipes will remain online, btw)

I'm left-liberal British and am therefore expected to support the BBC I guess, but why in heaven this publicly-funded organisation has one recipe site, let alone two, baffles me.

BBC Good Food is funded by BBC Worldwide, the commercial arm of the BBC. It is not funded by licence fee money. The website is a companion (or extension) of their BBC Good Food magazine.

The BBC Food website, on the other hand, is funded by the licence fee. Recipes from many of their food programmes are published here, but much more too. It's grown to be much more than just a companion to their food programmes. But now, with little support from the public, they've decided to close the site.

Re: Show HN: BBC Good Food Scraper in Go

#22

Post by sleepychu » Thu, Jun 23, 2016, 4:18 PM UTC

krat0sprakhar wrote: post #2

Goquery is pretty great - used it last semester to write the crawler for a side-project of mine. https://github.com/prakhar1989/bekanjoos

bekanjoos looks nice!

I was thinking today about making a more generic "when this thing on that website changes" notify me sort of thing. It looks like you're running on AWS. Would you be willing to share how much the bill for that runs to?

Re: Show HN: BBC Good Food Scraper in Go

#23

Post by Chris2048 » Thu, Jun 23, 2016, 5:47 PM UTC

IMHO the real challenge for scraping (other than at-scale issue like spawning many processes, crawling, proxies etc), is a scraping framework that allows you to change you mind about what needs scraping, without having to redo the entire scrape. Also, re-scraping for updates.

Between "remember every bit of HTML", and "only remember parsed data" is perhaps "remember every bit of HTML, but notice base-html patterns so it can be massively compressed. "dynamic" content like java-script/AJAX content, rendered dates complicate this...

Re: Show HN: BBC Good Food Scraper in Go

#24

Post by khaerudin » Thu, Jun 23, 2016, 5:50 PM UTC

oke thanks for the informations https://pariwisatakepulauanseribu.wordpress.com

Re: Show HN: BBC Good Food Scraper in Go

#25

Post by khaerudin » Thu, Jun 23, 2016, 5:53 PM UTC

khaerudin wrote: post #24

oke thanks for the informations https://pariwisatakepulauanseribu.wordpress.com

info wisata pulau seribu http://travelwisata.asia http://pulautidungpewe.com http://tidungpaketmurah.com http://birowisatapulauseribu.com http://pariwisatakepulauanseribu.wordpress.com

Re: Show HN: BBC Good Food Scraper in Go

#26

Post by jameshart » Thu, Jun 23, 2016, 6:40 PM UTC

Doctor_Fegg wrote: post #20

ryck wrote: post #17

BBC Good Food is not going anywhere. So you can stop the scraping... It's BBC Food (different name, different website) the one who is going to close (even the recipes will remain online, btw)

I'm left-liberal British and am therefore expected to support the BBC I guess, but why in heaven this publicly-funded organisation has one recipe site, let alone two, baffles me.

Before the internet was a thing, cookery programs used to end with the information that the recipes featured in the program could be found on Ceefax, or you could send a self-addressed envelope into BBC TV center and they would mail the recipes out to you. It seems like an obvious and natural use of the web for the BBC to provide the same information on a web page that it used to provide.