Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: An API for scraping recipe web pages

choppingboard.recipes

Post reply on HN
Print view View original

81–90 of 113 posts

Re: Show HN: An API for scraping recipe web pages

#81

Post by mmccaff »

oelmekki wrote: post #70
Freak_NL wrote: post #3

It looks good and seems to work with some arbitrarily picked recipes on the usual large recipe websites, although more obscure links cause some bugs¹. Is the source code on GitHub? Are you handling specific websites such as allrecipes.com with bespoke code? Technically the project is interesting, but if you want to offer a commercial API you might run into copyright and fair use issues (as with any scraping tool). No…

I'm actually amazed it manages to scrap minor websites. Scrappers usually have to take in consideration every single website they're targeted at, and they break each time there's a major redesign (or sometimes, a minor one). @brad0 : how did you manage that? Was thespruce.com in your targets?

I'm also curious what the approach was to being able to scrape sites in a way where the code is not coupled to a known, fixed markup structure using css selectors, xpath etc.

What are your plans for the project? Will you be open sourcing the code on GitHub?

Re: Show HN: An API for scraping recipe web pages

#82

Post by tinebak »

JshWright wrote: post #79
Dowwie wrote: post #34

This falls under the category of unethical scraping

While it doesn't address your concerns about ethics, recipes are explicitly not covered by US copyright law (obviously there are other factors involved here, but that's certainly the big one). There may be ethical questions here, but there shouldn't be any legal concerns (IANAL, just a guy in the process of building a site in this space).

Recipes aren't but images always are. Collections of recipes (such as a cookbook) are also protected.

Re: Show HN: An API for scraping recipe web pages

#84

Post by madamelic »

Freak_NL wrote: post #6
oneeyedpigeon wrote: post #4

This really seems to be an exercise in 'structuring recipe data' rather than the ins-and-outs of scraping. Seems like a much-needed task; is there anything approaching a 'standard' for recipe data already? "ingredients": [ "600g pineapple, peeled, chopped" ] This seems like a prime candidate for improvement; something like the following would seem to be more useful: "ingredients": [{ "ingredient": "pineapple", "quant…

Whenever I read a recipe online I get this nagging feeling that there should be some widely used open standard for describing recipes that can do all sorts of awesome stuff (like a web crawler such as DuckDuckGo being able to answer "I have a pineapple, cream, and the usual pantry basics, what can I cook?"), but this seems to go directly against the business models of the big recipe sites, so it will not likely see m…

Check out PantryPlan: https://pantryplan.io. It does exactly that and not just meal-by-meal but entire menus.

Disclaimer: I made it. :) Let me know your thoughts.

Re: Show HN: An API for scraping recipe web pages

#85

Post by nkrisc »

mikepurvis wrote: post #69
oneeyedpigeon wrote: post #4

This really seems to be an exercise in 'structuring recipe data' rather than the ins-and-outs of scraping. Seems like a much-needed task; is there anything approaching a 'standard' for recipe data already? "ingredients": [ "600g pineapple, peeled, chopped" ] This seems like a prime candidate for improvement; something like the following would seem to be more useful: "ingredients": [{ "ingredient": "pineapple", "quant…

I tried to build a little recipe DB along these lines years ago. What I ended up finding is that as I improved in my own food preparation skills, my interest in super-rigid recipes of this form diminished. Basically, there's just a lot of slush and room for experimentation in cooking. Think about making a stir fry— how many people would level off exact masses of different vegetables to combine in perfect proportion?…

I 100% agree with you. Perhaps this kind of structure would be better suited to baking recipes? This sort of precision (amounts, timing, even order of ingredients) can be very important when baking.

Re: Show HN: An API for scraping recipe web pages

#86

Post by benyami »

philbarr wrote: post #16
Freak_NL wrote: post #6

Earlier quoted context omitted.

Whenever I read a recipe online I get this nagging feeling that there should be some widely used open standard for describing recipes that can do all sorts of awesome stuff (like a web crawler such as DuckDuckGo being able to answer "I have a pineapple, cream, and the usual pantry basics, what can I cook?"), but this seems to go directly against the business models of the big recipe sites, so it will not likely see m…

I thought once about doing a "flavour graph" where people could go on and add what they thought were good flavours to mix together. The more people added the same flavours, the stronger the edges between nodes would get. I'd probably have to separate it somehow by geographic region, as tastes vary, but in the end you should have a good approximation of "what goes with what".

The Flavor Bible is a published book version of what you're describing.

Re: Show HN: An API for scraping recipe web pages

#87

Post by k3oni »

tinebak wrote: post #58
pjc50 wrote: post #54

Earlier quoted context omitted.

Nice :) Is there some artefact of the sorting that puts only pizza recipes on the front page?

That's funny! I'm not seeing any pizza recipes now. It shows the latest shared recipes that have images. There are trends, though. One person wrote and complained that vegans were taking over the website, but the next day the Community was back to having lots of meat. The app got a bad review in the app store because this person said that Copy Me That was always showing pot recipes!

Well not sure what it might be, but as the previous poster got pizza all over i'm into salmon apparently, in 2 pages and 95% recipes are all about salmon :)

Re: Show HN: An API for scraping recipe web pages

#89

Post by tinebak »

k3oni wrote: post #87
tinebak wrote: post #58

Earlier quoted context omitted.

That's funny! I'm not seeing any pizza recipes now. It shows the latest shared recipes that have images. There are trends, though. One person wrote and complained that vegans were taking over the website, but the next day the Community was back to having lots of meat. The app got a bad review in the app store because this person said that Copy Me That was always showing pot recipes!

Well not sure what it might be, but as the previous poster got pizza all over i'm into salmon apparently, in 2 pages and 95% recipes are all about salmon :)

How weird, it IS all salmon! :) The front page can have a bit of a theme if someone quickly copies in a lot of similar recipes, for example, if he/she is looking for the best salmon recipes from around the web. However, the front page is limited to three recipes from any given person's recipe box. The front page might also show several copies of the same recipe if that recipe suddenly becomes very popular. However, then the recipes would all be showing the exact same salmon recipe. But this is lots of different people copying in salmon recipes from lots of different websites! The world has gone bonkers for salmon.

The front page (Community page) looks different if you're logged in because it updates more often. Maybe I need to re-think the time frame because now the website is stuck on all salmon for two hours for non-logged in users (unless they do a search).

Re: Show HN: An API for scraping recipe web pages

#90

Post by solatic »

mikepurvis wrote: post #69
oneeyedpigeon wrote: post #4

This really seems to be an exercise in 'structuring recipe data' rather than the ins-and-outs of scraping. Seems like a much-needed task; is there anything approaching a 'standard' for recipe data already? "ingredients": [ "600g pineapple, peeled, chopped" ] This seems like a prime candidate for improvement; something like the following would seem to be more useful: "ingredients": [{ "ingredient": "pineapple", "quant…

I tried to build a little recipe DB along these lines years ago. What I ended up finding is that as I improved in my own food preparation skills, my interest in super-rigid recipes of this form diminished. Basically, there's just a lot of slush and room for experimentation in cooking. Think about making a stir fry— how many people would level off exact masses of different vegetables to combine in perfect proportion?…

> I think treating food preparation as equivalent to a Vagrantfile would really lose something along the way.

That's a neat idea, though, in a different direction. If you could implement some kind of standard recipe API and build a business around being a broker for commercial kitchens, wherein a customer inputs a recipe that follows a certain schema and an address, and the commercial kitchens respond saying either "no, we can't make that - we don't have the necessary ingredients / we're cooking at capacity / etc." or "sure, here's how much time it would take and here's how much it would cost." You would lose economies of scale found in most commercial kitchens, but customers would gain much more freedom over what they're ordering compared to pick-from-a-list traditional menus. Users could also search for highly rated recipes from around the world and see if somebody local could make it for them etc.

Post reply on HN