Scraping Recipe Websites
21–30 of 204 posts
Re: Scraping Recipe Websites
#22Is this post illegal as it contains information on how to commit crimes such as copyright infringement?
Copyright does not protect recipes. It does more likely than not protect the images being scraped, though.
Re: Scraping Recipe Websites
#23Neat write-up, and thanks for putting me on to jsonld.js - looks useful.
I'm building https://simplescraper.io and we're trying to create heuristics to update CSS selectors whenever a website changes. People become unhappy when a scrape task that ran smoothly on Monday suddenly returns nothing on Tuesday so while it's a tough nut to crack it's super important.
We use a combination of XPath, historical data and data type (the value may change but the type and length often remain the same or similar) to narrow down the options.
Of course there's more sophisticated methods using Machine learning etc. but it's fun to try different approaches to solve this problem.
Re: Scraping Recipe Websites
#24Earlier quoted context omitted.
Scraping is LEGAL, all search engines scrape to some degree for example, there is a fair use component, so you can't "scrape" 100% of a site and stick it on your domain, but you can still scrape more than zero. In general it is leaning more acceptable than less.
Yeah I understand that part, my question is about showing the scraped data to your users. https://www.yummly.com/ used to have a paid API for recipe search and currently still lets users search their index. Did they have to go and get permission from each site that they index or is it fair use?
It looks a shade more detailed than google's recipe cards, they link back to the original source for the instructions, I would bet they didn't get permission, and that they count as a fair-use search engine. The law isn't (can't be) perfectly prescriptive here, there's some line that you have to sue about to know if it has been crossed.
Re: Scraping Recipe Websites
#25Next level : Shazam for cooking shows.
Re: Scraping Recipe Websites
#26Earlier quoted context omitted.
Having thought about this and my own use of cookbooks as loose inspiration rather than actually to follow in detail, I have come to the conclusion that the "fluff" is what most of the recipe-reading public want. In particular, the fluff has value even if you never cook the recipe! Which saves a lot of time and inconvenience on your part while still giving the same warm fluffy feelings. Actual "I have these things and…
I have come to the conclusion that the "fluff" is what most of the recipe-reading public want. I've often heard the prevailing reason why this happens on the web is because of SEO and 'bounce rates'. More time spent on the site improves ranking, so the actual recipe is pushed below the fold so users have to scroll down thereby adding more time on the site. Have often wondered if any SEO wonks with the inside baseball…
I can't validate this but have been told this by multiple authors and SEO professionals. So it's anecdotal. One blogger apologized to me and told me she was embarrassed doing it but it's an industry practice. She explained it's because other recipe sites use automated scraping tools and republish their recipes in an effort to outrank original authors. The personal fluff helps slow them down. Also, a lot of recipes are bullshit, they manually steal them from elsewhere and change a few variables to evade copyright claims. Although, I guess changing a few variables is how cuisines evolve.
Re: Scraping Recipe Websites
#27Paprika 3 (I use the iOS version, but I believe the Mac version has the same function) has a fantastic web scraper for recipes. I've had to correct maybe 1-2 errors across 100 recipes I've brought in from a bunch of different sites. It's super helpful to look through them in a standardized way (and you can sort by ingredient/category) to figure out what to make.
Tried this out and I have to say I'm impressed on the first recipe. Scraped it correctly (albeit from the BBC which has a reasonably sane layout) and, since I've only got 75g of dessicated coconut instead of the 85g required, I wanted to scale it by 75/85 ... which worked. I just typed in 75/85 and it worked. Amazing.
Re: Scraping Recipe Websites
#28Earlier quoted context omitted.
It's a running joke in our house. I start off wanting to make some mashed potatoes, and time and time again, I have to suffer through someone's life story--the camping trip in North Dakota when Susan's husband first discovered his love of homemade sour cream--etc. Makes me wonder if a super barebones recipe site that literally just has recipes and absolutely no fluff would be something people would gravitate towards.
Having thought about this and my own use of cookbooks as loose inspiration rather than actually to follow in detail, I have come to the conclusion that the "fluff" is what most of the recipe-reading public want. In particular, the fluff has value even if you never cook the recipe! Which saves a lot of time and inconvenience on your part while still giving the same warm fluffy feelings. Actual "I have these things and…
It happened to me. When I quizzed my grandmother for her beloved chocolate chip cookies recipe, I felt like being entrusted with a huge family secret.
Nope. Toll House cookie recipe (which is on the back of every package) with an extra quarter cup of flour XD.
Re: Scraping Recipe Websites
#29There is markup specifically for recipes. I wonder why it isn't more often used. EDIT: Yes, the article mentions it, but doesn't give a clue why it isn't more prevalent.
Probably to make stuff like this less effective - the person who posted the recipe only makes money from it when people actually visit the page and see the ads.
Re: Scraping Recipe Websites
#30Earlier quoted context omitted.
I have come to the conclusion that the "fluff" is what most of the recipe-reading public want. I've often heard the prevailing reason why this happens on the web is because of SEO and 'bounce rates'. More time spent on the site improves ranking, so the actual recipe is pushed below the fold so users have to scroll down thereby adding more time on the site. Have often wondered if any SEO wonks with the inside baseball…
I can't validate this but have been told this by multiple authors and SEO professionals. So it's anecdotal. One blogger apologized to me and told me she was embarrassed doing it but it's an industry practice. She explained it's because other recipe sites use automated scraping tools and republish their recipes in an effort to outrank original authors. The personal fluff helps slow them down. Also, a lot of recipes ar…
Recipes aren't copyrightable. It's bad manners to copy and republish one without attribution, but not illegal.
The introductory fluff is copyrightable, and that's one reason for it.