Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How to scrape and is it legal?

news.ycombinator.com

Post reply on HN
Print view View original

1–2 of 2 posts

Ask HN: How to scrape and is it legal?

#1

Post by newyorker »

I want to put together a website with a collection of data from other websites. I'm simply putting all the info in one place instead of many.

What's the best way to scrape this data? UPDATES: What if the data changes, will I be able to set it to rescrape or do I have to do it manually? Is it legal? I know its someone else's data, but is it the same as copy and paste?

*I have no background in scraping and only a little to make the web app, so any advice would be very helpful!

Thanks

Re: Ask HN: How to scrape and is it legal?

#2

Post by apedley »

Most websites have terms and conditions saying that automated retrieval of their content is prohibited.

It is generally all up to the Terms and Conditions of the site.

If a site wanted to share it's data it would normally give you an API.

You could email them and ask for permission if they do prohibit it, depending on your purpose.

And scraping does have its drawbacks such as changing site design and having to change your code with it.

The best and only way to scrape is just to retrieve the webpage in your application and parse out the results you want. You could do things such as XSS but it all depends on the complexitiy of the data. Normally the split and substring and indexof functions are what you will use.

Post reply on HN