Live data from Hacker News

Using Sitemaps in Web Scraping

1–3 of 3 posts

Post by webrobots » Mon, Mar 25, 2019, 11:33 AM UTC

Re: Using Sitemaps in Web Scraping

Post by HackerDalius » Mon, Mar 25, 2019, 11:37 AM UTC

Good article, useful general tips that work for many scraping tools and platforms.

Post by akriukas » Mon, Mar 25, 2019, 11:41 AM UTC

Very useful for some big eshops if they list their sitemaps. However not all stores have sitemaps and even robots.txt site...