Ask HN: How you scrape websites?

11–17 of 17 posts

Re: Ask HN: How you scrape websites?

#11

Post by hawkweed » Mon, May 15, 2017, 12:59 PM UTC

JSoup[0] works fine for me.

#12

Post by motyar » Mon, May 15, 2017, 3:00 PM UTC

#13

Post by espiii » Mon, May 15, 2017, 6:07 PM UTC

curiousgal wrote: post #2

Scrapy

This tool felt like it descended from the gods of software. It's a brilliant tool.

#14

Post by Madeindjs » Mon, May 15, 2017, 8:23 PM UTC

Ruby & Nokogiri gem. You can also use Anemone gem to crawl the complete website.

#15

Post by bbcbasic » Tue, May 16, 2017, 10:33 AM UTC

tjalfi wrote: post #6

wget -mrnp https://www.example.com

| grep ... | sed ... | uniq ... | etc ...

#16

Post by egfx » Wed, May 17, 2017, 12:31 AM UTC

Try with another. The qkast web clipper http://qkast.com

#17

Post by jetti » Thu, May 18, 2017, 5:09 PM UTC

It has been awhile since I last did this but I used HTMLAgilityPack for .NET.

Post reply on HN