Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Flyscrape – A standalone and scriptable web scraper in Go

github.com

Post reply on HN
Print view View original

11–20 of 57 posts

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#11

Post by cxr »

xyzzy_plugh wrote: post #9

I like web scraping in Go. The support for parsing HTML in x/text/html is pretty good, and libraries like github.com/PuerkitoBio/goquery go a long way to matching ergonomics in other tools. This project uses both, but then also goes on to use github.com/dop251/goja, which is a JavaScript VM and it's accompanying nodejs compatability layer and even esbuild, in order to interpret scraping instruction scripts . I mean,…

Your comment was posted 4 minutes ago. That means you still have enough time to edit your comment to change it so it contains real URLs that link to the project repos for the packages mentioned:

https://github.com/PuerkitoBio/goquery>

https://github.com/dop251/goja>

(Please do not reply to this comment of mine—if you do, I won't be able to delete it once the previous post is fixed, because the existence of the replies will prevent that.)

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#12

Post by mdaniel »

figmert wrote: post #6
snake117 wrote: post #5

Looks interesting, and thank you for sharing this! One common issue with scraping web pages is dealing with data that is dynamically loaded. Is there a solution for this? For example, when using Scrapy, you can have Splash running in Docker via scrapy-splash ( https://github.com/scrapy-plugins/scrapy-splash ).

Can't you load the URL that is being dynamically loaded directly within your scraper?

Not only can you, in my experience it is substantially less drama and arguably less load on the target system since the full page may make many many other requests that a presentation layer would care about that I don't

The trade-offs usually fall into:

- authing to the endpoint can sometimes be weird

- it for sure makes the traffic stand out since it isn't otherwise surrounded by those extraneous requests

- it, as with all good things scraping, carries its own maintenance and monitoring burden

However, similar to those tradeoffs, it's also been my experience that a full page load offers a ton more tracking opportunities that are not present in a direct endpoint fetch. I mean, look how many "stealth" plugins out there designed to mask the fact that a headless browser is headless

But, having said all of that: without question the biggest risk to modern day scraping is Cloudflare and Akamai gatekeeping. I do appreciate the arguments of "but ddos!11" and yet I would rather only actors that are actually exhibiting bad behavior[1] be blocked instead of everyone trying with a copy of python who have set reasonable rate limits

1 = this setting aside that "bad behavior" can be defined as "downloading data that the site makes freely available to Chrome but not freely available to python"

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#13

Post by sunshadow »

These days, I'm not even using Go for scraping that much, as the webpage changes makes me crazy and JS code evaluation is a lifesaver, so I moved to Typescript+Playwright. (Crawlee framework is cool, while not strictly necessary).

Its been 8+ years since i started scraping. I even wrote a popular Go web scraping framework previously: (https://github.com/geziyor/geziyor).

My favorite stack as of 2023: TypeScript+Playwright+Crawlee(Optional) If you're serious in scraping, you should learn javascript, thus, playwright should be good.

Note: There are niche cases where lower-level language would be required (C++, Go etc), but probably only <%5

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#14

Post by cheapgeek »

cxr wrote: post #11
xyzzy_plugh wrote: post #9

I like web scraping in Go. The support for parsing HTML in x/text/html is pretty good, and libraries like github.com/PuerkitoBio/goquery go a long way to matching ergonomics in other tools. This project uses both, but then also goes on to use github.com/dop251/goja, which is a JavaScript VM and it's accompanying nodejs compatability layer and even esbuild, in order to interpret scraping instruction scripts . I mean,…

Your comment was posted 4 minutes ago. That means you still have enough time to edit your comment to change it so it contains real URLs that link to the project repos for the packages mentioned: https://github.com/PuerkitoBio/goquery > https://github.com/dop251/goja > (Please do not reply to this comment of mine—if you do, I won't be able to delete it once the previous post is fixed, because the existence of the repl…

[flagged]

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#15

Post by mikercampbell »

sunshadow wrote: post #13

These days, I'm not even using Go for scraping that much, as the webpage changes makes me crazy and JS code evaluation is a lifesaver, so I moved to Typescript+Playwright. (Crawlee framework is cool, while not strictly necessary). Its been 8+ years since i started scraping. I even wrote a popular Go web scraping framework previously: ( https://github.com/geziyor/geziyor ). My favorite stack as of 2023: TypeScript+Pla…

Have you seen Crul??

I love the JS flow, but I thought crul was an interesting newer tool!!

But I agree, you gotta get in there and it’s easier with JS

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#16

Post by [deleted] »

sunshadow wrote: post #13

These days, I'm not even using Go for scraping that much, as the webpage changes makes me crazy and JS code evaluation is a lifesaver, so I moved to Typescript+Playwright. (Crawlee framework is cool, while not strictly necessary). Its been 8+ years since i started scraping. I even wrote a popular Go web scraping framework previously: ( https://github.com/geziyor/geziyor ). My favorite stack as of 2023: TypeScript+Pla…

[deleted]

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#17

Post by [deleted] »

xyzzy_plugh wrote: post #9

I like web scraping in Go. The support for parsing HTML in x/text/html is pretty good, and libraries like github.com/PuerkitoBio/goquery go a long way to matching ergonomics in other tools. This project uses both, but then also goes on to use github.com/dop251/goja, which is a JavaScript VM and it's accompanying nodejs compatability layer and even esbuild, in order to interpret scraping instruction scripts . I mean,…

[deleted]

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#18

Post by [deleted] »

snake117 wrote: post #5

Looks interesting, and thank you for sharing this! One common issue with scraping web pages is dealing with data that is dynamically loaded. Is there a solution for this? For example, when using Scrapy, you can have Splash running in Docker via scrapy-splash ( https://github.com/scrapy-plugins/scrapy-splash ).

[deleted]

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#19

Post by hipadev23 »

sunshadow wrote: post #13

These days, I'm not even using Go for scraping that much, as the webpage changes makes me crazy and JS code evaluation is a lifesaver, so I moved to Typescript+Playwright. (Crawlee framework is cool, while not strictly necessary). Its been 8+ years since i started scraping. I even wrote a popular Go web scraping framework previously: ( https://github.com/geziyor/geziyor ). My favorite stack as of 2023: TypeScript+Pla…

How does that help you mitigate when a site changes? If you’re fetching some value in a given under a long XPATH and they decide to change that path?

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#20

Post by xyzzy_plugh »

cxr wrote: post #11
xyzzy_plugh wrote: post #9

I like web scraping in Go. The support for parsing HTML in x/text/html is pretty good, and libraries like github.com/PuerkitoBio/goquery go a long way to matching ergonomics in other tools. This project uses both, but then also goes on to use github.com/dop251/goja, which is a JavaScript VM and it's accompanying nodejs compatability layer and even esbuild, in order to interpret scraping instruction scripts . I mean,…

Your comment was posted 4 minutes ago. That means you still have enough time to edit your comment to change it so it contains real URLs that link to the project repos for the packages mentioned: https://github.com/PuerkitoBio/goquery > https://github.com/dop251/goja > (Please do not reply to this comment of mine—if you do, I won't be able to delete it once the previous post is fixed, because the existence of the repl…

Even if I saw this post in time, I wouldn't have edited it. They are all proper Go package names.
Post reply on HN