Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Flyscrape – A standalone and scriptable web scraper in Go

github.com

Post reply on HN
Print view View original

1–10 of 57 posts

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#4

Post by philippta »

bryanrasmussen wrote: post #3

Looks like it doesn't have the possibility of running it as a particular browser etc. Which I guess makes it fine for a lot of pages, but also a lot of scraping tasks would be affected. Am I right or did I miss something?

Yes, this is correct. As of right now there is no built-in support for running as a browser.

What is possible though, is to use a service like ScrapingBee (not affiliated) and set it as the proxy. This would render the page on their end, in a browser.

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#5

Post by snake117 »

Looks interesting, and thank you for sharing this! One common issue with scraping web pages is dealing with data that is dynamically loaded. Is there a solution for this? For example, when using Scrapy, you can have Splash running in Docker via scrapy-splash (https://github.com/scrapy-plugins/scrapy-splash).

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#6

Post by figmert »

snake117 wrote: post #5

Looks interesting, and thank you for sharing this! One common issue with scraping web pages is dealing with data that is dynamically loaded. Is there a solution for this? For example, when using Scrapy, you can have Splash running in Docker via scrapy-splash ( https://github.com/scrapy-plugins/scrapy-splash ).

Can't you load the URL that is being dynamically loaded directly within your scraper?

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#7

Post by philippta »

snake117 wrote: post #5

Looks interesting, and thank you for sharing this! One common issue with scraping web pages is dealing with data that is dynamically loaded. Is there a solution for this? For example, when using Scrapy, you can have Splash running in Docker via scrapy-splash ( https://github.com/scrapy-plugins/scrapy-splash ).

Thanks! As mentioned in another comment, currently there is no build in support for this yet.

As a workaround one could use a service like ScrapingBee (not affiliated) as a proxy, that renders the page in a browser for you.

Surely, relying on a service for this is not always ideal. I am also working on a small wrapper that turns Chrome into an HTTPS proxy, which you could plug right into flyscrape. Unfortunately it is very experimental still and not public yet. I have not yet decided if I release it as part of flyscrape or as a separate project.

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#9

Post by xyzzy_plugh »

I like web scraping in Go. The support for parsing HTML in x/text/html is pretty good, and libraries like github.com/PuerkitoBio/goquery go a long way to matching ergonomics in other tools. This project uses both, but then also goes on to use github.com/dop251/goja, which is a JavaScript VM and it's accompanying nodejs compatability layer and even esbuild, in order to interpret scraping instruction scripts.

I mean, at this point I am not sure Go is the right tool for the job (I am actually pretty confident that it is not).

A pretty neat stack of engineering, sure! This is cool, niely done. But I can't help but feel disturbed.

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#10

Post by [deleted] »

xyzzy_plugh wrote: post #9

I like web scraping in Go. The support for parsing HTML in x/text/html is pretty good, and libraries like github.com/PuerkitoBio/goquery go a long way to matching ergonomics in other tools. This project uses both, but then also goes on to use github.com/dop251/goja, which is a JavaScript VM and it's accompanying nodejs compatability layer and even esbuild, in order to interpret scraping instruction scripts . I mean,…

[deleted]
Post reply on HN