Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Flyscrape – A standalone and scriptable web scraper in Go

github.com

Post reply on HN
Print view View original

31–40 of 57 posts

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#31

Post by erhaetherth »

sunshadow wrote: post #27
8n4vidtmkvmk wrote: post #26

Earlier quoted context omitted.

I started putting data-testid attributes in my web app for automated testing using playwright. Prevents me from breaking my own script but it sure would make me more scrapable if anyone cared. Well.. I guess I only do it on inputs, not the rendered page which is what scrapers care most about.

Unless you start a war against scrapers, you don't need to worry about that as I'll always find a way to scrape your site as long as its valuable to 'me'. Even if it requires Real browser + OCR :)

Oh I know I couldn't prevent it. But if you wanted to scrape me, you'd have to pay the monthly subscription because everything is behind a pay wall/login. And then you'd only have access to data you entered because it's just that kind of app :-)

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#32

Post by _lvbh »

philippta wrote: post #4
bryanrasmussen wrote: post #3

Looks like it doesn't have the possibility of running it as a particular browser etc. Which I guess makes it fine for a lot of pages, but also a lot of scraping tasks would be affected. Am I right or did I miss something?

Yes, this is correct. As of right now there is no built-in support for running as a browser. What is possible though, is to use a service like ScrapingBee (not affiliated) and set it as the proxy. This would render the page on their end, in a browser.

Try tls-client. It gets around TLS fingerprinting by Cloudflare

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#33

Post by mdaniel »

reyostallenberg wrote: post #22
mikercampbell wrote: post #15

Earlier quoted context omitted.

Have you seen Crul?? I love the JS flow, but I thought crul was an interesting newer tool!! But I agree, you gotta get in there and it’s easier with JS

Can you add a link to it?

I'm sorry to hear that your searches for that very specific name didn't provide the information you were looking for

its show hn: https://news.ycombinator.com/item?id=34970917

tfl: https://www.crul.com/

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#35

Post by fyzix »

What happens if 'find()' returns a list and you call '.text()'. Intuition tells me it should fail but maybe it implicitly gets the text from the first item if it exists.

Either way, I think you create a separate method 'find_all()' that returns a list to make the API easier to reason about.

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#36

Post by aynyc »

nurettin wrote: post #28
hipadev23 wrote: post #19

Earlier quoted context omitted.

How does that help you mitigate when a site changes? If you’re fetching some value in a given under a long XPATH and they decide to change that path?

Don't know about the poster, but I try to find divs and buttons in a fuzzy way. Usually via element text. Sometimes it mitigates changes, sometimes it doesn't. It's a guessing game. Especially when they start using shadow elements or iframes in the page. If I'm looking for something specific like a price or dimensions, I can sometimes get away with it by collecting dollar amounts or X x Y x Z from the raw text.

iframes have been a pain the butt to scrape against. I see it more and more in websites now.

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#37

Post by gymbeaux »

sunshadow wrote: post #13

These days, I'm not even using Go for scraping that much, as the webpage changes makes me crazy and JS code evaluation is a lifesaver, so I moved to Typescript+Playwright. (Crawlee framework is cool, while not strictly necessary). Its been 8+ years since i started scraping. I even wrote a popular Go web scraping framework previously: ( https://github.com/geziyor/geziyor ). My favorite stack as of 2023: TypeScript+Pla…

What are some examples of needing a lower-level language?

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#38

Post by moehm »

Interesting. Can you compare it to colly? [0]

Last time I looked it was the most popular choice for scraping in Go and I have some projects using it.

Is it similar? Does it have more/less features or is it more suited for a different use case? (Which one?)

[0] https://github.com/gocolly/colly

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#39

Post by philippta »

moehm wrote: post #38

Interesting. Can you compare it to colly? [0] Last time I looked it was the most popular choice for scraping in Go and I have some projects using it. Is it similar? Does it have more/less features or is it more suited for a different use case? (Which one?) [0] https://github.com/gocolly/colly

Colly is a great scraping library if you are a Go developer.

Flyscrape on the other hand is a ready-made CLI tool that aims to be easy to use even for someone who is a little familiar with JavaScript. It just happens to be written in Go, but that should not matter to the end user.

It does not have full feature parity with Colly but most use cases should be covered.

Post reply on HN