Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Flyscrape – A standalone and scriptable web scraper in Go

github.com

Post reply on HN
Print view View original

21–30 of 57 posts

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#21

Post by sunshadow »

hipadev23 wrote: post #19
sunshadow wrote: post #13

These days, I'm not even using Go for scraping that much, as the webpage changes makes me crazy and JS code evaluation is a lifesaver, so I moved to Typescript+Playwright. (Crawlee framework is cool, while not strictly necessary). Its been 8+ years since i started scraping. I even wrote a popular Go web scraping framework previously: ( https://github.com/geziyor/geziyor ). My favorite stack as of 2023: TypeScript+Pla…

How does that help you mitigate when a site changes? If you’re fetching some value in a given under a long XPATH and they decide to change that path?

You don't use XPath&CSS selectors at all (Except if you dont have choice). You rely on more generic stuff, e.g, "the button that has 'Sign in' on it":

    await page.getByRole('button', { name: 'Sign in' }).click();
See playwright locators: https://playwright.dev/docs/locators

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#22

Post by reyostallenberg »

mikercampbell wrote: post #15
sunshadow wrote: post #13

These days, I'm not even using Go for scraping that much, as the webpage changes makes me crazy and JS code evaluation is a lifesaver, so I moved to Typescript+Playwright. (Crawlee framework is cool, while not strictly necessary). Its been 8+ years since i started scraping. I even wrote a popular Go web scraping framework previously: ( https://github.com/geziyor/geziyor ). My favorite stack as of 2023: TypeScript+Pla…

Have you seen Crul?? I love the JS flow, but I thought crul was an interesting newer tool!! But I agree, you gotta get in there and it’s easier with JS

Can you add a link to it?

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#23

Post by [deleted] »

xyzzy_plugh wrote: post #20
cxr wrote: post #11

Earlier quoted context omitted.

Your comment was posted 4 minutes ago. That means you still have enough time to edit your comment to change it so it contains real URLs that link to the project repos for the packages mentioned: https://github.com/PuerkitoBio/goquery > https://github.com/dop251/goja > (Please do not reply to this comment of mine—if you do, I won't be able to delete it once the previous post is fixed, because the existence of the repl…

Even if I saw this post in time, I wouldn't have edited it. They are all proper Go package names.

[deleted]

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#24

Post by sunshadow »

mikercampbell wrote: post #15
sunshadow wrote: post #13

These days, I'm not even using Go for scraping that much, as the webpage changes makes me crazy and JS code evaluation is a lifesaver, so I moved to Typescript+Playwright. (Crawlee framework is cool, while not strictly necessary). Its been 8+ years since i started scraping. I even wrote a popular Go web scraping framework previously: ( https://github.com/geziyor/geziyor ). My favorite stack as of 2023: TypeScript+Pla…

Have you seen Crul?? I love the JS flow, but I thought crul was an interesting newer tool!! But I agree, you gotta get in there and it’s easier with JS

Crul looks nice, though, you cannot imagine how many startups that I've seen failed doing a very similar thing as Crul. Wouldn't rely on it. The problem is complex: Humans generating messy pages

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#25

Post by [deleted] »

xyzzy_plugh wrote: post #20
cxr wrote: post #11

Earlier quoted context omitted.

Your comment was posted 4 minutes ago. That means you still have enough time to edit your comment to change it so it contains real URLs that link to the project repos for the packages mentioned: https://github.com/PuerkitoBio/goquery > https://github.com/dop251/goja > (Please do not reply to this comment of mine—if you do, I won't be able to delete it once the previous post is fixed, because the existence of the repl…

Even if I saw this post in time, I wouldn't have edited it. They are all proper Go package names.

[deleted]

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#26

Post by 8n4vidtmkvmk »

sunshadow wrote: post #21
hipadev23 wrote: post #19

Earlier quoted context omitted.

How does that help you mitigate when a site changes? If you’re fetching some value in a given under a long XPATH and they decide to change that path?

You don't use XPath&CSS selectors at all (Except if you dont have choice). You rely on more generic stuff, e.g, "the button that has 'Sign in' on it": await page.getByRole('button', { name: 'Sign in' }).click(); See playwright locators: https://playwright.dev/docs/locators

I started putting data-testid attributes in my web app for automated testing using playwright. Prevents me from breaking my own script but it sure would make me more scrapable if anyone cared. Well.. I guess I only do it on inputs, not the rendered page which is what scrapers care most about.

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#27

Post by sunshadow »

8n4vidtmkvmk wrote: post #26
sunshadow wrote: post #21

Earlier quoted context omitted.

You don't use XPath&CSS selectors at all (Except if you dont have choice). You rely on more generic stuff, e.g, "the button that has 'Sign in' on it": await page.getByRole('button', { name: 'Sign in' }).click(); See playwright locators: https://playwright.dev/docs/locators

I started putting data-testid attributes in my web app for automated testing using playwright. Prevents me from breaking my own script but it sure would make me more scrapable if anyone cared. Well.. I guess I only do it on inputs, not the rendered page which is what scrapers care most about.

Unless you start a war against scrapers, you don't need to worry about that as I'll always find a way to scrape your site as long as its valuable to 'me'. Even if it requires Real browser + OCR :)

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#28

Post by nurettin »

hipadev23 wrote: post #19
sunshadow wrote: post #13

These days, I'm not even using Go for scraping that much, as the webpage changes makes me crazy and JS code evaluation is a lifesaver, so I moved to Typescript+Playwright. (Crawlee framework is cool, while not strictly necessary). Its been 8+ years since i started scraping. I even wrote a popular Go web scraping framework previously: ( https://github.com/geziyor/geziyor ). My favorite stack as of 2023: TypeScript+Pla…

How does that help you mitigate when a site changes? If you’re fetching some value in a given under a long XPATH and they decide to change that path?

Don't know about the poster, but I try to find divs and buttons in a fuzzy way. Usually via element text. Sometimes it mitigates changes, sometimes it doesn't. It's a guessing game. Especially when they start using shadow elements or iframes in the page. If I'm looking for something specific like a price or dimensions, I can sometimes get away with it by collecting dollar amounts or X x Y x Z from the raw text.

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#29

Post by [deleted] »

sunshadow wrote: post #21
hipadev23 wrote: post #19

Earlier quoted context omitted.

How does that help you mitigate when a site changes? If you’re fetching some value in a given under a long XPATH and they decide to change that path?

You don't use XPath&CSS selectors at all (Except if you dont have choice). You rely on more generic stuff, e.g, "the button that has 'Sign in' on it": await page.getByRole('button', { name: 'Sign in' }).click(); See playwright locators: https://playwright.dev/docs/locators

[deleted]

Re: Show HN: Flyscrape – A standalone and scriptable web scraper in Go

#30

Post by latchkey »

sunshadow wrote: post #21
hipadev23 wrote: post #19

Earlier quoted context omitted.

How does that help you mitigate when a site changes? If you’re fetching some value in a given under a long XPATH and they decide to change that path?

You don't use XPath&CSS selectors at all (Except if you dont have choice). You rely on more generic stuff, e.g, "the button that has 'Sign in' on it": await page.getByRole('button', { name: 'Sign in' }).click(); See playwright locators: https://playwright.dev/docs/locators

This is where you just train an LLM so you can write:

'get button named "sign in" and click'

Then on the back end, it generates your example code.

Post reply on HN