51–57 of 57 posts
Earlier quoted context omitted.
You don't use XPath&CSS selectors at all (Except if you dont have choice). You rely on more generic stuff, e.g, "the button that has 'Sign in' on it": await page.getByRole('button', { name: 'Sign in' }).click(); See playwright locators: https://playwright.dev/docs/locators
This is where you just train an LLM so you can write: 'get button named "sign in" and click' Then on the back end, it generates your example code.
Earlier quoted context omitted.
What are some examples of "most real world sites". What are some examples of sites that are not "most real world sites". Is HN a "real world site". What percentage of sites submitted to HN are "most real world sites". (IME, it's a minority fraction.) Why not just delete or change the user-agent line in scrape.go before compiling. (Personal experience: I have been successfully retrieving information from the www for d…
I was scraping a Wordpress site a few months ago using Go and I had to spoof my user agent to get results. So it definitely happens
IME, Wordpress sites do not require a user agent header. Contrast, for example, with Squarespace sites which do require a user agent header.
IME, if send a user agent header with a particular value, then some sites will block, depending on the value. Whereas if _do not send_ a user agent header, then almost all sites will accept.
The so-called "developers" who publish code and commentary about "scraping" almost always include a user agent header. Usually they use fake values. They try to guess the "correct" values to send.
Im not referring to sending fake values. Im referring to not sending the header at all. No "spoofing" is involved. This works for me, for decades, across thousands of websites.
Earlier quoted context omitted.
Your comment was posted 4 minutes ago. That means you still have enough time to edit your comment to change it so it contains real URLs that link to the project repos for the packages mentioned: https://github.com/PuerkitoBio/goquery > https://github.com/dop251/goja > (Please do not reply to this comment of mine—if you do, I won't be able to delete it once the previous post is fixed, because the existence of the repl…
Even if I saw this post in time, I wouldn't have edited it. They are all proper Go package names.
Earlier quoted context omitted.
Your comment was posted 4 minutes ago. That means you still have enough time to edit your comment to change it so it contains real URLs that link to the project repos for the packages mentioned: https://github.com/PuerkitoBio/goquery > https://github.com/dop251/goja > (Please do not reply to this comment of mine—if you do, I won't be able to delete it once the previous post is fixed, because the existence of the repl…
Even if I saw this post in time, I wouldn't have edited it. They are all proper Go package names.
1. https://news.ycombinator.com/item?id=38232101>
Earlier quoted context omitted.
Have you seen Crul?? I love the JS flow, but I thought crul was an interesting newer tool!! But I agree, you gotta get in there and it’s easier with JS
Crul looks nice, though, you cannot imagine how many startups that I've seen failed doing a very similar thing as Crul. Wouldn't rely on it. The problem is complex: Humans generating messy pages
Earlier quoted context omitted.
What are some examples of "most real world sites". What are some examples of sites that are not "most real world sites". Is HN a "real world site". What percentage of sites submitted to HN are "most real world sites". (IME, it's a minority fraction.) Why not just delete or change the user-agent line in scrape.go before compiling. (Personal experience: I have been successfully retrieving information from the www for d…
I've not looked at the source code, but if GP is correct, then absent JS rendering means there's little added value for me (a dude who scrapes a lot). Real world example, I was looking at scraping unjobs.org for a friend the other day. The need for JS rendering turned the job from 15 minutes of requests and beautifulsoup into a full-blown session with selenium, geckodriver etc. I'm not saying the linked framework isn…
Below is a simple demonstration using only common UNIX utilties and minimising the number of TCP conections. No browser. No Javascript. No Selenium. No Geckodriver. No proxies.
Step 1 requires 40 TCP connections and completes in under a minute. Step 2 requires one TCP connection and completes in 10min. (NB. The connection minimisation used here, what RFCs used to call "web etiquette", is not possible using a popular headless graphical browser.) 1.htm is 1.5M, 2.htm is 40M
# step 1
x="some user-agent string"
# e.g., https://raw.githubusercontent.com/51Degrees/Device-Detection/master/data/20000%20User%20Agents.csv
n=1;while true;do
test $n -le 40||break # unjobs.org only shows listings 1-1000
case $n in 9|17|25|33)sleep 10;esac # need a delay after every 8 requests
echo "GET /new/$n HTTP/1.0@Host: unjobs.org@User-Agent: $x@" \
|tr @@ '\r\n' \
|openssl s_client -connect unjobs.org:443 -ign_eof -servername unjobs.org
n=$((n+1));
done > 1.htm
# step 2
n=1000;
grep -Eo -m$n /vacancies/1[0-9]\{12\} 1.htm \
|tail -$n \
|sed "\$!s>.*>GET & HTTP/1.1@Host: $x@Connection: keep-alive@>;
\$s>.*>GET & HTTP/1.1@Host: $x@Connection: close@>" \
|tr @@ '\r\n' \
|openssl s_client -connect unjobs.org:443 -ign_eof > 2.htm