Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google begins requiring JavaScript for Google Search

techcrunch.com

Post reply on HN
Print view View original

61–70 of 158 posts

Re: Google begins requiring JavaScript for Google Search

#61

Post by lozenge »

marginalia_nu wrote: post #55
supriyo-biswas wrote: post #53

Earlier quoted context omitted.

A major player in this space is apparently looking for people experienced in scraping without using browser automation. My guess is that not running a browser results in using far fewer resources, thus reducing their costs heavily. Running a headless browser also means that any differences in the headless environment vs. a "headed" one can be discovered, as well as any of your Javascript executing within the page, wh…

My experience is that headless browsers use about 100x more RAM, and at least 10x more bandwidth and 10x more processing power, and page loads take about 10x as long time to finish (vs curl). Though these numbers may be a bit low, there are instances you need to add another zero to one or more of them. There's also considerably more jank with headless browsers, since you typically want to re-use instances to avoid in…

Is it possible to pause a VM just after the browser has started up? Then map it as copy-on-write memory and spin up many VMs from that "image".

Re: Google begins requiring JavaScript for Google Search

#62

Post by fweimer »

supriyo-biswas wrote: post #53
palmfacehn wrote: post #42

Earlier quoted context omitted.

My impression is that there's less effort for them to go directly to headless browsers. There are several foot guns in using a raw HTML parsing lib and dispatching HTTP requests. People don't care about resource usage, spammers even less and many of them lack the skills.

A major player in this space is apparently looking for people experienced in scraping without using browser automation. My guess is that not running a browser results in using far fewer resources, thus reducing their costs heavily. Running a headless browser also means that any differences in the headless environment vs. a "headed" one can be discovered, as well as any of your Javascript executing within the page, wh…

The change rate for Chromium is also so high that it's hard to spot the addition of code targeting whatever you are doing on the client side.

Re: Google begins requiring JavaScript for Google Search

#63

Post by palmfacehn »

marginalia_nu wrote: post #55
supriyo-biswas wrote: post #53

Earlier quoted context omitted.

A major player in this space is apparently looking for people experienced in scraping without using browser automation. My guess is that not running a browser results in using far fewer resources, thus reducing their costs heavily. Running a headless browser also means that any differences in the headless environment vs. a "headed" one can be discovered, as well as any of your Javascript executing within the page, wh…

My experience is that headless browsers use about 100x more RAM, and at least 10x more bandwidth and 10x more processing power, and page loads take about 10x as long time to finish (vs curl). Though these numbers may be a bit low, there are instances you need to add another zero to one or more of them. There's also considerably more jank with headless browsers, since you typically want to re-use instances to avoid in…

On the other hand you need to be able to do basics like match the headers, sometimes request irrelevant resources, handle malformed documents, catch changing form parameters, and other gotchas. Many would just copy the request from the browser console.

Re: Google begins requiring JavaScript for Google Search

#65

Post by gnfargbl »

marginalia_nu wrote: post #46
palmfacehn wrote: post #42

Earlier quoted context omitted.

My impression is that there's less effort for them to go directly to headless browsers. There are several foot guns in using a raw HTML parsing lib and dispatching HTTP requests. People don't care about resource usage, spammers even less and many of them lack the skills.

Most black hat spammers use botnets, especially against bigger targets which have enough traffic to build statistics to fingerprint clients and map out bad ASNs and so on, and most botnets are low powered. You're not running chrome on a smart fridge or an enterprise router.

True, but the bad actor's code doesn't typically run directly on the infected device. Typically the infected router or camera is just acting as a proxy.

Re: Google begins requiring JavaScript for Google Search

#66

Post by dotancohen »

can16358p wrote: post #31

Honestly I wouldn't be surprised that if Google requires some Proof-of-work done on browser's host's CPU/GPU to validate search results and make it infeasible for bots therefore.

That brings up an interesting conundrum. If PoW were implemented, could known-valid (i.e. goodstanding for over a decade) accounts be switched over to PoS instead? Or paying accounts?

PoW could be written into infrequent pages such as the registration page and reset password page. It could run while the user fills in the form. I might implement this on some sites that get attacked.

Re: Google begins requiring JavaScript for Google Search

#67

Post by _factor »

supriyo-biswas wrote: post #50
marginalia_nu wrote: post #25

To be fair, if my search engine is anything to go on, about 0.5-1% of the requests I get are from human sources. The rest are from bots, and not like people who haven't found I have an API, but bots that are attempting to poison Google or Bing's query suggestions (even though I'm not backed by either). From what I've heard from other people running search engines, it looks the same everywhere. I don't know what Googl…

I run a semi-popular website hosting user-generated content, although it's not a search engine; the attacks on it have surprised me, and I've eventually had to put in the same kinds of restrictions on it. I was initially very hesitant to restrict any kind of traffic, relying on ratelimiting IPs on critical endpoints that needed low friction, and captchas on the higher friction with higher intents, such as signup and…

The problem is accountability. Imagine starting a trade show business in the physical world as an example.

One day you start getting a bunch of people come in to mess with the place. You can identify them and their organization, then promptly remove them. If they continue, there are legal ramifications.

On the web, these people can be robots that look just like real people until you spend a while studying their behavior. Worse if they’re real people being paid for sabotage.

In the real world, you arrest them and find the source. Online they can remain anonymous and protected. What recourse do we have beyond splitting the web into a “verified ID” web, and a pseudonymous analog? We can’t keep treating potential computer engagement the same as human forever. As AI agents inevitably get cheaper and harder to detect, what choice will we have?

Re: Google begins requiring JavaScript for Google Search

#68

Post by ptx »

Joeri wrote: post #41
gazchop wrote: post #2

How else are you going to load a hideously incorrect AI summary block without your initial page latency being through the roof?

You could probably get it working with declarative shadow dom, streaming in the AI generated content at the end of the html document and slotting it into place. There are no doubt a lot of gotchas but at first glance it seems feasible. Here’s a demo I found of something like that: https://github.com/dgp1130/out-of-order-streaming

The example repo is a little confusing to me, since it seems to use client-side JS to demonstrate that it doesn't need client-side JS: "It bootstraps a service worker and [...] No client-side JavaScript!"

But I guess the point is that the code in the service worker could have been on the server instead?

The trick seems to be using a template element with a slot and then slotting in the streamed content at the end. But you could probably also do it using just CSS to reposition the content from the bottom to the top, similarly to how many websites handle navigation menus, assuming that the client supports CSS.

Post reply on HN