Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Codex scraped the ICM website and discovered 2026 Fields Medal winner list

phemex.com

Post reply on HN
Print view View original

61–70 of 119 posts

Re: Codex scraped the ICM website and discovered 2026 Fields Medal winner list

#61

Post by chasd00 »

foobarbecue wrote: post #11
resonious wrote: post #10

Earlier quoted context omitted.

I've also seen Google indexing pages with random values in the path that don't get linked to statically (server asks for the URL then redirects to it immediately). I'm pretty sure they index straight out of the Chrome address bar.

Holy crap I hope that's not true. I've also had unguessable pages indexed, though, and don't have an explanation.

This is why Chrome begs you to login constantly and will do it automatically when you login to Gmail through Chrome. Everything you do in the browser (bookmarks, settings, address bar) is data about you sent to Adsense. No need for cookies when you control the browser and know who is using it.

Edit: also private browsing isn’t exactly private when you’re logged in to the browser.

Re: Codex scraped the ICM website and discovered 2026 Fields Medal winner list

#62

Post by [deleted] »

EGreg wrote: post #41
inigyou wrote: post #37

Earlier quoted context omitted.

There's no reason to think it isn't true. It matches every pattern of behavior observed from every tech company.

Why don’t they also read your gmail and get your bank passwords? And maybe have access to EVERY site actually, with “forgot password” type stuff in addition to providing oauth tokens…

[deleted]

Re: Codex scraped the ICM website and discovered 2026 Fields Medal winner list

#63

Post by whalesalad »

This is like when a news site throws up a paywall and hides half the article. Open inspector. Select the body, delete the overflow/scroll capture styles, delete the masks... and boom there is the entire article. Only some sites are smart enough to actually truncate the content server-side.

Re: Codex scraped the ICM website and discovered 2026 Fields Medal winner list

#64

Post by dataviz1000 »

edoceo wrote: post #8

I've been working on a site. It's new, domain is only a few weeks old. It's got SSL, so all the bots know it exists. It's never had any sub-pages exposed, just the placeholder lander, no links. Somehow in Google search one of the unguessable pages is indexed. We have used Claude and Gemini to assist with some design aspects. I'm thinking some aggressive data ingestion/indexing is happening by all the bots in the ques…

The tin hat guess. Did you include Google analytics embedded in the pages? Do you navigate to pages and Google analytics sends that data home? 10 years ago I discovered that Google analytics would send the equivalent amount as organic users; meaning if we sent an email newsletter with links to articles, Google would send almost 1:1 ratio the same number of people from search results. They are tracking everything and using it for more than just reporting.

Do you use a CMS or other tools that auto generate sitemap.xml? Perhaps you unknowingly told Google about those sub-pages.

Re: Codex scraped the ICM website and discovered 2026 Fields Medal winner list

#66

Post by stavros »

edoceo wrote: post #8

I've been working on a site. It's new, domain is only a few weeks old. It's got SSL, so all the bots know it exists. It's never had any sub-pages exposed, just the placeholder lander, no links. Somehow in Google search one of the unguessable pages is indexed. We have used Claude and Gemini to assist with some design aspects. I'm thinking some aggressive data ingestion/indexing is happening by all the bots in the ques…

It's indexed some unlisted draft blog posts of mine that were never touched by AI or published anywhere. I use a static site generator so there's no earthly way they ever found the pages by scraping, at most I visited the pages once or twice from my browser.

Re: Codex scraped the ICM website and discovered 2026 Fields Medal winner list

#67

Post by [deleted] »

pixl97 wrote: post #58
EGreg wrote: post #42

Earlier quoted context omitted.

Why don’t they also capture information you enter into forms on Chrome? They control the entire browser surface, technically they can know everything, even TLS and E2E encrypted data, that they silently phone home… If you think this is silly, consider that Microsoft Recall had been observing everything on people’s entire SCREENS and phoning home much of it. That is how a guy was caught recently: https://x.com/t3chfal…

>Why don’t they also capture information you enter into forms on Chrome? For some reason people are downvoting you, but yea, one day we'll likely see a lawsuit where they do exactly that.

[deleted]

Re: Codex scraped the ICM website and discovered 2026 Fields Medal winner list

#68

Post by animex »

edoceo wrote: post #8

I've been working on a site. It's new, domain is only a few weeks old. It's got SSL, so all the bots know it exists. It's never had any sub-pages exposed, just the placeholder lander, no links. Somehow in Google search one of the unguessable pages is indexed. We have used Claude and Gemini to assist with some design aspects. I'm thinking some aggressive data ingestion/indexing is happening by all the bots in the ques…

I've always wondered if Chrome leaks these URLs too.

Re: Codex scraped the ICM website and discovered 2026 Fields Medal winner list

#69

Post by Lomlioto »

fragmede wrote: post #24
pohuing wrote: post #22

Earlier quoted context omitted.

There's a couple avenues besides just stealing what's in your URL bar. If you don't use wildcard certs all of your subdomains can be scraped from the certificate transparency logs. Additionally, any domain+cert using HSTS with preload enabled end up in a big list at Google to speed up the initial connection from browser to site.

CT logs just explain how they found the domain. T doesn't explain how they could have found unlinked content on the domain itself. If I put up secret-example.com/asdf-1234567.html, how does that page get found if there are no public links to it?

Don't underestimate people not knowing were they share stuff by accident.

Creating Sitemaps, sharing it somewere public, putting the url in some 3th party service, server logs, some indirect path in javascript.

But if you never mention that url, it will not be found if not leaked by your server.

Re: Codex scraped the ICM website and discovered 2026 Fields Medal winner list

#70

Post by st_goliath »

brookst wrote: post #28
st_goliath wrote: post #18

Earlier quoted context omitted.

Well, the angle is kind of important here. The company gets their name in the news, they have a reasonable explanation why they were scraping around, and we end up with a story about innovative tech company whiz-kids who made a funny discovery, while it was the webdevs on the other side that goofed up. Imagine a private individual just scraped the website (or simply clicked 'view source') for no reason in particular…

The fact that an egregious case happened once, decades ago, is probably not sufficient grounding to act like every bit of equally trivial “hacking” always results in massively disproportionate law enforcement response. Sucks it happened. But we all know that is not the typical scenario.

> But we all know that is not the typical scenario.

Back in the day, you could read a stories on Slashdot practically every other week that usually went something like this: Company/institution does something stupid, somebody finds out, tries to be a good citizen and tells them. The organization then throws a tamper tantrum in the media, fires the legal department on all cylinders, screaming "hacker!" and throwing the book at them. The most egregious cases usually happened in the US, the CFAA happens to be a particularly strong book to throw.

People eventually got the hint and either talked to the press instead, or organizations like the CCC (at least in this part of the world) and let them deal with the organization and not talk to them directly.

At least in my perception/memory, it started improving over the 2010s, but stories like this are now starting to pop up again in recent years. I guess we have a new crop of computer enthusiasts who need to learn the same lessons again.

Of the top of my head, the CTF group in Malta comes to mind who gave a talk at (last years?) CCCongress. A badly worded E-mail asking about a bug bounty resulted in several arrests, house searches and ultimately a presidential pardon (https://timesofmalta.com/article/pardon-issued-students-lect...).

Post reply on HN