Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Watching o3 guess a photo's location is surreal, dystopian and entertaining

simonwillison.net

Post reply on HN
Print view View original

31–40 of 453 posts

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#31

Post by _the_inflator »

cenamus wrote: post #8
speedgoose wrote: post #4

Earlier quoted context omitted.

If you are a professional geoguessr player, you play so many games that it’s not unrealistic to get very good guesses once in a while. But I wouldn’t be surprised if some form of cheating is happening.

Not just cheating, but just lots of meta, from the blurred number plates color/format, to the stitching of the image at the bottom, mounting location of the camera on the google car. There are list which variant of car was used in which countries and so on. Which is still impressive but not quite the same as just guessing from the image

Nice examples.

Every attribute is of importance. A PhD put you in a 1-3% pool. What data do you have, what is needed to hit a certain goal. Data Science can be considered wizardry when exercised on seemingly innocent and mundane things like a photo.

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#32

Post by bslanej »

blueprint wrote: post #11
OutOfHere wrote: post #10

Why is it dystopian? It's a nice utility. The tool is just intelligence. Intelligence itself is not dystopian or utopian. It's what you use it for that makes it so.

please accidentally post an identifying photo of your neighborhood...

How do you “accidentally post a photo”?

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#33

Post by thegeomaster »

hashemian wrote: post #21

To those argue that LLMs might cheat by using EXIF, I saw a post recently on twitter ( https://x.com/tszzl/status/1915212958755676350 ) and out of curiosity, screen-captured the photo and passed it to O3. So no EXIF. You can read the chat here: https://chatgpt.com/share/680a449f-d8dc-8001-88f4-60023323c7... It took 4.5m to guess the location. The guess was accurate (checked using Google Street View). What was amazing…

See my other comment: https://news.ycombinator.com/item?id=43804041

4o can do it almost as well in a few seconds and probably 10-50x fewer tokens: https://chatgpt.com/share/680ceeff-011c-8002-ab31-d6b4cb622e...

o3 burns through what I assume is single-digit dollars just to do some performative tool use to justify and slightly narrow down its initial intuition from the base model.

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#34

Post by AstroBen »

simonw wrote: post #25

I added a section just now with something I had missed: o3 DOES have a loose model of your location fed into it, which I believe is intended to support the new search feature (so it can run local searches). The thinking summary it showed me did not reference that information, but it's still very possible that it used that in its deliberations. I ran two extra example queries for photographs I've taken thousands of mi…

I can't see the new images uploaded (it just says "Uploaded an image" in ChatGPT for me) but it seems it's identifying well known locations there? That certainly takes away from your message - that it's honing in on smaller details

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#35

Post by AIPedant »

simonw wrote: post #22
AIPedant wrote: post #20

Earlier quoted context omitted.

There have been a few cases where the LLM clearly did look at the EXIF, got the answer, then confabulated a bunch of GeoGusser logic to justify the answer. Sometimes that's presented as deception/misalignment but that's a category error: "find the answer" and "explain your reasoning" are two distinct tasks, and LLMs are not actually smart enough to coherently link them. They do one autocomplete for generating text th…

Do you have links to any of those examples?

I have one link that illustrates what I mean: https://chatgpt.com/share/6802e229-c6a0-800f-898a-44171a0c7d... The line about "the latitudinal light angle that matches mid‑February at ~47 ° N." seems like pure BS to me, and in the reasoning trace it openly reads the EXIF.

A more clear example I don't have a link for, it was on Twitter somewhere: someone tested a photo from Suriname and o3 said one of the clues was left-handed traffic. But there was no traffic in the photo. "Left-handed traffic" is a very valuable GeoGuesser clue, and it seemed to me that once o3 read the Surinamese EXIF, it confabulated the traffic detail.

It's pure stochastic parroting: given you are playing GeoGuesser honestly, and given the answer is Suriname, the conditional probability that you mention left-handed traffic is very high. So o3 autocompleted that for itself while "explaining" its "reasoning."

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#36

Post by parsimo2010 »

I’m sure there are areas where the location guessing can be scary accurate, like the article managed to guess the exact town as its backup guess.

But seeing the chain of thought, I’m confident there are many areas that it will be far less precise. Show it a picture of a trailer park somewhere in Kansas (exclude any signs with the trailer park name and location) and I’ll bet the model only manages to guess the state correctly.

Before even running this experiment, here’s your lesson learned: when the robot apocalypse happens, California is the first to be doomed. That’s the place the AI is most familiar with. Run any location experiments outside of California if you want to get an idea of how good your software performs outside of the tech bubble.

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#37

Post by simonw »

bslanej wrote: post #32
blueprint wrote: post #11

Earlier quoted context omitted.

please accidentally post an identifying photo of your neighborhood...

How do you “accidentally post a photo”?

A selfie with a snippet of building in the background might give away your location even if you think there's no way it could be locatable.

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#38

Post by simonw »

AstroBen wrote: post #34
simonw wrote: post #25

I added a section just now with something I had missed: o3 DOES have a loose model of your location fed into it, which I believe is intended to support the new search feature (so it can run local searches). The thinking summary it showed me did not reference that information, but it's still very possible that it used that in its deliberations. I ran two extra example queries for photographs I've taken thousands of mi…

I can't see the new images uploaded (it just says "Uploaded an image" in ChatGPT for me) but it seems it's identifying well known locations there? That certainly takes away from your message - that it's honing in on smaller details

You should be able to see slightly cropped versions of those images if you scroll through the "thinking" text a bit.

My key message here is meant to be "try it out and see for yourself".

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#39

Post by thegeomaster »

For all of the images I've tried, the base model (e.g. 4o) already has a ~95% accurate idea of where the photo is, and then o3 does so much tool use only to confirm its intuition from the base model and slightly narrow down. For OP's initial image, 4o in fact provides a more accurate initial guess of Carmel-by-the-Sea (d=~100mi The clue is in the CoT - you can briefly see the almost correct location as the very first reasoning step. The model then apparently seems to ignore it and try many other locations, a ton of tool use, etc, always coming back to the initial guess.

For pictures where the base model has no clue, I haven't seen o3 do anything smart, it just spins in circles.

I believe the model has been RL-ed to death in a way that incentivizes correct answers no matter the number of tools used.

[0]: https://chatgpt.com/c/680d011a-9470-8002-97a0-a0d2b067eacf

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#40

Post by simonw »

AIPedant wrote: post #35
simonw wrote: post #22

Earlier quoted context omitted.

Do you have links to any of those examples?

I have one link that illustrates what I mean: https://chatgpt.com/share/6802e229-c6a0-800f-898a-44171a0c7d... The line about "the latitudinal light angle that matches mid‑February at ~47 ° N." seems like pure BS to me, and in the reasoning trace it openly reads the EXIF. A more clear example I don't have a link for, it was on Twitter somewhere: someone tested a photo from Suriname and o3 said one of the clues was lef…

Yes! Great example, it's clearly reading EXIF in there. Mind if I link to that from my post?
Post reply on HN