Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Watching o3 guess a photo's location is surreal, dystopian and entertaining

simonwillison.net

Post reply on HN
Print view View original

21–30 of 453 posts

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#21

Post by hashemian »

To those argue that LLMs might cheat by using EXIF, I saw a post recently on twitter (https://x.com/tszzl/status/1915212958755676350) and out of curiosity, screen-captured the photo and passed it to O3. So no EXIF.

You can read the chat here: https://chatgpt.com/share/680a449f-d8dc-8001-88f4-60023323c7...

It took 4.5m to guess the location. The guess was accurate (checked using Google Street View).

What was amazing about it:

    1. The photo did not have ANY text

    2. It picked elements of the image and inferred based on those, like a fountain in a courtyard, or shape of the buildings.
All in all, it's just mind-blowing how this works!

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#22

Post by simonw »

AIPedant wrote: post #20
hughes wrote: post #7

> I’m confident it didn’t cheat and look at the EXIF data on the photograph, because if it had cheated it wouldn’t have guessed Cambria first. If I was cheating on a similar task, I might make it more plausible by suggesting a slightly incorrect location as my primary guess. Would be interesting to see if it performs as well on the same image with all EXIF data removed. It would be most interesting if it fails, since…

There have been a few cases where the LLM clearly did look at the EXIF, got the answer, then confabulated a bunch of GeoGusser logic to justify the answer. Sometimes that's presented as deception/misalignment but that's a category error: "find the answer" and "explain your reasoning" are two distinct tasks, and LLMs are not actually smart enough to coherently link them. They do one autocomplete for generating text th…

Do you have links to any of those examples?

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#23

Post by AIPedant »

segmondy wrote: post #18
suddenlybananas wrote: post #17

Earlier quoted context omitted.

Why didn't he do that then for this post?

Even better, edit it and place a false location.

This is a good test - the salient point is that it is fine if the LLM is confused, or even gets it wrong! But what I suspect would happen is that it would confabulate details which aren't in the photo to justify the incorrect EXIF answer. This is not fine.

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#24

Post by tantalor »

rafaelmn wrote: post #12

The fact that they give these models low res photos but don't provide them with built in tools for querying more details feels suboptimal. Executing python to crop an image is clever from model and a facepalm from the implementation side.

I don't follow. Are you suggesting full Blade Runner enhance mode?

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#25

Post by simonw »

I added a section just now with something I had missed: o3 DOES have a loose model of your location fed into it, which I believe is intended to support the new search feature (so it can run local searches).

The thinking summary it showed me did not reference that information, but it's still very possible that it used that in its deliberations.

I ran two extra example queries for photographs I've taken thousands of miles away (in Buenos Aires and Madagascar) - EXIF stripped - and it did a convincing job with both of those as well: https://simonwillison.net/2025/Apr/26/o3-photo-locations/#up...

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#27

Post by sorcerer-mar »

AIPedant wrote: post #20
hughes wrote: post #7

> I’m confident it didn’t cheat and look at the EXIF data on the photograph, because if it had cheated it wouldn’t have guessed Cambria first. If I was cheating on a similar task, I might make it more plausible by suggesting a slightly incorrect location as my primary guess. Would be interesting to see if it performs as well on the same image with all EXIF data removed. It would be most interesting if it fails, since…

There have been a few cases where the LLM clearly did look at the EXIF, got the answer, then confabulated a bunch of GeoGusser logic to justify the answer. Sometimes that's presented as deception/misalignment but that's a category error: "find the answer" and "explain your reasoning" are two distinct tasks, and LLMs are not actually smart enough to coherently link them. They do one autocomplete for generating text th…

> Sometimes that's presented as deception/misalignment but that's a category error: "find the answer" and "explain your reasoning" are two distinct tasks

Right but if your answer to "explain your reasoning" is not a true representation of your reasoning, then you are being deceptive. If it doesn't "know" its reasoning, then the honest answer is that it doesn't know.

(To head off any meta-commentary on humans' inability to explain their own reasoning, they would at least be able to honestly describe whether they used EXIF or actual semantic knowledge of a photography)

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#28

Post by ksec »

I wonder What happened if you put fake EXIF information and asking it to do the same. ( We are deliberately misleading the LLM )

I am also wondering if we have any major breakthrough (comparatively speaking) coming out of LLM. Or non-LLM AI R&D.

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#29

Post by oortoo »

tantalor wrote: post #24
rafaelmn wrote: post #12

The fact that they give these models low res photos but don't provide them with built in tools for querying more details feels suboptimal. Executing python to crop an image is clever from model and a facepalm from the implementation side.

I don't follow. Are you suggesting full Blade Runner enhance mode?

No, the LLM can only "see" a lower res version of the uploaded photo. It has to crop to process finer details, and they are suggesting its silly this isn't a built in feature and instead relies on python to do this.
Post reply on HN