Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Watching o3 guess a photo's location is surreal, dystopian and entertaining

simonwillison.net

Post reply on HN
Print view View original

271–280 of 453 posts

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#271

Post by SamPatt »

charcircuit wrote: post #251
cco wrote: post #187

Earlier quoted context omitted.

Meh, meta is so boring and uninteresting to me personally. Knowing you're in Kenya because of the snorkel, that's just simple memorization. Pick up on geography, architecture, language, sun and street position; that's what I love. It's clearly necessary to compete at the high level though.

How is stuff like geography, architecture, or language not memorization either?

It's a valid question.

My guess is the actual objection is the artificial feeling of the Google specific information. It cannot possibly be useful in any other context to know what the Street View car in Bermuda looked like when they did their coverage.

Whereas knowing about vegetation or architecture feels more generally useful. I think it's a valid point, but you're right that it is all down to memorization at some point.

Though some memorization is "vibes" where you don't specifically know how you know, but you just do. That only comes with repetition. I guess it feels more earned that way?

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#272

Post by matthewdgreen »

SamPatt wrote: post #63

I play competitive Geoguessr at a fairly high level, and I wanted to test this out to see how it compares. It's astonishingly good. It will use information it knows about you to arrive at the answer - it gave me the exact trailhead of a photo I took locally, and when I asked it how, it mentioned that it knows I live nearby. However, I've given it vacation photos from ages ago, and not only in tourist destinations eit…

I was absolutely gobsmacked by the three minute chain of reasoning this thing did, and how it absolutely nailed the location of the photo based on plants, the color of a fence, comparison with nearby photos, and oh yeah, also the EXIF data containing the exact lat/long coordinates that I accidentally left in the file. https://bsky.app/profile/matthewdgreen.bsky.social/post/3lnq...

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#273

Post by brk »

brundolf wrote: post #165
SamPatt wrote: post #63

I play competitive Geoguessr at a fairly high level, and I wanted to test this out to see how it compares. It's astonishingly good. It will use information it knows about you to arrive at the answer - it gave me the exact trailhead of a photo I took locally, and when I asked it how, it mentioned that it knows I live nearby. However, I've given it vacation photos from ages ago, and not only in tourist destinations eit…

I find this type of problem is what current AI is best at: where the actual logic isn't very hard, but it requires pulling together and assimilating a huge amount of fuzzy, known information from various sources They are, after all, information-digesters

FWIW, I do a lot of talks about AI in the physical security domain and this is how I often describe AI, at least in terms of what is available today. Compared to humans, AI is not very smart, but it is tireless and able to recall data with essentially perfect accuracy.

It is easy to mistake the speed, accuracy, and scope of training data for "intelligence", but it's really just more like a tireless 5th grader.

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#274

Post by gf000 »

imatworkyo wrote: post #265
fire_lake wrote: post #177

Earlier quoted context omitted.

Which also fits with how it performs at software engineering (in my experience). Great at boilerplate code, tests, simple tutorials, common puzzles but bad at novel and complex things.

how often are we truly writing actual novel programs that are complex in a way AI does not excel at? There are many types of complex, and many times complex for a human coder, are trivial for AI and its skillset.

Depends on the field of development you do.

CRUD backend app for a business in a common sector? It's mostly just connecting stuff together (though I would argue that an experienced dev with a good stack takes less time to write it as is than painstakingly explaining it to an LLM in an inexact human language).

Some R&D stuff, or even debugging any kind of code? It's almost useless, as it would require deep reasoning, where these models absolutely break down.

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#275

Post by westmeal »

jdiff wrote: post #259
fire_lake wrote: post #177

Earlier quoted context omitted.

Which also fits with how it performs at software engineering (in my experience). Great at boilerplate code, tests, simple tutorials, common puzzles but bad at novel and complex things.

Definitely matches my experience as well. I've been working away on a very quirky, non-idiomatic 3D codebase, and LLMs are a mixed bag there. Y is down, there's no perspective distortion or Z buffer, there are no meshes, it's a weird place. It's still useful to save me from writing 12 variations of x1 = sin(r2) - cos(r1) while implementing some geometric formula, but absolutely awful at understanding how those fit in…

I gotta ask what are you actually doing because it sure sounds funky

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#276

Post by SamPatt »

matthewdgreen wrote: post #272
SamPatt wrote: post #63

I play competitive Geoguessr at a fairly high level, and I wanted to test this out to see how it compares. It's astonishingly good. It will use information it knows about you to arrive at the answer - it gave me the exact trailhead of a photo I took locally, and when I asked it how, it mentioned that it knows I live nearby. However, I've given it vacation photos from ages ago, and not only in tourist destinations eit…

I was absolutely gobsmacked by the three minute chain of reasoning this thing did, and how it absolutely nailed the location of the photo based on plants, the color of a fence, comparison with nearby photos, and oh yeah, also the EXIF data containing the exact lat/long coordinates that I accidentally left in the file. https://bsky.app/profile/matthewdgreen.bsky.social/post/3lnq...

Lol it's very easy to give the models what they need to cheat.

For my test I used screenshots to ensure no metadata.

I mentioned this in another comment but I was a part of an AI safety fellowship last year where we created a benchmark for LLMs ability to geolocate. The models were doing unbelievably well, even the bad open source ones, until we realized our image pipeline was including location data in the filename!

They're already way better than even last year.

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#277

Post by gf000 »

olex wrote: post #131
joenot443 wrote: post #122

Earlier quoted context omitted.

Super cool, man. Watching pro Geoguessr is my latest break-time activity, these geo-gods never cease to impress me. One thing I'm curious about - in high level play, how much of the meta involves knowing characteristics about the photography/equipment/etc. that Google used when they shot it? Frequently I'll watch rainbolt immediately know an African country from nothing but the road, is there something I'm missing?

In the stream commentary for some of competitive Geoguessr I've watched, they definitely often mention the color and shape of the car (visible edges, shadow, reflections), so I assume pro players know which cars were used where very well.

That sounds exactly like shortcut learning.

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#278

Post by simonw »

brk wrote: post #273
brundolf wrote: post #165

Earlier quoted context omitted.

I find this type of problem is what current AI is best at: where the actual logic isn't very hard, but it requires pulling together and assimilating a huge amount of fuzzy, known information from various sources They are, after all, information-digesters

FWIW, I do a lot of talks about AI in the physical security domain and this is how I often describe AI, at least in terms of what is available today. Compared to humans, AI is not very smart, but it is tireless and able to recall data with essentially perfect accuracy. It is easy to mistake the speed, accuracy, and scope of training data for "intelligence", but it's really just more like a tireless 5th grader.

Something I have found quite amusing about LLMs is that they are computers that don't have perfect recall - unlike every other computer for the past 60+ years.

That is finally starting to change now that they have reliable(ish) search tools and are getting better at using them.

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#279

Post by dalmo3 »

SamPatt wrote: post #271
charcircuit wrote: post #251

Earlier quoted context omitted.

How is stuff like geography, architecture, or language not memorization either?

It's a valid question. My guess is the actual objection is the artificial feeling of the Google specific information. It cannot possibly be useful in any other context to know what the Street View car in Bermuda looked like when they did their coverage. Whereas knowing about vegetation or architecture feels more generally useful. I think it's a valid point, but you're right that it is all down to memorization at some…

It's not only about usefulness. People play gg recreationally to fantasize about being in those places, so of course real world knowledge is where the fun is. Camera meta is a turn off.

Re: Watching o3 guess a photo's location is surreal, dystopian and entertaining

#280

Post by simonw »

gf000 wrote: post #274
imatworkyo wrote: post #265

Earlier quoted context omitted.

how often are we truly writing actual novel programs that are complex in a way AI does not excel at? There are many types of complex, and many times complex for a human coder, are trivial for AI and its skillset.

Depends on the field of development you do. CRUD backend app for a business in a common sector? It's mostly just connecting stuff together (though I would argue that an experienced dev with a good stack takes less time to write it as is than painstakingly explaining it to an LLM in an inexact human language). Some R&D stuff, or even debugging any kind of code? It's almost useless, as it would require deep reasoning,…

Have you tried debugging using the new "reasoning" models yet?

I have been extremely impressed with o1, o3, o4-mini and Gemini 2.5 as debugging aids. The combination of long context input and their chain-of-thought means they can frequently help me figure out bugs that span several different layers of code.

I wrote about an early experiment with that here: https://simonwillison.net/2024/Sep/25/o1-preview-llm/

Here's a Gemini 2.5 Pro transcript from this afternoon where I'm trying to figure out a very tricky bug: https://gist.github.com/simonw/4e208ab9edb5e6a814d3d23d7570d...

Post reply on HN