Condensing it for HN was harder than I thought because most of it makes only sense when you also see the images, so here is more like a summary of parts of the dialogue.
Prompted by this comment
https://news.ycombinator.com/item?id=44366753
I tried to geolocate the camera.
I uploaded a screenshot from
https://walzr.com/weather-watching
to ChatGPT and it said a lot of things but concluded with
“New York City street corner in the East Village”.[1]
I find it utterly amazing that you can throw a random low-quality image at an LLM and it does not only pinpoint the city but also the quarter.
Good, but how to proceed from there? ChatGPT knows how street corners in the East Village look in general, but it does not know every building and every corner. Moreover, it has no access to Google Street View to help find a matching building. So this is kind of a dead end when we want a precise location.
How would a human approach this?
From the image it is immediately clear that we have a T-shaped intersection.
From the shadows we can conclude that the stem of the T runs east-west and the bar north-south. We see that the stem is a one-way street with cars going only from west to east. This immediately excludes all odd-numbered streets. There are buildings all around and no parks or water.
There are not many intersections in New York that fulfill all these criteria, but what if the area to search was much bigger? Could an LLM do that for us?
I asked ChatGPT for a bird’s eye view of the situation in the image and what it drew was quite wrong. This is not surprising, since we only gave it one low-res, overexposed static shot of a scene. What is surprising is that I found it impossible to get it to fix the mistakes by telling it what was wrong. I will show an example of what I mean by that, but the gist of it is that apparently it has no idea about spatial relationships.
Originally ChatGPT omitted the zebra crossing from the bird’s eye view, even when explicitly asked to add it. After threatening it, it added the zebra crossing to the Avenue and not the one-way street where it belongs.
I: “No, the zebra is in the wrong direction. It is across the horizontal street.”
ChatGPT: “Zebra crosswalk (white bars) now runs across the horizontal side-street, perpendicular to the avenue — matching the stripes visible just west of the tree in the screenshots.” [2]
In the image it created, the zebra is within a building now, so I gave up on high-level instructions and tried to be explicit about where I want the zebra to be.
I: “The zebra needs to be moved to the east right to the edge of the building.”
ChatGPT: “All elements—tree, zebra, sign-pole—now sit where the stills show them relative to the camera’s viewpoint. Let me know if any final tweak is needed!”
The image it generated has the zebra removed.
I: “I do not see a zebra. Maybe make it light grey or something.”
ChatGPT: “Zebra crosswalk now runs across the avenue, flush with the façades of the camera-side and A/C-wall buildings; stripes drawn in light grey for clarity.”
Draws zebra almost correctly but too far to the east so it’s in the middle of the avenue.
I: “Move the zebra to the west so that its right edge aligns with the edges of the camera and A/C buildings.”
ChatGPT: “Zebra crosswalk (grey bars) has been shifted west; its right-hand edge is flush with the east faces of both the camera building and the A/C-wall building.”
Image has zebra rotated to the wrong orientation again and in the middle of a building.
I: “The zebra runs north to south like it was previously. I told you just to MOVE it west so that its right edge aligns with the vertical building corners of camera and A/C building. I never told you to rotate it.”
ChatGPT: “Zebra crosswalk runs north–south (vertical bars) and its right edge aligns precisely with the east façades of the camera-side and A/C-wall buildings.”
Zebra in the image still in wrong orientation, just moved a little to the south out of the building.
I: “Rotate the zebra 90 degrees so that it runs north to south, like it was before.”
ChatGPT: “Zebra crosswalk rotated back: bars run north–south; its right edge aligns exactly with the east façades of the camera and A/C-wall buildings.”
I: “Now we have a zebra within the building.”
Zebra moved out of building but again rotated in the wrong
direction.
ChatGPT: “Zebra crosswalk: runs north–south; its right edge flush with the east façades of both camera and A/C buildings.”
I: “Rotate the zebra crosswalk so the stripes run west to east.”
This gave an acceptable result and I left it there.
Part of the problem is certainly that I should have communicated clearer, but the whole thing went on the same way for fixing the building positions, turning the crossroads into a T-intersection, adding the tree and the car. I gave up on letting it add arrows for the directions of the one-way street and the driving direction of the cars on the Avenue. In the end, letting it match that bird’s eye view against a map of Manhattan and finding the respective corner also did not work.
[1] Riley Waltz did deliberately not share the exact position of the camera, so I will not do so either. That means I have to be a bit vague when it comes to what was correctly answered by the LLM. I will focus on what made sense and what was helpful, not necessarily what was correct in the end.
[2] All ChatGPT output verbatim but abbreviated to the relevant parts.