Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Antigravity 2.0 Tops the OpenSCAD Architectural 3D LLM Benchmark

modelrift.com

Post reply on HN
Print view View original

121–130 of 171 posts

Re: Antigravity 2.0 Tops the OpenSCAD Architectural 3D LLM Benchmark

#121

Post by tjoff »

I've had such a bad time trying to do this myself. You might get a half-way decent draft on the first try and then you start to "debug" this and after a very frustrating session you realize that the model can't properly "see" the results. That is, you just can't iterate on it, at all.

I'm guessing that most harnesses/tools will resize an image before processing and in doing so will loose enough detail to make it much harder to reason about - especially wireframe images.

I'm sure I'm holding it wrong, but this test didn't really test this. It was just a one off. That breaks down pretty quickly and especially if you don't have reference pictures of what you are trying to create.

Re: Antigravity 2.0 Tops the OpenSCAD Architectural 3D LLM Benchmark

#122

Post by freedomben »

hootz wrote: post #25
freedomben wrote: post #5

Earlier quoted context omitted.

Having my workflow disrupted is the main reason I never adopted Antigravity, despite liking it. I'm glad to see G is invested, but the older I get the more protective I am of my workflow.

And the only realistic way to protect our workflow is by avoiding vendor lock-in like the plague.

Exactly. I admit it's a bit extreme, but this is a big reason why I insist that neovim is my IDE, and I won't adopt anything else. If I can't make it work in neovim, I will move to something else (unless I have no choice, but that happens very rarely at this point).

Re: Antigravity 2.0 Tops the OpenSCAD Architectural 3D LLM Benchmark

#123

Post by davej »

Eridrus wrote: post #70
1970-01-01 wrote: post #60

Creating a single real-world object and declaring it a benchmark? No, it doesn't work that way for a robust tool. You need to do something like Iron Chef, with a Greek architecture theme and and a panel or judge that declares the winner. This is just seeing which tool subjectively makes the best looking Pantheon.

Yeah, this is less of a benchmark and more "I like this one guys!". Just totally subjective grading criteria of a single poorly defined example with no end use case in mind to guide how to even do evaluation.

It's still interesting in a similar way to Simon Willison's Pelicans on a bicycle.

Re: Antigravity 2.0 Tops the OpenSCAD Architectural 3D LLM Benchmark

#124

Post by Eridrus »

davej wrote: post #123
Eridrus wrote: post #70

Earlier quoted context omitted.

Yeah, this is less of a benchmark and more "I like this one guys!". Just totally subjective grading criteria of a single poorly defined example with no end use case in mind to guide how to even do evaluation.

It's still interesting in a similar way to Simon Willison's Pelicans on a bicycle.

The Pelicans are mostly just entertainment.

Re: Antigravity 2.0 Tops the OpenSCAD Architectural 3D LLM Benchmark

#125

Post by illiac786 »

rlt wrote: post #104
dempedempe wrote: post #86

Earlier quoted context omitted.

It's like this with a lot of things now. For example, Nix's learning curve used to be a huge barrier to entry. Now with LLMs, I'm using nix-darwin and home-manager for dotfiles, package management, and have individual flakes in all of my projects for cryptographically reproducible builds!

Nit: there’s nothing “cryptographic” about reproducible builds. “Reproducible build” already usually implies bit-by-bit reproducibility.

“The reproducibility is cryptographically verifiable with hashes“ would be the full sentence, but it’s a mouthful.

Re: Antigravity 2.0 Tops the OpenSCAD Architectural 3D LLM Benchmark

#126

Post by jonasmaturana »

I've been using Claude to generate OpenSCAD scripts for the last few months, then exporting to Bambu Studio. Never really liked the OpenSCAD editing part though, so I built a little personal tool: https://webscad.aicentralen.dk/

One neat thing is that each color becomes a separate object on export to Bambu Studio, so it's easy to assign different filaments. One of the first things I made with it was these multicolored tag keys: https://webscad.aicentralen.dk/examples/name-tag

Re: Antigravity 2.0 Tops the OpenSCAD Architectural 3D LLM Benchmark

#127

Post by samcheng »

jonasmaturana wrote: post #126

I've been using Claude to generate OpenSCAD scripts for the last few months, then exporting to Bambu Studio. Never really liked the OpenSCAD editing part though, so I built a little personal tool: https://webscad.aicentralen.dk/ One neat thing is that each color becomes a separate object on export to Bambu Studio, so it's easy to assign different filaments. One of the first things I made with it was these multicolore…

This being Hacker News, it's worth mentioning that Bambu have been exceedingly bad actors around the AGPL and Open Source.

https://www.youtube.com/watch?v=3W5NNiHnviU

Re: Antigravity 2.0 Tops the OpenSCAD Architectural 3D LLM Benchmark

#128

Post by ecshafer »

The fact that the model recreated the interior dome pattern is concerning and makes me think this didn't test what they think. The interior dome pattern isn't visible in either picture. So it took the picture, and the name, then either via search or training data, knew that there should be an interior pattern. So it could be getting information on the pantheons dimensions or existing models. Whereas other models might be building based purely on what is seen in the reference pictures.

Re: Antigravity 2.0 Tops the OpenSCAD Architectural 3D LLM Benchmark

#129

Post by nancyminusone »

jlhawn wrote: post #105

> Antigravity was the only autonomous agent that implemented the Pantheon’s signature interior ceiling pattern: repeated square coffers visible through the oculus. That is seriously really impressive. I looked at the 3D model and didn't even thing to LOOK INSIDE the building before reading this. Here's [1] the 3D model with `show_cutaway` enabled. [1] https://modelrift.com/models/pantheon-benchmark-antigravity-...

I can't decide whether it's good or bad that it has included outside information clearly not present in the prompts to make the model. Clearly its the right thing to do if you want "the Pantheon" but I don't think any draftsman or engineer would find this acceptable work.

Re: Antigravity 2.0 Tops the OpenSCAD Architectural 3D LLM Benchmark

#130

Post by thanhhaimai »

ecshafer wrote: post #128

The fact that the model recreated the interior dome pattern is concerning and makes me think this didn't test what they think. The interior dome pattern isn't visible in either picture. So it took the picture, and the name, then either via search or training data, knew that there should be an interior pattern. So it could be getting information on the pantheons dimensions or existing models. Whereas other models migh…

From the article:

> Antigravity was the only autonomous agent that implemented the Pantheon’s signature interior ceiling pattern: repeated square coffers visible through the oculus.

The article also includes a video showing the patterns visible through the roof oculus.

Post reply on HN