Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The First Fully General Computer Action Model

si.inc

Post reply on HN
Print view View original

51–60 of 85 posts

Re: The First Fully General Computer Action Model

#51

Post by nee1r »

cs702 wrote: post #31

At first glance, this looks incredible to me. The authors train one model on 40K hours of computer-use video, previously labeled by contractors with keyboard and mouse actions, then use that model, in effect, to label 11M hours of computer-use video, which they use to train the computer-action model. The key advance is in compression. Quoting from the OP: > [previous models] burn a million tokens to understand just o…

yeah! i love the BCO paper, i think its extremely intuitive and these methods are really interesting in a time where data without labels is abundant. i especially like the idea of iteratively making the inverse dynamics better—might lean closer to that in the future

Re: The First Fully General Computer Action Model

#52

Post by nee1r »

nextzck wrote: post #48

I think you guys are on the right track here. I’d love to learn more about the math behind the FDM. I don’t think folks realize how behind we are on vision, thank you for your work here.

thanks! the math and architecture of the FDM (no video encoder) is pretty simple, its a regular transformer with next-token predictions but with frames interleaved.

Re: The First Fully General Computer Action Model

#53

Post by nee1r »

sp1nningaway wrote: post #30

May I suggest a driving demo in a parking lot with a mannequin instead of a real world video where it drives way too close to a pedestrian? Otherwise, very cool and exciting!

safety was important for the demo, the model didn't have access to the brake or accelerator.

Re: The First Fully General Computer Action Model

#55

Post by mcint »

Congratulations! I’ll be interested to see the next steps in alignment. Do you plan to start selling access, or collect more data to train bigger & better? What tasks or benchmarks are your biggest guide stars, or what was unexpectedly tricky—a few are hinted in the post.

It would be pretty interesting to see activation maps for the encoder on video, confidence building to see the compression derived from so much training.

Re: The First Fully General Computer Action Model

#56

Post by nee1r »

mcint wrote: post #55

Congratulations! I’ll be interested to see the next steps in alignment. Do you plan to start selling access, or collect more data to train bigger & better? What tasks or benchmarks are your biggest guide stars, or what was unexpectedly tricky—a few are hinted in the post. It would be pretty interesting to see activation maps for the encoder on video, confidence building to see the compression derived from so much tra…

we have an alignment blog post dropping soon! scaling up in the next couple of months, then hopefully opening up an API or licensing it.

Benchmarks are really fun—lots of secret ones. Our main thesis is that you should be using the same benchmarks to measure human ability to use a computer, as you would an AI model. Definitely a suite of continuous long term planning tasks (games) and things such as marking emails as spam etc.

definitely! we are looking into more interp + visualizations in general as we scale up.

Re: The First Fully General Computer Action Model

#57

Post by rao-v »

nee1r wrote: post #8
ennucore wrote: post #4

How do you tokenize the mouse inputs?

good question! we use exponential binning (map the mouse movements onto a plane with exponentially increasing tick marks https://si.inc/fdm1/exponential_binning.webp ) but tried a bunch of other methods (linear creates too many tokens for the model to learn well). Polar coordinates seem like a better solution but empirically didn't work well because the tokens got too coarse too fast.

It’s interesting that you invest in mouse movements vs just targeting a click at X in Y milliseconds. CAD and video games are of course a great reason for this, but I wonder how much typical tool use can be modeled by just next click events.

I’d love to see this sort of thing paired with eye tracking and turned into a general purpose precog predictive tool for computer use … but you probably have many better use cases for your world model!

Re: The First Fully General Computer Action Model

#59

Post by segmondy »

Nice, I have always felt the computer was the ultimate environment and screen capture the ultimate training data. Nice to see it in practice, now we have to wait to see if folks are going to argue on if your model could really learn a world model. I'm surprised this post doesn't have more comments, their site is worth checking out. Rooting for them, they are gritty, checkout their storage buildout story.

Re: The First Fully General Computer Action Model

#60

Post by lambdaloop »

This is fascinating! Having a really strong video encoder model and then a simpler decoder from that reminds me of the recent D4RT from DeepMind as well: https://d4rt-paper.github.io/

I think we'll see more of these video encoder models in the coming years, they truly seem like magic.

Post reply on HN