Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The First Fully General Computer Action Model

si.inc

Post reply on HN
Print view View original

31–40 of 85 posts

Re: The First Fully General Computer Action Model

#31

Post by cs702 »

At first glance, this looks incredible to me. The authors train one model on 40K hours of computer-use video, previously labeled by contractors with keyboard and mouse actions, then use that model, in effect, to label 11M hours of computer-use video, which they use to train the computer-action model. The key advance is in compression. Quoting from the OP:

> [previous models] burn a million tokens to understand just one minute of 30 FPS computer data. Our video encoder encodes nearly 2 hours of video in the same number of tokens—that’s 50x more token-efficient than the previous state-of-the-art and 100x more token-efficient than OpenAI’s encoder.

While I was already aware that there are people working on new, more efficient "world models," this is the first one I've seen in action. I'm a bit in shock at how good it is, quite frankly.

I've added the OP, as well as a related 2018 paper on Behavioral Cloning from Obervation (BCO) to my reading list.[a] So far, I've only skimmed the 2018 paper, but it's already evident that it's well-written. I'm no expert in deep RL, and I can understand it. BTW, "Behavioral Cloning from Obervation" is a really good name, with an easy-to-remember acronym.

Thank you for sharing this on HN.

[a] https://arxiv.org/abs/1805.01954

Re: The First Fully General Computer Action Model

#33

Post by AndrewKemendo »

nee1r wrote: post #5

Hey guys! I’m Neel, been holed up in our south park office for the past year working on model training. excited to share our research! This is a preview of a very different type of computer use model—we train on the internet. Specifically we have 11 million hours of computer video stored on our storage cluster (previously shared https://news.ycombinator.com/item?id=45438496 !) and the model can work in 30 FPS. Since…

This looks like a really promising approach

In particular the Forward rollout module is very important. It aligns your (effectively) world model with what it expects from the world, and keeping those in sync I think gives this the power it needs to be able to generate the state action pairs to continuously train semi supervised

Re: The First Fully General Computer Action Model

#34

Post by dangoodmanUT »

nee1r wrote: post #5

Hey guys! I’m Neel, been holed up in our south park office for the past year working on model training. excited to share our research! This is a preview of a very different type of computer use model—we train on the internet. Specifically we have 11 million hours of computer video stored on our storage cluster (previously shared https://news.ycombinator.com/item?id=45438496 !) and the model can work in 30 FPS. Since…

11 million hours of data is a lot, did you have to synthesize it at all, or was it purely collected?

Re: The First Fully General Computer Action Model

#39

Post by sheepscreek »

nee1r wrote: post #24
152334H wrote: post #23

holy crap, this is so good. How did it get buried?

real

Are you guys affiliated with Meta’s ex-CTO in any way? I remember he famously implied that LLMs hyped. The demos are very impressive. Does this use an attention based mechanism too? Just trying to understand (as a layman) how these models handle context and if long contexts lead to weaker results. Could be catastrophic in the real world!

Re: The First Fully General Computer Action Model

#40

Post by sheepscreek »

sheepscreek wrote: post #39
nee1r wrote: post #24

Earlier quoted context omitted.

real

Are you guys affiliated with Meta’s ex-CTO in any way? I remember he famously implied that LLMs hyped. The demos are very impressive. Does this use an attention based mechanism too? Just trying to understand (as a layman) how these models handle context and if long contexts lead to weaker results. Could be catastrophic in the real world!

I think in the long run, we may need something like a batch job that compresses context from the last N conversations (in LLMs) and applies that as an update to weights. A looser form of delayed automated reinforcement learning.

Or make something like LoRA mainstream for everyone (probably scales better for general use models shared by everyone).

Post reply on HN