What sort of fine tuning data was needed to allow the model to self-drive? One hour of video of someone driving, or extra labeling?
The First Fully General Computer Action Model
11–20 of 85 posts
Re: The First Fully General Computer Action Model
#12What sort of fine tuning data was needed to allow the model to self-drive? One hour of video of someone driving, or extra labeling?
i actually drove the car (with arrow keys) around south park for around ~45 minutes as finetuning data, no extra labelling other than that. think the car line graph is super cool because you actually see the videegame prior working
Re: The First Fully General Computer Action Model
#13I rly liked the point about ctrl-c only being able to be labelled retrocausally. I do think that with enough past context you should be able to know what was copied - in some sense the past does encode the future - but also an agentic decision is precisely the kind where the future is more informative than the past for reconstructing that decision. It does make me wonder if you should have the inverse dynamics model…
yeah we actually had some wacky ideas with ctc + a reverse-causal mask but diffusion does just make it all a bit more simple
Re: The First Fully General Computer Action Model
#14What sort of fine tuning data was needed to allow the model to self-drive? One hour of video of someone driving, or extra labeling?
relevant note is that we finetuned by having the human also use arrow keys which keeps it in-distribution but also slower to collect
Re: The First Fully General Computer Action Model
#15what tasks can the model do out of the box? was each of the examples a different fine tuned model?
Re: The First Fully General Computer Action Model
#16what tasks can the model do out of the box? was each of the examples a different fine tuned model?
it's a pretty general policy but this is all super early, it's great at exploring websites so fuzzing was easy, for CAD it has good enough base rates with the few-shot prompt when we do the repetitive stuff, and we gave it checkpoints on each step, the other stuff in the mosaic are just some of our favorite clips from internal evals
Re: The First Fully General Computer Action Model
#17[deleted]
Re: The First Fully General Computer Action Model
#18[dead]
Re: The First Fully General Computer Action Model
#19[dead]
no finetuning data for the blender task! we actually think its the opposite, there are a lot of video tutorials for complex tasks like onshape/blender/fusion360 but not as much of people idly browsing.
but also at the 11M hour scales it still sees a substantial amount of data
Re: The First Fully General Computer Action Model
#20The video compression is very cool. And the small tricks like binning the mouse movements.
Wonder how much data is generalizable across different UIs? ie how good will the model be at using Figma if it’s never seen it before but has seen a lot of Photoshop