RoboCat – A Self-Improving Robotic Agent

I have a very opinionated opposing view of this research. A lot of research in this direction is working on raising the floor. Basically they just want the robot to handle a large variety of simple tasks and environments. Fair enough most industrial robots can't handle the smallest changes. But many times they implicitly make the assumption that raising the floor will also raise the ceiling. They assume, if it can generalize at 90%, it might also be able to do far more dextrous tasks that humans can. I think this is completely false, at best if we could do dextrous tasks in 1 environment this can transfer them to other environments with a presumably lower efficiency.

On the other hand, I think a more promising direction is to raise the ceiling of robot arm manipulation sky high. OpenAI kind of did this with Dactyl but I would like to see more of it. Can we get robotic arms to tie a shoelace, knit, perform pottery etc (with an arm like morphology, no special mechanisms). I think this can actually then lead to large scale generalization, kind of what we are seeing happening with NERF's now. I would like a robot arm NERF, overfit to one hard task but reproduces it with human like precision and dexterity. Deepminds approach to me (with GATO, robocat) seems like a red herring, they will never reach the kind of results we want from our arms.

There is FEETECH on ali-express. About 0.5-0.25 times Dynamixel price. I've never tried it though. You're right very few of us want High Torque and High Accuracy which is the problem

It could, but I bet it won't. The UI is not the limiting factor - the SaaS providers are. Automation like this should be easy with regular APIs and code (or no-code tools). It isn't, because service providers don't want you to do it. A skilled dev can get their way with Selenium or some other way of browser automation - this is tolerated, because approximately no one actually does it.

If anyone could easily get an LLM to automate this, you can bet service providers will scramble to find ways to defeat it (justifying it with some bullshit reason like "security", so it's not blatantly obvious they're artificially constraining their product, to prevent you walking around their most profitable toll booths). In general, it doesn't matter what new technology is promising in the abstract - if anything, it's just marketing noise. The technology will deliver only the things its wielders can make money on, and will not deliver things that don't have a good business case. It's the standard tech industry bait-and-switch.

Want proof that this is more than just cynical rambling? Take a look at IFTTT, Zapier, et al. See what you can actually do with them. Notice that it's entirely limited by what the integrated services expose to those automation services. See how carefully those integrations are crafted, so that you can't exactly do much with them. If you want anything non-trivial, you'd better be a business, because individuals aren't supposed to empower themselves with technology. And if you're a business, well, anything is possible if you're willing to pay enough.

The few of us who know how to operate advanced automation tools, and whose frustration outweighs the cost of micromanaging brittle DIY hacks, can beat SaaS tools into submission. Sometimes. But only as long as it stays below the noise floor of vendors' financial metrics.

Hostility to automation is a consequence of the “free to use” double-sided market trap that have been flourishing since Web 2.0.

If you are paying for a service, which adequately charges you at least how much it costs to run + margin, the service should not care whether you are giving them your eyeballs, and you are entitled to request features like automation friendliness and APIs and e2e encryption because you can vote with your wallet by changing suppliers.

If advertisers are the ones paying, and specifically paying for your eyeballs, then naturally your interests are at odds.

(There are edge cases where the service may in fact charge you, but 1) their billing structure does not let them cover operational costs, 2) they are afraid to alienate users by raising prices or otherwise hostage to current billing structure, and 3) they are incapable of optimizing performance to turn profit with current billing structure; in those cases crippled API and automation-hostile GUI may serve as a throttle.)

Surprisingly, I don't believe it has anything to do with advertising. SaaS model is, by its very nature, hostile to its users - even the paying ones.

> If you are paying for a service, which adequately charges you at least how much it costs to run + margin, the service should not care whether you are giving them your eyeballs,

They may not care about your eyeballs, but their understanding of "adequate charge" is "what the market can bear". More specifically, if they allow for some automation, and that automation saves their customers a noticeable amount of money, then they'll try to capture as much of those savings as possible. This is business as usual - the entire market economy works like this. The problem with SaaS is...

> and you are entitled to request features like automation friendliness and APIs and e2e encryption because you can vote with your wallet by changing suppliers.

...that you can't actually vote with your wallet. Software resists commoditization, and SaaS companies are especially good at this. Whatever the service you use, it's highly likely there is no good alternative for it. There may be some competitors whose offering partially overlaps with what you use now, but it's highly likely there's some unique aspect that, combined with hassle of moving your data and overall inertia, will make switching service providers an act of last resort.

Those sticking points often aren't software-related at all! E.g. Netflix and HBO Max and Disney+ aren't really substitutes, because their catalogs do not overlap. Tech-wise, Spotify is mostly a shitty audio player (that gets worse every update) - their entire value was always in the deals they signed with the labels and artists. Etc.

Combine this with your typical SaaS targeting a global market, and you can see the power imbalance. They don't give a shit what you want. You ain't gonna "vote with your wallet". Hardly anyone does. And you're just one user of a hundred thousand, or a million - a rounding error. You're not worth caring about, not unless you're in a position to start a cascade of users leaving - which might happen if they try to screw you over and you're quick to Twitter, but which definitely won't happen just because they made their UI suck on purpose, or put basic automation behind a high price tier.

> 3) they are incapable of optimizing performance to turn profit with current billing structure; in those cases crippled API and automation-hostile GUI may serve as a throttle

That's a possibility in some cases, but I think that, if the "normal" mode of user interaction involves a website or a mobile app written using modern frameworks and development techniques, then it's probably a wash - increased utilization of the core service may be canceled out by reduction in the flood of requests their bloated GUI generates.