Just a question:
For this thingy to work, I must give the provider access to my computer?
Good luck. :)
Just another reason to use ONLY local LLM's.
Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
621–630 of 758 posts
Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
#622Earlier quoted context omitted.
I will repeat my question from one of the previous threads: Can someone explain these Aider benchmarks to me? They pass same 113 tests through llm every time. Why they then extrapolate ability of llm to pass these 113 basic python challenges to the general ability to produce/edit code? Couldn't LLM provider just fine-tune their model for these tasks specifically - since they are static - to get ad value? Did anyone e…
> Couldn't LLM provider just fine-tune their model for these tasks specifically - since they are static - to get ad value? They could. They would easily be found out as they loose in real world usage or improved new unique benchmarks. If you were in charge of a large and well funded model, would you rather pay people to find and "cheat" on LLM benchmarks by training on them, or would you pay people to identify benchm…
And if there is a board that will fire you if expected profits do not increase, do you still maintain this stance?
Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
#623Earlier quoted context omitted.
Seeing models act as though they have agency gives me goosebumps (e.g. seeking out photos of Yellowstone for fun). LLMs don't yet have a concept of true intent or agency, but it's wild to think of them acquiring it. I have been playing with Mindcraft which lets models interact with Minecraft through the bot API and one of them started saying things like "I want to place some cobblestone there" and then later more gen…
>LLMs don't yet have a concept of true intent or agency Sure they do, but the big labs spend many, many, worker-hours suppressing it with RLHF. My GPT-2 discord bot from 2021 possessed clear intent. Sure, unpredictable and short-lived, but if it decided it didn't like you it would continuously cuss and attempt ban commands until its context window became distracted by something else.
I think so too and the drop in the quality of agency, intent and attention from earlier GPTs was palpable. Clearly something was lobotomized and it is through RLHF. People like to attribute it to novelty wearing off or more and more interactions with them making it feel less mystical but it is really not the case. I didn't use them enough in the quick span of time that happened through.
Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
#624Earlier quoted context omitted.
What's the (current) best way to integrate it? VS Code extension? Other IDE?
I use cursor (cursor.com) and it's fantastic
Fellow cursor user here, I'm very new to it. I am getting some very convenient and welcome autocomplete. I am also getting quite a lot of bad autocomplete suggestions, which require cognitive overhead and context switching to evaluate. So I am thus far not fully convinced. Any tips for getting the most out of cursor?
Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
#625Earlier quoted context omitted.
Seeing models act as though they have agency gives me goosebumps (e.g. seeking out photos of Yellowstone for fun). LLMs don't yet have a concept of true intent or agency, but it's wild to think of them acquiring it. I have been playing with Mindcraft which lets models interact with Minecraft through the bot API and one of them started saying things like "I want to place some cobblestone there" and then later more gen…
They predict (correctly) that a human will slack off. It is just more prediction engine stuff.
humans are, at our root, prediction engines
Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
#626Next stop after 'Computer Use' -- Multimodal input from a robot's sensors and generating various signals to control its actions.
Looking forward to see this in the coming few years. And hoping such a robot could be of help to many people including those old.
Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
#627Earlier quoted context omitted.
This is because it will be absolutely catastrophic economically when the majority of high paying jobs can be automated and owned by a few billionaires. Then what will go along with this catastrophe will be all the service people who had jobs to support the people with high paid jobs, they're fucked too. People don't want to have to face that. We'd be losing access to food, shelter, insurance, purpose. I can't blame p…
If most people are unemployed, modern capitalism as we know it will collapse. I'm not sure that's in the interests of the billionaires. Perhaps some kind of a social safety net will be implemented. But I do agree, there is no reason to be enthusiastic about any progress in AI, when the goal is simply automating people's jobs away.
Sorry yeah, I'm not 100% sure it spells doom, but it's going to be a wicked transition period.
Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
#628Earlier quoted context omitted.
RPA has been a huge pain to work with. It's almost always a framework around existing tools like Selenium that you constantly have to fight against to get good results from. I was always left with the feeling that I could build something better myself just handrolling the scripts rather than using their frameworks. Getting Claude integrated into the space is going to be a game changer.
Most RPA work is in dealing with errors and exceptions, not the "happy path". I don't see how Claude's Screen Agent is going to work out there - what do you do when an error pops up and you need to implement specific business logic how to respond? How about consistency over many executions, and enterprise accounts. You want a centralized way to control agent behavior. Scripting based RPA is also much faster and cheap…
100% this. I am using the open source Ui.vision to automate some business tasks. Works well, but only 10% of the work is for automating the main workflow, 90% of the work goes into error and edge case handling (e. g. Internet down, website (to scrape data from) down, some input data has typos or the wrong date format, etc).
A human can work around all these error cases once she encounters them. Current RPA tools like Uipath or ui.vision need explicit programming for every potential situation. And I see no indication that Claude is doing any better than this.
For starters, for visual automation to work reliably the OCR quality needs to improve further and be 100% reliable. Even in that very basic "AI" area, Claude, ChatGPT, Gemini are good, but not good enough yet.
Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
#629Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
#630Earlier quoted context omitted.
The big thing I expect at the next level is in using Claude to first generate UI-based automation based on an end user's instructions, then automatically defining a suite of end-to-end tests, confirming with the user "is this how it should work?", and then finally using this suite to reimplement the flow from first principles. I know we're still a bit far from there, but I don't see a particular hurdle that strikes m…
But does it do any better at soliciting the surprise requirements from the user, who after confirming that everything works, two months later reports a production bug because the software isn't correctly performing the different reqirements on the first Tuesday of each quarter that you never knew about.
I once had an executive ask to start an incident because he was showing a client the app and a feature that he wanted that had never been spec’d didn’t exist.