111–120 of 287 posts
If anyone would like to try it for coding in VSCode, I just added it to http://double.bot on v93 (AI coding assistant). Feels quite strong so far and got a few prompts that I know failed with gpt4o. fyi for anyone testing this in their product, their docs are wrong, it's claude-3-5-sonnet-20240620, not claude-3.5-sonnet-20240620.
Before I read your comment I was looking for a solution to use Claude as co-pilot in Neovim. I've seen in Double's website FAQ that it's not supported yet. Do you have an idea if this feature is expected to land anytime soon?
Earlier quoted context omitted.
I'm actually working on an open source product to solve this. For a long while I've wanted a good "pro" UI that can connect to multiple different llm APIs Convenient editing and branching is one of the items in my roadmap already, what else do you think I could include?
You might be looking for "LLM Web-UI"s. I searched for a while until I found this thread with recommendations: https://old.reddit.com/r/LocalLLaMA/comments/1847qt6/llm_web...
Maybe something like what I'm talking about exists already, but I think I'll still try and make my own open source version to fulfill my personal requirements.
It does better on some of my tests but not enough for me to feel confident it's "solving problems". One thing I like to do is pose a problem and give it a bunch of objects it could use, some more helpful than others. I have found language models fail at discerning which of the tools are useful and which are red herrings, opting to use everything in some way. My go to test is the boat and goat: "You are on the shore o…
Sometimes it's funny to me how we can have such a feeling the responses are so obviously wrong in some way but then don't even see it the same way between ourselves. Imagine someone strikes up a conversation with you saying they've got a truck & a sofa with them and they want to know how to get to Manhattan. You say "just drive the sofa over the bridge" and they say "Good, but wrong. I don't need the sofa to get to Manhattan". You'd probably say "okay... so what are you going to do with this sofa you said you had with you"?
Of course, like you point out, LLMs sometimes take those associations a little to far and where your average person would say "Okay, they're saying they are with all of these things but probably because it's a list of what's around not a list of what they need to cross with" the LLMs are eager to answer in the form "Oh he's with all of these things? Alright - let's figure out how to use them all for them regardless of how odd it may be!".
Earlier quoted context omitted.
What I understand is that it's GPT 6 that just went into training, and that GPT 5 is complete and being delayed until after the U.S. election.
And after GPT-5's release, what would be the plan for subsequent elections? This seems to be a temporary play in delaying AI regulation if public sentiment further becomes that AI can have a strong influence in the elections.
OpenAI must be cooking something huge for them to not be releasing products far ahead of competitors
Idk about that, they did that weird event pre google IO and still haven't released the features they announced. Wouldn't be surprised if the only thing cooking is OpenAI itself.
Earlier quoted context omitted.
I'm actually working on an open source product to solve this. For a long while I've wanted a good "pro" UI that can connect to multiple different llm APIs Convenient editing and branching is one of the items in my roadmap already, what else do you think I could include?
Good history search (including non "main" conversation branches) and convenient conversation management (bookmarking, folders, maybe something smarter) would be great. Also, maybe some convenient way to create message templates? I don't know how I'd implement this, I just know that I often write one long prompt that I reuse multiple times, with multiple minor tweaks/edits, and it'd be amazing to have a convenient too…
I just followed you on Twitter (I'm @NamanyayG there as well), I'll definitely ping you when I have something to test.
I'm excited to test this out! I've been building an open source TypeScript agentic AI platform for work (DevOps related with an autonomous agent and software engineer workflow). The Claude 3 models had an influence on the design with their tuning on using XML and three levels of capabilities, and have been my preferred models to use. I remember having moments looking at the plans Opus generated and being impressed wi…
Agentic Coding Claude 3.5 Sonnet solves 64% of problems on an internal agentic coding evaluation, compared to 38% for Claude 3 Opus. Our evaluation tests a model’s ability to understand an open source codebase and implement a pull request, such as a bug fix or new feature, given a natural language description of the desired improvement. For each problem, the model is evaluated based on whether all the tests of the codebase pass for the completed code submission. The tests are not visible to the model, and include tests of the bug fix or new feature. To ensure the evaluation mimics real world software engineering, we based the problems on real pull requests submitted to open source codebases. The changes involve searching, viewing, and editing multiple files (typically three or four, as many as twenty). The model is allowed to write and run code in an agentic loop and iteratively self-correct during evaluation. We run these tests in a secure sandboxed environment without access to the internet.
% of problems which pass all tests Claude 3.5 Sonnet 64% Claude 3 Opus 38% Claude 3 Sonnet 21% Claude 3 Haiku 17%
Opus remained better than GPT for me, even after the release of GPT-4o. VERY happy to see an even further improvement beyond that, Claude is a terrific product and given the news that GPT-5 only began its training several weeks ago I don't see any situation where Anthropic is dethroned in the near term. There are only two parts of Anthropic's offering I'm not a fan of: - Lack of conversation sharing: I had a conversa…