Earlier quoted context omitted.
GPT4o is way behind sonnet 3.5
Huh I guess all the benchmarks are wrong then
Agreed.
Codestral Mamba
61–70 of 148 posts
Re: Codestral Mamba
#62Earlier quoted context omitted.
Most the 7b instruct models are very bad outside very simple queries. You can run a 7b on most modern hardware.How fast will vary. To run 30-70b models you're getting in the realm of needing 24gb or more of vRAM.
I'm looking to run something on a 24gb GPU for the purpose of running wild with agentic use of LLMs. Is there anything worth trying that would fit on that amount of vRAM? Or are all the open-source PC-sized LLMs laughable still?
You can run the llama 70b based models faster than 10 tkn/s on 24gb vram. I've found that the quality of this class of LLMs is heavily swayed by your configuration and system prompting and results may vary. This Reddit post seems to have some input on the topic:
https://www.reddit.com/r/LocalLLaMA/comments/1cj4det/llama_3...
I haven't used any agent frameworks other than messing around with langchain a bit so I can't speak to how that would effect things.
Re: Codestral Mamba
#63Is there a good explanation of the Mamba architecture?
Re: Codestral Mamba
#64Earlier quoted context omitted.
What made you switch?
Claude Projects which allow attaching a bunch of files to fill up the 200k context. I wrote up a script to dump a bunch of code and documentation files to markdown as context and I add them to a bunch of Claude projects on a per topic basis. For example, I'm currently working on a Rust/Qt desktop app so I have a project with the whole Qt6 book attached to ask questions about Qt, a project with my SQL schema and ORM/S…
I was under the impression that with LLMs, in order to get high-quality answers, it's always best to keep context short. Is that not the case anymore?
Does Claude under this usage paradigm not struggle with very long contexts in ways as for example described in the "lost in the middle" paper (https://arxiv.org/abs/2307.03172)?
Re: Codestral Mamba
#65Is there a good explanation of the Mamba architecture?
There's a paper: https://arxiv.org/abs/2312.00752
I haven't seen any good non-paper explainers yet.
Re: Codestral Mamba
#66Does anyone have a favorite FIM capable model? I've been using codellama-13b through ollama w/ a vim extension i wrote and it's okay but not amazing, I definitely get better code most of the time out of Gemma-27b but no FIM (and for some reason codellama-34b has broken inference for me)
Re: Codestral Mamba
#67Earlier quoted context omitted.
Claude Projects which allow attaching a bunch of files to fill up the 200k context. I wrote up a script to dump a bunch of code and documentation files to markdown as context and I add them to a bunch of Claude projects on a per topic basis. For example, I'm currently working on a Rust/Qt desktop app so I have a project with the whole Qt6 book attached to ask questions about Qt, a project with my SQL schema and ORM/S…
I was under the impression that with LLMs, in order to get high-quality answers, it's always best to keep context short. Is that not the case anymore? Does Claude under this usage paradigm not struggle with very long contexts in ways as for example described in the "lost in the middle" paper ( https://arxiv.org/abs/2307.03172 )?
I don't have the time to evaluate the effects of context length on my use cases so I have no idea. There might be some degradation when I attach the Qt book which is probably already in Claude's training data but when using it against my private code base, it's not like I have any other choice.
The UX of drag and dropping a few monolithic markdown files to include entire chunks of a large project outweighs the downsides of including irrelevant context in my experience.
Re: Codestral Mamba
#68What are the steps required to get this running in VS Code? If they had linked to the instructions in their post (or better yet a link to a one click install of a VS Code Extension), it would help a lot with adoption. (BTW I consider it malpractice that they are at the top of hacker news with a model that is of great interest to a large portion of the users where and they do not have a monetizable call to action on t…
If you can run this using ollama, then you should be able to use https://www.continue.dev/ with both IntelliJ and VSCode. Haven’t tried this model yet - but overall this plugin works well.
Unrelated, all my devices freeze when accessing this page, desktop Firefox and Chrome, mobile Firefox and Brave.
Is this the best alternative to access code ai helpers besides the GitHub Copilot and Google Gemini on VSCode?
Re: Codestral Mamba
#69Is there a good explanation of the Mamba architecture?
Re: Codestral Mamba
#70The first sentence is wrong. The website says: > As a tribute to Cleopatra, whose glorious destiny ended in tragic snake circumstances but according to Wikipedia this is not true: > When Cleopatra learned that Octavian planned to bring her to his Roman triumphal procession, she killed herself by poisoning, contrary to the popular belief that she was bitten by an asp.
Maybe Octavian was the snake?