What is the SOTA for benchmarking all of the models you can run on your local machine vs a test suite?
Surely this must exist, no? I want to generate a local leaderboard and perhaps write new test cases.
My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)
81–90 of 415 posts
Re: My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)
#82Apart from using a Mac, what can you use for inference with reasonable performance? Is a Mac the only realistic option at the moment?
The top 3 approaches I see a lot on r/localllama are:
1. 2-4x 3090+ nvidia cards. Some are getting Chinese 48GB cards. There is a ceiling to vRAM that prevents the biggest models from being able to load, most can run most quants at great speeds
2. Epyc servers running CPU inference with lots of RAM at as high of memory bandwidth as is available. With these setups people are getting like 5-10 t/s but are able to run 450B parameter models.
3. High RAM Macs with as much memory bandwidth as possible. They are the best balanced approach and surprisingly reasonable relative to other options.
Re: My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)
#83Aside that space invaders from scratch is not representative for real engineering, it will be interesting to see what the business model for Anthropic will be if I can run a solid code generation model on my local machine (no usage tier per hour or week), let’s say, one year from now. At $200 per month for 2 years I can buy a decent Mx with 64GB (or perhaps even 128GB taking residual value into account)
How come it's "not representative for real engineering"? Other than copy-pasting existing code (which is not what an LLM does), I don't see how you can create a space invaders game without applying "engineering".
>> Other than copy-pasting existing code (which is not what an LLM does)
I'd like to see someone try to prove this. How many space invaders projects exist on the internet? I'd be hard to compare model "generated" code to everything out there looking for plagiarism, but I bet there are lots of snippets pulled in. These things are NOT smart, they are huge and articulate information repositories.
Re: My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)
#84Most likely its training data included countless Space Invaders in various programming languages.
This comment is ~3 years late. Every model since gpt3 has had the entirety of available code in their training data. That's not a gotcha anymore. We went from chatgpt's "oh, look, it looks like python code but everything is wrong" to "here's a full stack boilerplate app that does what you asked and works in 0-shot" inside 2 years. That's the kicker. And the sauce isn't just in the training set, models now do post-tra…
You are reading too much into my comment. My point was that the test (a Space Invaders clone) used to asses the model is irrelevant for some time now. I could have gotten a similar result with Mistral Small a few months ago.
Re: My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)
#85Earlier quoted context omitted.
How come it's "not representative for real engineering"? Other than copy-pasting existing code (which is not what an LLM does), I don't see how you can create a space invaders game without applying "engineering".
>> Other than copy-pasting existing code (which is not what an LLM does) I'd like to see someone try to prove this. How many space invaders projects exist on the internet? I'd be hard to compare model "generated" code to everything out there looking for plagiarism, but I bet there are lots of snippets pulled in. These things are NOT smart, they are huge and articulate information repositories.
Go for it. https://www.google.com/search?client=firefox-b-1-d&q=github+... has a bunch of results. Here's the source code GLM-4.5 Air spat out for me on my laptop: https://github.com/simonw/tools/blob/main/space-invaders-GLM...
Based on my mental model of how these things work I'll be genuinely surprised if you can find even a few lines of code duplicated from one of those projects into the code that GLM-4.5 wrote for me.
Re: My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)
#86Earlier quoted context omitted.
is GLM 4.5 better than Qwen3 coder??
For what? It's really hard to say what model is "generally" better then another, as they're all better/worse at specific things. My own benchmarks has a bunch of different tasks I use various local models for, and I run it when I wanna see if a new model is better than the existing ones I use. The output is basically a markdown table with a description of which model is best for what task. They're being sold as gener…
coding? they are coding models? what specific tasks is one performing better than the other?
Re: My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)
#87Most likely its training data included countless Space Invaders in various programming languages.
I find the visual similarity to breakout kind of interesting.
Re: My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)
#88Aside that space invaders from scratch is not representative for real engineering, it will be interesting to see what the business model for Anthropic will be if I can run a solid code generation model on my local machine (no usage tier per hour or week), let’s say, one year from now. At $200 per month for 2 years I can buy a decent Mx with 64GB (or perhaps even 128GB taking residual value into account)
What about power used and support hardware ? Also card going down means you are down until you get warranty service.
Re: My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)
#89Earlier quoted context omitted.
I've had little success with Agentic coding, and what success I have had has been paired with hours of frustration, where I'd have been better off doing it myself for anything but the most basic tasks. Even then, when you start to build up complexity within a codebase - the results have often been worse than "I'll start generating it all from scratch again, and include this as an addition to the initial longtail spec…
Bingo, it's magical but the learning curve is very very steep. The METR study on open-source productivity alluded to this a bit. I am definitely at a point where I am more productive with it, but it took a bunch of effort.
The subjects in the study you are referencing also believed that they were more productive with it. What metrics do you have to convince yourself you aren't under the same illusionary bias they were?
Re: My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)
#90Alas, my 3 year old Mac has only 16 GB RAM, and can barely run a browser without running out of memory. It's a work-issued Mac, and we only get upgrades every 4/5 years. I must be content with 8B parameters models from Ollama (some of which are quite good, like llama3.1:8b).
Odd. My MBP has 16 GB and I routinely have 5 browsers windows open. Most of them have 5-20 tabs. I'm also routinely running vi vscode and editing videos with davinci resolve without issue.
My only memory issue that I can remember is an OBS memory leak, otherwise these MBPs incredible hardware. I wish any other company could actually deliver a comparable machine.