31–40 of 415 posts
Earlier quoted context omitted.
I also wouldn't be surprised if labs were starting to mix in a few pelican SVGs into their training data.
Even "accidentally" it makes sense that "SVGs of pelicans riding bikes" are now included into datasets used for training as it has spread as a wildfire on the internet, making it less useful as a simple benchmark. This is why I keep all my benchmarks private and don't share anything about them publicly, as soon as you write about them anywhere publicly they'll stop being useful in some months.
This is also why, if I were an artist or anyone commercially relying on creative output of any kind, I wouldn't be posting anything on the internet anymore, ever. The minute you make anything public, the engines will clone it to death and turn it into a commodity.
Alas, my 3 year old Mac has only 16 GB RAM, and can barely run a browser without running out of memory. It's a work-issued Mac, and we only get upgrades every 4/5 years. I must be content with 8B parameters models from Ollama (some of which are quite good, like llama3.1:8b).
Most likely its training data included countless Space Invaders in various programming languages.
This comment is ~3 years late. Every model since gpt3 has had the entirety of available code in their training data. That's not a gotcha anymore. We went from chatgpt's "oh, look, it looks like python code but everything is wrong" to "here's a full stack boilerplate app that does what you asked and works in 0-shot" inside 2 years. That's the kicker. And the sauce isn't just in the training set, models now do post-tra…
Even then, when you start to build up complexity within a codebase - the results have often been worse than "I'll start generating it all from scratch again, and include this as an addition to the initial longtail specification prompt as well", and even then... it's been a crapshoot.
I _want_ to like it. The times where it initially "just worked" felt magical and inspired me with the possibilities. That's what prompted me to get more engaged and use it more. The reality of doing so is just frustrating and wishing things _actually worked_ anywhere close to expectations.
Is probably more correct to say - my 2.5 year laptop can RETELL space invaders. Pretty sure it cannot write a game it has never seen, so you can even say - my old laptop can now do this fancy extraction of data from a smart probabilistic blob, where the original things are retold in new colours and forms :)
If you haven't seen that yourself yet I suggest firing up the free, no registration required GLM-4.5 Air on https://chat.z.ai/ and seeing if you can prove yourself wrong.
Is probably more correct to say - my 2.5 year laptop can RETELL space invaders. Pretty sure it cannot write a game it has never seen, so you can even say - my old laptop can now do this fancy extraction of data from a smart probabilistic blob, where the original things are retold in new colours and forms :)
If I understand correctly, the author is managing to run this model on a laptop with 64GB of RAM? So a home workstation with 64GB+ of RAM could get similar results?
Similar in quality, but CPU generation will be slower than what macs can do.
What you can do with MoEs (GLMs and Qwens) is to run some experts (the shared ones usually) on a GPU (even a 12GB/16GB will do) and the rest from RAM on CPU. That will speed things up considerably (especially prompt processing). If you're interested in this, look up llama.cpp and especially ik_llama, which is a fork dedicated to this kind of selective offloading of experts.
Either with SDL2+C, or even TCL/Tk, or Pythn with TKInter.
"2.5 year old laptop" is potentially the most useless way of describing a 64GB M2, as it could be confused with virtually any other configuration of laptop.
The thing I find most notable here is that this is the same laptop I've used to run every open weights model since the original LLaMA. The models have got so much better without me needing to upgrade my hardware.
No need to overly quantize our headlines.
"64GB M2 makes Space Invaders-- can be bought for under $xxxx"
Is probably more correct to say - my 2.5 year laptop can RETELL space invaders. Pretty sure it cannot write a game it has never seen, so you can even say - my old laptop can now do this fancy extraction of data from a smart probabilistic blob, where the original things are retold in new colours and forms :)