171–180 of 199 posts
Earlier quoted context omitted.
How would destroying the GPUs prevent the model weights from leaking? By the time you get your hands on them the memory is powered off for a long enough time that a cold-boot style attack is impossible.
Would you bet your trillion dollar company on that? Or would you smash up the garbage [to you] memory chips to be sure.
Impressive work. But the problem is not the 30 tok/s which is fine for agentic coding and chat. It's prefill; slow prefill kills agentic workloads dead. If you have 100,000 tokens at ~150tok/s per the OP, you're looking at: You have: 100000 / (150/s) You want: hms 11 min + 6.6666667 sec Which is quite a wait indeed.
The V100 and the 4090 are based on vastly different architectures, the former uses the older Volta while the latter uses Ada. Last I checked you cannot meaningfully combine them. The 3090 is better than the V100, just get two 3090 and a NVLink.
Earlier quoted context omitted.
qwen3-coder-next runs fine on my consumer grade nvidia 4070. Performance is not spectacular, but it's only a little bit slower than a properly-fit model.
What are your settings and tokens/second? Even with 2 GPUs (MI100, RX 6600 XT 8GB) and 32GB of RAM it was running at a snails pace for me. I didn't try a sched_spread with a 3090 and the MI100 which would provide 56GB ram
The machine:
CPU: 24 × AMD Ryzen 9 9900X 12-Core Processor
RAM: 128gb
GPU: NVIDIA GeForce RTX 4060 Ti 16gb (I typo'd the GPU above)
(This is via Ollama on Ubuntu.)
But 1-3 tokens per second is much faster than a lot of other high end models I've tried, so I was pretty pleased with it. Obviously other models run much faster on this hardware though.
> The way it works is that a vision encoder (similar to what ChatGPT and Claude use) takes image pixels and translates them into the LLM’s token embedding space. The model does not “see” the image the way a human does. Instead, the vision encoder compresses the image into a sequence of vectors that live in the same mathematical space as text tokens. The LLM then processes those vectors as if they were just another se…
Earlier quoted context omitted.
> Where do you think llms learned to write that way? Not from individual human content, that's for sure - maybe MLM marketing copy? Sleazy 4AM ads? I mean, every time this response comes up, I keep asking the person to point at something written prior to 2022 that gets 80%+ on the LLM detectors, and yet no one can find anything. Maybe you, postalrat, can find something written in this style that was published prior t…
I have written the blog post. I know empirically that I have used 0% AI while writing it. I also know LLM detectors are total BS and they don't really work. I have tried a couple on this exact blog post, and QuillBot, for example, gave me 0% AI detected on it. I have then used a blog post of mine from 2021. QuillBot gave me 8%... The King James version of the Bible came out at almost 100% AI generated a while ago. It…
I mean, seriously, which human says "the compute"?
Earlier quoted context omitted.
That line was the exact moment I also realized the post was AI written. I kept reading though, but I am left constantly guessing at which key details might be pure hallucinations.
FYI, not a single line was AI written. If there is a hallucination, it’s fully mushy brain sourced.
I think for me it was mainly the superlative "genuinely surprising" that made me wonder.
Earlier quoted context omitted.
You can also look at past posts by the same author (before LLM usage proliferated) if you’re curious. The project is still very cool, but it’s a little less enjoyable to read when everything sounds the same. It would be just as annoying for people to manually write in a corporate/marketing style, because humanity is what makes the small web interesting. https://blog.tymscar.com/posts/privategithubcicd/
I’m glad I’ve started this blog before the AI wave so I can prove people I’m just weird at writing. It grinds my gears how so many people just talk about my writing style instead of the content.
But yeah, probably feels sucky to have your style analyzed for AI writing. FWIW, the datacenter GPU post was great! I went to look at the ebay postings.
Earlier quoted context omitted.
I’m glad I’ve started this blog before the AI wave so I can prove people I’m just weird at writing. It grinds my gears how so many people just talk about my writing style instead of the content.
What's interesting about the older post is that all the sentences are long, compared to the current datacenter GPU post which contains lots of short sentences. But yeah, probably feels sucky to have your style analyzed for AI writing. FWIW, the datacenter GPU post was great! I went to look at the ebay postings.
I did get feedback on all sorts of things over the years.
One of them was to do with sentence lengths.