Viewing profile — ryan_glass

I'm sure this is a wonderful breaktheough but I'd rather have the current, or a recent, generation of semiconductor technology available at a reasonable price.

For a fraction of the price of 96GB vram, I built a desktop based on a supermicro server mobo and EPYC 9 series CPU, with just under 400GB rdimm ram (approx $4500 all in but this w…

"Make the button red" probably doesn't need an LLM at all.

[dead]

Coding, my own proprietary code hence my desire for local hosting, a decent amount of legacy code. General troubleshooting of anything and everything from running Linux servers to …

Basically it comes down to memory bandwidth of server CPUs being decent. A bit of oversimplification here but... The model and context have to be pulled through RAM (or VRAM) every…

To be honest I haven't used o3 or Sonnet as the code I work with is my own proprietary code which I like to keep private, which is one reason for the local setup. For troubleshooti…

The quality on Gemma 27B is nowhere near good enough for my needs. None of the smaller models are.

It might be 5 to 10 times slower than a hosted provider but that doesn't really matter when the output is still faster than a person can read. Context wise, for troubleshooting I h…

Thank you for making the dynamic quantisations! My setup wouldn't be possible without them and for my personal use, they do exactly what I need and are indeed excellent.

No hard numbers I'm afraid in that I don't monitor the power draw. But the machine uses a standard ATX power supply: a Corsair RM750e 750W PSU and the default TDP of the CPU is 280…

You are right that I haven't been rigorous - it's easy to benchmark tokens/second but quality of output is more difficult to nail down. I couldn't find any decent comparisons for U…

Prompt eval time varies a lot with context but it feels real-time for short prompts - approx 20 tokens per second but I haven't done much benchmarking of this. When there is a lot …

I run Deepseek V3 locally as my daily driver and I find it affordable, fast and effective. The article assumes GPU which in my opinion is not the best way to serve large models lik…

Great idea - I can see this being popular. Small heads-up on responsiveness needing fixed on tablet in portrait view.

It would be interesting to see graphs of deaths for other reasons to compare numbers. For example deaths from starvation/malnutrition are likely increasing in India due to lockdown…

Scotland has a right of access to land (including beaches) as well as inland water throughout the country. This is known as 'the right to roam' and is a good example of a jurisdict…

This article really, really makes me want to give Go a shot - more than anything I've read about the language before.

One possible issue with this article is that its focus is on students - their 'job' being to learn as opposed to producing or delivering something, which is the aim once their skil…

Nice post - thanks. I'm guilty of sometimes splitting and other times not, making the choice mostly on feel. Never modifying local state when splitting functions is an easy rule of…