41–50 of 132 posts
Really a pity that Oracle killed off SPARC. They already had 32-core CPUs almost a decade ago but Oracle never really understood the value that SPARC and Solaris brought to the table.
I found this part particularly interesting: > Also, with 512 gigs of RAM and a massive CPU, it can run a 405 billion parameter Large Language Model. It's not fast, but it did run, giving me just under a token per second. If you're serious about running LLMs and you can afford it, you'll of course want GPUs. But this might be a relatively affordable way to run really huge models like Llama 405B on your own hardware. T…
It's really slow. Like, unusably slow. For those interested in self-hosting, this is a really good resource: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...
Really a pity that Oracle killed off SPARC. They already had 32-core CPUs almost a decade ago but Oracle never really understood the value that SPARC and Solaris brought to the table.
Ironically, Oracle seems to be the only cloud compute offering Ampere currently.
[1] https://azure.microsoft.com/en-us/blog/azure-virtual-machine...
Earlier quoted context omitted.
When was the last time you saw a server with 10GB Ram no matter the number of cores\threads?
>When was the last time you saw a server with 10GB Ram no matter the number of cores\threads? Is that a lot or a little? I have a bunch that only have 8gb, it just depends on what they are being used for.
Earlier quoted context omitted.
Ironically, Oracle seems to be the only cloud compute offering Ampere currently.
Azure seems to be offering Ampere-based offerings[1], as well as Hetzner[2]. [1] https://azure.microsoft.com/en-us/blog/azure-virtual-machine... [2] https://www.hetzner.com/press-release/arm64-cloud/
Earlier quoted context omitted.
To be fair, utilizing 192 cores for a single process or operation is often exceedingly difficult. Scheduling and coordination and resource sharing are all really hard with thread counts that high, so you're probably best operating in terms of smaller clusters of 4-16 threads instead. Lots of algorithms stop scaling well around the 8-32 range.
I use xargs -P every weekend to back up my (Oracle) database. It has 510 files that are around 2gb, and the parallel script uses rman to make a datafile copy on each one, lzip it, then scp it to my backup server. I have xargs set to run 10 at once. Could I increase to 192? Yes.
I found this part particularly interesting: > Also, with 512 gigs of RAM and a massive CPU, it can run a 405 billion parameter Large Language Model. It's not fast, but it did run, giving me just under a token per second. If you're serious about running LLMs and you can afford it, you'll of course want GPUs. But this might be a relatively affordable way to run really huge models like Llama 405B on your own hardware. T…
It's really slow. Like, unusably slow. For those interested in self-hosting, this is a really good resource: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...