61–70 of 231 posts
Earlier quoted context omitted.
Yeah, and that doesn't even do the nutty IO on these things justice. A 16x CS2 cluster like they describe is like a huge Nvidia cluster in terms of throughput, but more like a single Nvidia node structurally.
It’s unbelievable stuff. Does anyone know how much a single box costs? They are selling them it looks like.
Earlier quoted context omitted.
They're dynamically scaled and something must be broken. If you inspect source you can find the raw images, here's a few: https://www.cerebras.net/wp-content/uploads/2023/03/Downstre... https://www.cerebras.net/wp-content/uploads/2023/03/Scaling-... https://www.cerebras.net/wp-content/uploads/2023/03/Scaling-... EDIT: Looks like it scores better with less training - up until it matches GPT-J/Pythia/OPT and doesn't ap…
Disabling ublock on Firefox fixes the issue for me. edit: scratch that, it seems the AJAX endpoint returns 504 more often that not.
Earlier quoted context omitted.
Serious question: is it typical to describe client-side computing as "on the edge"? I thought running something on the edge referred to running it in close network proximity to the user, rather than users having control and running things themselves.
Yes, "edge computing" can refer to both computing done as close to the user as possible geographically, or even on the device itself. If someone says "I wanna do edge computing" it's not clear enough to know if they just want to have servers they control as close to the user as possible, or do the computing on the device itself. I think Apple would say "edge computing" is on the actual device while CloudFlare would s…
It is kind of cyclical then is not?
By that I mean computers used to be shared and to log into it through a terminal.
Then the PC came around.
Then about 15 years ago Cloud computing became the rage (really an extension or more sophisticated system than the first time shared computers)
Now we're back to local computing. I even see more self hosting and moving away from cloud due to costs.
All that rant is to say is it's interesting.
Side note, getting this AI to be localized as much as possible I imagine will be really useful in the medical industry because it helps alleviate HIPAA requirements.
Comparing the 13B model here https://huggingface.co/cerebras/Cerebras-GPT-13B to LLaMA-13B https://github.com/facebookresearch/llama/blob/main/MODEL_CA... you can see that in all of the reasoning tasks Cerebras-GPT lags behind. Any reason to use Cerebras instead of LLaMA? Doesn't seem like it.
This is called a silver lining for some (in case you were worried about gpt taking your job). Privacy requirements alone will in the near term force major companies to run their own inference (if not training). The expertise required are nearly identical to that of running large scale distributed computational graphs.
This is an interesting diveragence from what happened with web. The backends started out simple before map-reduce and before deconstructing databases and processing distributed logs. With ML, we'll jump right into the complex backends in tandem with easy-picking early stage edge applications (which we see daily on HN).
Comparing the 13B model here https://huggingface.co/cerebras/Cerebras-GPT-13B to LLaMA-13B https://github.com/facebookresearch/llama/blob/main/MODEL_CA... you can see that in all of the reasoning tasks Cerebras-GPT lags behind. Any reason to use Cerebras instead of LLaMA? Doesn't seem like it.
This type of article (or press release, or whatever you want to call it) is exactly what makes the future so interesting. The cat is out of the bag, the genie is out of the bottle, the confetti has left the cannon[0]. It's tempting to see a world dominated by Google Bard, ChatGPT, Bing Search, etc. And no doubt, they will be huge players, with services that are far more powerful than anything that can be run on the e…
Yes, yes, and yes. I'm waiting for an actually open AI that can run on the edge, purely on commodity hardware like our laptops and phones - it's inevitable. I imagine this "cat out of the bag" situation, the democratization and commodification of powerful technology accessible and affordable to the public, is similar to what's happening with single-board computers and microcontrollers like Raspberry Pi, Arduino, ESP3…
These local models are absolutely critical for that to happen though. I'm hitting daily situations where I have to reconsider my use of ChatGPT because what I'm asking would leak very private personal information or somehow trip its morality filter.
Just swapped in a 2TB nvme for a separate /home mount and reinstalled Arch just so I could have enough disk space to test a dozen models locally. I'm so ready!
Earlier quoted context omitted.
Yes, "edge computing" can refer to both computing done as close to the user as possible geographically, or even on the device itself. If someone says "I wanna do edge computing" it's not clear enough to know if they just want to have servers they control as close to the user as possible, or do the computing on the device itself. I think Apple would say "edge computing" is on the actual device while CloudFlare would s…
I guess I've been out of the loop for a bit and didn't realize that "edge computing" became a term since cloud computing took off. It is kind of cyclical then is not? By that I mean computers used to be shared and to log into it through a terminal. Then the PC came around. Then about 15 years ago Cloud computing became the rage (really an extension or more sophisticated system than the first time shared computers) No…
> By that I mean computers used to be shared and to log into it through a terminal.
> Then the PC came around.
> Then about 15 years ago Cloud computing became the rage (really an extension or more sophisticated system than the first time shared computers)
There's a really neat article called "The Eternal Mainframe"[1] that you might be interested. It explores this idea in greater depth.
---
1. http://www.winestockwebdesign.com/Essays/Eternal_Mainframe.h...
Does the chinchilla recipe still hold today? I got the impression that the LLaMA paper proposed a different result where throwing far more tokens at the problem had a very meaningful impact, or did I misunderstand that?
Tldr; Chinchilla isn’t wrong, it’s just useful for a different goal than the llama paper.
There’s 3 hyper parameters to tweak here. Model size (parameter count), number of tokens pre trained on, and amount of compute available. End performance is in theory a function of these three hyperparameters.
You can think of this as an optimization function.
Chinchilla says, if you have a fixed amount of compute, here’s what size and number of tokens to train for maximum performance.
A lot of times, we have a fixed model size though though, because size impact inference costs and latency. Llama operates in this territory. They choose to fix the model size instead of the amount of compute.
This could explain gaps in performance between Cerebras models of size X and llama models of size X. Llama models of size X have way more compute behind them