131–140 of 199 posts
Does anyone know if a torrent is available? I think it would take quite a while to download 1.5tb from their servers.
It's funny that we've finally returned to tanh activation functions, time is a circle.
From the paper (page 6 with a comparison to GLU and SwiGLU) they are not using tanh directly (i.e. f(x) = tanh(x)) but: f_gate(b,x) = b * tanh(x / b) * sigmoid(x) f_up(b,x) = b * tanh(x / b) Looking at the graph I wonder if this is to try and get the best of both GLU (better representation at higher values of x >~ 5) and SwiGLU (the value bump just before 0).
f_gate(x, y) = f(x) * y
f_gate(x) = 4 * tanh(x / 4) * sigmoid(x)
Earlier quoted context omitted.
Exactly. I wish people would just stop saying "open source" regarding models. Even "open weight" is disingenuous. "self-hostable" would be more honest.
It's a gradient almost, with some steps. So far, I think you could categorize every single released so far as one of: - Proprietary - No access beyond remote endpoints - Downloadable - You can run it, but there are restrictions and training data/code isn't public and/or under FOSS license, nor are the weights under a FOSS license - Open weights - The weights are under a FOSS license and downloadable without restricti…
Any guess on the pre-training tokens/flops they consumed?
Back of the envelope calculation (could be off, correct me if I am) If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 paral…
We've already saved money compared to last years token cost on Claude/Gemini
What would it take to get an American open model to compete with this?
Earlier quoted context omitted.
I wonder how they'll figure out who to target for litigation when this license is violated.
How many companies host and serve models via API and have a $20M+ revenue? Going to be pretty straightforward to catch offenders.
Earlier quoted context omitted.
It's a gradient almost, with some steps. So far, I think you could categorize every single released so far as one of: - Proprietary - No access beyond remote endpoints - Downloadable - You can run it, but there are restrictions and training data/code isn't public and/or under FOSS license, nor are the weights under a FOSS license - Open weights - The weights are under a FOSS license and downloadable without restricti…
"Downloadable" should be "weights available" by analogy to "source available". K3 is weights available because it includes usage restrictions. K2.6 is open weights, because the only restriction is a weaker version of the "obnoxious BSD advertising clause", which is accepted as FOSS.
Earlier quoted context omitted.
Yeah but you don't need two such people, or even one, dedicated to this single rack. A company of the size that this is worthwhile for, probably has dedicated devops on staff already and can add this rack to the inventory with no additional staff.
Who is going to upgrade the models ? Who is going to fix it when the api does something weird ? Who is going to proactively make sure it’s not overheating? Chat GPT has enterprise contracts for a reason.
> A company of the size that this is worthwhile for, probably has dedicated devops on staff already
Earlier quoted context omitted.
I think internal use is allowed at any scale in the license? > 4. The requirements set forth in Sections 2 and 3 do not apply to: (a) internal use of the Software, defined as any use that does not make the Software, its outputs, or its underlying capabilities available to third parties; [...]
It's very hard to make sure none of the outputs are ever made available to third parties. Source code can end up widely distributed (e.g. client-side js, open source). Prose will frequently get shared across organization boundaries (e.g. emails, websites, documents).