151–160 of 301 posts
Earlier quoted context omitted.
20 tokens per second for eval time is the killer here. It means you can't use this to process any meaningful amount of text. A GPU typically processes close to 1000 tokens/s during eval.
I'm pretty sure eval time is token generation time where it's actually outputting new tokens. If you're getting a thousand per second on that, I'd love to know on what.
Earlier quoted context omitted.
this is sorta like saying that being able to run your blog on your laptop will completely implode the cloud business
This is actually what happens. I run my word processing software on my apple 2 (a total joke of a computer) instead of running it on the WANG. I run my book keeping software on visicalc instead of the IBM. I run my simulation software on my IBM PC (I even paid for the 8087!) instead of the VAX. Moore's law has, at least so far, allowed the pioneers with toy computers to grow their toys big enough to solve "big boy" p…
If a vendor can SaaS a solution, then enterprise is generally happy (they don't want to have to hire folks for maintenance), and that completely locks out any ability to run locally.
Between enterprise's ambivalence and the obvious financial incentive to vendors, you get SaaS-only products.
Earlier quoted context omitted.
> (purple on black is really hard to read) Noted, and agree (it looks like it has also already been clicked, which I dislike). I honestly I need to redo the themes. > You say it runs "at reading speed". Have you benchmarked it? At some point a few weeks ago, yes I think so, but I didn't write it down for some reason... so I'll have to find a time when it's not busy and do it again without a noisy system. Right now th…
20 tokens per second for eval time is the killer here. It means you can't use this to process any meaningful amount of text. A GPU typically processes close to 1000 tokens/s during eval.
It's probably too small for the timings to be taken seriously.
We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.
We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.
Earlier quoted context omitted.
It should be closer to 85W on load. And it's incredibly silent on even a low end cooler. I rarely get above 50° Celcius.
85W for the whole system?! The specifications for the CPU mention a TDP of 85W [1]. [1] https://www.intel.com/content/www/us/en/products/sku/92986/i...
Earlier quoted context omitted.
I'm pretty sure eval time is token generation time where it's actually outputting new tokens. If you're getting a thousand per second on that, I'd love to know on what.
From the prompt timings above, it seems like 'prompt eval time' is the equivalent to 'processing time for input tokens'. Hyperscalers can perform this evaluation very quickly because evaluation can be significantly parallelized. The layer `i` output of token `j` only requires access to the layer `i-1` output of all previous tokens, so a parallel frontier develops. Token (0,0) [(token, layer)] is processed first, then…
As someone doing this for fun on a windows 11 machine (96gb ram, 5090 24gb) I wonder if I need any flags to keep the model in memory and avoid swapping to ssd? I use LM studio and qwen3.5 35B - but never figured out if it is swapping or not. Om am unrelated note, does anyone know a model that can help with this use case: https://news.ycombinator.com/item?id=48301635