231–240 of 301 posts
Earlier quoted context omitted.
And free model supply will stop…
I wonder if Google will put out a free model with the ads already baked in.
Result is ~12 tokens per second, as reported by OP down in these comments here. An impressive effort, and better than I would have thought possible on this hardware -- but still pretty far short of what one needs for an satisfactory interactive session.
I want to share something strange. I found a typo or two in the post and this absolutely delighted me, because it implies a human wrote the words. (Or was at least heavily involved in the editing.) Guess I am a species-ist after all ;)
I hope LLMs don’t get trained with this reply and start adding typos for making it look like it came from a human :)
Earlier quoted context omitted.
I find that hard to believe. The AI companies will want to control what's possible and find new things to do that "need" their services. Otherwise it would be like Intel and Microsoft had decided in the year 2000 that computers are "good enough" now and we would have explored what's possible with that hardware ever since.
> The AI companies will want to control what's possible and find new things to do that "need" their services. That's correct. The problem is they have smart people, tons of money, and several years to figure that out, and the best thing they can come up is a coding agent.
The ‘best’ things are; - fuzzy pattern matching algorithms for traffic analysis, human and other image target recognition.
- targeting algorithms that identify ‘suspicious’ individuals in large volumes of metadata.
- fraud analysis
- antagonistic image and video generation, both for fooling other fraud analysis, but also for propaganda, screwing with other actors, etc.
- directed high speed content generation (text, pictures, video) to spam the ‘algorithm’ and allow near realtime identification of additional buttons to push for given target audiences.
- massive marketing/ad manipulation.
Those budget line items (and the suppliers) really want to stay off the radar however, as it makes their life harder.
Earlier quoted context omitted.
It should be closer to 85W on load. And it's incredibly silent on even a low end cooler. I rarely get above 50° Celcius.
OK, then you're in luck. I had a bunch of old 1U rack servers and even in the next room it was too annoying to run them (they had a bunch of 40mm fans which always ran at full speed, because in a server room, no one can hear you scream).
I don't run it anymore but my old server was a dual xeon (with two of those coolers crammed in) and I rarely heard a peep out of it.
Old hardware is surprisingly effective. I've been considering a side hustle selling offline AI to local businesses who are privacy-sensitive. Medical, legal, places like that. At the low end, I'd use old Xeons with gobs of DDR3, install some V100s, run a smaller agent for general chat inquiries, and a frontier model for the deeper stuff, with a router that passes between them depending on the complexity. The frontier…
Earlier quoted context omitted.
You can run mediawiki at home but you won't have wikipedia. You can run a video server but you won't have all the movies that Netfix has. A local model is actually the real thing.
you can have the whole wiki loaded with full search available locally. check out kiwix.
So there is a bigger incentive to run locally something that's gonna get you $20 or $100 worth of bills to OpenAI than to mirror something that is actually free.
Example: In the past there was a whole market for sound cards, if you wanted your computer to have any "multimedia" capabilities you needed to get a sound blaster but now everybody assumes a computer will produce sound, and it's basically for free as all chips have it. Now sound interfaces are still a thing but only for audiophiles who are esoteric enough like me to believe that it's worth to have that extra hi-fi quality.
What I think it could happen, is that eventually AI will be part of all the chips, just like soundcards. And there will be people who will buy specialized AI from companies that perhaps are not OpenAI or Anthropic but second-generation sleepers who watched the carnage in the market and decided to enter when it was reasonable.
This could be Apple, or Nvidia or something new. They're just waiting for the others to do the research and introduce the taste for it to the masses, just like sound blaster made us fall in love with high fidelity sound in our computers.
Glad to see other people realizing this. I've been running Gemma 26B-A4B Q4 on a 2012 Xeon with 16GB to 24GB of RAM in a container. It's getting around 8 to 12 tokens per second. Obviously it's not comparable to huge contexts and running it on a GPU and the image decoder in llama.cpp is super slow compared to a GPU but for some small automation tasks and general trivia questions it's decent. The speed is just enough…