131–140 of 346 posts
Sadly until china scales production of hardware it really isn’t economical to run this stuff yourself. It is good it exists though to put pressure against the labs. Honestly imo this is just proof apple will win in the end. Eventually a phone will be able to run a model good enough to do most things and it then is game over.
Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive at…
That's going to hit first amendment grounds pretty quick, the same way that software in general did.
The modern version of the decss flag will be a character that says "I think good weights are {...weights go here...}"
They could, however, ban any payment to a chinese entity, or any entity owned by a chinese entity for inference/ai services/etc
Earlier quoted context omitted.
I was in this state a few weeks ago. I spent a bit of time familiarizing myself then wrote up my learnings as a series of exercises. If you think of docker as "kinda like vms except not really" and k8s as "kinda like deploying and composing docker containers but not really", this may be for you: https://ojensen.net/infra/understanding-k8s-1 It's actually really neat, i wish i had bothered to learn it years ago.
I appreciate the effort and I'm in your target audience, but that document didn't help me. It seems to dive into the details of installing and running k8s without saying much about the purpose. From my very ignorant standpoint, K8s seems to be about running a "cluster", but I don't know why I would want to do that.
It allows you to stop caring about the individual machines, and just treat them as combined compute, which starts mattering if you leave a single machine setup and need to start thinking about scaling in and out and gluing the individual parts together. Then you have known abstractions to do it.
Of course you can do everything kubernetes does using a bespoke solution, and the concepts aren't new, but having a widely supported technology has a lot of advantages and creating something with even half the feature has a high chance of just being worse.
Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive at…
And that list of tasks grows smaller every day
> Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive attempt to ban them by origin.
Historically just asking it about tianment square or getting some random answers turn into chinese (as latest interation of online deepseek likes to do recently) is enough
> Now there are solutions for that latter problem. But they are all ugly and restrictive. Making a DRM-like license protection system mandatory can be a solution.
I am very worried that's where consumer hardware will go to. All so AI companies can license local use of their stuff, and once that happens, less of an incentive to even have model be open.
Possibly even have DRM that counts number of computation done per model in pay per use model
Sadly until china scales production of hardware it really isn’t economical to run this stuff yourself. It is good it exists though to put pressure against the labs. Honestly imo this is just proof apple will win in the end. Eventually a phone will be able to run a model good enough to do most things and it then is game over.
https://www.amd.com/en/developer/resources/technical-article...
At 7 months of claude code subscription per node, the cluster pays for itself in 28 months. On a 5 year (60 month) depreciation schedule, you can buy two of those clusters for basically break even, so you get two concurrent request streams (each of which can batch, etc).
The next generation hardware has already been announced, and should ship roughly two Moore’s law doublings later. It’s likely its steady state price is So, once the bubble pops (because the financial machinations eventually will come to an abrupt halt), and the labs stop buying hardware for data centers, local inference will be extremely practical and cheaper than a subscription.
My main question is, when that happens, will UNIX Surplus be selling inference servers for pennies on the dollar (like after the dotcom crash), or are the power requirements too exotic for home use?
Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive at…
> “The source described leading AI labs or their allies approaching the administration every 3-5 months with an idea to ban open-source models.” That's going to hit first amendment grounds pretty quick, the same way that software in general did. The modern version of the decss flag will be a character that says "I think good weights are {...weights go here...}" They could, however, ban any payment to a chinese entity…
Earlier quoted context omitted.
I'm using qwen 3.6 35B unsloth 4 bit with my 5950x (128 gb memory) and a 3060 12 gb gpu with a self made harness. At 10k context I get about 40 tps generation and 500 tps prefill. At 100k context I get about 25 tps generation and 400 tps prefill. It works, but I often use gpt or claude to make a detailed enumerated plan of what I want to do first, then have qwen follow it. I'm not sure if it is economical or not, but…
> The biggest benefit for me is that it's all done locally, and I know the harness is not uploading anything or sending telemetry to someone else. Are there any articles you’d recommend for this? I have Qwen running on an HP Z8. Very nice platform. I have mine in a sandbox, due to privacy fears. Your solution sounds more elegant.
I really just iterated over the harness over and over for about two weeks with opencode until I was sort of satisfied (still lots to do there :).
For the llama.cpp I asked claude fable to optimize it for my hardware and iterated a few times. In the end I landed on the following: https://pastebin.com/2PpJFUC0
Earlier quoted context omitted.
Most unmature markets aren't subsidized to the point that LLM market is, most market have some level of baseline profitablity, this market doesn't, that's because most market subsidized the marketing or the capex but this market doesn't hold the opex, the capex not the amount of marketing let alone all of this together
Most new markets are funded by initial investment capital. Early entrants operate at a loss as they grow. This isn’t as unusual as some people are trying to make it sound. This has been happening since the dawn of finance. I thought this would be less foreign to everyone since we just went through this whole conversation for a decade with Uber and Lyft. Their demise was predicted from the start from everyone who thou…
Because that just absolutely murders any competition that manages to not get that level of free money. You're not pouring money in to make it happen at all at that point, you are pouring money in so nobody else can get the part of the pie.
Which is great for investors, bad for everyone else