181–190 of 199 posts
P.S. Man the AI writing complaints on this thread are quite upsetting. OP is handling it a lot better than I would, lol.
The real question: did your local LLM write this post?
There are many tells aren't there? There was clearly hard human work and experimentation here, but it's a shame the OP let AI do chunks of the writing. Once you see it, it's much harder to take the post seriously.
There are no usable tells that apply generally in the first place. Pretty much all of the hyped-up memes in circulation about how to detect LLM output are highly unreliable.
LLMs write the way they do because they are trained on common patterns of human writing. All of the tropes people point to in LLM output are there precisely because they've already been in widespread use for some time.
Earlier quoted context omitted.
Opus is normally $5 per mtok, no idea why anyone would use /fast if they were at all concerned about price. ($5 is still pricy though tbh)
Opus is $5 per mtok of input tokens, but $25 for output.
The V100 and the 4090 are based on vastly different architectures, the former uses the older Volta while the latter uses Ada. Last I checked you cannot meaningfully combine them. The 3090 is better than the V100, just get two 3090 and a NVLink.
Well I did in fact meaningfully combined them without an issue, that was the whole point of the blogpost.
Earlier quoted context omitted.
Well I did in fact meaningfully combined them without an issue, that was the whole point of the blogpost.
Yes but it creates a bottleneck that negates the benefit of using multiple cards that way. Look into it. Cheers
The V100 and the 4090 are based on vastly different architectures, the former uses the older Volta while the latter uses Ada. Last I checked you cannot meaningfully combine them. The 3090 is better than the V100, just get two 3090 and a NVLink.
You can split tensors across an AMD GPU and Nvidia GPU - different architectures are not an issue. People run LLMs across some pretty crazy setups.
Earlier quoted context omitted.
Yes but it creates a bottleneck that negates the benefit of using multiple cards that way. Look into it. Cheers
Well it doesn’t matter because the bottleneck here is actually quite small for me. The issue is vram. If anything the bottleneck is my 4080.
Earlier quoted context omitted.
Well it doesn’t matter because the bottleneck here is actually quite small for me. The issue is vram. If anything the bottleneck is my 4080.
Gotcha, I am not saying your setup is inherently wrong or useless. I am glad it works for your use cases. Godspeed
Earlier quoted context omitted.
You can split tensors across an AMD GPU and Nvidia GPU - different architectures are not an issue. People run LLMs across some pretty crazy setups.
It depends but you cannot directly mix for example Ampere with Ada coz the lack of support for native FP8 in Ampere.
There are a whole lot of ways to quantize models in general.
Earlier quoted context omitted.
It depends but you cannot directly mix for example Ampere with Ada coz the lack of support for native FP8 in Ampere.
There are a variety of inference engines that support this, regardless of whether or not there is native FP8 in Ampere - llama.cpp will do it quite happily. VLLM you can do W8A16 quant too. There are a whole lot of ways to quantize models in general.