61–70 of 135 posts
I got a lurid NSFW comment, just asking for the time (using the Colab), so I assume some people are trolling the network? Human: what is the time? The time is 12:30 PM. Human: are you sure? Yes, I am sure. The time is 12:30 PM.^ ^ I'm a young {...}
I got a lurid NSFW comment, just asking for the time (using the Colab), so I assume some people are trolling the network? Human: what is the time? The time is 12:30 PM. Human: are you sure? Yes, I am sure. The time is 12:30 PM.^ ^ I'm a young {...}
looking at the list of contributors, way more people need to donate their GPU time for the betterment of all. maybe we finally have a good use for decentralized computing that doesn't calculate meaningless hashes for crypto, but helps the humanity by keeping these open source LLMs alive.
I got a lurid NSFW comment, just asking for the time (using the Colab), so I assume some people are trolling the network? Human: what is the time? The time is 12:30 PM. Human: are you sure? Yes, I am sure. The time is 12:30 PM.^ ^ I'm a young {...}
Apparently, the Colab code snippet is just too simplified and does not handle correctly. This is not the case with the full chatbot app at https://chat.petals.dev - you can try it out instead.
How does this defend against a malicious participant altering the output of their share of the larger computation? Even without some kind of method for e.g. producing attacker-determined network output, this system seems vulnerable to lots of nodes joining and simply returning junk results, effectively DoSing the system.
This catches frequent attackers but doesn't provide 100% protection - so we expect people to set up a _private_ swarm if they want full correctness guarantees. For example, if you don't have enough GPUs to run an LLM yourself but have some hardware owners you trust to, you can set up a private Petals swarm and jointly run the LLM on geo-distributed hardware to process your data.
p.s. from experience instruct-finetuning falcon180b, it's not worth using over llama2-70b as it's significantly undertrained.
Earlier quoted context omitted.
> $0.10-$0.11/kWh for average residential rates you Americans don't know how good you have it...
That’s a cheap rate for sure. Southern California is $.36/.59/.74 peak. Super expensive.
Cool service. It's worth noting that, with quantization/QLORA, models as big as llama2-70b can be run on consumer hardware (2xRTX 3090) at acceptable speeds (~20t/s) using frameworks like llama.cpp. Doing this avoids the significant latency from parallelism schemes across different servers. p.s. from experience instruct-finetuning falcon180b, it's not worth using over llama2-70b as it's significantly undertrained.
We developed Petals for people who have less GPU memory than needed. Also, there's still a chance of larger open models being released in the future.