Viewing profile — borzunov

Hi, a Petals dev here. You're right, there's no point in using Petals if your machine has enough GPU memory to fit the model and you're okay with the quantization quality. We devel…

Hi, a Petals dev here. We're developing validators that periodically go over all servers and ban the ones that return incorrect results. Additionally, clients can run data through …

Hi, a Petals dev here. means "end of sequence" for LLMs. If a model generates it, it forgets everything and continues with an unrelated random text (I'm sorry to hear that the mode…

We've moved to a new domain, the chat is now at https://chat.petals.dev

Chat web app: http://chat.petals.ml Colab: https://colab.research.google.com/drive/1uCphNY7gfAUkdDrTx21... Project description: https://github.com/bigscience-workshop/petals

Consider making a Streamlit or Gradio app and hosting it at Hugging Face Spaces: https://huggingface.co/spaces/launch This allows you to make a simple frontend using Python only, t…

Sure! The point system is being developed, we'll ship it soon. Once it's ready, you'll be able to spend points on high-priority requests to increase the speed.

Right now most CPUs are orders of magnitude slower than GPUs for doing forward/backward passes, so you're unlikely to get a similar speed. Some kind of pruning may help though.

A Hivemind/Petals dev here. As far as I understand, most federated learning methods can't efficiently train very large models (with billions of parameters) because they repeat some…

A Petals dev here. FlexGen is good at high-throughput inference (generating multiple sequences in parallel). During single-batch inference, it spends more than 5 sec/token in case …

A Petals dev here. Recent models indeed outperform BLOOM with less parameters (for English). However, the largest LLaMA still doesn't fit into one consumer-grade GPU, and these mod…

A Petals dev here. We say up front that "Single-batch inference runs at ≈ 1 sec per step (token)". In turn, "parallel inference" refers to the high-throughput scenario when you gen…

While I agree that throughput-focused scenarios exist and this work may be valuable for them, I still think that the repository can be improved to avoid "overselling". The fact tha…

I'm afraid that, unlike proprietary APIs and Petals, this system can't be used for single-batch inference of 175B models with interactive speeds - the thing you actually need for r…

Note that the authors report the speed of generating many sequences in parallel (per token): > The batch size is tuned to a value that maximizes the generation throughput for each …

During the training, participants only exchange tensors (embeddings, gradients) and never send code to each other. No other peer can execute arbitrary code on your computer - they …

It varies from time to time. You can also switch to the few-shot mode to try machine translation, code generation, or other tasks involving longer responses

Chat bot interfaces are only a small part of what can be done with large LMs. You can use and fine-tune them to solve almost all existing natural language processing tasks: machine…

Theoretical best-case for RAM offloading is 5.5 sec/token, for SSD offloading - 22 sec/token. Implementations we've tested are not faster than 10 sec/token though. See details in o…

In case of offloading, the computations are usually still performed on GPU, but the model is hosted in RAM/SSD instead of the GPU memory (and its chunks are copied to the GPU memor…

BLOOM is a large LM, and Petals is a tool for running large LMs (not necessarily BLOOM). People using Petals should still follow the model's terms of use regardless of how the tool…