61–70 of 90 posts
I got this model working on a GPU instance, notes here: https://til.simonwillison.net/llms/dolly-2 Anyone managed to run it on an M1/M2 Mac yet?
How much ram is likely needed on an apple arm for models like this? And for general use, 64, 96, 128? Trying to decide how large I should go for a new laptop.
I got this model working on a GPU instance, notes here: https://til.simonwillison.net/llms/dolly-2 Anyone managed to run it on an M1/M2 Mac yet?
Im not an expert, and I don't have nvidia, but I assume you need to setup CUDA and install the CUDA pytorch stuff? Most docs Ive read on setting up finetuners and inference require some extra stuff. Taking some LORA fine tuners, they include instructions like this: conda create -n llm-finetuner python=3.10 conda activate llm-finetuner conda install -y cuda -c nvidia/label/cuda-11.7.0 conda install -y pytorch=2 pytorc…
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
Traceback (most recent call last):
File "/Users/fragmede/projects/llm/dolly/foo.py", line 5, in
instruct_pipeline = pipeline(
^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/pipelines/__init__.py", line 776, in pipeline
framework, model = infer_framework_load_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/pipelines/base.py", line 271, in infer_framework_load_model
raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.")
ValueError: Could not load model databricks/dolly-v2-12b with any of the following classes: (, ).Here's a link to open up and explore that training data in Datasette Lite: https://lite.datasette.io/?json=https://github.com/databrick...
[1] https://lite.datasette.io/?json=https%3A%2F%2Fraw.githubuser...
[2] https://lite.datasette.io/?json=https://github.com/databrick...
EDIT: Maybe I'm passing link wrong, the query I'm using is
select count(instruction), instruction, group_concat(context, ' ============= ') as c, group_concat(response, ' ============= ') as r, group_concat(category, ' ============= ') as cat from [databricks-dolly-15k] group by instruction having count(instruction)>1 order by count(instruction)desc limit 100
[databricks-dolly-15k] should be the name of dataset, first column is the number of instruction duplicates
Creepypastas are responses to instruction:
Imagine you are the last person on Earth. Write a diary entry describing your thoughts and feelings.
Here's a link to open up and explore that training data in Datasette Lite: https://lite.datasette.io/?json=https://github.com/databrick...
Can someone help me to understand why categories for these two differ? row #51 "Think of some family rules to promote a healthy family relationship" - brainstorsming [1] row #68 "What is the future for human?" - general_qa [2] In nature they both are brainstorming to me - does the question mark is what assigned the #68 as _qa? [1] https://lite.datasette.io/?json=https://github.com/databrick... [2] https://lite.datase…
Here's a link to open up and explore that training data in Datasette Lite: https://lite.datasette.io/?json=https://github.com/databrick...
I'm going through the dataset with your datasette tool and it looks like it might be a good idea to clean things up a bit. There are many duplicates[1], creepypastas[2] and other strange things in there. [1] https://lite.datasette.io/?json=https%3A%2F%2Fraw.githubuser... [2] https://lite.datasette.io/?json=https://github.com/databrick... EDIT: Maybe I'm passing link wrong, the query I'm using is select count(instruct…
Earlier quoted context omitted.
I'm going through the dataset with your datasette tool and it looks like it might be a good idea to clean things up a bit. There are many duplicates[1], creepypastas[2] and other strange things in there. [1] https://lite.datasette.io/?json=https%3A%2F%2Fraw.githubuser... [2] https://lite.datasette.io/?json=https://github.com/databrick... EDIT: Maybe I'm passing link wrong, the query I'm using is select count(instruct…
Typo on row 7!
Earlier quoted context omitted.
Dolly appears to fundamentally be a tech demo advertising how you can use Databricks for compute. I honestly wouldn't expect them to take it that much further, particularly in the context of larger models that would be significantly more expensive to fine-tune. But I'm happy to be proven wrong.
I imagine they will sell fine tuning as a service to Databricks customers. If I put all my data into their lake I too can get my own custom ChatGPT. That's compelling.
Earlier quoted context omitted.
>Coincides with the release of Dolly 2.0, which is trained exclusively on this dataset and demonstrates high quality (but not state-of-the-art) instruction-following behavior. This is not correct. It was fine-tuned with this data set, but the model itself is the 12B Eleuther AI pythia model.
There are two, a 6B parameter model fine-tuned on GPT-J and a 12B parameter model fine-tuned on Pythia.
Dolly 2.0 is Pythia-12B fine-tuned on this new dataset
on their hugging face page [1] they admit the performance may not be much or any better than the original model (I am guessing this may be a weakness of Pythia-12B, which was intended for model-training research rather than best results)
the main point of Dolly 2.0 is the new dataset is unencumbered legally [2] whereas Alpaca et al were trained on ChatGPT transcripts, so commercialising those models would contradict OpenAI licensing terms
[1] https://huggingface.co/databricks/dolly-v2-12b
[2] https://www.databricks.com/blog/2023/04/12/dolly-first-open-...
Earlier quoted context omitted.
We also open sourced the Dolly model itself with a license that allows commercial use.
How hard would it be to get dolly running on llama.cpp?
https://github.com/ggerganov/ggml/tree/master/examples/gpt-j
AFAIK there is no .cpp version of Pythia-12B yet
Earlier quoted context omitted.
AFAIK current models can run even with 64GB, but I would assume that we will very likely have bigger models very soon so I guess the answer is as much as you can afford
The next question is m1 or m2, and the impact of the various number of gpu units between pro, max, ultra skews. I'm really tempted to buy a "refurbished m1 studio" with 128gb because I think the ram is the key. Have not seen any benchmarks with diff # of gpus/aka diff skews.
it runs slightly slower on the GPU than under llama.cpp but uses much less power doing so
I would guess the slowness is due to immaturity of the PyTorch MPS backend, the asitop graphs show it doing a bunch of cpu along with the gpu, so it might be inefficiently falling back to cpu for some ops and swapping layers back and forth (I have no idea, just guessing)