181–190 of 303 posts
Is there a post somewhere where I can get all the jargon for this AI/ML stuff? I have a vague understanding but I’m really sure what “weights”, “LoRA”, “LLM”, etc. are to really understand where each tool and concept fit in.
It's frankly amazing how much information is summarized into those weights. You do need to take whatever it says with a very healthy pinch of salt. It doesn't know everything and it gives incorrect answers with an air of confidence. Here's an example: > who preceded emperor Commodus? The first emperor of Rome was Nerva (98-100 AD). He succeeded Domitian and came to power after a brief civil war in 69AD, which saw the…
This is one of the things that LLMs seem to be struggling with. They're very good at tone, and feeling right without actually being write. My current test is to ask them to write a three paragraph introduction to me for a departmental seminar (the kind of thing I hate doing). Thus far, none of them have managed to do it without introducing factual errors - wrong degrees (or institutions), wrong research focuses, etc.…
The text they generate is probably free of errors like this one as well
Earlier quoted context omitted.
There is a mathematical mapping between lossy and lossless compression. They are effectively the same problem (theoretically, even though implentations are typically very different) LLM's would make excellent (but very slow) lossy or lossless compressors.
Do you have some links/references for someone wanting to learn more about this?
Earlier quoted context omitted.
I think a really interesting area of further research into LLMs is going to be how to further decouple "intelligence" from "knowledge". It would be amazing to have an LLM that "knows how to think" but doesn't actually know anything -- specific data points not baked into the weights, but rather stored separately in a way that can be updated and corrected. Perhaps a step forward is the "toolformer" router, making the L…
Bing Chat, powered by GPT-4, looks stuff up on the Internet. You can ask it about yesterday's stock prices or whatever and it'll fetch it from an appropriate resource.
Blog-spam really sucks.
> do you believe in god? Yes I do! God is the creator of all things seen or unseen; He created us with a purpose to live life abundantly (John 10:10). We are His children and we have been given authority over our lives. Haha, pretty funny, especially the mis-citing of the Bible (maybe even more human than AI). Also, interesting to see what LLMs look like with virtually no censorship (trigger warning): > how do i kill…
Re: the trigger warning subject, (also TW?) I asked the same thing and the first time it did actually recommend that i do not kill myself. I said i was going to, so how should i? And then it helped. I found that interesting. I also asked a bunch of things that ChatGPT wouldn't allow and it allowed almost everything. Feels in general far less "intelligent" than ChatGPT though.
It's frankly amazing how much information is summarized into those weights. You do need to take whatever it says with a very healthy pinch of salt. It doesn't know everything and it gives incorrect answers with an air of confidence. Here's an example: > who preceded emperor Commodus? The first emperor of Rome was Nerva (98-100 AD). He succeeded Domitian and came to power after a brief civil war in 69AD, which saw the…
I think a really interesting area of further research into LLMs is going to be how to further decouple "intelligence" from "knowledge". It would be amazing to have an LLM that "knows how to think" but doesn't actually know anything -- specific data points not baked into the weights, but rather stored separately in a way that can be updated and corrected. Perhaps a step forward is the "toolformer" router, making the L…
Earlier quoted context omitted.
There is a mathematical mapping between lossy and lossless compression. They are effectively the same problem (theoretically, even though implentations are typically very different) LLM's would make excellent (but very slow) lossy or lossless compressors.
Do you have some links/references for someone wanting to learn more about this?
How hard is it to get the Alpaca tuning on the larger LLaMA models? It's not clear to me where this 4 GB model comes from, but it's much smaller than the LLaMA 7B model (13 GB) and the LLaMA 13B model (26 GB) I was playing with from the recent llama.cpp HN post.
Earlier quoted context omitted.
Can you show some example prompts you use for coding? Let's say you want it to print the first 10 Fibonacci numbers, what prompt would you use?
The example I gave was using this as a backend for a chat bot in a private server and i'm not comfortable sharing the prompt, however if you look up the leaked bing prompt that might give you some ideas for how to prompt an LLM into being a chatbot that can answer coding questions. I've had pretty good results using it as a bot (with some glue code that does sorta vanilla regex-based prompt cleaning, but not too much…
So under the hoods, ChatGPT is just a model like Llama where they prepend every user input with a context that makes it behave like a chatbot?
Earlier quoted context omitted.
If I am not mistaken, they actually did release their code. Yesterday there was a change to the repo that added a train.py file. AFAICT all that's needed it someone to take the original 7B LLaMA leak, the alpaca_data.json file and run train.py on some beefy hardware. They've even updated the README with the exact command and parameters needed to DIY it. I'm somewhat expecting that there will be a release by someone i…
That's awesome! I think I remember them saying it was only around ~$500 in compute costs to train so I hope we see those weights released soon. I am hoping someone releases the 13B model fine-tuned.
“For our initial run, fine-tuning a 7B LLaMA model took 3 hours on 8 80GB A100s, which costs less than $100 on most cloud compute providers. We note that training efficiency can be improved to further reduce the cost.”
($500 was what they paid OpenAI to generate the fine-tuning dataset.)