181–190 of 315 posts
> LLMs can generate valid JSON 100% of the time If that seems surprising, it is worth doing a course like Karpathy's zero to hero NN, and have all the magic peeled away a layer at a time. The reason you can do this is because LLMs don't just generate the next word or token, it produces a probability distribution over all tokens. A JSON parser can give you a list of next valid tokens. The tokens in each case might be…
[deleted]
So to explain this another way: After each token generated by the LLM you update the logit bias “mask” to only allow the next token to be a valid json token? Very slick!
edit: They describe this more carefully in the paper.
> LLMs can generate valid JSON 100% of the time If that seems surprising, it is worth doing a course like Karpathy's zero to hero NN, and have all the magic peeled away a layer at a time. The reason you can do this is because LLMs don't just generate the next word or token, it produces a probability distribution over all tokens. A JSON parser can give you a list of next valid tokens. The tokens in each case might be…
Maybe it's just me, but I'm not doing anything that calls itself 'zero to hero'. Would love some good resources (preferably textbook, or at least written) on LLMs though. I don't even understand the link to 'generative' image/video AI, which seems to have exploded at roughly the same time and surely isn't a coincidence. I studied a little (literally 'intro to') ML at university, about enough to grok it as an applicat…
I can understand that you might be interested in book form only, I was lile this for the longest time, until I bumped into some really high quality video series that changed my mind to be a bit more flexible.
Also Important to note that LLM development is moving at a very fast pace recently so a book form might not be ideal. The basic ideas might be same, most of it might be out of date 6-12 months from now. I dont see how anyone could write a quality book that covers everything on this.
Earlier quoted context omitted.
Care to explain how a string replacement algorithm relates to nudging the logits of a ML model? I don't see the "rip off", the paper you cite requires a complete document to work on while this work is for guiding the generation of tokens
Both papers use the phrase "regular expressions" and there the resemblance ends. The linked manuscript uses regular expression to realize a grammar and then memoizes logic masks. I want to know why FlashText failed to cite: Baeza-Yates, Ricardo A., and Gaston H. Gonnet. "Fast text searching for regular expressions or automaton searching on tries." Journal of the ACM (JACM) 43.6 (1996): 915-936. Eltabakh, Mohamed Y.,…
Earlier quoted context omitted.
Both papers use the phrase "regular expressions" and there the resemblance ends. The linked manuscript uses regular expression to realize a grammar and then memoizes logic masks. I want to know why FlashText failed to cite: Baeza-Yates, Ricardo A., and Gaston H. Gonnet. "Fast text searching for regular expressions or automaton searching on tries." Journal of the ACM (JACM) 43.6 (1996): 915-936. Eltabakh, Mohamed Y.,…
Your comment here doesn’t feel like it’s in good faith, but there’s a good chance I’m misreading it.
I don't think it's fair of you to criticize the authors for not citing some obscure preprint, when that manuscript itself neglected to cite decades of prior, relevant work.
Earlier quoted context omitted.
The problem with Llama 2 chat versions is that they have been RLHF-ed to death. You can't ask questions without getting a sermon of how your question may be inappropriate for this or that reason. I think it's worse on the smaller models, but still present in the 70B one.
Apologies if you’d already seen this and were only trying to make a point, but you might like this article from a week or 2 ago that talks about how to run Llama 2 “uncensored” locally, and it seems to do a decent job of mitigating the sermons! Article: https://ollama.ai/blog/run-llama2-uncensored-locally Discussion: https://news.ycombinator.com/item?id=36973584
I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine times out of ten isn't good enough. Occasionally it will hallucinate responses like this: {"key1": "value1", "key2": "value2" for i in range(n)} Re-prompting with the parsing error message is usually enough to get it on the second try. But escaping double-quote…
Earlier quoted context omitted.
How do you imbue XML with meaning?
XML Elements themselves: their naming, their attributes, comments, indentation. There's more opportunity at every level of the hierarchy to demarkate and establish meaning. Having closing-tags as well, I've found, is a massive boon; LLMs can better understand what "finishing" looks like if its delimited in a semantic way - with a name.
I'm actually partial to CSV these days though, it can really cut down on response times just not needing to return all the extra tokens for JSON/XML delimiters
Earlier quoted context omitted.
Can anyone recommend a good, and trusted UI so I can use it via the API? I don't want to pay monthly for it, but would be nice to use occasionally. I keep meaning to do this!
OpenAI has its own playground where you can test all models (I believe GPT-4 is not available to everyone yet): https://platform.openai.com/playground Monthly subscription is only for ChatGPT. When you use the APIs you pay per token.
I still don't have access, except through the regular ChatGPT interface, which is mildly annoying. It would be interesting to experiment with the API.