Earlier quoted context omitted.
My calculator manages
Your calculator is deterministic. Humans and AI are not.
LLMs are deterministic. We just sample the results, no? Also, no reason AI needs not be deterministic.
Transformers Can Do Arithmetic with the Right Embeddings
151–160 of 215 posts
Re: Transformers Can Do Arithmetic with the Right Embeddings
#152Earlier quoted context omitted.
>> deductive reasoning is just drawing specific conclusion from general patterns. This is according to whom, please?
> Deductive reasoning is the process of drawing valid inferences. An inference is valid if its conclusion follows logically from its premises, meaning that it is impossible for the premises to be true and the conclusion to be false. https://en.wikipedia.org/wiki/Deductive_reasoning >
That's not the definition used by the comment above.
Re: Transformers Can Do Arithmetic with the Right Embeddings
#153Earlier quoted context omitted.
Doesn’t seem as complicated as, say, coding a lexer for C. And why shouldn’t tokenisation use lexers or an equivalent?
Good old software development. :( Recent case studies: - llama.cpp wasn't tokenizing properly, and it came to a head with llama3. Essentially every local model before May 2024 is soft-deprecated, new ones have to indicate the proper tokenizer, and that currently only covers a small subset of popular models - I recently had to review 41 Phi-3 and Llama 3 models, only 3 had the right tokenizer set Not saying it's impos…
Meanwhile, I just wrote a custom tokeniser for my fan control experiment.
It features such amusements as: - Tokens representing the current time of day and day of week, with half-hour granularity. [14:30][Monday], as the debugger reports. - An entirely separate set of numeric tokens for CPU usage and such, on a logarithmic scale. Also features tokens for digit position, measured from the right. - A hardcoded text tokeniser for executable paths. [/nix/store](..cut..)/bin/executable name. I didn't feel like using the usual approach, so I built a huffman compressor to generate the tokens for arbitrary text, because why not. - Tokens representing program state - "just started", "long-running", etc. - Tokens representing the fact that the following text is from `tail -f ~/.bash_history`. - Start-of-segment tokens for each of the above, and also for GPU and CPU core complex power usage.
It's not that many tokens in total, and the input is structured data, so why not represent it as such? I still had sixty-five thousand tokens for the text tokeniser.
Re: Transformers Can Do Arithmetic with the Right Embeddings
#154I like to see more focus on the input embeddings.
It's basically the same as feature engineering in pre-deep machine learning: constructing features with high information content can significantly reduce the amount of data and computation needed to fit a useful model. And sometimes it's impossible to fit a useful model without careful feature engineering, either because the model itself is constrained in some way or because there isn't enough data or both.
It's analogous to making a choice of inductive bias within the model itself. We literally could not do LLMs without the carefully-constructed transformer architecture. Why should we expect to make further progress without paying more attention to the embeddings?
Re: Transformers Can Do Arithmetic with the Right Embeddings
#155I like these kinds of fixes. It's like realizing your child has vision problems and getting them glasses.
Re: Transformers Can Do Arithmetic with the Right Embeddings
#156Re: Transformers Can Do Arithmetic with the Right Embeddings
#157Wow, a lot of grumpiness in here. If it's true that adding like 20 or so tokens to encode column location / decimal spot triples math performance in out of band tasks, that's a big deal. It's a simple fix, it improves performance A LOT, and they even indicate it's not just a party trick, in that the LLM can use the information to do better on related tasks like sorting and list making. This is basically free to add,…
engineering vs. science -> scientist-types find such hacks ugly whereas engineers have to pay bills and get things moving fast.
Re: Transformers Can Do Arithmetic with the Right Embeddings
#158Earlier quoted context omitted.
> Problem is the current systems can’t reason about things Sounds like the AGI argument trap: They're not able to reason, but we can't succintly define what it is. I don't come with a reasoning chip. Whatever I call reasoning happens as a byproduct of my neural process. I do think that the combination of a transformer network and calls to customized reasoning chips (systems that search and deduce answers, like Wolfra…
> They're not able to reason, but we can't succintly define what it is. For transformer-based LLMs, and most LLMs there's an obvious class of problems that they cannot solve. LLMs generally perform bounded computation per token, so they cannot reason about computational problems that are more than linearly complex, for a sufficiently large input instance. If you have a back-and-forth (many shot) your LLM can possibly…
Humans have the same limitation and use same solution: showing your work and taking notes. There's no blocker here.
Re: Transformers Can Do Arithmetic with the Right Embeddings
#159I just wonder if numbers were written right to left, llms would be much better at arithmetic. You can 'predict' the least significant digit by reusing the already written digits in the computation, but to generate most significant ones, you generally need to do the entire computation in one go.
Left to right is fine, but it takes more work (multi shot) to do carries.
17 + 14 = 20 + 11 = 30 + 1 = 31
vs 17 + 14 = 10 + 10 + 10 + 1 = 31
Re: Transformers Can Do Arithmetic with the Right Embeddings
#160Earlier quoted context omitted.
Exactly
The point is don't ask an LLM to do tasks that a calculator can do. Ask if to use the calculator, just like most humans would.
The point is that you shouldn't need to ask. If it's actually a general-purpose system it will do it automatically.