"I’m teaching myself LLM internals by re-implementing the stack from first principles." - curious what resources you're using? Any books or courses, or just building it straight up? Great work!
Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
31–40 of 79 posts
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#32Kudos, I think (in the short term at least) there is a large amount of perf. optimization to be found by coding parts of the whole AI/ML infrastructure in C++ like this one, not as a rewrite (god no!) but drop in and fix key bottlenecks. Anytime I see someone (seems Chinese engineers are good at this) put something out in C++, good chance some solid engineering tradeoffs have been made and dramatic improvement will b…
And while we’re at it, let’s move away from Python altogether. In the long run it doesn’t make sense just because it’s the language ML engineers are familiar with.
No! This is not good.
Iteration speed trumps all in research, most of what Python does is launch GPU operations, if you're having slowdowns from Pythonland then you're doing something terribly wrong.
Python is an excellent (and yes, fast!) language for orchestrating and calling ML stuff. If C++ code is needed, call it as a module.
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#33Earlier quoted context omitted.
Agreed. A former mentor of mine told me a nice way of viewing software development: 1. Make it work. 2. Make it fast. 3. Make it pretty. Transformers & LLMs have been developed to a point where they work quite well. I feel as though we're at a stage where most substantial progress is being made on the performance side.
Heh, seems people I've been learning from been biased away from beauty, as I know that as "Make It Work, Make It Right, Make It Fast".
What's the difference between make it work and make it right? Aren't they the same thing?
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#34Earlier quoted context omitted.
Heh, seems people I've been learning from been biased away from beauty, as I know that as "Make It Work, Make It Right, Make It Fast".
What's the difference between make it work and make it right? Aren't they the same thing?
Yeah, if it's not right, it doesn't work.
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#35Earlier quoted context omitted.
Heh, seems people I've been learning from been biased away from beauty, as I know that as "Make It Work, Make It Right, Make It Fast".
What's the difference between make it work and make it right? Aren't they the same thing?
Making it work can be a hacky, tech debt laden implementation. Making it right involves refactoring/rewriting with an eye towards maintainability, testability, etc etc
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#36Earlier quoted context omitted.
What's the difference between make it work and make it right? Aren't they the same thing?
Yeah, if it's not right, it doesn't work.
In ML, often it does work to a degree even if it's not 100% correct. So getting it working at all is all about hacking b/c most ideas are bad and don't work. Then you'll find wins by incrementally correcting issues with the math / data / floating point precision / etc.
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#37Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#38"I’m teaching myself LLM internals by re-implementing the stack from first principles." - curious what resources you're using? Any books or courses, or just building it straight up? Great work!
[deleted]
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#39Earlier quoted context omitted.
What's the difference between make it work and make it right? Aren't they the same thing?
Yeah, if it's not right, it doesn't work.
Not true. Things can work with hacks. Your standards might consider it unacceptable to have hacks. So you can have a “make it right” stage.
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#40Earlier quoted context omitted.
Heh, seems people I've been learning from been biased away from beauty, as I know that as "Make It Work, Make It Right, Make It Fast".
What's the difference between make it work and make it right? Aren't they the same thing?
> make it work and make it right?
My mentor used say it is the difference between a screw and glue.
You can glue some things together and prove that it works, but eventually you learn that anytime you had to break something to fix it, you should've used a screw.
It is trade off in coupling - the glue binds tightly over the entire surface but a screw concentrates the loads, so needs maintenance to stay tight.
You only really know which is "right" it if you test it to destruction.
All of that advice is probably sounding date now, even in material science the glue might be winning (see the Tesla bumper or Lotus Elise bonding videos - every screw is extra grams).