Viewing profile — matthewolfe

What exactly doesn't "just work" anymore?

I believe SchemeFlow [0] is working on solving some of these problem, particularly with the insane reporting requirements. But of course, that still leaves the unions... [0] https:…

For argument's sake, suppose we live in a world where many high-quality models can be run on-device. Is there any concern from companies/model developers about exposing their propr…

To echo the other replies, the tokenizer is definitely not the bottleneck. It just happens to be the first step in inference, so it's what I did first.

haha, I thought about it.

Fair chance I'm remembering it wrong :D

Alright, 0.1.1 should now be a true drop-in replacement. I'll write up some examples soon.

Should be the same. Both use Byte-Pair Encoding (BPE) as underlying algo.

Ah good catch. Updating this right now.

Modal's GPU glossary is a good overview about how GPUs work [0]. Karpathy's LLM overview is a good high level overview on LLMs [1]. 3b1b's video (and subsequent videos) on transfor…

I'm working on incremental re-tokenizing next. Then I'll run some benchmarks against this crate too.

Agreed. A former mentor of mine told me a nice way of viewing software development: 1. Make it work. 2. Make it fast. 3. Make it pretty. Transformers & LLMs have been developed to …

Cool! I've reached out to the guy who maintains Tiktoken to talk about this.

A lot of model-specific tokenizers have reference implementations ([0], [1]). Underlying them is a core algorithm like SentencePiece or Byte-pair encoding (BPE). Tiktoken and Token…

I probably will. Was hesitant initially, because adding PCRE2 as a dependency might cause issues to existing projects. I believe this was discussed briefly in a closed PR with othe…

The output should be identical, assuming no bugs. The Tiktoken implementation takes a collection of all special tokens upon initialization and compiles them into a regex by joining…

Thanks, I clarified it.