Viewing profile — mcyc

Lichess has a checkmate captcha that I think is cute. It requires you to solve a mate-in-one puzzle to, e.g., post on the forums. (Sorry, don't have a better link, there wasn't any…

Yeah, it is interesting to me that it is coming from the _city_'s government. I've seen sovereign AI things at the country level, but this is the first municipal one I have seen.

You are right about most tokenizers being heavily biased towards English, but the situation is not so bad for Portuguese. Here are some results on the Goldfish corpus [1] with a fe…

Today, I launched blunder.clinic, a daily chess puzzle app that provides realistic positions for you to try to not blunder on. These are similar to traditional chess puzzles (i.e.,…

This is a fantastic guide! I did a lot of work on structured generation for my PhD. Here are a few other pointers for people who might be interested: Some libraries: - Outlines, a …

This is a nice attitude. I think HN is overall pretty nice for geeking out and also hearing other people geek out, but there is still a strain of elitism (not like StackExchange th…

You can cross whitespace boundaries by setting flag `--split-on-whitespace` to false (it's true by default). https://github.com/google/sentencepiece/blob/master/doc/opti...

Just a minor nit: SentencePiece is a library, not a tokenization algorithm. It implements two tokenization algorithms, Unigram and BPE. BPE builds vocabularies from the base up so …

People may also be interested in Pynini [1], a python wrapper (+ a lot of additional ease-of-use functionality) of OpenFst [2] (a really great library for transducers). There are s…

You might be interested in the Recurse Center ( https://www.recurse.com/ ) and the experiences of people who have gone through it (they heavily encourage blogging about your time t…

Thanks! Our paper [1] is kind of a goofy adversarial thing where we thought "here's this cool metric, how can we break it?". The tokenizers we propose are definitely not tokenizers…

It's two problems: 1) the sequence length increases too much. Idk what the average token length is for Llama, but imagine it's like 5+ bytes. Using individual bytes as tokens immed…