Viewing profile — cschmidt

Send me an email (my address is in my profile)

Send me an email (my address is in my profile)

Send me an email (my address is in my profile)

I’m not sure why you think this is ai slop. I work on tokenization research full time. My name is Craig Schmidt and I have a number of papers in the field. This researcher has done…

Can I say this seems to be fantastic work. I cloned your repo earlier today after seeing it on the tokenization discord. I know everyone in the tokenization community wants to abso…

I suggest Scholar Inbox. https://www.scholar-inbox.com/landing It is a recommendation system for new papers that come out each day. If you train it a bit by specifying what you lik…

It all depends on where you get your news. You need somewhere that puts things into context so you can understand why things are happening. Personally I really like the Economist. …

It does seem like those 20% are exactly on the nasty side. So even more impressive.

The "Quantitative Display of Information", which I just checked, is using Monotype Bembo. So still Bembo, but a different version.

Those are not global students. Those are people who are already living in the state. Foreign students typically pay the most tuition possible with no financial aid, subsidizing eve…

Looks great. I just ordered it. Thanks for the recommendation.

There are equal weight S&P ETFs, which avoid having a handful of stock dominating. However, they do have to do a lot more rebalancing to keep things in line.

There is other research that works with pixels of text, such as this recent paper I saw at COLM 2025 https://arxiv.org/abs/2504.02122 .

I worry how often that is happening already on Spotify.

I’m not sure about this masters program, but the undergrad program seems to be proper ORMS.

I think in this context Management Science is an older term that was synonymous with operations research. The flagship journal of Informs (the institute for operations research and…

Attention does help, which is why it can learn arithmetic, even with arbitrary tokenization. However, if you put it in a standard form, such as right-to-left groups of 3, you make …

Math operations go right to left in the text, while we write them left to right. So if you see the digits 123... in an autoreressive manner, you don't know really anything, since i…

And in regard to utf-8 being a shitty biased tokenizer, here is recent paper trying to design a better style of encoding https://arxiv.org/abs/2505.24689

Virtually all current tokenization schemes do work at the raw byte level, not the utf-8 character. They do this to avoid the Out of Vocabulary (OOV) or unknown token problem. In ol…

I suppose it is. There is a lot to tokenization - pre-tokenization, how to handle digits, the tokenization training approach - that is about adding cleverness. In the long run, the…

This paper has a good solution: https://arxiv.org/abs/2402.14903 You right to left tokenize in groups of 3, so 1234567 becomes 1 234 567 rather than the default 123 456 7. And if y…

Gurobi does have a cloud service where you pay by the hour. A full non-academic license is pricy.

I'm just saying that these systems don't work for me. I write ML/AI conference papers in LaTeX, and I think that use case will be tough to dislodge. I can see this being very attra…