Live data from Hacker News

Made a GPT-3.5-Turbo and GPT-4 Tokenizer

typeblock.co

1–2 of 2 posts

Re: Made a GPT-3.5-Turbo and GPT-4 Tokenizer

#2
For the longest time I haven't been able to find a tokenizer online similar to the one for GPT-3 for other models. So I made one and put it up online for free for anyone.

It's built on top of the tiktoken library and is basically just a lambda function in the backend.

You could just download the library directly but I hope this saves someone time if they need the token ids for logit bias.