Made a GPT-3.5-Turbo and GPT-4 Tokenizer
typeblock.co
Made a GPT-3.5-Turbo and GPT-4 Tokenizer
1–2 of 2 posts
Re: Made a GPT-3.5-Turbo and GPT-4 Tokenizer
#2For the longest time I haven't been able to find a tokenizer online similar to the one for GPT-3 for other models. So I made one and put it up online for free for anyone.
It's built on top of the tiktoken library and is basically just a lambda function in the backend.
You could just download the library directly but I hope this saves someone time if they need the token ids for logit bias.