Calculate the number of language model tokens for a string

Post by sva_ » Wed, Feb 05, 2025, 2:01 PM UTC

  import tiktoken
  enc = tiktoken.encoding_for_model('o1-mini')
  len(enc.encode('some string'))

Post by topicseed » Wed, Feb 05, 2025, 3:18 PM UTC

NitpickLawyer wrote: post #3

sva_ wrote: post #2

import tiktoken enc = tiktoken.encoding_for_model('o1-mini') len(enc.encode('some string'))

Does that work on anything other than openai?