This is a Java implementation of a GPT3/4 tokenizer, loosely ported from Tiktoken with the help of ChatGPT. ...that all 3.5-turbo models released after 0613 now have ...

gpt-tokenizer is a Token Byte Pair Encoder/Decoder supporting all OpenAI's models (including GPT-3.5, GPT-4, GPT-4o, and o1). It's the fastest, smallest and lowest footprint GPT tokenizer available ...