tiktoken-rs: GPT tokenizer library for Rust

A ready-made, batteries-included tokenizer library for working with GPT and tiktoken.

Find the source at https://github.com/zurawiki/tiktoken-rs.