Expand description
Tokenizer gateway for encoding and decoding text using tiktoken.
This module provides token counting functionality which is useful for:
- Managing context window limits
- Estimating API costs
- Debugging tokenization issues
- Optimizing prompt engineering
Structsยง
- Tokenizer
Gateway - Gateway for tokenizing and detokenizing text using tiktoken.