Count tokens with the model and context in view

Keep the tokenizer attached to the sequence

A tokenizer maps text into vocabulary units and corresponding identifiers. Different tokenizers can produce different sequences for the same text. Special tokens and model-specific formatting can add structure beyond the visible words. Use the tokenizer and formatting expected by the selected model instead of relying on a universal characters-to-tokens ratio.

For persisted tokenized data, record the tokenizer identity, configuration and relevant version. Preserve sequence order. An integer identifier selects a vocabulary entry; its numerical magnitude does not measure importance. If the model expects a mask or special markers, keep those representations aligned with the identifiers. A dataset can be a valid rectangular array and still be incompatible with the model when its tokenizer or formatting assumptions differ.

Budget the complete request

Plan context using the full input the model will receive, including task instructions, conversation history and selected evidence. Reserve room for the requested output according to the selected model’s documented context and output rules. An output allowance is a cap, not a guarantee that the task will finish within it.

1. Build the final request representation before estimating its size.
2. Count with the appropriate tokenizer or provider-supported counting method.
3. Select evidence that preserves the information needed for the task.
4. Apply an explicit policy for inputs that exceed the budget.
5. Inspect the observed completion or stop condition before accepting a result.

Prefer a controlled rejection or a documented reduction strategy to silent truncation. Removing the last paragraph can also remove the evidence that determines the answer.

Separate estimates from observed usage

Store planned input size and output allowance separately from usage returned by the provider. Keep the model identifier and request attempt with those values. A local estimate can help admission control, while the observed response describes what the provider reports for that execution. Field names and accounting categories vary, so interpret them using the relevant interface contract.

Track retries as separate attempts under one logical task. Otherwise, resource reporting can hide work repeated after a timeout. For variable-length batches, inspect both useful token positions and padding, because rectangular shape can increase computation without adding source content. Use representative documents to evaluate changes in context selection. A lower count is valuable only when the reduced request still supplies what the task needs to succeed.