What is Token?
The smallest unit an LLM processes — roughly ¾ of a word in English.
// Explanation
Models don't read whole words but fragments called tokens. A short word can be 1 token, a long one can be 2 or 3. Spaces and punctuation count too. That's why API prices are measured in 'per million tokens'. This definition is roughly 50 tokens.
// Example
The word 'automatically' might be 2-3 tokens. 'AI' is 1. So writing in English is slightly cheaper in tokens than Spanish (lucky you).