{"id":89823,"date":"2026-05-28T17:25:51","date_gmt":"2026-05-28T15:25:51","guid":{"rendered":"https:\/\/webraketen.space\/?post_type=glossar&amp;p=88843"},"modified":"2026-05-28T17:25:51","modified_gmt":"2026-05-28T15:25:51","slug":"token","status":"publish","type":"glossar","link":"https:\/\/webraketen.space\/en\/glossar\/token-ki-llom\/","title":{"rendered":"Token (AI\/LLM)"},"content":{"rendered":"<p class=\"wp-block-paragraph\">A token is the smallest unit with which a large language model (LLM) processes text. These are not whole words but mostly word parts, character sequences, or individual punctuation marks. Tokens are the basic building blocks with which models read, generate, and bill for language.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How it works<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Before an LLM can process text, the input text is broken down into tokens by a tokenizer. Each token is assigned a numerical ID from the model's vocabulary. Therefore, the model does not work with letters or words, but with these numbers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Common tokenizer methods include Byte-Pair Encoding (BPE), WordPiece, and SentencePiece. They strike a balance between two extremes: individual characters (too granular) and whole words (too large vocabulary). Common words like \u201eand\u201c usually form a single token. Rare or compound words are broken down into multiple parts.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The number of tokens a specific text produces can be directly counted \u2013 for example, with the OpenAI Tokenizer or the library <code>tiktoken<\/code> for GPT models.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Examples<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The English word \u201etokenization\u201c is used in <code>token<\/code> + <code>ization<\/code> disassembled \u2014 two tokens.<\/li>\n\n\n\n<li>The German word \u201eDonaudampfschifffahrtskapit\u00e4n\u201c breaks down into about a dozen tokens. However, the splits don't follow the word components (Donau, Dampf, Schiff...), but rather statistical fragments: <code>D | one | ud | amp | fs | ch | if | ff | ahr | ts | kap | it | ain<\/code>.<\/li>\n\n\n\n<li>Punctuation, spaces, and emojis also count as separate tokens.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">As a rough estimate: 1,000 tokens correspond to approximately 750 words in English. In German, due to longer word compositions, it's often fewer, usually in the range of 500 to 600 words. The exact values depend on the text and the tokenizer.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Practical significance<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Tokens are important for three reasons:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Context window<\/strong> Each model has a maximum number of tokens it can process at once (e.g., 8,000, 128,000, or 1 million). This limit includes both input and output.<\/li>\n\n\n\n<li><strong>Costs<\/strong> API providers like OpenAI, Anthropic, or Google charge per token, separately for input and output tokens.<\/li>\n\n\n\n<li><strong>Performance:<\/strong> More tokens mean higher computational costs and longer response times.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Demarcation<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A token is not the same as a word or a syllable. The segmentation follows statistical patterns from the training data, not linguistic rules.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Different models use different tokenizers. Therefore, the same text results in a different token count depending on the model. For example, the sentence \u201eArtificial intelligence is changing the world of work.\u201c yields:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>GPT-4 \/ GPT-3.5<\/strong> Tokenizer <code>cl100k_base<\/code>): 15 Tokens<\/li>\n\n\n\n<li><strong>GPT-4o<\/strong> Tokenizer <code>o200k_base<\/code>): 11 Tokens<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The same sentence, the same provider ecosystem. Yet a difference of one-third because the newer tokenizer uses a larger vocabulary and combines German word parts more coarsely. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For cost and context window calculations, the tokenizer of the specific model is therefore always decisive, not a general rule of thumb.<\/p>","protected":false},"excerpt":{"rendered":"<p>Ein Token ist die kleinste Einheit, mit der ein Large Language Model (LLM) Text verarbeitet. Dabei handelt es sich nicht um ganze W\u00f6rter, sondern meist um Wortteile, Zeichenfolgen oder einzelne Satzzeichen. Tokens sind die Grundbausteine, mit denen Modelle Sprache lesen, erzeugen und abrechnen. Funktionsweise Bevor ein LLM Text verarbeiten kann, wird der Eingabetext durch einen [&hellip;]<\/p>\n","protected":false},"featured_media":0,"template":"","class_list":["post-89823","glossar","type-glossar","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/webraketen.space\/en\/wp-json\/wp\/v2\/glossar\/89823","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/webraketen.space\/en\/wp-json\/wp\/v2\/glossar"}],"about":[{"href":"https:\/\/webraketen.space\/en\/wp-json\/wp\/v2\/types\/glossar"}],"version-history":[{"count":0,"href":"https:\/\/webraketen.space\/en\/wp-json\/wp\/v2\/glossar\/89823\/revisions"}],"wp:attachment":[{"href":"https:\/\/webraketen.space\/en\/wp-json\/wp\/v2\/media?parent=89823"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}