Decoding Strategies & the LLM Landscape

A language model does not give one answer; it gives a probability for every possible next token. How we pick from those probabilities is called decoding, and it decides whether the output is boring, creative, repetitive or random. Then we will take a tour of today’s major LLMs.

01Greedy and beam search

Suppose the prompt is “The best time to drink chai is”. The model scores every token. The simplest strategy, greedy decoding, always picks the single most likely token. It is fast and repeatable, but can be dull and sometimes gets stuck repeating itself.

Beam search keeps the top B partial sentences (beams) at every step and finally returns the one with the highest total probability. It finds better overall sentences than greedy, which helps translation and summarisation, but it is slower and tends to produce safe, generic text.

02Sampling: temperature, top-k and top-p

Chatbots usually sample: they pick the next token randomly, but in proportion to its probability. Three knobs control this randomness:

• Temperature divides the scores before softmax. Below 1 makes the top choice even stronger (more focused); above 1 flattens the distribution (more creative, more risky). Temperature close to 0 behaves like greedy.
• Top-k keeps only the k most likely tokens and samples among them.
• Top-p (nucleus) keeps the smallest set of tokens whose probabilities add up to at least p (for example 0.9), so the number of choices adapts to how confident the model is.

Repetition penalty

Models sometimes loop: “chai is good, chai is good, chai is good…”. A repetition penalty (for example 1.1–1.3) lowers the score of tokens that have already appeared, so the model moves on. Similar settings are called frequency and presence penalties in some APIs.

# model and tok loaded as in Session 23
inputs = tok("The best time to drink chai is", return_tensors="pt").to(model.device)

greedy   = model.generate(**inputs, max_new_tokens=20, do_sample=False)
beam     = model.generate(**inputs, max_new_tokens=20, num_beams=4, early_stopping=True)
creative = model.generate(**inputs, max_new_tokens=20, do_sample=True,
                          temperature=0.9, top_k=50, top_p=0.9, repetition_penalty=1.2)

for out in (greedy, beam, creative):
    print(tok.decode(out[0], skip_special_tokens=True))

03The LLM landscape

There are two big groups: closed models you use through an API, and open-weight models you can download and run yourself. This field changes every few months, so always check the latest versions on each provider’s site.

04Reasoning, context, multimodality and tools

05Summary

• Greedy picks the top token; beam search keeps several candidate sentences.
• Temperature controls randomness; top-k and top-p limit the choices; repetition penalty stops loops.
• Low temperature for facts and code; higher for creative tasks.
• Choose between closed APIs and open weights based on quality, cost, privacy and control.
• Compare models on reasoning, context length, multimodality and tool calling.

06Quick quiz

07References

• Holtzman et al., The Curious Case of Neural Text Degeneration (nucleus sampling) (2019)
• Hugging Face, Text generation strategies
• Leaderboards to compare models: LMArena, Artificial Analysis