Google

Search algorithms, also known as decoding algorithms, are foundational methods for LLM inference that navigate the vast space of possible output sequences to select the most probable one. Most of these algorithms operate through a level-by-level search process, building sequences token by token. While many of these techniques are now central to LLMs, some have their roots in earlier sequence-to-sequence models, whereas others are more recent developments specifically associated with modern large-scale models.

Search (Decoding) Algorithms for LLM Inference

Due to the computational infeasibility of an exhaustive search across an exponentially large space of sequences, LLM inference relies on heuristic algorithms. These practical methods use pruning strategies to navigate the vast output space efficiently. Pruning involves avoiding the exploration of low-quality or improbable sequences at each decoding step, which trades guaranteed optimality for a manageable computational cost. Common heuristic techniques include greedy search and sampling-based search.

Heuristic Search Algorithms for LLM Inference

Stopping criteria are essential rules within LLM inference that determine when the text generation process should conclude. These conditions are necessary to signal the end of decoding, prevent indefinite output, and manage practical considerations like decoding cost and verbosity by avoiding overly long sequences.

Stopping Criteria in LLM Inference

The complete search space for potential output sequences in LLMs grows exponentially with the length of the generated text. This rapid expansion makes a direct or exhaustive search for the optimal sequence computationally infeasible for all but the shortest outputs.

Computational Infeasibility of Exhaustive Search in LLM Decoding

Imagine a language model is tasked with generating a 10-word sentence. At each of the 10 steps, it considers the 5 most likely words to add next. To find the single best 10-word sentence, why is it computationally impractical for the model to evaluate every single possible combination of these words? Explain the core challenge that this scenario illustrates for text generation.

The Challenge of Generating Optimal Text

A language model is given the same prompt twice but uses a different method for generating the text each time, resulting in the two outputs below. Based on your understanding of how text generation works, analyze the fundamental difference in the approaches that likely produced Output A versus Output B. Specifically, comment on how each approach likely navigates the space of possible next words at each step of the generation process.

Analyzing Text Generation Behavior

A language model is given the prompt 'The capital of France is'. Internally, the model's calculations show that the single most probable next word is 'Paris'. However, the model ultimately generates the sequence 'The capital of France is a beautiful city'. Which statement best analyzes the reason for this discrepancy?

Learn Before

Related