Google

The most straightforward version of beam search relies on a single hyperparameter called the beam size, denoted as $$k$$. The beam size specifies the exact number of top candidate sequences that the algorithm maintains at each generation time step, providing a flexible trade-off between the accuracy of the output and the overall computational cost.

Beam Size in Beam Search

The selection of the beam width parameter, K, in beam search requires balancing search efficiency with output accuracy. A larger K allows the algorithm to explore more candidate sequences, which can improve accuracy but at a higher computational cost. Conversely, an excessively large K may not provide significant benefits. For LLM inference tasks, practical experience shows that smaller values, such as K=2 or K=4, often achieve a satisfactory level of performance efficiently.

Balancing Efficiency and Accuracy with Beam Width (K)

A development team is fine-tuning a text generation model. They run two experiments on the same input prompt, adjusting a single numerical parameter, `K`, which dictates how many of the most likely next-word sequences are kept for consideration at each step of the generation process. Analyze the results below.

Analyzing Generation Algorithm Performance

A developer is configuring a text generation model that uses a search algorithm to produce output. They must choose a value for a parameter that determines how many of the most likely next-word sequences are kept for consideration at each step. Compare the likely impact of setting this parameter to a very low value (e.g., 2) versus a very high value (e.g., 20) on both the computational resources required and the quality of the final generated text.

Analyzing Parameter Impact on Text Generation

At the first time step, beam search selects the $$k$$ tokens with the highest predicted probabilities from the vocabulary to serve as the initial $$k$$ candidate sequences. At each subsequent time step, the algorithm considers the $$k$$ candidate sequences from the previous step and extends them with all possible tokens, creating $$k\left|\mathcal{Y}ight|$$ possible combinations. From these combinations, it evaluates and selects the $$k$$ extended sequences with the highest overall predicted probabilities to carry forward to the next step.

Process of Beam Search

An engineer is using a text generation model that employs a search algorithm where a parameter, `K`, determines the number of top candidate sequences kept at each step. The engineer observes that with `K=1`, the generated text is often repetitive and predictable. To improve the diversity and potential quality of the output, which of the following adjustments to `K` is the most logical next step?

Learn Before

Related