Google

To handle sequences of varying lengths within a single batch, a technique called padding is often used. This involves adding special `⟨pad⟩` tokens to the shorter sequences until they match the length of the longest sequence in the batch. This creates a uniform tensor shape, which is necessary for efficient parallel computation in many deep learning frameworks. The image shows this by adding three `⟨pad⟩` tokens to the shorter of the two sequences.

Padding in Sequence Batching

This diagram illustrates a common scenario in LLM serving where a batch contains sequences of varying lengths, such as `⟨SOS⟩ I think this movie is better` and `I really like`. To create a uniform batch for processing, the shorter sequence is prepended with a start-of-sequence token and padded with `⟨pad⟩` tokens, resulting in `⟨pad⟩ ⟨pad⟩ ⟨pad⟩ ⟨SOS⟩ I really like`. Crucially, the image also depicts how these sequences' data blocks are stored in non-contiguous physical memory, visualizing the problem of memory fragmentation that arises from dynamic allocation.

Example of Padded Sequences in Fragmented Memory

A data scientist is preparing a batch of text data for a model. The batch contains two tokenized sentences: `['The', 'cat', 'is', 'on', 'the', 'mat']` (length 6) and `['Birds', 'fly']` (length 2). When they attempt to combine these into a single tensor, they encounter an error: `RuntimeError: Tensors must have same size at dimension 1`. Based on this scenario, what is the most likely cause of the error, and what specific technique should be applied to the shorter sequence to resolve it?

Debugging a Batch Processing Error

A machine learning engineer is preparing a batch of text sequences of different lengths for processing. They decide to skip the step that makes all sequences in the batch have the same length. Explain the primary computational issue this will cause for most deep learning frameworks and why enforcing a uniform length resolves this issue.

Consequences of Non-Uniform Sequence Lengths

The efficiency of batch processing is significantly influenced by the lengths of the sequences within the batch. When sequences have similar lengths, the amount of padding required to create a uniform tensor is minimized. This reduction in padding leads to less wasted computation, as the model does not have to perform operations on non-informative `⟨pad⟩` tokens, thereby maximizing computational efficiency and throughput.

Efficiency of Batching Sequences with Similar Lengths

When batching input sequences of varying lengths for Large Language Model inference, their lengths must be standardized. Left padding is specifically used to add dummy tokens to the beginnings of shorter sequences. This ensures that all sequences within the batch have the identical length required for the prefilling stage.

Left Padding in LLM Batching

A deep learning model is being prepared to process the following three text sequences together in a single batch: `['The', 'cat', 'sat']`, `['A', 'quick', 'brown', 'fox']`, and `['On', 'the', 'mat']`. To ensure all sequences have a uniform length for efficient computation, a special `⟨pad⟩` token is added to the end of the shorter sequences. Which of the following options correctly represents the batch after this process is applied?

Learn Before

Related