Google

This diagram illustrates the second iteration in a continuous batching process, which follows the initial prefilling of requests x1 and x2. In this step, the scheduler directs the inference engine to perform a single decoding operation for the entire batch. This concurrently generates the first output token for both request x1 and request x2, demonstrating how the system transitions from the prefilling phase to the iterative decoding phase for a group of requests.

Example of the First Decoding Step in Continuous Batching (Iteration 2)

This diagram illustrates the third iteration in the continuous batching example, continuing from the first decoding step. In this stage, the scheduler again directs the inference engine to perform a single decoding operation for the batch containing requests x1 and x2. This action generates the second output token for each of the two requests, demonstrating the ongoing, iterative nature of the decoding phase.

Example of the Second Decoding Step in Continuous Batching (Iteration 3)

An LLM inference system is processing two user prompts, 'Prompt A' and 'Prompt B', together in a batch. The system has just completed the initial processing of the input text for both prompts and has now performed exactly one subsequent generation step. Describe the state of both 'Prompt A' and 'Prompt B' at this moment.

State of Batched Requests After First Generation Step

An inference engine is processing two user prompts simultaneously. The engine has just completed the initial processing phase where it has understood both prompts. It now executes a single, unified generation operation for both prompts together. Based on how batched processing works in this context, what is the most likely state of the generated output for each prompt immediately after this operation, and why?

Inference Engine State Analysis

An inference engine is processing user requests. It has just completed the initial processing for two separate requests, 'Request A' and 'Request B', loading them into memory. Both requests are now ready for the next stage of generation. What is the most likely immediate next action the engine will take to operate efficiently, and what will be its result?

Learn Before

Related