Google

In a typical LLM inference system, the process begins with a pool of incoming user requests. A scheduler manages this pool, grouping individual requests into a batch. This batch is then dispatched to the inference engine, which executes the model to process the entire batch and produce the corresponding predictions.

Request Processing Workflow in LLM Inference

Arrange the following stages of a typical request processing workflow in a Large Language Model (LLM) inference system into the correct chronological order, from the initial arrival of a request to the final output.

This diagram illustrates the high-level workflow of an LLM inference system. It begins with a 'Request Pool' containing user inputs (e.g., x1, x2, x3). A 'Scheduler' selects requests from this pool and groups them into a 'batch'. This batch is then sent to the 'Inference Engine' for processing. Finally, the engine executes the model on the batch and returns the corresponding 'Predictions' (e.g., y2, y1, y3).

Diagram of the LLM Inference Workflow

A development team is deploying a new LLM-powered, real-time customer service chatbot. They are considering two different strategies for handling incoming user queries. Analyze the primary trade-off between these two strategies and recommend the most suitable one for this specific application, justifying your choice.

LLM Inference Scheduling Strategy

An operations team monitors an LLM inference system and notices that the hardware responsible for model execution is consistently underutilized, even when there is a continuous stream of user requests waiting to be processed. This leads to lower-than-expected overall system throughput. In a standard workflow where requests are grouped into batches by a scheduler before being processed, what is the most probable explanation for this specific performance issue?

Learn Before

Related