An LLM inference engine processes requests in iterative cycles. Arrange the following events to show the correct sequence for a single cycle where the active batch of requests is modified.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Comprehension in Revised Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Removing Completed Sequences in Continuous Batching
Adding New Requests in Continuous Batching
Maintaining an Unchanged Batch in Continuous Batching
Overhead of Dynamic Batch Reorganization in Continuous Batching
An LLM inference system is processing a batch of user requests. An observer notes the following: At the start of one processing step, the active batch contains requests {A, B, C, D}. Immediately before the next processing step begins, the active batch contains requests {A, C, E}. Based on this observation, what is the most fundamental principle of this system's batch management strategy?
Inference Batch Management Scenario