Google

In practical applications, scheduling systems can be designed with custom priority policies that go beyond simple prefill/decode prioritization. These policies allow practitioners to account for specific operational needs and constraints, such as meeting request deadlines or giving precedence to requests based on user-defined importance levels.

Custom Priority Policies in LLM Scheduling

Based on the scenario provided, evaluate the current 'first-come, first-served' scheduling policy in the context of the company's business goals. Propose a more effective custom priority policy and justify your choice, including a discussion of its primary trade-off.

Evaluating Scheduling Policies for a Multi-Tenant LLM Service

An organization uses a large language model for two main tasks: generating complex financial reports with strict deadlines and answering real-time, interactive queries from business analysts. To ensure deadlines are met, they implement a custom scheduling policy that gives absolute priority to any report-generation request with a deadline approaching within the next hour. Analyze the potential negative consequences of this scheduling policy on the overall system performance and user experience. What specific issues might arise for the interactive queries?

Analyzing Trade-offs in Deadline-Aware LLM Scheduling

An inference serving system for a large language model must handle requests from two user tiers: 'Premium' users who pay for guaranteed low latency, and 'Standard' users. The system also runs internal, non-urgent 'Analytics' jobs that can tolerate high latency. The primary business goal is to retain Premium users by meeting their low-latency expectations, while still processing requests from other tiers. Which custom scheduling policy would be the most effective for achieving this business goal?

Learn Before

Related