Google

In speculative decoding, a speculated token is accepted or rejected by comparing its probability from a draft model ($$ q $$) with that from a target model ($$ p $$). If the draft model's probability for a token, $$ q(\hat{y}_{i+t}) $$, is greater than the target model's probability, $$ p(\hat{y}_{i+t}) $$, the token is rejected with a probability of $$ 1 - \frac{p(\hat{y}_{i+t})}{q(\hat{y}_{i+t})} $$. In the alternative case, where $$ q(\hat{y}_{i+t}) \le p(\hat{y}_{i+t}) $$, the token is accepted outright.

Acceptance-Rejection Mechanism for Speculative Decoding

In the speculative decoding process, after each token in the drafted sequence is evaluated for acceptance or rejection, a key step is to determine the maximum number of tokens that have been accepted consecutively from the beginning of the sequence. This count establishes the length of the valid prefix that can be appended to the final output.

Determining the Maximum Number of Consecutively Accepted Tokens in Speculative Decoding

In a text generation process, a small, fast model proposes the next token as 'learning' with a probability of 0.8. A larger, more accurate model then evaluates this same token and assigns it a probability of 0.6. Based on the standard acceptance-rejection procedure used in this context, what is the outcome for the token 'learning'?

A text generation system uses a small 'draft' model and a large 'target' model to speed up output. The draft model proposes a sequence of tokens, and an acceptance-rejection mechanism decides whether to keep them. For each proposed token, analyze the provided probabilities and determine if the token is (A) accepted outright, or (B) subject to a probabilistic check. If it's subject to a probabilistic check, calculate the specific probability of *rejection*.

Evaluating Proposed Tokens in a Generation Process

In the acceptance-rejection step of speculative decoding, $$r_t$$ is a random variable sampled from a uniform distribution between 0 and 1, denoted as $$U(0, 1)$$. For each candidate token at position $$t$$, a new value of $$r_t$$ is drawn and compared against the probability ratio of the target and draft models to decide whether to accept or reject the token.

Role of the Uniformly Distributed Random Variable ($$r_t$$) in Speculative Decoding

In a text generation process that uses a draft model and a target model, if the draft model assigns a higher probability to a proposed token than the target model does, that token is automatically rejected.

Learn Before

Related