Token Generation After Speculative Acceptance
In the scenario below, describe the immediate next step the system takes to continue generating the text. Specify which model performs this action and what sequence of tokens serves as its input.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Formula for Next Token Generation After Acceptance in Speculative Decoding
A text generation system using speculative decoding has the confirmed output 'The cat sat on the'. A draft model then proposes the four-token sequence: 'mat and then slept'. The main verification model evaluates this draft and accepts the first two tokens ('mat', 'and'). What is the correct, immediate next action for the system to take to continue the generation process?
In a single step of a speculative decoding process, after the main model has compared its own probabilities with those of the draft model for a sequence of candidate tokens, what is the correct order of operations to finalize the output for that step?
Diagram of Post-Acceptance Token Prediction in Speculative Decoding