Google

This technique frames soft prompt learning as a context compression problem solved using knowledge distillation. In this process, the knowledge from a lengthy, standard prompt is distilled from a teacher model into a compact set of 'pseudo tokens'. The embeddings for these pseudo tokens, which are appended to the user's input sequence, are then optimized to replicate the predictions of the teacher model, effectively capturing the essence of the original, more complex prompt.

Soft Prompt Learning as Context Compression via Knowledge Distillation

The optimal soft prompt, denoted as $\hat{\sigma}$, can be determined by finding the prompt that minimizes the dissimilarity between the model's predictions with and without the full context. This is expressed by the formula: $$\hat{\sigma} = \arg \min_{\sigma} s(\hat{\mathbf{y}}, \hat{\mathbf{y}}_{\sigma})$$ Here, $s$ is a function measuring the dissimilarity (e.g., distance) between the prediction from the full context, $\hat{\mathbf{y}}$, and the prediction using the soft prompt, $\hat{\mathbf{y}}_{\sigma}$. This method aligns the compact prompt's behavior with that of the original, more descriptive prompt.

Formula for Soft Prompt Optimization by Minimizing Prediction Dissimilarity

An engineer proposes replacing the long policy document with a compact, learnable set of vectors. These vectors would be optimized to make the chatbot generate the same high-quality answers as it did with the full document. Based on the principles of training such vectors, evaluate the feasibility and primary benefit of this proposal. Explain the core principle that would allow these learned vectors to effectively stand in for the lengthy text.

Optimizing Language Model API Costs

A team is training a set of learnable, continuous parameters to serve as a compact substitute for a long, detailed textual instruction set for a language model. The goal is for these compact parameters to guide the model to produce the same quality of output as the original long instructions when given any user input. Which of the following best describes the core objective of this training process?

In the framework of knowledge distillation, the teacher model is characterized as being more powerful and larger, whereas the student model is intentionally designed to be smaller and more computationally efficient for practical applications.

Characteristics of Teacher and Student Models in Knowledge Distillation

In the framework of learning a soft prompt via knowledge distillation to compress a longer context, match each component with its corresponding role in the process.

Learn Before

Related