Google

In the context of few-shot learning, demonstrations are example input-output pairs included in a prompt. These samples serve to teach a large language model how to perform a specific task by showing it concrete examples of the desired behavior.

Demonstrations in In-Context Learning

A developer wants to use a language model to classify user feedback into two categories: 'Bug Report' or 'Feature Request'. They have created two potential sets of example input-output pairs to include in the prompt.

**Set A:**
1. Input: 'The app crashes when I click the save button.' -> Output: 'Bug Report'
2. Input: 'It would be great if you could add a dark mode.' -> Output: 'Feature Request'

**Set B:**
1. Input: 'The login button is broken and doesn't respond.' -> Output: 'Bug Report'
2. Input: 'The export-to-PDF feature is not working correctly.' -> Output: 'Bug Report'

Which set of examples is more effective for teaching the model this specific classification task, and why? Justify your reasoning.

Evaluating Prompt Demonstrations

A developer is using a large language model to classify customer support tickets. The goal is to categorize each ticket as 'Billing Inquiry', 'Technical Support', or 'General Question'. The model is performing poorly, frequently misclassifying the tickets. Analyze the example input-output pairs provided in the prompt below. Identify at least two significant flaws in these examples and explain how these flaws likely contribute to the model's poor performance. Finally, suggest specific improvements.

Evaluating and Improving Prompt Demonstrations

In few-shot prompting, Large Language Models learn from the provided demonstrations of problem-solution pairs. These demonstrations teach the model not only the underlying problem-solving logic but also the specific way to format its generated output. For instance, a model learns to use special tokens for calculation annotations (like $$\ll ... \gg$$) and to demarcate the final answer (like ####) by observing these patterns in the few-shot examples.

Learning Output Formatting from Demonstrations

A developer is trying to get a language model to extract product codes from customer emails. They provide the following examples in the prompt before asking the model to process a new email:

Example 1:
Input: 'Hi, my SuperWidget model SW-1000 is broken.'
Output: 'SW-1000'

Example 2:
Input: 'I need a replacement part for my SuperWidget Pro, model number SW-2500.'
Output: 'SW-2500'

New Email:
Input: 'My GigaGadget GG-500 won't turn on.'

The model incorrectly outputs 'SW-500'. Based on an analy

Learn Before

Related