Google

Dot product attention is a fundamental type of multiplicative attention based on dot similarity. Geometrically, the dot product measures the alignment between vectors: if a query and key share a similar direction, their dot product is higher, whereas orthogonal vectors yield a dot product of $$0$$. This implies that keys which are more conceptually related to the current query will receive larger attention scores. One notable characteristic of pure dot product attention is that it does not introduce any additional learnable parameters, relying entirely on the existing vector representations. The attention score is calculated mathematically using the standard query-key notation as: $$ a(\mathbf{q}, \mathbf{k}_i) = \mathbf{q}^	op \mathbf{k}_i $$ where $$\mathbf{q}$$ represents the query vector and $$\mathbf{k}_i$$ represents the key vector.

Dot Product Attention

This example presents a language modeling task where the goal is to predict masked tokens, denoted by `[M]`, based on the surrounding context, `[C]`. The specific structure provided is: `[C] The kitten [M] playing the [M] .`

Example of Predicting Masked Words: Kitten Playing

This example illustrates a masked language modeling task using the sentence `[C] The kitten is chasing the ball .` as the original text. The model is provided with an input where some words are replaced by a mask token, such as `[C] The kitten [M] [M] [M] ball .`. In this structure, `[C]` denotes the context and `[M]` represents the masked tokens. The model's objective is to predict the original words that were masked, which are 'is', 'chasing', and 'the'.

Example of Masked Language Modeling: Kitten Chasing Ball

In context-based prediction, a model generates text based on a given context. For instance, if the model is provided with the context `[C] The kitten is`, its objective is to predict the subsequent text to form a complete sentence, such as `[C] The kitten is chasing the ball .`. The target prediction can also be represented as a simple sequence of words or an indexed sequence, like `chasing1 the2 ball3 .`, where numbers indicate a specific prediction order.

Example of Context-Based Prediction: Kitten Chasing Ball

Based on the mathematical form of Function A, explain the most likely reason for its lower performance on complex tasks compared to Function B.

Evaluating Attention Mechanisms in Machine Translation

In a sequence-to-sequence model using an attention mechanism, the current decoder hidden state is represented by the vector `h'_t = [0.5, 1.0, -0.2]`. The model is evaluating an encoder hidden state represented by the vector `h = [0.6, 0.8, 0.1]`. Using the dot product scoring function, calculate the attention score for this pair of vectors. Show your calculation.

Calculating a Dot Attention Score

In a sequence-to-sequence model, an attention mechanism calculates a score for three input vectors (A, B, and C) relative to a single output vector (D). The scoring function is the simple dot product between the output vector and each input vector. You are given the following geometric relationships:

- Vector A points in a very similar direction to Vector D.
- Vector B is orthogonal (at a 90-degree angle) to Vector D.
- Vector C points in the opposite direction of Vector D.

Which input vector 

Learn Before

Related