Google

Here on the image you can see the structure of the decoder which is very similar to the encoder part we just described with only difference that in this case we pass K, V from the input to the each decoder attention layer(encoder -decoder layer) and now we take queries from previous decoder layers and compare the decoder query with encoder keys just like in the usual se2seq model. Also the difference is that we have here a layer of so called masked self-attention. It is just the layer at each time stamp we do not compare the query with future keys

Transformer Decoder

A Transformer decoding network, often denoted as `Dec(·)`, is fundamentally constructed from an embedding network and a series of stacked layers. The embedding network handles the initial input processing, while the stacked layers, each comprising a self-attention module and a Feed-Forward Network (FFN), perform the main sequence processing.

Core Components of a Transformer Decoding Network

Masked self-attention is a crucial component of the Transformer decoder, enabling autoregressive text generation. Unlike standard self-attention, it restricts each position from attending to subsequent, or 'future,' positions in the sequence. This is implemented by applying a mask to the attention scores before the softmax function, effectively zeroing out the weights for future tokens. Consequently, the query for a given token can only interact with keys from its own position and all preceding positions, ensuring that the prediction for the current step depends only on the known past.

Masked Self-Attention in Transformer Decoders

Within a single decoder block of a standard Transformer architecture, information is processed through three main computational sub-layers. Arrange these sub-layers in the correct operational sequence.

In the Transformer decoder's encoder-decoder attention layer, what is the decoder query compared with?

What is the defining rule of the masked self-attention layer in the Transformer decoder?

A standard Transformer decoder block contains two distinct attention sub-layers. Which statement accurately differentiates the roles and data sources for these two sub-layers?

Learn Before

Related