University of Michigan - Ann Arbor

To preserve autoregression in the Transformer decoder, the prediction generated for position $$i$$ must depend exclusively on known outputs at positions strictly less than $$i$$. This constraint is enforced by two synchronized mechanisms: modifying the decoder's self-attention sub-layer to mask out subsequent positions and shifting the output embeddings to the right by one position.

Autoregressive Dependency Guarantee in Transformer Decoders

To preserve autoregression in the Transformer decoder, what dependency constraint must be satisfied by the prediction generated for position $i$?

What positional adjustment must be applied to the output embeddings to help enforce the autoregressive constraint in the Transformer decoder?

Explain how the Transformer decoder prevents future position information from influencing predictions, detailing the necessary mechanisms and the operational condition they uphold.

Learn Before

Related