The structure of the Encoder is shown in the figure

Northeastern University (US)

The Transformer is an instance of the encoder-decoder architecture that fundamentally relies on self-attention. Unlike attention mechanisms used in standard sequence-to-sequence learning, the Transformer adds positional encoding to both the input (source) and output (target) sequence embeddings before feeding them into the encoder and decoder, respectively.

Transformer Architecture Overview

A very influential paper that introduced the concept of Transformer model.
https://arxiv.org/abs/1706.03762

Attention Is All You Need

The structure of the Decoder is shown in the figure

Decoder Structure  of Transformer

In Transformer models, the Feed-Forward Network (FFN) sub-layer plays a crucial role by introducing non-linearities into the representation learning process. This function is vital for preventing the representations learned by the self-attention mechanism from degenerating. Structurally, a standard FFN consists of two fully connected layers. The first layer typically uses a non-linear activation function like ReLU, while the second is a linear layer.

Purpose and Structure of the Feed-Forward Network (FFN) in Transformers

The self-attention mechanism, a core component of Transformer models, is a significant contributor to the challenges and complexity encountered during the inference process.

Learn Before

Related