Learn Before
Encoder-decoder networks
The abstracted architecture of the twisted autoregressive generation is referred to as the encoder-decoder architecture.
It consists of three parts:
- encoder: accepts a sequence as its input and generates a corresponding sequence of contextualized representations (hidden states);
- context vector: is a function of the vector of contextualized representations generated from the encoder and conveys the essence of the input to the decoder;
- decoder: takes context vector as input and generates an arbitrary length sequence of hidden states, therefore obtains a corresponding sequence of output states.
The encoder and decoder networks are typically implemented with the same architecture, often using recurrent networks, but there are some other possibilities in each part.
0
4
Contributors are:
Who are from:
References
Speech and Language Processing (3rd ed. draft)
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Tags
Data Science
Foundations of Large Language Models Course
Computing Sciences
Learn After
Encoder
Decoder
Context vector
Encoder-Decoder with Transformers
Multi-lingual Pre-training for Encoder-Decoder Models
Mathematical Formulation of an Encoder-Decoder Model
Seq2seq Models for Text Generation
Auto-Regressive Decoding in Machine Translation
Applying Encoder-Decoder Architectures to NLP via the Text-to-Text Framework
A sequence-to-sequence model is designed to translate English sentences into French. When given the English input, 'The quick brown fox jumps over the lazy dog,' the model produces the French output, 'Où est la bibliothèque?' ('Where is the library?'). The generated French sentence is grammatically perfect and fluent, but it is completely unrelated to the meaning of the English input. Based on this specific failure, which component of the underlying architecture is most likely the primary source
Diagnosing an Architectural Flaw in a Summarization Model
Arrange the following events to accurately describe the flow of information in a standard encoder-decoder architecture for a sequence-to-sequence task.
Your team is pretraining an internal T5-style enco...
Your company wants one internal model to support m...
Your team is pretraining an internal T5-style mode...
Your team is building a single internal T5-style t...
Diagnosing a T5-Style Model That Ignores Task Prefixes After Span-Denoising Pretraining
Choosing Between Span-Denoising Pretraining and Task-Specific Fine-Tuning in a T5-Style Text-to-Text System
Designing a Unified Text-to-Text Model and Pretraining Objective for Multiple NLP Features
Root-Cause Analysis of a T5-Style Model Producing Fluent but Unfaithful Outputs
Selecting an Architecture and Pretraining Objective for a Unified Internal NLP Service
Post-Pretraining Data Formatting Bug in a T5-Style Text-to-Text Service
Pre-training Encoder-Decoder Models via Masked Language Modeling