Google

- For the decoder, autoregressive generation is used to output a sequence, an element at a time, until an end-of-sentence marker appears.
- Typically, use an LSTM or GRU-based RNN, where the context vector consists of the final hidden state of the encoder and is used to initialize the first hidden state of the decoder. 
- To avoid the fading influence of context vector during the decoding process, a solution is to add the context vector as a parameter to the computation of the current hidden state.
- In order to keep track of what has already been generated and what hasn’t, condition the output on three parts, the newly generated hidden state, the output generated at the previous state, and the encoder context.
- Beam search is used to optimize the output, preventing the unreliable result by independently choosing the argmax over a sequence.

Decoder

In the context of machine translation, the auto-regressive decoding process generates each token of the translated text (target language) by conditioning on both the complete source language sequence and the tokens that have already been generated in the target language.

Auto-regressive Decoding in Machine Translation

An autoregressive decoder generates an output sequence one element at a time. At each step, the decision for the next element is typically conditioned on three key inputs: 1) the element generated in the previous step, 2) the decoder's current hidden state, and 3) the context derived from the entire input sequence. Explain the distinct purpose of each of these three inputs in this process.

Decoder Input Analysis

Given the case study, what is the most probable architectural reason for the decline in translation quality towards the end of the generated sentence?

Diagnosing Translation Degradation

Teacher forcing is a common training strategy for sequence models where the ground-truth token from a prior time step is used as input, rather than the model's own generated prediction. In an encoder–decoder architecture, this involves feeding the original target sequence directly into the decoder. Specifically, a special beginning-of-sequence token (e.g., `` `<bos>` ``) is prepended to the target sequence, excluding its final token. The decoder is then trained to predict the original target sequence shifted by one time step, ending with an end-of-sequence token (e.g., `` `<eos>` ``). This shifting method for self-supervised learning closely resembles standard language model training.

Teacher Forcing

Beam search is a sequence decoding strategy that strikes a compromise between the computational efficiency of greedy search and the optimality of exhaustive search. Instead of greedily picking the single most likely token or exploring all possible paths, beam search evaluates and retains a predetermined number of the most promising candidate sequences at each step of the generation process.

Beam Search Strategy in Sequence-to-Sequence Models

An autoregressive sequence generation model is tasked with producing an output. At each step, it calculates the probability for every possible next element and selects the single element with the highest probability before moving to the next step. What is the primary limitation of this step-by-step selection strategy?

Learn Before

Related