Google

A significant improvement for multilingual pre-trained models, such as mBERT, involves incorporating bilingual data into the pre-training process. In contrast to training on separate monolingual corpora, this approach explicitly models the relationships between tokens from two different languages. This method equips the model with inherent cross-lingual transfer abilities, making it more readily adaptable to new languages.

Bilingual Pre-training for Multilingual Models

Proposed by Lample and Conneau (2019), Cross-Lingual Language Models (XLMs) are a specific approach to pre-training that leverages bilingual data. An XLM can be trained using either a causal language modeling (CLM) or a masked language modeling (MLM) objective. When using the MLM approach, the training objective is identical to BERT's, where the model, treated as an encoder, learns to predict randomly selected tokens that have been masked, replaced, or left unchanged in the input.

Cross-Lingual Language Models (XLM)

A specific technique used in bilingual pre-training involves sampling a pair of aligned sentences from two different languages. These sentences are then concatenated to form a single, combined sequence that is used as a training input. This method directly exposes the model to parallel data within a single training instance, facilitating cross-lingual learning.

Bilingual Sentence Packing for Pre-training

During the pre-training of a bilingual model, a phenomenon known as interference can occur. This means that after a certain amount of training, the model's overall performance may begin to decline rather than continue improving.

Performance Degradation due to Interference in Bilingual Pre-training

A language model can be pre-trained for multilingual tasks in two ways: (A) on a large collection of documents where each document is in a single language (e.g., English or German), or (B) on a collection of sentence pairs, where each pair consists of a sentence and its direct translation (e.g., an English sentence and its German translation). Analyze why approach (B) is generally more effective for developing a model with strong cross-lingual transfer abilities.

Analyzing Pre-training Strategies for Multilingual Models

Based on the following scenario, which pre-training dataset should the research team prioritize to best achieve their goal? Justify your choice by explaining the mechanism through which the selected data helps the model develop the required cross-lingual capabilities.

Pre-training Strategy for Zero-Shot Cross-Lingual Transfer

An NLP team is developing a model for a Spanish-to-Portuguese translation service. They are considering two different pre-training strategies before fine-tuning the model on a specific translation dataset.

Strategy 1: The model is trained on a large corpus containing millions of Spanish documents and a separate, equally large corpus of Portuguese documents. During each training step, the model processes text from only one of the two languages.

Strategy 2: The model is trained on a large corpus

Learn Before

Related