Google

The pre-training of large language models relies on vast and varied text corpora. Key sources for these datasets include webpages, books, conversational text, software code, Wikipedia, and news articles, in addition to other materials like scientific papers and content from question-and-answer (Q&A) platforms.

Common Data Sources for Pre-training LLMs

A technology company is developing a new large language model intended for public use. A significant portion of its pre-training data is sourced from a massive crawl of the public internet, including forums and social media. Critically evaluate the decision to rely heavily on this type of data. In your response, you must weigh the potential benefits for the model's general capabilities against the significant ethical and performance-related risks.

Evaluating Data Sources for LLM Pre-training

A development team is building a language model specifically to help software engineers by generating high-quality technical documentation. Given the case study below, identify the TWO most critical data sources for pre-training this specialized model and justify your choices. Your justification should explain how each selected source directly contributes to the model's intended function.

Data Source Selection for a Specialized LLM

GPT-3 is a massive Transformer-decoder model that scales up the GPT-2 architecture by approximately two orders of magnitude in both model size and training data, utilizing $$300$$ billion pretraining tokens. It retains the foundational architecture of GPT-2 but incorporates sparser attention patterns at alternating layers. GPT-3 thoroughly validated the in-context learning paradigm, proving that few-shot performance rapidly improves as model capacity increases.

GPT-3

Falcon is a family of large language models. A notable version is Falcon-180B, which contains 180 billion parameters and was pre-trained on 3.5 trillion tokens derived from diverse sources including webpages, books, conversations, code, and technical articles.

Falcon

LLaMA2 is a family of large language models introduced in 2023. It includes prominent versions such as the 65-billion-parameter LLaMA2-65B, which was pre-trained on between 1.0 trillion and 1.4 trillion tokens. The training data for LLaMA2 comes from a diverse mix of public sources, including webpages, software code, Wikipedia, books, academic papers, and question-and-answer content.

LLaMA2

PaLM-450B is a large language model introduced in 2022. It was pre-trained on a massive corpus of 0.78 trillion tokens, utilizing diverse data sources that include webpages, books, conversations, code, Wikipedia, and news articles.

PaLM-450B

Gemma-7B is a large language model pre-trained on a massive dataset of 6 trillion tokens. Its pre-training data is sourced from webpages, mathematics content, and code.

Gemma-7B

A newly developed large language model demonstrates high fluency and generates grammatically perfect, conversational text. However, it frequently provides outdated information, struggles to generate well-structured, long-form content like reports, and often fabricates details when asked about events from the last year. Based on these specific performance characteristics, which of the following descriptions most likely represents the composition of its pre-training dataset?

Learn Before

Related