Google

In Large Language Models, symbolic input tokens are not directly processed. Instead, each token is converted into a numerical representation called an embedding. These embeddings, learned by a token embedding model, serve as the actual input that the core LLM operates on.

Input Embeddings in LLMs

In Large Language Models, symbolic input tokens are first converted into numerical representations called token embeddings. Prompt tuning modifies this initial input by prepending a sequence of trainable vectors, known as pseudo-embeddings ($$p_0, \dots, p_n$$), to the standard token embeddings of the user's text. These pseudo-embeddings are not associated with any specific words in the vocabulary but are learned directly to guide the model's behavior for a specific task. The resulting concatenated sequence of pseudo-embeddings and token embeddings serves as the complete input to the LLM.

Mechanism of Prompt Tuning at the Embedding Layer

A user provides a sentence as input to a large language model. Arrange the following initial processing steps in the correct chronological order before the model begins its main computational tasks.

A large language model is given the input sentence: 'The scientist discovered a new subatomic particle, the 'quarkino'.' The word 'quarkino' is a neologism and does not exist in the model's original training vocabulary. Based on your understanding of how text is prepared for processing, explain the most likely method the system uses to create a numerical representation for the unfamiliar word 'quarkino' and contrast this with how it would represent a common word like 'scientist'.

Handling Unfamiliar Words in LLM Inputs

A large language model is processing the following two sentences: 'The cat sat on the mat' and 'The feline rested on the rug'. Assuming the model has been well-trained, which of the following statements best analyzes how the initial numerical representations for the tokens 'cat' and 'feline' would relate to each other before being processed by the main model layers?

Learn Before

Related