Google

In mathematical expressions involving parameterized models, it is a common convention to simplify the notation by omitting explicit parameters. For instance, superscripts like $$\mathbf{W}$$ (representing Softmax weights) and $$\theta$$ (representing encoder parameters) may be dropped from probability distributions for brevity, even though the dependency on these parameters is still implied.

Omission of Parameter Superscripts in Language Model Probability Notation

A colleague is reading a research paper abstract that states: 'We introduce a novel neural network for a task. The model is trained to maximize the conditional probability p(y|x), where x is the input and y is the output.' Based on this, your colleague concludes that the model must be non-parametric (i.e., has no learnable parameters) because the probability expression does not explicitly list any parameters. Evaluate your colleague's conclusion and justify your reasoning.

Interpreting Model Notation in a Research Context

In the context of parameterized machine learning models, the mathematical expression `p(y|x)` indicates that the probability of output `y` given input `x` is calculated without relying on any learned model parameters.

A research paper on language models presents the probability of an output token `y` given an input context `x` in two different ways:

Expression 1:  `p(y | x; W, θ)`
Expression 2:  `p(y | x)`

Assuming both expressions refer to the same underlying model where `W` and `θ` are the model's parameters, what is the most accurate interpretation of the relationship between them?

Learn Before

Related