Google

One example of implementing pointwise scoring for segments involves using a powerful Large Language Model (LLM) to automate the rating process. The LLM generates scores for sequences, and the score for a particular segment can then be derived from the difference between these LLM-generated scores.

Automated Segment Scoring via LLM-Generated Ratings

When automating the rating of segments using a strong language model, it can be challenging to obtain human preferences directly at the segment level. One strategy is to use the language model to assign a cumulative rating score to the sequence up to the current segment, and then subtract the score of the sequence up to the previous segment. The score for a specific segment $$\bar{\mathbf{y}}_k$$ is thus calculated as the difference between the LLM's score for the sequence ending at segment $$k$$ and the score for the sequence ending at segment $$k-1$$. The relationship is formulated as:
$$s(\bar{\mathbf{y}}_k) = s(\bar{\mathbf{y}}_1...\bar{\mathbf{y}}_k) - s(\bar{\mathbf{y}}_1...\bar{\mathbf{y}}_{k-1})$$
where $$s(\cdot)$$ represents the rating score assigned by the language model.

Segment Score as Difference of Sequence Scores

Based on the scoring method described in the case study, what is the individual quality score assigned specifically to the second paragraph? Explain how you arrived at your answer.

Calculating Segment Score from a Language Model

A team is using a large language model to rate text segments by calculating the change in the total sequence score after each segment is added. They observe that a well-written, grammatically correct segment receives a negative score. Based on this scoring methodology, explain a plausible reason why a high-quality segment might receive a negative score.

Interpreting Negative Segment Scores

A team is using a large language model to automate the quality rating of a narrative. The model generates a cumulative quality score for the text after each segment is added. The score for an individual segment is then determined by the increase in the cumulative score that its addition causes. If the calculated score for 'Segment C' is significantly higher than for 'Segment A' and 'Segment B', what is the most precise conclusion one can draw?

Learn Before

Related