Combining LLMs with human judgement by means of Prediction-Powered Inference (PPI)

In enterprise analytics, calculating the Web Promoter Rating (NPS) sometimes includes guide information annotation from staff. Some might imagine to make use of machine studying fashions to label the info, nonetheless this doesn’t have the theoretical ensures we get from human labeled information. Enter Prediction-Powered Inference (PPI), a brand new statistical method that mixes human and machine labeled information to create confidence intervals which might be information environment friendly and theoretically assured.

This text explores the instinct behind PPI and emphasizes why you’d need to use it. We then bounce right into a code walkthrough of methods to use it for 2 metrics: NPS and buyer suggestions.

PPI is a statistical method proposed by Angelopoulos et al. [1]. The objective is to reinforce confidence intervals by combining human and machine labeled information. Let’s stroll by means of some steps to encourage its usefulness.

In our use case we need to estimate the true NPS rating given a set of buyer critiques. Sometimes, an worker will manually learn every overview and assign a rating from 1 to 10, a dependable however time-inefficient methodology. When coping with quite a few critiques it could be handy to have a extra automated methodology.

To handle this, we are able to leverage a machine studying mannequin. A Giant Language Mannequin (LLM) is an efficient candidate to resolve this drawback as a result of they generalize effectively to new duties. The mannequin is prompted to learn the overview and output a rating. That is handy, however the mannequin comes with errors and imperfections. When making a call, we’d like to ensure our information is aligned with human judgement.

Contemplating the restrictions of each approaches, what if we might mix them? We are able to with Prediction-Powered Inference (PPI)! PPI is a framework that leverages the theoretical ensures of human-labeled information for confidence intervals and the effectivity of machine-labeled information. With PPI, we purpose to profit from the strengths of each strategies.