PEGASUS: evaluation driven development for GenAI

Who is PEGASUS for?

• Data Scientists & AI Developers: for tuning, ablation and CI style test gates
• Model Validators: for rigorous, repeatable validation workflows
• Business users: for actionable, decision grade insights

By standardising metrics and methodology, PEGASUS helps technical teams compare models apples to apples, gives validators audit ready artefacts, and equips product owners with numbers that matter to operations and risk. 

What to measure: metrics (exact vs. subjective)

Two families of evaluation exist:

• Exact (objective) metrics produce judgment with minimal ambiguity, e.g., BLEU, ROUGE, METEOR (token overlap based) and BERTScore, BLEURT (semantic similarity). These are well defined in open source NLP libraries and are not the emphasis of PEGASUS. We use them when appropriate, but they’re readily available from standard open-source NLP toolkits.
• Subjective (judgment based) metrics are where PEGASUS focuses, akin to essay grading. Scores can vary by rater; therefore, clear rubrics and anchored scales are essential for fairness and repeatability. PEGASUS formalises this via explicit criteria, Likert style anchors, optional business context (the guidance parameter), and support for chain of thought rationales to make the “why” behind a score auditable. 

Current metric families in PEGASUS (v1.4):

Below are the core metric families and representative metrics in current release. 

• Retrieval Augmented Generation: Answer Correctness, Answer Relevancy, Context Precision, Context Recall. These assess whether retrieved context is relevant and whether the answer is grounded in that context.
• Summarisation (including PDF aware variants for multimodal sources): Faithfulness, Accuracy, Clarity, Coherence, Completeness, Conciseness. 
• Prompt quality (task agnostic): Goal Alignment, Context Relevance, Information Adequacy, Instruction Clarity and Specificity, Overall Quality. These provide a shared language for prompt quality across teams.
• Ethics / Safety (where required): Bias, Toxicity, Hallucination/Fact risk signals—used in concert with our Responsible AI controls. 

Design intent: “What to measure” starts from common needs (RAG, summarisation, prompt quality, safety) and extends to use‑case‑specific demands via a bespoke namespace (e.g., NL2SQL code correctness or conversational smoothness for chatbots).

How to measure: methodology (AI as Judge, Likert rubrics)

AI as Judge / LLM as Judge (LaaJ)

Rather than relying solely on human panels (slow, costly, inconsistent and difficult to scale) or overlap metrics (often misaligned with human preferences), LaaJ uses an LLM to score outputs against explicit criteria by utilising structured prompts. Two common modes are:

• Pointwise (single output) scoring: judge assigns a score to one output given the input (optionally a reference).
• Pairwise (or listwise) comparison: judge picks a “winner” among multiple outputs; robust for A/B or A/B/n tests.

Quality can be improved with chain of thought (CoT) judging, in context exemplars, position swapping (to reduce ordering bias), and careful prompt engineering of the evaluator. It is also important to validate the result of LaaJ against human panels.

Why Likert style scales

Subjective judgments benefit from anchored rating scales (e.g., 5 or 7 point; we expose a 1–10 with decimals for sensitivity). Anchors provide consistent interpretation across raters (or judge prompts) and make scores analysable over time. PEGASUS implements anchored numeric scales with descriptors to limit rater drift and enable thresholding in Continuous Integration feedback loop. 

PEGASUS implements LaaJ-based scoring across the various metric families described above, with evaluation rubrics encoded directly into the evaluator prompt. The framework integrates seamlessly with evaluation approaches inspired by G-Eval, DeepEval, and RAGAS-style checks, while remaining model-agnostic and compatible with both Vertex and open-source models via the platform. Below is a pointwise example of “Summarisation Coherence” rubric, judge prompt and code demo: