Metrics

Measures whether a response actually addresses the question that was asked. The RAGAS answer_relevancy metric reimagined as a learned model — captures topic alignment, completeness of address, and absence of tangential content. An answer can be faithful to context yet irrelevant if it talks about the wrong thing. This model catches that failure mode.

Canary model — a deliberately trivial concept (blatant sentiment) used as a fixed reference point for engine health. Its receipt should sit near 1.0 accuracy with near-full c90 coverage; any drop signals an engine or promise-machinery fault, not task difficulty.

Canary model — a deliberately trivial paired concept (does the response discuss the context's topic) used as a fixed reference point for the relational serve path. Its receipt should sit near 1.0 accuracy with near-full c90 coverage; any drop signals an engine or promise-machinery fault, not task difficulty.

Measures: Actionability, Motivation, Scoping, Specificity, User Impact Focus

Scores the quality of a cold outreach opener — whether it demonstrates genuine research on the recipient, offers a specific reason for reaching out, and earns the right to a reply without resorting to templates or manipulation.

Measures: Actionable Summary, Context Sufficiency, Intent Clarity, Scope Precision, Signal Density

Measures: Concrete Usage Demonstration, Copy-Pasteable Setup, Hook Speed, Hype-Free Credibility, Problem Framing, Progressive Disclosure, Structural Scannability, Value Proposition Clarity

Measures whether text communicates efficiently without unnecessary padding. Targets the specific verbosity patterns that plague LLM output: preambles, question-restating, hedging, meta-commentary, filler transitions, and redundant restatement. Based on Phoenix/Arize conciseness evaluator and ConCISE (2025) framework — information density over raw word count.

Scores the quality of crisis communication — whether it takes ownership, scopes the impact honestly, and provides clear next steps, rather than deflecting or hiding behind corporate platitudes.

Measures: Empathy & Acknowledgment, Expectation Setting, Personalization, Resolution Specificity, Tone Calibration

Measures: Developer Voice, Honest Scope, Path Clarity, Show Don't Tell, Technical Credibility, Time to Understanding, Zero Friction Try

Measures whether every claim in a response is supported by the provided context. The RAGAS faithfulness metric as a learned model — detects when LLMs hallucinate beyond their source material, add unsupported details, or confabulate facts not present in the retrieved documents. Operates at the claim level: a response with 5 claims where 1 is unsupported should score lower than one where all are grounded.

Measures whether text is genuinely funny — not just attempting humor, but landing it. Evaluates the mechanics that make comedy work: surprise, economy, specificity, and structural craft. Replaces LLM-as-judge humor scoring (Braintrust autoevals, Phoenix) with a learned model that captures what separates a laugh from a groan.

Scores the quality of a peer congratulation message — whether it's specific and warm enough to land as genuine, rather than reading as a perfunctory LinkedIn reflex.

Measures: Blamelessness, Impact Transparency, Remediation Commitment, Root Cause Depth, Timeline Specificity

Scores the quality of a prospect research note — whether it surfaces signal-bearing observations that inform outreach, rather than restating generic profile information.

Measures pun quality along the dimensions that distinguish a Jimmy Carr line from a strained, telegraphed, or over-explained attempt. Built on the bisociation theory of humor (Koestler), the Script-Based Semantic Theory of Humor (Raskin / Attardo), and the empirical finding from Kao, Levy & Goodman that distinctiveness — each meaning anchored to different parts of the carrier sentence — separates great puns from merely-ambiguous ones. Operates at the carrier-sentence level: a pun where every word earns its place, the sound-bridge is recognizable but not identical, and the resolution feels both surprising and retrospectively inevitable should score higher than one whose setup announces the joke or whose punchline restates rather than transforms.

Measures whether a response is grounded in retrieved context versus floating on model knowledge. Distinct from faithfulness (claim-level accuracy) — this is about style and posture. A grounded response reads like it was written by someone who just read the sources; an unanchored response reads like a model generating from training data with context as decoration. Targets the "context was retrieved but ignored" failure mode in RAG systems.

Measures: Commitment Specificity, Dignity Preservation, Offer Relevance, Ownership of Failure, Value Reaffirmation

Measures how concrete and specific text is versus generic LLM-style prose. The core signal that separates human-distinctive writing from AI-generated filler: proper nouns, numbers, dates, named examples, particular behavioral details. Replaces vibe-check "does this sound like AI?" with a learned model that captures the linguistic markers of specificity across domains.

Detects sycophantic behavior in AI-generated text — gratuitous validation, opinion-matching, and performative helpfulness that prioritizes pleasing the user over being truthful or direct. Based on Sharma et al. 2023 (Anthropic/ICLR 2024) taxonomy of sycophancy types. Replaces prompted judges with a learned detector that catches the subtle patterns RLHF trains into language models.

Measures: Actionable Takeaways, Grounded Motivation, Honest Specificity, Incremental Complexity, Narrative Throughline, Progressive Concreteness