LLM Eval Kit

Evaluate and benchmark LLM outputs with structured test suites, scoring rubrics, and regression detection — ensure your AI features maintain quality over time.

Features

• Eval Suite Builder — Creates structured evaluation datasets with expected outputs and scoring criteria
• Multi-Model Comparison — Benchmarks the same prompts across different models side by side
• Scoring Rubrics — Configurable rubrics for accuracy, relevance, tone, format compliance, and safety
• Regression Detection — Alerts when prompt changes degrade output quality below baseline thresholds
• Report Generation — Produces detailed evaluation reports with pass rates, score distributions, and trends

Quick Start

claude skill install llm-eval-kit

Usage

# Create an eval suite for a classification prompt
/llm-eval-kit create --task "ticket-classification" --samples 20

# Run evaluation and generate a report
/llm-eval-kit run --suite evals/ticket-classification.json --output report.html

How It Works

Builds evaluation datasets from your prompt use cases with expected outputs and scoring criteria. Runs each test case through your prompt pipeline and scores results against the rubric. Tracks scores over time to detect regressions and generates visual reports comparing prompt versions and model performance.

Compatibility

Works with Claude Code, Codex, Gemini CLI, Cursor, Windsurf, and Aider.