EvalCore
EvalCore is an open-source snapshot testing tool for AI behavior, licensed under Apache-2.0. It is distributed as a single dependency-free binary that can be installed on macOS, Linux, and CI runners. The tool records requests and responses from AI models into a local SQLite cassette, then replays them deterministically in CI environments without requiring network access or API keys. Users define evaluation suites in YAML, specifying targets (any HTTP or shell command, including OpenAI-compatible APIs), datasets in JSONL format, and scorers such as string containment or LLM-as-judge. During the initial live run, EvalCore captures every interaction and computes a summary of tokens and cost. Subsequent runs can replay from the cassette, ensuring identical results for identical requests and enabling fast, free, and flake-free regression testing. EvalCore supports advanced features like baseline comparison (e.g., against the main branch), agent tracing via OpenTelemetry, and historical reporting. It integrates with any programming language because targets communicate via standard HTTP or shell interfaces. The tool gates CI pipelines by exit code and highlights only regressions, making it suitable for automated workflows. Built in Rust, EvalCore is designed for performance and simplicity. It requires no sign-up, telemetry, or server component. The entire evaluation configuration lives alongside the codebase, and the cassette files can be committed to version control. This makes AI behavior testing reproducible, portable, and zero-cost in CI.
- Category
- AI Testing
- Pricing
- Free
Problem solved
EvalCore solves the challenge of detecting regressions in AI applications after modifying prompts, models, or dependencies. It provides deterministic, offline snapshot testing that eliminates network flakiness and API costs in CI, ensuring that changes do not inadvertently degrade response quality before users notice.
Use cases
- Regression testing for prompt changes
- Validating model swaps or updates
- Testing dependency upgrades for AI services
- Offline CI evaluation without API keys
- Baseline comparison across git branches
- Tracking token usage and cost per evaluation
Audience
- Developers
- ML Engineers
- Data Scientists
Pros
- Deterministic offline replay eliminates network flakiness
- Free and open-source with no API costs in CI
- Works with any language via HTTP or shell targets
- Single binary, easy to install and integrate
- Baseline comparison gates PRs only on regressions
Cons
- Requires an initial live run to record interactions
- Snapshot testing may miss nuanced quality changes if not properly scored
- LLM-as-judge reliability depends on the scorer model
- No hosted or managed service option
Official website: https://evalcore.cc/
Canonical LambdaBase page: https://www.lambdabase.com/tools/evalcore