EvalCore is an open-source snapshot testing tool for AI behavior, licensed under Apache-2.0. It is distributed as a single dependency-free binary that can be installed on macOS, Linux, and CI runners. The tool records requests and responses from AI models into a local SQLite cassette, then replays them deterministically in CI environments without requiring network access or API keys. Users define evaluation suites in YAML, specifying targets (any HTTP or shell command, including OpenAI-compatible APIs), datasets in JSONL format, and scorers such as string containment or LLM-as-judge. During the initial live run, EvalCore captures every interaction and computes a summary of tokens and cost. Subsequent runs can replay from the cassette, ensuring identical results for identical requests and enabling fast, free, and flake-free regression testing. EvalCore supports advanced features like baseline comparison (e.g., against the main branch), agent tracing via OpenTelemetry, and historical reporting. It integrates with any programming language because targets communicate via standard HTTP or shell interfaces. The tool gates CI pipelines by exit code and highlights only regressions, making it suitable for automated workflows. Built in Rust, EvalCore is designed for performance and simplicity. It requires no sign-up, telemetry, or server component. The entire evaluation configuration lives alongside the codebase, and the cassette files can be committed to version control. This makes AI behavior testing reproducible, portable, and zero-cost in CI.
Solves
EvalCore solves the challenge of detecting regressions in AI applications after modifying prompts, models, or dependencies. It provides deterministic, offline snapshot testing that eliminates network flakiness and API costs in CI, ensuring that changes do not inadvertently degrade response quality before users notice.