schemafit
schemafit is an open-source command-line tool that lints JSON schemas against the documented constraints of LLM providers such as OpenAI, Anthropic, Gemini, Mistral, and Cohere. It ensures that structured output and tool definition schemas will not cause 400 errors when sent to these APIs in production. Unlike runtime libraries that normalize schemas on the fly, schemafit performs static analysis during development or CI. It uses versioned, declarative rule packs that encode each provider's exact restrictions—such as unsupported keywords, structural limitations, and formatting requirements. Each violation is reported with the precise JSON Pointer path and the offending keyword, along with a human-readable explanation. A repair command automatically rewrites schemas into provider-valid variants where possible, while a lint command exits with a non-zero status to block merges in CI. The tool requires no API keys, makes no network calls, and has zero runtime dependencies, making it trivial to vendor and audit. Optional live-verify can detect when rule packs drift from actual provider behavior. Integrations include SARIF output for GitHub code scanning, and the tool is distributed as a single pip installable package. It fills the gap left by runtime clients like Instructor and BAML by providing pre‑ship, build‑time validation over the raw JSON schemas that teams already author.
- Category
- LLM
- Pricing
- Free
Problem solved
LLM API providers enforce different and often undocumented constraints on JSON schemas for structured outputs and tool calls. A schema that is valid on one provider may be rejected by another with an opaque 400 error, causing runtime failures and tedious debugging. schemafit catches these incompatibilities statically in CI, before deployment, by checking schemas against per‑provider rule packs, giving immediate, actionable feedback on violations.
Use cases
- Validate LLM tool schemas against provider constraints before deployment
- Enforce provider-specific schema compliance checks in CI pipelines
- Automatically repair JSON schemas to meet a target provider's requirements
- Detect when static rule packs are outdated relative to live provider APIs
- Prevent production 400 errors from incompatible structured output schemas
- Achieve multi‑provider schema compatibility without per‑provider manual tuning
Audience
- Developers
- ML Engineers
- Data Scientists
Pros
- Static analysis prevents runtime API errors
- Zero runtime dependencies, works fully offline without API keys
- Drift detection keeps rule packs aligned with live provider documentation
- SARIF output integrates with GitHub code scanning for security workflows
- Lightweight and CI‑enforceable with non‑zero exit codes
Cons
- Repairs are best‑effort and may alter schema intent in edge cases
- Only supports five LLM providers; no extensibility for others
- Does not offer runtime schema normalization or dynamic repair
- Requires manual authoring of JSON schemas; no generation features
- Static rule packs may lag behind rapid API changes without live‑verify runs
Official website: https://www.danmercede.com/works/schemafit
Canonical LambdaBase page: https://www.lambdabase.com/tools/schemafit