How We Built a Deterministic Eval Engine in FastAPI

API Request
  |
  v
Suite Load
  |
  v
Case Selection
  |
  v
Check Execution
  |
  v
Persist Results
  |
  v
Report + Metering

The check engine is the heart of the system. It loads a suite, materialises the cases, executes the checks, and persists the results in a form that can be compared later. Each step is intentionally explicit so a failure can be traced back to the exact boundary where the behavior changed.

Tenant
  |
  v
Project
  |
  v
Suite
  |
  v
Case
  |
  v
Run
  |
  v
Check Result

EvalKit is built as a multi-tenant SaaS, so the data model has to keep project-scoped records isolated without making the app hard to operate. The main product objects are deliberately small: tenants own projects, projects own suites, suites own cases, and runs own the check results that fall out of execution.

One of the more annoying implementation issues came from the boundary between SQLAlchemy and Neon around enum handling. The practical fix was to make the enum contract explicit instead of relying on implicit inference to do the right thing across ORM, migration, and database layers.

Worth calling out because it changes how you design the rest of the system. Once a schema edge has bitten you once, you stop treating type inference as a convenience and start treating it as a place where hidden coupling shows up.

• Use a dual auth strategy so trusted service calls and tenant-scoped requests do not share the same assumptions.
• Meter usage at the point where the product can explain what consumed a run, not after the fact in a vague aggregate.
• Keep Stripe webhook handling idempotent so subscription state remains stable under retries.
• Treat billing state as part of the SaaS contract, not a side effect hidden in the UI.

This integration work matters because SaaS credibility isn't only about the core engine. If the product can't handle access control, usage accounting, and billing state cleanly, a good evaluation layer alone isn't enough.

The deployment choice was deliberately pragmatic. Railway gave us a straightforward path to production without requiring Docker in the first iteration. That kept the build moving and reduced the amount of infrastructure overhead the team had to carry while the product shape was still changing.

The important part isn't the platform itself — it's choosing not to add deployment ceremony when the real work is still proving the engine, the tenancy model, and the SaaS plumbing all behave correctly together.

• Unit tests cover individual checks and deterministic scoring behavior.
• Route tests exercise the API contract and failure responses.
• Webhook tests verify billing state changes stay idempotent.
• Regression tests guard the enum fix and other schema-sensitive paths.

The test strategy is meant to mirror the product model: if the engine is deterministic, the tests should be too. If the SaaS contract includes webhooks and tenant boundaries, the tests need to make those edges visible instead of assuming the happy path covers it.

Taken together, the series shows the same idea from three angles: why the product exists, why the problem matters, and what it takes to build the engine cleanly.