A focused technical case study within World Publishing Houses covering whether AI workflows are accurate, safe, reliable, fast, and financially sustainable.
This is the AI Evaluation video from the four-part WPH technical video series.
An LLM workflow can return a plausible answer while failing on source grounding, safety, cost, latency, or repeatability. I built a combined evaluation and telemetry layer to make those failure modes visible.
I designed deterministic fixtures, trace propagation, cost calculation, usage aggregation, budget enforcement, admin APIs, dashboards, tests, and documentation for the platform.
Checks whether statements are supported by retrieved evidence and whether source context is preserved.
Flags unsafe acquisition language, unsupported rights claims, misleading completeness claims, and other prohibited patterns.
Uses repeatable fixtures to compare prompt and agent behavior before promotion or rollback.
Request-level trace propagation connects users, workspaces, prompts, agents, models, evaluations, latency, tokens, and failures.
Database-backed pricing supports per-request cost calculation, daily usage summaries, and workspace budget enforcement.
Admin APIs and dashboards expose cost, token, latency, failure, prompt, agent, and model trends plus trace-level details.