AI Costs Dashboard

At a glance

Who it serves

Builders who need to inspect model usage and cost before choosing where to optimise.

What I owned

Standalone metering, cached-eval routing and dashboard library.

Key decision

Make cost, task quality and fallback behaviour inspectable together.

Current state

Public library and seeded demo; illustrative product usage is not live telemetry from my apps.

Evidence and gap

Repository, tests and demo. Real product savings, production latency and cross-app deployment are not established here.

Problem

AI product cost usually becomes visible too late. A team ships useful model calls across several products, then the bill arrives split across Google, OpenAI, Anthropic, or another provider with no clean answer to the product question: which feature created the spend, which model was used, and whether that model was necessary.

The second problem is routing discipline. Without an explicit quality-and-cost policy, teams often default to one familiar model per task. Easy jobs may use unnecessarily expensive models, while harder structured-output jobs may use a model whose fitness has not been established.

The product opportunity was to meter calls, normalise cost by product and feature, cache eval scores per task/model pair, and make routing policy inspectable. The preferred route is the cheapest candidate that clears the configured score; fallback rules apply when none passes or scores are missing.

The tool is deliberately standalone and provider-agnostic. It is not wired into private products in the repo; it is the reusable core that can be dropped into products such as nsso, Dreamsea, 24Seven, or any AI feature that needs cost and quality to be governed together.

Architecture

Model: there is no generation model inside the core. The package is provider-agnostic and governs whichever model your product already calls. The core Meter object wraps pricing, storage, routing, budgets, and reporting behind one small product-facing API, and model choice is treated as a decision the router makes rather than something hard-coded per product. The stack underneath is Python, SQLite, an argparse CLI, editable JSON pricing, unittest coverage, and an optional Streamlit/Pandas dashboard. There is no model SDK dependency in the core, because products call their own providers and the tool only meters, routes, and reports around those calls.

Context: the context layer is the Store, a local SQLite database by default with three tables: usage_events, eval_scores, and budgets. usage_events is append-only and records timestamp, product, feature, task_type, provider, model, input tokens, output tokens, computed cost, latency, and status. eval_scores caches quality scores by task_type and model so a routing decision never has to wait on a fresh eval run. budgets stores product-level limits and alert thresholds. Sitting next to the Store is the pricing layer, which loads provider and model rates from ai_costs_dashboard/pricing.json and estimates cost from input and output tokens, normalising Google, Anthropic, OpenAI, or any future provider into the same reporting model while keeping pricing editable since rates change often.

Orchestration: the Meter records every call as it happens, the router picks the model for the next one, and the CLI gives the whole loop a working operational surface through aicd seed-demo, report, route, and budget. The demo command seeds eval scores for tasks like summarise and extract-json, records usage across dreamsea, 24seven, and nsso, and prints spend by product, so the governance loop can be exercised end to end without ever connecting a real provider account. The dashboard.py surface is a Streamlit reader over that same Store, showing total spend, product count, models in use, spend by product, spend by model, spend by feature, budget progress, and cached eval scores. It is intentionally a thin observability layer over the append-only usage store, not a second source of truth.

Governance: the router sorts candidates by estimated cost and selects the first cached score that clears the threshold. If none clears, it selects the best-scoring candidate; with no eval data, it selects the most expensive candidate as a capability proxy. Those fallbacks do not establish that the quality bar was met. This is an eval-informed policy, not a fail-closed release gate. The store records usage, cached scores, budgets, and alerts; it does not supply missing model-quality evidence. Provider imports, per-user attribution, a shared database adapter, and a hosted cockpit remain future work.

Human: product judgment still sets the terms the system runs on: the quality threshold a model has to clear for a given task, the design of the evals that produce that score, the budget limit per product, and what counts as an acceptable cost/quality tradeoff in the first place. The tool does not try to automate that judgment away. It gives it somewhere to live as data, so the same judgment gets applied consistently on every call instead of being re-decided by whoever happens to be looking at the bill that week.

Why this approach

The routing policy considers cached eval scores before cost alone. It prefers a candidate above the threshold but still returns a fallback when the threshold cannot be established. A product requiring a hard quality gate would need to hold or escalate that unresolved case for review.

Eval scores are cached rather than computed during every route call. That keeps routing fast, deterministic, and cheap. The product team can run evals on its own schedule, then let production code use those scores as a decision table.

The tool starts as a library rather than a hosted SaaS dashboard because the first job is instrumentation. Product teams need a small Meter they can import at the point of model use, not another external dashboard that still depends on clean events being sent into it.

SQLite is a pragmatic default. It keeps the repo installable in minutes, makes tests deterministic, and proves the schema before a team pays the operational cost of a shared database. For portfolio purposes, that matters: the product value is the governance pattern, not cloud complexity.

The Streamlit dashboard is intentionally secondary. The durable surface is the event model and routing logic; the dashboard is a quick way to inspect it. That keeps the architecture honest and makes the hosted cockpit a future presentation layer rather than the core product.

Tradeoffs

The first tradeoff is SQLite as the default store instead of a production-grade database. That makes the repo installable in minutes and keeps every test deterministic, but it is not the right store for a multi-product team with high-volume traffic, multiple environments, user-level attribution, and long retention needs. The schema is shaped so a Postgres or Supabase adapter can replace it later without changing the Meter API, but that migration has not happened yet.

The second tradeoff is a bundled, editable pricing table instead of a live pricing feed. Keeping pricing.json as a flat file means there is no dependency on a provider billing API and the numbers stay easy to inspect and override, but it also means the reported spend is only as accurate as however recently a team updated that file. A model price change that is not reflected in pricing.json quietly produces a stale cost figure.

The third tradeoff is trusting cached scores and returning a fallback even when no candidate clears the threshold. Choosing the highest score, or the highest price when scores are absent, keeps the workflow moving but cannot establish fitness for the task. It may raise cost without improving quality. A hard hold or human-review policy is a future change, not current behaviour.

Demo / proof

The public GitHub repository includes the Python package, README, editable pricing registry, CLI, Streamlit dashboard, demo script, tests, pyproject metadata, and MIT license.

The local test suite was run against the attached codebase and passed 13 tests covering pricing load/estimation, unknown model handling, router decisions, budget alerts, eval-score upserts, spend rollups, invalid dimensions, and since filters.

The demo script proves the core behavior: summarise routes to a cheap model that clears the eval threshold, extract-json skips cheaper models that fail and moves to the cheapest model that passes, then usage is recorded and reported by product.

Demo product names, usage events, and eval scores are seeded examples. They are not live telemetry, measured savings, or production quality results from nsso, Dreamsea, or 24Seven.

The package exposes an installable CLI through the aicd command, with seed-demo, report, route, and budget subcommands.

The README is explicit about honest scope: this is a standalone reusable core, not a private-product integration pretending to be live telemetry.

What I would improve

The tradeoffs above point directly at what is next. Replacing SQLite with a Postgres or Supabase Store adapter would resolve the first tradeoff without giving up the speed that made local development easy. Multiple products and environments could write to the same governed cost ledger without changing the Meter API.

Provider import would close the gap the pricing tradeoff creates. Pulling historical usage straight from Google, OpenAI, and Anthropic billing dashboards would keep spend accurate even when pricing.json falls behind, and would make the tool useful before every product has instrumented its own Meter calls.

The router dependence on whatever eval scores it is given is also the clearest place to strengthen the governance layer. Richer eval metadata, such as eval version, dataset name, sample size, date run, owner, and regression status, would turn a single cached score into something closer to real model governance, rather than a routing decision that looks disciplined without being auditable.

The dashboard itself should grow past reporting spend and start doing the Human layer actual job: connecting cost to outcomes. Cost per successful task, cost per user journey, cost per accepted AI action, latency by feature, failure rate by provider, and drift after model changes would show not just what was spent, but whether it was worth it.

The hosted cockpit should eventually replace the Streamlit dashboard for public-facing use, but only once the library has proven the event model and routing decisions in real product traffic, the same library-before-SaaS sequencing laid out in Why This Approach above.