A scorecard for the AI age

OpenAI’s New Scorecard Gives Developers a Practical Way to Measure AI ROI

For months, developers and engineering leaders have struggled to answer a deceptively simple question: Is the AI we’re using actually worth what we’re paying for it? Token counts, latency benchmarks, and raw throughput tell you how fast the model runs, but they don’t tell you whether it’s making your application better, your users happier, or your team more productive. That gap between technical metrics and business value has made it hard to compare AI tools, justify costs, and decide where to invest next.

OpenAI’s CFO Sarah Friar recently addressed this head-on with a blog post introducing a practical AI scorecard. The framework moves beyond vague productivity claims and focuses on four concrete dimensions: useful work, cost per successful task, dependability, and return on compute. For developers building with AI, this scorecard isn’t just a nice-to-have — it’s a tool that can transform how you evaluate models, choose between providers, and communicate value to stakeholders.

What’s in the Scorecard?

The scorecard is deliberately designed to be model-agnostic and practical. Let’s break down each of the four metrics and what they mean for someone shipping code that depends on AI.

• Useful work — This measures the actual output that solves a real problem, rather than raw token generation. A model that produces 10,000 tokens of irrelevant text is doing less useful work than one that produces 500 tokens and answers the user’s question correctly. For developers, this means shifting from “how many tokens can we generate per dollar” to “how many correct, actionable responses can we produce per dollar.”
• Cost per successful task — Instead of reporting a flat price per token, this metric ties cost directly to an outcome. A successful task might be a correctly classified email, a bug-free snippet of generated code, or a customer support query that resolves without a human needing to intervene. This is the metric that allows you to directly compare a cheaper, less reliable model against a more expensive, more dependable one. Use it to decide which model is truly cheaper for your specific workload.
• Dependability — This captures consistency, reliability, and predictability of the model’s output. A model that nails 80% of tasks but fails in wildly unexpected ways on the other 20% may be less useful overall than one that gets 75% right but fails gracefully every time. For production systems, dependability often outweighs raw accuracy when it comes to user trust and maintenance overhead.
• Return on compute — This ties the value of useful work to the compute resources consumed. It encourages developers to think about whether the GPU cycles spent on a task could be better allocated elsewhere, or whether a smaller, faster model could deliver an acceptable result at a fraction of the cost.

Why This Matters for Developers

The AI landscape is flooded with claims. Every model provider publishes benchmark scores, latency figures, and price-per-token tables. But these surface-level numbers rarely translate directly to the real-world scenarios developers face. Your application might need high dependability for a finance chatbot, but low latency for a real-time translator. A model that wins on the MLPerf leaderboard could be a terrible fit for your use case if its cost per successful task is too high.

The scorecard forces a shift from commodity thinking to value-based thinking. Instead of asking “which model has the lowest cost per million tokens,” you start asking “which model gives me the lowest cost per successful task in my specific environment.” This is a fundamentally developer-empowering perspective: it puts the evaluation criteria in your hands, rather than relying on vendor benchmarks that may not reflect your workload.

To make this concrete, consider a hypothetical example. You’re building a code review assistant that suggests fixes for security vulnerabilities. Model A costs $0.15 per million tokens and catches 70% of vulnerabilities, but it also frequently suggests incorrect fixes that need human review. Model B costs $0.50 per million tokens and catches 65% of vulnerabilities, but its suggestions are almost always correct — meaning the cost per successful task is actually lower because you don’t waste time verifying hallucinations. A simple cost-per-token comparison would lead you to choose Model A, while a cost-per-successful-task analysis leads to Model B. The scorecard surfaces that trade-off.

Practical Steps to Build Your Own Scorecard

The OpenAI scorecard is a framework, not a library — you’ll need to adapt it to your own systems. Here’s how to start implementing these ideas today.

• Define what a “successful task” is for each of your AI features. Be specific. For a summarization tool, success might mean the summary contains all key points without hallucinating facts. For a code generation tool, success might mean the code compiles and passes unit tests. Document these criteria so you can measure consistently.
• Instrument your application to track the four metrics. Log the input, output, and cost for every AI call. Track whether the output was used directly, needed manual correction, or was discarded. Over time, you’ll build a dataset that lets you compute useful work and dependability — not just for the model as a whole, but per use case.
• Use cost-per-token as a starting point, then adjust for success rate. OpenAI’s LLM API cost calculator is a good place to get a rough cost baseline. But the real insight comes when you divide that baseline by the fraction of successful tasks. A model with a lower token cost but a 50% success rate may actually be twice as expensive as a model with a higher token cost and a 90% success rate.
• Factor in dependability as a separate dimension. Don’t just average success rates — look at variance. If your AI-powered feature occasionally fails in unpredictable ways (e.g., generates an offensive response or reveals private data), the cost of those failures can dwarf the cost of the successful calls. Dependability helps you quantify that risk.

You can also use the scorecard as a lens to compare different model providers. The LLM API pricing reference gives you the raw numbers, but the scorecard tells you which numbers actually matter. For instance, you might find that a frontier model’s higher dependability justifies its premium price for a medical diagnosis assistant, while a lightweight model’s lower cost per successful task is the right choice for a content generation pipeline with human review.

How This Fits Into the Wider Tooling Landscape

The scorecard arrives at a time when developer tooling around AI evaluation is rapidly maturing. We already have logging frameworks (like LangSmith and Weights & Biases), prompt testing suites, and quality monitoring dashboards. What has been missing is a simple, shared vocabulary for talking about business impact. The four metrics — useful work, cost per successful task, dependability, return on compute — provide that vocabulary.

It also aligns with a broader trend toward economic reasoning in AI engineering. Early adopters treated model costs as an afterthought, prioritizing raw capability. As AI becomes embedded in production, the conversation is shifting to efficiency, reliability, and measurable return. The scorecard gives developers a way to lead that conversation within their teams, rather than having it imposed from above by finance or product leaders who may not understand the technical nuances.

What This Means for Your Next Model Choice

There is no single “best” model, and the scorecard makes that explicit. The right model depends on how your application defines success. If you’re building a high-volume, low-stakes tool (like a marketing email generator), cost per successful task will dominate your decision. If you’re building a legal document analyzer, dependability and return on compute might be the critical axes.

The scorecard also encourages you to re-evaluate over time. As models improve and pricing changes, a model that was a poor fit six months ago might now dominate on the same four dimensions. By tracking these metrics continuously, you can make data-driven migration decisions instead of relying on vendor announcements or hype.

Takeaways for Developers and Builders

• Stop comparing only on cost per token or raw accuracy. Start tracking cost per successful task in your actual production environment.
• Build internal dashboards that capture useful work and dependability, not just throughput. This will help you justify model choices to stakeholders and catch regressions early.
• Treat the scorecard as a starting point. Customize it for your domain. A “successful task” for a chatbot will look very different from one for a code assistant. The framework’s value is in forcing that definition, not in providing a universal formula.
• Share your results. As more teams adopt similar evaluation criteria, the industry will develop better norms for comparing AI tools on the dimensions that actually matter to users and businesses.

The OpenAI scorecard is a timely contribution to an industry that has been drowning in raw metrics while starving for meaningful ones. It shifts the conversation from how much AI can you access to how much value can you extract. For developers who want to build reliable, cost-effective AI-powered products, that shift is long overdue.

Source: OpenAI. Details as reported; verify specifics at the source.