In the dynamic landscape of AI-assisted development, the way we evaluate the effectiveness of coding agents warrants a significant shift. When tasked with coding, these AI agents engage in a comprehensive workflow that includes planning, code generation, testing, and iterative refinement. This multifaceted process often consumes a vast number of tokens—10 to 50 times more than a straightforward chat prompt. As these agents become more adept, the frequency of their activity increases, leading to substantial costs under per-token pricing models. The implications of this are becoming apparent; for example, Uber reportedly depleted its entire AI budget for 2026 early in the year due to increased reliance on advanced AI tools like Claude Code. Furthermore, GitHub Copilot has transitioned from flat-rate pricing to a credit-based model, while Gartner predicts that by 2028, the cost of AI coding tokens may surpass the average developer’s salary.
Rather than imposing strict usage caps or seat limits—which can inadvertently stifle valuable workflows—organizations should adopt a more nuanced metric: the cost per merged pull request. This metric reflects the total development expenditure divided by the number of approved changes, thereby providing a clearer insight into the efficiency and quality of the software produced. By tracking rework and defects in conjunction with this metric, organizations can ensure that lower costs do not come at the expense of quality. The ultimate goal is to ascertain whether rising AI usage translates into more substantial software output without sacrificing standards.
To improve this cost-effectiveness, two crucial strategies should be implemented: optimize the use of advanced models based on task requirements and explore ownership of AI inference resources. Not every coding task necessitates the most powerful models; simpler tasks can often be handled by open-weight models managed in-house. This approach allows organizations to leverage premium models only for the tasks that truly warrant it, optimizing costs while maintaining high-quality output. Additionally, owning inference capacity can reduce per-call expenses, transforming AI usage into a more sustainable investment. Organizations can maintain sensitive code securely while harnessing the capabilities of AI, facilitating a balance between innovation and governance. As companies navigate the complexities of AI-assisted development, the focus should shift from merely counting tokens to measuring the tangible software outcomes achieved through this technology.
Source: Stop counting tokens, start counting what ships via MIT Technology Review