Print Icon
 

May 16, 2026

AI inference costs are turning up on earnings calls as enterprises try to get a handle on token costs as they ramp AI agents. The bigger question is whether the context layer is the way to keep a lid on AI costs, or if you just kick it old school with budgeting.


First, AI inference is increasingly becoming a budget item. Based on an analysis of earnings call transcripts, it's clear that AI inference and token costs are top of mind for the technology companies most likely to be running AI agents at scale. Here's a sampling of AI inference cost comments from the most recent earnings calls of technology companies, which are probably a quarter or two ahead of the curve.


Cognizant CEO Ravi Kumar said the company is using token metering on projects and at the individual level. Kumar said:


"As our AI productivity capabilities mature, we are increasingly applying token metering at a project or an individual level to provide early insights into usage patterns, model management, and optimization of inference costs. Token metering is a reality at both the project and individual levels. We have token metering for both fixed-price and time-and-material programs. For fixed price programs, we have the opportunity to reduce the cost and keep the margin with us."


Kumar said customers are becoming more sophisticated in assessing the costs of metering tokens and analyzing the human and digital efforts delivered by AI. Enterprises are finding that they can meter token usage for frontier model companies like Anthropic and OpenAI, but increasingly want to hand off the economics to Cognizant.


Spotify Co-CEO Gustav Söderström said the company is integrating AI across the company and it is "accelerating how we build and deliver at a pace we haven't seen before." Söderström added that Spotify is shipping faster, with greater efficiency and lower cost per feature, but inference costs are a real budget item. "You can see some of the inference costs behind that acceleration in our OpEx, but we have tremendous confidence in what we're building," he said.


Twilio CFO Aidan Viggiano said the company is seeing AI productivity gains, but he's also watching costs. "I'd say while it's an area of spend that we're watching, our inference costs are manageable," he said.


ServiceNow CFO Gina Mastantuono said during the company's Investor Day that AI reasoning accounts for "less than 10% of our cost to serve." "Customers aren't paying us for tokens. They're paying for a resolved outcome. Reasoning is one input. Workflow orchestration, governance, context, cross-system action, that's where the other 90% of the value and costs sit," she said.

Bill McDermott, CEO of ServiceNow, added that "a lot of customers are getting a little bit surprised on the tokenization of models and how that is surprising their budget landscape." McDermott said those unpleasant budget surprises are good for ServiceNow's business and hybrid seat and consumption plans.


Can context layers help with AI costs?


Like ServiceNow, Atlassian sees itself as a fix to AI inference costs. Atlassian CEO Michael Cannon-Brookes recently outlined the company's traction in using its platform as a context layer with its Teamwork Graph. In other words, the right context at the right time with the right latency is going to lower your AI inference costs.


"We showed a stat that talked about how we were getting 44% of better quality in terms of output and 48% cheaper rate. You're using 48% fewer tokens to get to that same or better outcome. That's a conversation with your CIO. That's a conversation with your CFO that's going to resonate. And so, I think both things are true," said Cannon-Brookes. "I'm not sure most CFOs really like the token maxing meme, to be honest. I think they are starting to see and pay bills that this magical revenue pile is not coming from nowhere."

Is the context layer the cure for your potentially runaway AI inference and token costs? Perhaps. Enterprise technology buyers will need some way to evaluate the returns on context layers. After all, every vendor — Atlassian, ServiceNow, Boomi, Microsoft, Salesforce, Workday, SAP, Celonis, and a bevy of others — now has a context layer.


I could run a dozen slides of various context layers, but here’s Atlassian’s play.

What follows is a framework for evaluating context layers. The working theory here — and it is early — is that a context layer should save you money by getting you the right answers faster and with as few tokens as possible. There won't be a single context-layer winner, but you should be able to maximize returns without a dozen vendors playing the context game.


There are caveats with boiling context layers down to a metric. For starters, a context layer is really an enterprise AI infrastructure. The context layer is there to provide AI systems with the right business meaning, data, permissions, memory, and provenance at the right time, with the lowest costs and risk. Ultimately, a context layer must provide AI agents with the right answers to enable accurate, low-rework action.


Context layers also need to deliver precision, recall, a strong hit rate with minimal noise, and good sourcing. On the technical level, a context layer should be evaluated based on the groundedness of its answers, citation accuracy, and human acceptance rate. Databricks' RetrievalGroundedness judge is worth a look.


With those table stakes checked off, context layers will differentiate based on accuracy, adherence to the system of record, ontology, and complete data lineage. The money round for context layers will likely boil down to a scorecard of metrics. This context layer scorecard could include the following:

  • Task completion rates.
  • Percent of cases resolved without human intervention.
  • Reduced rework.
  • Cycle time reduction.
  • Decision accuracy.
  • Token cost and cost per successful task.
  • Cost avoidance, where AI costs are compared to a baseline human cost.

The costs boil down to context layer platforms, graph infrastructure, data integration, ontology work, governance and security, observability, model and token costs, and maintenance and human review.

What will make or break context layers will be the cost per accepted outcome. Other metrics, like the number of AI queries or tokens processed, are merely vanity plays. 


Will on-prem or open weight models provide the answer?


One of the more interesting riffs of the week at Boomi World 2026 was CEO Steve Lucas' take on open models and economics.


The short version:

  • Proprietary models at the present moment are sucking you in. Prices will go up. And you'll be locked in before you know it.
  • That concern was partly what drove Boomi's acquisition of Lunar.dev. The idea is that Boomi's platform will need to route traffic to the most cost-effective models.
  • In the end, open weight models win, argued Lucas.

"They cost money to run from a compute standpoint, but they are profoundly less expensive. Do we continue to pay 10 times what we paid last year for a proprietary frontier model? Or do we start doing what enterprises always do, which is to pursue the lowest-cost option with the highest rate of return? ROI always wins. It will profoundly supersede AI at all times," said Lucas.

Open weight models are also closing the gap on frontier models. "You'll need enterprise-grade intelligent prompt routing that distributes prompts, requests, actions, and activities to the right model with the most economically productive outcome," said Lucas.

And don't forget the on-prem play with open weight models. Boomi is emphasizing Distributed Agent Runtime with a focus on hybrid cloud and on-prem deployments. Boomi Distributed Agent Runtime enables customers to run agents locally in their own infrastructure, solves for data residency, resiliency, and cloud cost concerns, and runs agents behind the firewall.

That announcement was overlooked, as was Boomi's partnership with Red Hat. Boomi said it will integrate Boomi Agentstudio with Red Hat AI and create an integrated stack for on-prem deployments. Rest assured, open-weight and low-cost models will be included in that Red Hat and Boomi bundle.


Why AI FinOps will explode


Whether it's context layers or platforms, as outlined by Boomi at Boomi World 2026, the AI budget is going to be scrutinized, and it's a bit fuzzy how the ROI equations will look. Today, the idea that AI agents will deliver returns is partly based on blind faith.


Mike McLarty, CTO of Boomi's Innovation Group, said customers are spending heavily on AI on migration, process, and development work, but the biggest wild card is token costs. "Token cost is probably the biggest variable and the biggest risk factor," said McLarty.


The biggest challenge for AI costs will be the end of what is a "heavily subsidized market right now for tokens," said McLarty. Subsidized? How can you be subsidized for token costs when you're paying Anthropic 10x more than you were a year ago?


Circular deals in the AI industry between hyperscalers, chipmakers, and foundational model players are subsidizing token costs. At some point, that subsidy will run out. The industry's bet is that enterprises will be locked in before costs really surge.


"Who knows where the token costs are going to go, but we know it's going to be a big line item for CFOs," said McLarty.

     

NOTEBOOK


📒 For the latest on Cisco earnings, Boomi World 2026, and more, see Insights and Notebook.


🔀 SAP at Sapphire 2026 is making the case that it should be the platform enterprises use to become autonomous as companies overhaul workflows and processes for agentic AI.


🚀 Celonis said it has acquired Ikigai Labs and launched the Celonis Context Model (CCM) to meld process intelligence, decision intelligence, and foundational models for operations.


🫱🏽‍🫲🏾 Siebel will return as C3.ai’s CEO, and Stephen Ehikian, who had been CEO after Siebel stepped down due to health issues, will remain as the company's president.


🚀 OpenAI launches OpenAI Deployment Company, acquires Tomoro.


☁️ Alibaba's fourth-quarter earnings were disappointing, but the cloud unit continues to surge. The cloud intelligence group reported 40% revenue growth, with AI-related products accounting for 30% of that revenue.


💰 Anduril raised $5 billion in a Series H round and now has a valuation of $61 billion. The defense contractor, founded in 2017, is considering an IPO in 2027.


🔔 Quantinuum’s IPO: What you need to know.


👨🏽‍💻 Google Cloud said it will be hiring more forward-deployed engineers (FDEs) as part of its new AI-focused organization. In a LinkedIn post, Google Cloud CEO Thomas Kurian outlined the hiring plans.


🛍️ Amazon ditched its Rufus shopping AI assistant in favor of Alexa for Shopping.


🚀 Anthropic launched Claude for Small Business.

     

Insights Archive

Like getting our weekly round-up? Subscribe to Constellation Insights for consistent news and analysis on technology trends, earnings, enterprise updates and more.

Want to work with Constellation analysts? Our Sales team would love to get in touch!

You are receiving this email as you signed up for our newsletters.

Want to change how you receive these emails?

You can Unsubscribe or Update your preferences