Your data center has a near-perfect PUE, and you still can't say whether AI is paying off. Now what? Dr. Serena Huang talks with Vernon Turner, IoT and Sustainability Advisor at Cognizant, and Val Bercovici of WEKA, about rethinking how we measure AI value.
PUE can tell you how well a data center uses power, but not whether AI is creating value. Dr. Serena Huang sits down with Vernon Turner, IoT and Sustainability Advisor at Cognizant, and Val Bercovici, Chief AI Officer at WEKA, to break down how leaders should be measuring AI. They talk about what to measure beyond PUE, where companies should start, and how to tell whether AI is actually paying off.
Timestamps:
0:18 - Meet Vernon and Val
0:44 - From watts to outcomes: why PUE falls short
5:51 - Token efficiency and cost per task
8:20 - Is every AI seat worth the same?
11:15 - The SaaSpocalypse and the costs nobody's counting
13:36 - Cloud inference and broken ROI promises
15:25 - A new framework: tokens per watt to tokens per workload
19:21 - Why neo clouds can't build for inference
21:51 - Where to start measuring AI workloads
27:22 - Getting finance, procurement, and ops to the table
29:38 - Trillions of tokens and the AI payback question
31:54 - Bubble wrap, not a bubble
32:44 - What outcomes per token really means
37:34 - Saturated benchmarks and the search for better evals
40:59 - Where we'll be in six months
Links:
[00:00:00] Val Bercovici: PUE of 1.5 or two or above kind of disqualifies you from even having a, an efficient data center conversation.
[00:00:07] Vernon Turner: I was there at the beginning helping to create the PUE framework. So it's, uh, ironic that, uh, 20 years later I'm there going like, you know, it may have served its time.
[00:00:25] Serena Huang: Welcome back to another episode of "Deep Geeks." I'm your host, Dr. Serena Huang. Today I'm joined by Vernon Turner from Cognizant and Val Bercovici from WEKA. Thank you so much for joining us today.
[00:00:40] Val Bercovici: Glad to be here. Really thrilled to be back, Serena.
[00:00:43] Serena Huang: Yes. Wel- welcome back. This is going to be a great follow-up conversation to our previous episode.
Um, we have spent the last couple decades getting very good at measuring data center efficiency with PUE, and today we're going to dive into that. I know this, in the past couple years we've gotten really good at measuring something else, AI activity, I will call them, right? How much compute, how many tokens even, how many users we have.
But that doesn't really tell us whether or not AI is creating value, so we're going to talk about that today all the way down the stack, from watts to tokens to workloads to outcomes. And then we're going to share with our listeners what leaders should actually be measuring in this new AI economy. So, so excited to get started.
Um, I'm gonna start with a tough one for Vernon, because you have said this very provocative statement that PUE, which we have been using for two decades, is just not enough. Tell us why.
[00:01:53] Vernon Turner: Yeah, and, and it is a tough one because, um, I was there at the beginning helping to create the PUE framework. So it's, uh, ironic that, uh, 20 years later I'm there going like, you know, it may have served its time.
And, um, that's not to say it's, it's still a bad metric. I think it's, uh, a metric for the time when we were looking at usage and trying to figure out how much density could we pack into a data center, um, but we're measuring the power outside coming into the data center. So if you think about that, that metric alone, how much can we pack into the data center over the last 20 years, has just gone off the scales.
It's not even Moore's law. It's, you know, it's a whole different dimension of how you should look at that. And that made me start to think about, well, you know, 20 years ago, we would measure the effectiveness of the power consumption, the power usage. In other words, how much power was coming into the building and how much was being used, and come up with a very straightforward, uh, user friendly metric.
And, um, just for the sake of history, um, if you had, uh, a metric that was, uh, 2.0 over two, then that probably meant that, um, you were only as half as good at what the energy coming in to you using it. In other words, you had leakage all over the place. And over the last 20 years, you know, our data center ops, our data center team, uh, everybody's gotten pretty good at cutting that down if you're a on-prem data center.
And if you're one of the large, uh, scalers, hyperscalers, you're probably running very efficiently anyway at, um, at, at some metric like, you know, 1.2, 1.1. And as a result of that, um, we're kind of there. W- how much more can we get out of the curve? How much more efficiency is gonna ma- matter? But, um, you know, Serena, you mentioned one thing that, that I think is important.
You talked about efficiency, and this is where I would push back on everybody saying it's about... the PUE is about usage. And today we really are talking about how efficient can we run a workload based on AI. And that gap is what brought me to, to this whole conclusion that PUE just may not be the way, way we should need to go.
That maybe we need these other metrics that measure how efficient we're coding, um, h- when, when we link it to AI up the stack all the way up to, um, the output itself. And, and one of the things that I-I-I talk about is this AI commerce. So in other words, what are you generating from AI, uh, either by, um, agents by themselves, and can you measure that cost that took it to the revenue that it created?
And therefore, that's a completely different discussion when we talk about the data center, uh, efficiencies. So...
[00:04:59] Serena Huang: I, I agree, and that sounds so, so much more challenging than just measuring PUE all the way, you know, the, the output. Yes, I think that's what most business They do want to get there, but it's very difficult.
So we'll dive in today in how do we get there. Um, Val, let me ask you, how does this resonate with you so far?
[00:05:24] Val Bercovici: Oh, completely. I mean, this is the most important foundation for just understanding are we even in the right ballpark, right? If we're looking at efficient AI and sustainable AI, we've got to at least start with a decent PUE in the, in the ranges that Vernon mentioned earlier on.
A PUE of 1.5 or two or above kind of disqualifies you from even having a, an efficient data center conversation. Uh, I proposed something earlier on, I don't think I brought it up last time, but about six months ago when I was getting into this more deeply in tokenomics and all that, the notion of token unit efficiency, right?
Uh, that was something that was an interesting notion, and it still is. It's still kind of a foundational way to approach this. There are no standards for measuring this yet, right? So Vernon, maybe it's something we can work on, but it... Because tokens are such a transparent metric in the industry, you know, differently, uh, on inference obviously than on training.
With training, it's weights and checkpoints, and that's a little bit too opaque for my liking. But inference dominating the AI spend nowadays is, is, is super transparent around input token costs, output token costs, and particularly cached read and cached input token cost as a hyper-efficient version of that.
So that's one way to take the conversation, but in hindsight now, given the rapid advances we've seen with agents and the mainstreaming, uh, just recently last week with Meta, Facebook coming out with a very, very popular Muse agent now, number one in the App Store, at least in the US, I think, that is a clear step function increase in utilization, you know, data center utilization for AI inference for this very heavy agentic workload that is now persistent, right?
Twenty-four/seven. It's no longer just event-driven. And I think the engineering concepts of goodput, of making sure that we're actually generating useful output for energy and, and other resources invested is more and more important right now. And one of the, the trendsetting, I think, indicators from OpenRouter is no longer just measuring very well as they do in their leaderboards, cost per token, KV cache hit rate, et cetera.
They're now talking about cost per session And they're, they're showing leaderboards for cost per session. And other folks outside of the actual economic side, uh, are, are really pointing towards newer kinds of benchmarks focused on cost per task and, and these new Pareto curves about how are these new models doing in terms of quality and performance, and then how are they doing in terms of cost per task?
Are they burning thousands of tokens per task or merely hundreds or tens? So that's where I'd love to see the conversation going, is not just token usage efficiency, which is, I think, the next step forward. It's session, AI session u-usage efficiency, and ultimately outcome-based, task-based usage efficiency, uh, where we should be taking this conversation.
[00:08:26] Vernon Turner: I, I, I think you really, really hit s- uh, something on the head there, and that's this, the idea of the session. And I know you talked about it a little bit in the past about is the, you know, first of all, is the PUE model broken, but is the SaaS-based model broken as well? And, and, and when I think-- when I think of what you just said there, um, I, I, I gotta believe that, that the SaaS model, not that I'm trying to put SaaS out of business, but what I'm saying is, is how you as, as a customer look at, at, at a model like, like that to measure your AI.
You know, because, um, you know, is every AI seat the same? And are you getting the same value out of any AI, any, each AI seat? And is one person more, more valuable than another person, depending on what they're doing? Uh, and, and so, I, I mean, I'm not necessarily saying, you know, let's go down this, this, this path here, but, uh, but it, it, it ties in with the whole framework of what we need to do with AI and how to measure the value that it brings by looking at, at both...
Well, let's start with, we looked at the infrastructure ever so slightly with, with the data center PUE. But now we have to look at the, at the SaaS part of it as well to say, you know, how do you charge me? I, I'm, I'm procurement, or I'm, uh, finance or, or worse than that, I have to do chargeback. Heaven forbid cost recovery comes into this model.
And I, and I would ask any in the audience is how many of you implemented a chargeback sys- system that works? And then you'll hear crickets in, in the audience or people will duck because it's, it's fundamental to, to... You know, Val and I probably remember the day, you know, for you to be able to provide new hardware, you had to cost recover the use of that hardware over time, and therefore you had to be able to charge back the cost of that to the, to the unit, business unit.
I think the same thing here, that's where this discussion with, with AI and tokens needs to move forward. And, and, um, I'm not sure if the software... Like, I'm not sure the software model is also ready for that type of thing. So, um, again, probably an opportunity for, for, uh, a lot of good work to be done there.
[00:10:36] Serena Huang: You just pointed out something that I don't think we quite talked about on the show as far as measurement. We've talked a lot about measurement, but I think we have stayed at the organization company level or system level in aggregate. What you're pointing out, and I think we all know this intuitively, is everyone consumes and uses AI in different ways, and therefore produces AI in, you know, there's good output and there's less good output.
Um, but the measurement, the pricing model, at least in the SaaS world, is that they are all identical, and that creates, to your point, probably a bigger problem now than traditional SaaS, right?
[00:11:18] Val Bercovici: I'd like to really pull on this thread, so I'm glad you brought it up, Vernon. So, um, the SaaSpocalypse has been discussed now most of this year, and it's funny because the conversation goes from, you know, these popular memes of, you know, it's so over to we're so back and it's so over.
There's been a lot of ups and downs in terms of just looking at Salesforce as a proxy, right, for the SaaS industry. Uh, I still think we're only talking about half the conversation because a- apparently most of the industry and other podcasters and, and influencers are happy to talk about have there, has there been a product market fit for agent-enhanced SaaS, and, and can these companies charge for some agentic features?
And they are starting to find, in fairness, product market fit for extra value, pr- presumably at extra revenue for these AI-powered features and, and Agent, Agentforce and all these names. But everyone keeps ignoring the costs. Everyone keeps ignoring the unsustainable, inefficient, you know, inference that's being used right now.
Most of, most of the inference is not leveraging cache reads. Most of the inference is doing redundant pre-fills, which is the biggest sin of inference right now that almost every inference provider is committing. So we're still having these very, very incomplete conversations, and there's even a very loose proxy, you know, with a highly anticipated Anthropic IPO and the S1 filing.
As people are just digging into that, they're realizing Anthropic's revenues are breathtaking. They're outstanding, their revenue growth. But their training costs are even bigger, right? And it seems like, uh, everyone wants to ignore these massive, unpredictable, kind of lumpy training costs when you actually factor in the overall profitability of a pretty high-profile business like Anthropic.
So w- we need a much more holistic view of what's happening in AI. We need a much more granular view, to your point, Serena, about we're seeing quite a bit of diversity, not just within different kinds of agents, but within the same agent swarms. There are background agents, there are, uh, you know, latency sensitive foreground agents.
There's all sorts of other parallel agents in an agent hierarchy. So, uh, it's, it's, it's a research-rich topic and field, and it's young and evolving quite dynamically and, and we need to be... we need to have that, that really complete holistic view of it all to begin to measure it, understand it, optimize it.
[00:13:41] Vernon Turner: I think, I think one of the, one of the unintended consequences of, of AI running on a cloud service provider, um, is, is exactly that, where you do your inferencing from. And if you use the old models of simply saying, "We're just gonna lift and shift my infrastructure hardware," then you're, you're, y- you're caught in a, in a really difficult situation here of, of that, that ROI that you thought you were gonna get out of migrating to the cloud Really starts to take on a bit of strain and pressure.
And, you know, there's a part of me thinks like it would be nice to just leave some of the infrastructure at home where I could do the inferencing, and then, uh, you know... Sorry, the training, and then do the inferencing up in, up in the cloud. Um, and, and again, that, that would help with some... Maybe that's part of the s- the software model that we, we see in the future, uh, for someone like Salesforce, whether they, you know, whether you have a hybrid model or whether you have an on-prem model and, and what, and how that all fits together with, with, with AI.
But, you know, that, that brings a lot of complexity and, and people's heads start to spin and hurt wh- when, when you go down that path. But I think it's... I think we're gonna have to face that reality, otherwise we're gonna have a situation of, of, um, broken promises. You know, whether it's a financial promise or a value promise, or even an infrastructure promise that said, you know, "This is, this was gonna bring me all these new products and all these new re- new savings of, uh, of, of efficiency."
Whereas it might bring a whole generation of, of new, um, IT professionals that, that could help us... the situation, so.
[00:15:25] Serena Huang: Yeah. Great, great conversation so far. Um, loving all of this. So now let's go back and I wanna make sure we give our listeners something concrete to take away. So if PUE is not the right path forward, what is?
Vernon, can you get us started, and maybe just paint a picture as well if you have something much more concrete inside the company, um, help us visualize what the better metric is.
[00:15:54] Vernon Turner: So, so I, I have a framework that, that I, I, I sort of played with. And my point is, I think PUE is a foundational part of all this journey that, that I drew a framework of.
And I, I think right, you know, r- right above that is, um, is the whole idea of if you know how much energy you're using, then how many tokens per watt is... it would be the next logical step for that. And the- therefore, you could begin to understand, okay, what type of, what type of, um, configuration am I doing?
You know, you know, Val, to your point about caching, about low IO, about, you know, the whole idea of, of s- of server and storage and network, um, efficiency and capacity. Um, but I think, I think we can get... we can then go to, you know, steps a little higher where you can start to go tokens per person, and that kind of dovetails into that whole software model, um, of shipping from, uh, from pure cons- uh, from, from subscription...
from a consumption model to a subscription model, should I say, um, and vice versa. Um, I, and then I think at that point you could start to look at workloads. And the reason why I bring in workloads is, is at, at IDC, um, I help run, um, the workloads, uh, research arena, which means in the old, let's call it the old days, you know, if you had file and print or transaction or batch or real time, whatever, you know, um, remote, et cetera, you could bucket those into workloads, and you can do the same nowadays on, on, on whether it's, you know, retail or, or it's finance or whatever, whatever application creates all put together to make a, a viable product or workload.
I think now at that point we, we could do the same with tokens per workload. And once you get to that area Then I think you're, you're on the golden journey of saying, "Now I see where the costs are going. Is it all in one particular vertical that I support? Is it all going to one particular application or workload that I'm doing?"
I... At that point, I can join together the lower stack and the middle to upper stack here and calculate the cost. So that's where I think we, we end up with. So, so instead of just looking at the PUE of somebody saying, "I'm really efficient. Look at me. I should get a prize from, you know, um, the lead, uh, consortia."
Um, versus standing up and saying, "I can look at my business units and say, 'This is how much your application's taking on AI.'" Because now I know the, the token costs per workload And I think once you get to that, then you can move to the whole i- idea of, of, of, of, uh, tokens per o- for the output, because then you can start to mix in the, the business outcomes part of that.
You know, what am I actually selling? What am I generating? And that's when you, that's when the sales and finance team start again. So, um, that's the kind of arc that I would, I would take if I just made the statement that PU is data. I, I would say no, but it's... This is, here are the ones that, that, that are built off that, you know, to be able to do that.
So, um, you know, Val, a- a- a- you know, any- anything that you see like, "I got you, Vernon. There's a whole big, a big massive hole in that, in that, in that framework." Um-
[00:19:23] Val Bercovici: No, again, there's
[00:19:24] Vernon Turner: a, there's- Sorry, trying to do Serena's job. Sorry.
[00:19:26] Serena Huang: That's okay. That's okay. I like it.
[00:19:28] Val Bercovici: Instead of a hole, there's actually a... Again, this is all important layers that we're adding together.
Uh, what I'm seeing is we need these kinds of metrics almost as financial instruments, because one of the problems I'm seeing right now is there's what I call the golden handcuffs of, of the training revenue in this industry. Financially, most investors are much more comfortable investing in a stable three-year GPU offtake contract.
So what are data center providers building? What are the GPU providers and neo clouds building? Very training optimized inf- sorry, training optimized AI factories that are not inference optimized. It's fundamentally a different reference architecture, different network design. So we're really hamstringing the opportunity to do AI more efficiently if the majority of AI is inference right now.
We need to create metrics, such as we're discussing here, for financial instruments to be available for investors to underwrite inference optimized clouds. That's actually not happening right now. I deal with most of the top neo clouds around the world on a, on a wei- weekly basis, sometimes daily, and they all literally can't implement the new inference optimized reference architectures because they've sold out their capacity to training GPU offtakers.
So the, the need for aligning these financial incentives better via better metrics, via, you know, uh, better instruments is critical right now. Otherwise, we're gonna be in this uncomfortable, maybe even trough of disillusionment for a while because- Oh, we are. Yeah ... we're not getting anywhere near the level of token and session and task-based usage, usage efficiency that's possible
[00:21:14] Serena Huang: Yeah.
That's a great point, and I, I think the connection back to workload is, um, something that I see missing in a lot of conversations right now. Um, we are measuring kind of at the beginning of the arc that you mentioned, Vernon.
[00:21:31] Vernon Turner: Yeah, yeah, yeah.
[00:21:32] Serena Huang: Frankly because it's easy, right? Because we, we know how to do that.
Yeah. And dashboards are ready. When we want to talk about workload, oh my gosh, now suddenly you're talking about, well, which apps, which teams? Right. And we don't have that ready. Now someone has to go do some engineering work to even have that metric. Yeah,
[00:21:54] Val Bercovici: yeah.
[00:21:54] Serena Huang: Yes, and no one loves to do that. Um, I- So- I'm curious, Vernon, if you have, um, something concrete for, for companies that do want to do this work, how do they even get started?
If I'm starting from zero, I was just e- measuring the easy stuff. Mm-hmm. How do I go from that to something more intelligent like workload?
[00:22:16] Vernon Turner: Right. So that's a very, very good question. And, and it's... Let, let me use the analogy, then I'll, I'll get how it, how it covers, uh, overlaps into workloads. Um, I once did a project for, uh, a very large, um, beverage company, um, that makes, uh, a cola product.
Um, and they wanted to move the workloads from, um, from, from into, into the cloud, and they just thought they would lift and shift everything, and, um, hope that everything would work. And, and my, my argument there was, "Have you done any prioritization in, in the workloads at all? Have you, have you looked to see...
To applications, have you seen to which applications are right for sh- uh, right for fit? Fit for it." Um, and be able to go down that path of saying, "Let's put some in... Let's put, let's put some smarts around the workload and applications that will give you the most benefit, first of all." And you would be really surprised at how few people have done a rank, rank scaling of, of, of applications based on consumption on whether it's hardware, uh, software, or what have you.
The same thing has to happen again with, with, with, uh, the AI workload. I think, I think you want to have a, a rough handle on which app- which AI applications are, are, are Yeah, which applications are consuming most. And, and, and it's important part here is, is, you know, Val talked about, you know, the, the training and inference.
What's important to you? Is it, is it better to be well done on, on the inference piece and be able to... Sorry, training piece, and be able to easily collect the information? The difference with workloads for AI versus workloads at traditional workloads is traditional workloads were bounded. They really were sort of like generally stuck in their own world, whereas AI workloads are, are like crabgrass.
You know, that's, that's the only way I can describe it. You know, they go out to everything, and they can impact a lot more areas that, that you didn't have to worry about before when you're trying to measure a particular application or a particular workload. Because the definition of, of what the AI is, is, is trying to do, um, you know, through agents and prompts and whatnot, even multi agent, is, is gonna really complicate that sizing effort for you.
So my, my take on it is, it's, and it sounds like motherhood and apple pie, but you have to just pick one or two just to get your hands around the idea of how to measure those, the, you know, the, the, the token count. But be prepared for the complexity curve to really ramp up, because that's, that, that's, that's one of the things that, um, I, I see.
The, the, the second part is, is the whole governance model for, for I'll call it AI workloads. There has to be a, a, a governance agenda built around the, the fact that who can write the application, who can run the application, and who sees the output for the application. And under AI rules, I think that, that, that all changes again, um, for all, for all the right reasons.
You know, I... Having a financial services background at, uh, Fidelity and State Street, you know, I, I know a lot about, um, auditing, and because you're, you know, if you're, if you're a, if you're a nickel short, something wants to know why, 'cause a lot of nickels over time make a lot of money. And, uh, so that's the reason why.
Um, so the same thing is, is here. I don't see enough governance being brought in. And, and the reason why I bring it up, you know, Serena, is it's all part of that workload measurement where, where the, as I say, the complexity curve for AI is going to be a real, really, uh, a difficult row to haul. And I, and I think you said it earlier, it's hard work.
And to be honest with you Um, Val may appreciate this, but not many people appreciate. I, I, I look at myself as a plumber, and, you know, it's a pretty boring job, but God help you when you need a plumber. You, you want a good one, you don't want a bad one. And, and I see people looking at all the stuff that, that, that we might be talking about today as, "Oh, that's really boring plumbing work."
But it isn't, because it's all to do with the financial output at the end of the day.
[00:26:52] Val Bercovici: Yeah.
[00:26:52] Vernon Turner: And, and that's where I, that's where I think that, uh- When you're starting with the AI workloads, you need the procurement, you need finance, you need security, you need ops, you need businesses all to be there to understand what it is that AI workload looks like, as opposed to just having a, having a capacity planner, which is where we used to do it in, in the banking industry.
Capacity planner would just simply say, "Oh, file and print. I can measure the number of pages printed, I can measure blah, blah, blah, number of printers," et cetera. It was pretty easy. And I'm showing my age.
[00:27:25] Serena Huang: Well, it's, it's, it's not enough ... so
[00:27:27] Vernon Turner: it could be printing stuff.
[00:27:28] Serena Huang: I, I like the, the focus on interdisciplinary team that you just mentioned there, 'cause I, I don't see that happening in most companies.
It's, it's a very small team that got to be having those conversations, or they have separate conversations. They don't talk to each other, and, and they really need to. Even if they don't speak the same language, they should be at the table, whether it's, you know, finance, procurement, and, um, and, and tech, and...
Well, actually I would say the people side- Yeah ... should be there as well, 'cause there's a lot of employee impact as well. Um, I think that workload, um, and, and measuring what is hard, um, is, is getting in the way for a lot of organizations, and I'm-
[00:28:11] Vernon Turner: Mm-hmm ...
[00:28:11] Serena Huang: hopeful that from today's conversation that our listeners see this is not boring and tedious work, but it is meaningful, and it will get us to that outcome, that intelligence at the end, which is what CEOs and the board care about.
When they talk about the ROI of huge AI investments, that's what they want to see.
[00:28:35] Vernon Turner: For, for my sins, I was one of the industry analysts that said the IoT market is gonna be many trillion dollars, many billions of sensors. I have s- I, I can, I can defend it, but, you know, that's water under the bridge. But what I think, Serena, is you, you, you scratched the surface there, is that IoT has actually helped, uh, bridge that gap between I- For, for example, um, AI in, in manufacturing is, is an issue where we've seen this, this conversation before between IT and OT operations people, and IT people who did not wanna talk, who think, who think differently even though they're looking at the same, I'll call it the same problem.
That's very loosely put. So I'm hopeful. I'm hopeful that the, the scars from IoT implementation in the early days for CIOs, um, can help this discussion of bringing people together who wanna talk, who talk different languages, but all have the same, all have the same goal. So I, so I'm, I'm, I'm hopeful for that, that, that small benefit from IoT is, is where I'm hoping.
[00:29:43] Val Bercovici: Uh, I, I love, um, the five, you know, the penny or the, the five-cent auditing angle, because we're talking about volumes right now Uh, of quadrillions of tokens per day that are being inferenced that we know of, right? And the ones that are easy to measure more transparently, for example, on OpenRouter, the, the top models individually are 15, 20 trillion tokens per day Any individual inference provider, whether it's the model, exactly, model host themselves, or whether it's the third-party inference providers, have a healthy chunk of that.
I know many of the top inference providers per top model and customer are doing over five trillion tokens a day per customer. So these pennies, these, you know, nickels really add up. If you don't, if you don't get the plumbing right, you get these very legitimate questions that a lot of Wall Street analysts are having right now is like, is this enormous CapEx of, for AI right now in the trillion plus ever going to be paid back?
Is there ever a revenue model that actually pays back for these unprecedented capital investments? And because we're not unifying a lot of these groups, w- because we're not tying these connections and connecting the dots imp- you know, well enough, directly enough, these remain open questions. And, uh, w- we're not gonna boil the ocean here, but I think the more we can educate the market, the more we can get different functions and different teams collaborating, coordinating on transparent metrics, on this observability, on understanding where the trillions of, of pennies are disappearing, which tends to add up to a lot of money, then, uh, quadrillions actually today, then, then that, then that'll make all the difference.
And that will, I think, at least, at the very least from a numerical perspective, answer a lot of legitimate financial questions which impact government policy and so forth on, is this all really worth it or not? It's, it's a, it's a disturbingly open question right now, and it, it can be a, it can be a lot, you know, a lot less vague.
[00:31:46] Vernon Turner: But, but how long can I s- how long can I keep going without somebody saying, "Stop. I want to see my ROI on my-" Mm. "... trillion-dollar data center investment"?
[00:31:57] Serena Huang: Yeah.
[00:31:58] Vernon Turner: I-
[00:31:59] Serena Huang: No.
[00:31:59] Val Bercovici: Everyone's watching for the bubble to burst, and I always like to say there's never gonna be a bubble bursting here, because if nothing else, the, the unprecedented growth of demand, which is still unending right now, shows that there's some kind of consumer backstop.
People, organizations, companies want this. The question is, of course, what are they willing to pay and what does it cost? Yes. So instead of a bubble, I like to say we're dealing with bubble wrap, right? There's gonna be some bubbles that will pop. We'll pop some ourselves. But, uh, there, there isn't- No, no ... one giant bubble- Yeah
that's gonna burst here. Yeah. There's just some bad businesses, there's some bad services, maybe bad model providers, infrastructure providers that are not sustainable, and many, many others- Mm ... you know, will be. And the, uh, the Uber, Airbnb analogies always apply. The use cases that we haven't even dreamed of are, are the ones that are gonna make the most money in the future.
[00:32:48] Serena Huang: Yeah. All right, well let's move on to, um, a couple other important topics before we wrap today. Um, one is the, the out- the metric that you land on, Vernon. Um, let's talk about the magical KPI called outcomes per token. Mm-hmm. And I think every CEO would love to know, is there a magic wand I can wave and I'll just have this pop up on my dashboard and I can see trend lines?
What, what does, um, outcome per token really mean in the business world? Where does it come from, and how can... Again, how can someone get started?
[00:33:27] Vernon Turner: Yeah. Um, good question. Um, and I think that there's a opportunity here for, um, a startup to, to, to come up with that metric. But, but let me, but, but let me, but let me describe it a little, little, uh, a little more...
Uh, put some clarity around it. If we follow up that stack and get to the idea of how many tokens it took in that workload, and therefore we can translate that to, um, a product, then, uh, I mentioned earlier about, uh, AI commerce. So one hand on, on, on... call it the left-hand side, w-we know how much it costs in terms of AI to get to that output product.
And on the second, we know we can measure, um, the AI journey that commerce took to fulfill either buying or selling that product. So, so I'm being very commercial here. Um, so I've gone away from the enterprise view of inside, in-inside, you know, you know, the CIO world to the SVP for sales who would stand back and say, "Look, this is how much we took in using AI as a prompt or an agent or multi agents that steered somebody on a website, for example, through to the completion of purchase of, of a product because of AI."
So on one hand, you've got the cost, on the other hand, you've got the revenue. And that's where the, that, that's where the output, um, really starts to come in, because then that dashboard hopefully would be able to show somebody the cost tokens and the revenue. Sorry, the tokens used to, to create it, and the tokens used to sell.
And at that point, I think nirvana. Then we might be able to stand back and say, "Was this worth it? Was it worth all this on the, on the left-hand side, um, to get this on the right-hand side?"
[00:35:29] Serena Huang: Could-- If I could push you a little bit, Vernon. It sounds like you want to, you want us to connect it to really customer outcomes, right?
Something that matters for the customers ultimately. Yeah. Does that mean the outcomes per token metric wouldn't matter for back office functions, I'll call them? You know, the finance, IT, HR, marketing of the world.
[00:35:53] Vernon Turner: Well, that- Well- No. I mean, it, it, it would be, it'd be almost the same in the sense that you'd be able to measure how many tokens per person have been used, how many people you have in a department, and you'd be able to, to measure that that way.
So, so you could, you'd have an organizational structure that, that was r- that was, uh, capable of, of... You're able to see, you know, is HR doing lots of,
[00:36:16] Val Bercovici: of,
[00:36:16] Vernon Turner: of AI work, looking for candidates through resume search, and then parsing and whatnot. Is it worth it? The... Is, is that worth doing that, that effort under AI?
Um, you know, the same with any of the back office, and finance as well. You know, it's, it is, it is very possible. It... I, I, I hate, I hate to say it, right? I don't wanna say it's like an ERP project, 'cause then you'll cut me off. Um, but, but, you know, I, I think it's a project for life, uh, if, if, if we have to go down that path.
On a serious note, I think it's a restructuring of how you organize your, your, your information flow across the company. And we've tried for years and years to have a common framework of, of... And, and Val, this is where you'll be like, "Yes, Vernon. Finally, the last two minutes we're talking about storage platforms."
Um, I think that, I think at that point, you know, you have to have that, that ability to see transparently across the company. And We're trying. We're trying really hard to get there. But, uh, it, it's, it's, uh, it's, it's a CIO, um, nirvana, uh, to be able to get there. So I, so I, I, I, you know, we, we talk about the outside, the commerce piece, but it's very much applicable inside as well, um, as long as we measure and tweak, um, the, the data flow.
[00:37:34] Serena Huang: Yeah. That's, that's very helpful. Thank, thank you. Um, Val, what would you add to that?
[00:37:40] Val Bercovici: My mind usually goes to the very provocative Alex Karp in Palantir, right? Because they've had a very good track record of taking a risk, getting into an organization's actual processes and workflows, capturing what's in people's heads, which is often how work gets done, and creating context graphs, creating decision traces.
I think the brand that, uh, Palantir uses is the ontology. And so once you have that, you're able to actually benchmark what it is your company does, what it is your various divisions, departments, individuals do. And when you get to that level of maturity, which is rare today, but will become more and more commonplace, you can actually measure it.
You can have agents, uh, take action, take correct action. You know, with these workflows, you, you, you know what good and bad is, what a successful outcome, and as importantly, or more importantly, a failed outcome is, to not repeat that. So those are the kinds of things we need to do, and I don't know if they're startups or not, they consider themselves that, but Whether it's artificial analysis that is measuring more and more, you know, task-based things or domain-specific, you know, legal, marketing, sales, manufacturing type of benchmarks.
There's a, a... I guess it is a startup, uh, called Arena AI, which is measuring more and more high-level tasks and works. There's one, of course, I like because of the name called Vals AI Index, which is just about evals, but it's vals for short, so, um, evaluations. Uh, the whole point is, um, we, we know that the industry now, especially the big model, the frontier labs, and even the open weights models, they do too much what we call benchmarksing, right?
These actual benchmarks are kind of saturated. So if you see like the MMLU, the Massive Multitask Language Understanding benchmark, really useful about a year, year and a half ago when the models couldn't saturate it. Today, every model saturates that, so it's literally a meaningless benchmark. Uh, GPQA, right, for a graduate level proof of Q&A, also very, very heavily saturated.
So we're always on the lookout, and that's why I think there is an opportunity for, for startups here. Always on the lookout for better evaluations, probably more domain-specific custom evaluations, more real world evaluations that are either hard to saturate or at least, you know, will live a good life, have a good half-life, and provide value before they're inevitably saturated.
That's, that's really important, uh, going forward here. And the good news is, um, because, you know, there's a vacuum and nature hates that, there definitely are some smart researchers and smart scientists. Uh, what's the controversial one right now, I think, that's trying to audit all these rogue agents, of course, that are escaping their sandboxes and seemingly attacking everyone.
So, uh, METR, MeTR, yeah, in Berkeley, METR. So that's another one that's trying to evaluate how these agents are working and how they're, they're, you know, to your point, Vernon, how they're breaking whatever governance there was, and maybe there isn't a governance, so it wasn't really, uh, known that they were misbehaving.
Uh, we're, we're, we're at the forefront. We're at the frontier of this right now. We can call it the, the, the jagged, I like to call it, bleeding edge of, of some of these technology advances, and we need to rein them in.
[00:40:56] Serena Huang: Yeah. Yeah, it- I wish we had more time to talk about governance. Maybe that's the s- the follow-up episode, 'cause it's so important.
Um, as, um, as we get close to the end today, um, I'm curious if we were to meet again in six months, the three of us, what are you hoping that the conversation will be like bet- between us? What, what do you hope that we'll be talking about that's different now?
[00:41:25] Val Bercovici: So the-- I've predicted, you know, that the, the expon-- Humans are bad, myself included, at predicting exponential progress correctly.
You know, the best in the world, the, the ones that are deepest involved in this exact research continue to really be bad at predicting the, the exponential progress. So we're at these models right now. We're at Opus 5.5 this week, which ge- is getting rave reviews for its efficiency and intelligence. We're at GPT-6 Astra.
We're at these really cost-effective Chinese open weight models, uh, that are progressing exponentially, that are, uh, some people don't like this term, but progressing at machine speed without a lot of humans in the loop, without a lot of human intervention. So just by the end of this year, we're at end of September, so Q4, we're gonna have these models at Christmas time that I don't think we can appreciate the power of.
So my point is that, you know, I hope in six months we actually do reach a next step level function of governance and AI maturity overall because we have to get much better understanding what these models do. We have to be able to get much better at controlling, you know, the negative externalities as well as the positive outcomes that we all desire here because we are...
You know, I'm not catastrophizing at all. I always look at it as, as a, the builder's perspective. But we are at this inflection point right now where these models are be- getting supremely powerful, and we, we can control them. This is just like adding seat belts and airbags and, you know, um, anti-lock braking to cars.
Of course, we're building these models. We're, we're calling them AI factories. We're manufacturing intelligence. It's not artificial at all. It's manufactured intelligence. We can control it. So it comes down to what are the incentives for creating, you know, observable, governable AI, safe AI? All we have to do is align those incentives because it's not like it's beyond our grasp.
We're, we're, we're builders. We're building these things. What, what are we incented to build? So that's what I hope in six months is that we realize we have put in at least the beginnings of some correct incentives, and we're now controlling these super intelligent, you know, uh, intelligent beings that we manufactured and, and we're manufacturing them correctly.
[00:43:32] Serena Huang: Love that.
[00:43:33] Vernon Turner: I, I-
[00:43:34] Serena Huang: Okay, Vernon.
[00:43:35] Vernon Turner: Yeah, I'll be, I'll be really quick. I, I think in six months we're gonna see, um... I think what's gonna happen here is, is that the economic climate is gonna put so much pressure on companies to make a decision about AI that they haven't done the, they haven't done their homework, and they're just gonna go down the road and, and hope to heck that nothing falls off b- between now and then.
Um, so I, I, I think that the, not the trough of disillusionment, I think the, the, the, the line of stupidity is gonna creep in, whereby people are just making stupid decisions because economically I think that they're, they're under pressure. I think the, the, um, some of our, some of our clients are seeing the pressure to use AI to reduce costs, and I don't know if that's the right business model they should be looking at as the first point of, of reference.
And as long as they keep going down that, they're still gonna make the same mistakes that we've seen so far. So, um, hopefully they'll, they'll be able to, like, like all teenagers, come out of that dark hole and then at the end, and, and safe. Um, having three, having three adult children, I know that, I know that really well.
Um, so I, so I, I really think in six months' time we're, we're, we're still gonna be in this turmoil of, of panic, um, of, of what to do with AI and how fast, how fast can, can they make a decision that's not well thought out. Um, so that's a brief of where I would be in six months.
[00:45:06] Serena Huang: Yeah. All right. Well, I think we should regroup in six months and, and see where we are.
See whose predictions come true. Even though humans are not good at it, um, but we do try, and I hope this episode brings everyone something new to consider as far as measurement, and, um, um, and stay tuned. Thank you both so much for joining us today, Vernon and Val. Thank you so much for your time.
[00:45:32] Val Bercovici: Thank you.
Ta, as always. Thanks.
[00:45:35] Serena Huang: Thanks for joining us on Deep Geeks, and thank you to Vernon and Val. The takeaway: start with one or two AI workloads, measure what they cost and what they produce, and bring finance, procurement, security, and ops in from the start. That's how you get outcomes per token. Find Deep Geeks on YouTube, Spotify, Apple Podcasts, or wherever you get your podcasts.
Until next time.