Why Quantum Benchmarks Still Don’t Agree

Summary

Tom Lubinski, Chief Software Architect for Quantum Circuits, Matt Langione, a partner at the Boston Consulting Group, and Pranav Gokhale, VP of Software at Infleqtion, are interviewed by Yuval Boger. Tom, Matt, Pranav and Yuval talk about quantum benchmarking – why benchmarks are important, what types of benchmarks exist, whether end-users should care, standardization efforts and much more.

Listen below or get the full transcript on the Quantum Computing Report site

Transcript

Yuval Boger: Hello, Tom. Hello, Matt. Hello, Pranav. Thanks for joining me today.

Tom Lubinski: Hey, Yuval.

Pranav Gokhale: Hello, Yuval.

Yuval: So Tom, who are you and what do you do?

Tom: Oh, I get to go first. Well, I am working as the Chief Software Architect for Quantum Circuits, a company working on a quantum computer.

For the last four years, I've also been involved in the QED-C, that's the Quantum Economic Development Consortium. I was the chairman of the Standards and Benchmarks Committee for three years, and recently, for the last two, I have taken ownership of the Quantum Computing Group within Standards and Benchmarking.

So I've been spending quite a bit of my time focusing on drawing together the community to develop a common understanding of benchmarks and standards that might be applicable for quantum technologies, in particular computing.

Yuval: Excellent. And Matt, who are you and what do you do?

Matt Langione: Sure. So I'm Matt Langione. I'm a partner at Boston Consulting Group, and I lead our quantum computing topic globally. The primary focus for me is really vendors, who we help with things like strategic assessments and business models; investors, who we help with diligences; end users, increasingly, who we help onboard quantum capabilities; as well as governments and other bodies that are thinking about national quantum strategies.

The benefit of the purview from my role is that we get to see all aspects of the ecosystem, and from that perspective have developed a view on what uses benchmarks can be put to in quantum computing.

Yuval: And Pranav, you know what's coming.

So who are you and what do you do?

Pranav: Yeah. So my name is Pranav, and I am the VP of Quantum Software at Infleqtion, which is the newly coined name for the umbrella that encompasses both ColdQuanta and Super.tech. Super.tech is a company that I co-founded, spun out of the University of Chicago, where we did research on connecting the dots from applications of quantum computing to the actual hardware. And benchmarking, as you might imagine, is a critical piece of connecting the dots from applications to hardware.

Yuval: So we're here to talk about quantum benchmarking.

So Tom, maybe I can ask you, who needs them and why? If I were to go and buy a gaming computer, I know what to look for. I may look at some popular benchmarks. But why do I need benchmarks for a quantum computer?

Tom: Well, first of all, I think it's important to understand that benchmarks exist at different levels, right? At the hardware vendor or provider level, the companies who are building the quantum computers obviously need to evaluate the components that make up their machines, and in fact, they have access to lots of academic work on doing that.

And they, of course, want to measure their own progress year to year and how their systems evolve. Now, one of the things that we found in the QED-C group is that the level of benchmarking done by the developers of the machines is often different from what users of the machines want to have access to.

Users need a different kind of benchmarking so that they can understand which of these various machines might actually help them solve or address a problem they have. So from my perspective, a very different type of benchmarking is required at the user level. And it's not so much just for evaluating or comparing different devices, but also to understand what quantum computers are even capable of at this point. Because they're very early in their development, understanding what they can actually do is another important thing to derive from benchmarks.

Yuval: And Matt, you see lots of different groups, as you mentioned: investors and vendors and end customers. Who are these benchmarks important for? Or if they're important for everyone, who are they most important for?

Matt: Well, that is how I would effectively break it down. The constituents I work with, as I mentioned, are vendors, investors, users, and governments.

From a vendor perspective, it's great to have a common measure of progress across the industry. You see what things like ImageNet did for AlexNet and for the development of machine learning. Of course, apples-to-apples comparisons often break down in quantum, but it is important, I think, to have a common target to strive for and to compare progress against.

I do think it stimulates competition in a productive way, so that would be the vendor case. From an investor perspective, it's really to get a measure of what companies are promising and give investors a little more certainty, as one of many inputs into the analysis that they do.

And don't get me wrong, things like talent, technical roadmap, and other things are at least as important as benchmarks, but benchmarks are an important figure in the calculation. For users, I think what's really important is to understand how to use benchmarking, not just in terms of what is the best technology today for my use case, but also the extent to which the technology roadmap can be benchmarked at a more generic level.

I think that was some of the work that the SupermarQ suite tackled to a certain extent, and the QED-C as well. Having application-defined benchmarking is, I think, particularly useful. The point there being that even if you're not comparing vendors, comparing, say, physical implementations and thinking about where you may want to invest resources if you're a pharma company versus a finance company is an important use case.

And then governments, I think, are more focused on ensuring that resources are deployed across the different research areas. Benchmarking can be helpful there as well. So that's the way I think things break down across the four constituents that we serve.

Yuval: I can understand why low-level benchmarks could be important for vendors. You know, this month I had this qubit fidelity, and in three months I have something better. But Pranav, do users care about such low-level benchmarks? After all, some would talk about fidelity, some would talk about connectivity, and some would talk about two-qubit gates.

So there are all these different parameters, and everyone can pick and choose what makes them look good, as opposed to the application level. What do you see when you talk to your customers?

Pranav: Yeah. Your question is really, in some sense, the motivation for why application benchmarking is so important.

It is worth acknowledging that the low-level benchmarks that preceded application benchmarking, the clock rate, the hertz, the number of qubits, things like that, do matter too. It's the same as when you go to Best Buy and buy the latest computer or the latest hard disk: you see on the sticker, "Oh, it's 2.6 gigahertz, it's one terabyte," and those things matter.

But if you're actually looking to get the best machine for your actual business use case, then in the classical world, which I think is a good analogy here, people doing machine learning look at MLPerf benchmarks, specifically image classification or language translation benchmarks. Or people who do video editing, photo editing, maybe podcast editing, look at performance benchmarks for Photoshop, et cetera.

And I think that's the link here that customers and users ultimately care about, too. Yes, it matters what clock speed you have; that influences things like latency. But if you're, to use Matt's example, a pharma company, low-level things like connectivity can certainly matter, but they can also sometimes be compiled away.

That's why it's much more actionable to have something like, "This machine can run VQE six times better than that machine." Maybe one thing I'll add is a time dimension here, too. "This machine did performance X last year, but this year it's doing performance 10X. I should be paying more attention to this machine, because if it's 10X again next year, that matters a lot."

So I'd say the short answer is that users care about both, but ultimately users with business needs are going to care about the application-level results the most.

Yuval: Tom, touching on the earlier point, because there are so many different low-level benchmarks and every vendor can pick the one that makes them look good, what is the focus of the QED-C benchmarking group?

Is it low level or high level, or everything? Where do you start?

Tom: Well, when we first began to work on this project, our charter in the QED-C was to identify ways that standards and benchmarks could help advance economic development. In other words, how can we help the builders of quantum devices actually succeed?

The conclusion we came to was that there was a wide variety of choices at the component level, but what was missing was some way for users to quickly get their hands on measurements of how these machines could perform. So we decided that the application level was more valuable to the community, because there was actually a dearth of choices there, right?

If somebody wanted to evaluate two machines side by side, they would essentially have to write the code themselves and exercise it on different machines in order to understand, and that can be a big job. So we decided we could fill that gap by making some easy-to-use benchmarking capabilities available to the community.

So basically, we're saving them time and money by providing these benchmarks to them. That was the focus of our effort.

Yuval: Pranav, you mentioned that some… Oh, go ahead, Tom. Sorry.

Tom: No, no, I'm done.

Yuval: Pranav, you mentioned that some of the problems could be compiled away, which I interpret as: there are different implementations. I can do VQE one way here and another way there, and what's best for one computer may not be best for the other.

How do you make benchmarks that no one is going to object to by saying, "Well, we can code the same thing better on our specific machine"?

Pranav: Yeah, good question. Let me first lean on one of the objectives of the podcast, which is educating and informing. So let me give an example of what I mean by compiling away, and then talk about how to address it.

If you have a system that only has qubits in a line, the lowest possible connectivity you can have, it's a fair critique that that quantum computer is worse than a quantum computer with full connectivity, where every qubit is connected to every other qubit. Now, that's often true, but it's not always the case.

Let me give the example of a sports game, a softball game or a basketball game. At the end, all the players high-five each other, right? If you think about it, that's also a line of qubits, well, players, who are all high-fiving every other player, and they only move in a line. That's an analogy to something known as a swap network. It's an idea that emerged in 2019 or 2016 that basically shows that even with linear connectivity, just a line of qubits, you can get your computer to run programs that simulate full connectivity, having every player high-five every other player, even though you only move one spot in the line at a time as you go down that train of high-fives.

So if I back up now, what we do in SupermarQ, the benchmark suite that we developed, as well as other benchmark suites out there, including QED-C, is to specify the problem not always directly, in the sense of, "This is the quantum circuit. You have to implement it exactly like this."

Instead, we specify it at the level of a more fundamental problem. For example, here is the chemistry problem that we want to solve. Then we leave it to the compiler: IBM gets to use its best compiler, any vendor gets to use its best compiler, and it's up to the quantum computer to figure out, "How do I want to address this specific problem?"

If you have low connectivity, you can insert this thing I mentioned called a swap network, the train of high-fives, to deal with that. If you have a fully connected device, you don't need that. So the short answer is that by specifying benchmarks at a level that gives hardware vendors the flexibility to figure out how they want to implement them, we try to resolve this problem. Essentially, for every machine, we're showing how it would run if you use the best software tools that vendor has.

We're not saying, "This is the very specific circuit, down to every qubit link, that we want to run." That would start to run up against issues of, "Well, my hardware has this capability that this one doesn't have," and vice versa. So the short answer is: define the benchmarks at a slightly more fundamental level.

Yuval: Matt, when you speak with customers, I'm sure you often have a broader scope than just quantum computing.

For instance, how quantum computing integrates with the rest of the IT infrastructure, with a high-performance computing center, and so on. Does the issue of compute speed also come up in discussions when comparing different computers?

Matt: For sure. Speed's an important piece of the puzzle.

Even if you take it out of the quantum realm and put it into, say, traditional deep neural networks, there are some problems that are compute bound and others that are memory bound, right? And there's all sorts of research into the right architecture to address those kinds of problems.

We've all read horrifying articles about the ballooning costs of training a lot of these neural networks, both to companies and to the environment. So certainly speed's important. I would also say the extent to which there are resource challenges, both at the compute level and, increasingly, in the FTEs required to handle the compute.

Those things are important as well. Within quantum, speed's an important thing. Quality and scale are other things that we focus on. IBM in particular has emphasized the importance of these three metrics: scale being something like the number of qubits; quality being the quality of circuits, so low operation errors; and then speed, where they've talked a lot about CLOPS.

Those three things are really critical within quantum. But if your question is more about what users are asking us about in the surround, that's really important. Often it's about costs, whether it's the cost of the hardware itself or the cost to operate the hardware.

And then also the timeline to implement any algorithm, right? That's partly a function of some of the things Pranav was talking about, namely the maturity of the hardware over time and how benchmarking allows you to roadmap that a bit.

And of course, part of it is the degree to which the problem itself is efficient for quantum computing in general, or the extent to which a lot of custom work needs to be done to make that translation from a business challenge to a quantum advantage solution, as it were.

Yuval: As a follow-up on that, Matt, I know you published an article about benchmarking a few months ago.

So far in this podcast, we've spoken about QED-C and about the Super.tech, now Infleqtion, benchmark. Are there other packages or vendors that you think users might be interested in looking at beyond these two?

Matt: Well, there are a number of different approaches to this. In the article, what we did was think about the landscape as a spectrum, from systems benchmarks on one end to application benchmarks on the other.

We've talked mainly about application benchmarks in this podcast, rightly, partly because Tom's QED-C benchmarks and Pranav's SupermarQ benchmarks are both on the application end of the spectrum. And I do think that the application end of the spectrum is the most useful, at least for users, right?

If you're a pharma company thinking about implementing quantum computing, taking an application approach is certainly what we see most doing, and, ex cathedra, I would assume it's the most useful for them. On the other hand, system benchmarks are important as well, especially in stimulating competition among vendors, or even just getting vendors to speak the same language so that some of their development efforts can benefit from advances from one vendor to another.

On that end of the spectrum, you see things I had started to mention from IBM, like quantum volume and CLOPS, which are a lot more like the traditional speed benchmarks you'd see, like TOPS, for example, in the classical world. Those are really important too.

They have quite different use cases, but they're also important. A couple of others I might mention would be the Sandia National Laboratories benchmark that uses mirror circuits. And somewhere splitting the difference a little is the Quantum LINPACK research out of UC Berkeley and Berkeley National Lab. I think those are also pretty meaningful efforts that sit somewhere between system benchmarking and application benchmarking.

Yuval: Tom, I wanted to ask you about the status of the standardization efforts for the various benchmarks we mentioned, whether qubit-level benchmarks, application-level, or something in between.

Which are the most advanced right now in terms of being ready for users? And related to that, do you expect a merging of the various benchmarking efforts, where everyone comes together under one set, or do you think there will continue to be half a dozen or even more different benchmarking suites?

Tom: Well, there's a lot to say. This is a very complicated and complex landscape. In terms of standards, and which benchmarking approaches might be ready for standards, I would say we're probably at zero. I don't think we're ready to standardize anything. The hardware is changing. Even the programming approaches are changing.

So I'd be reluctant to say that standards are around the corner. Obviously, people talk about LINPACK and whether we can get to that level. We may eventually, but it's really early right now. And so what you're seeing in the community is the surfacing of a whole bunch of different approaches.

There are component-level and system-level benchmarks, like the Sandia work. We looked at all of that when we put together our QED-C project. The QED-C is a consortium of all the different providers and vendors, and we tried to identify what we could do that would benefit all the members of the community.

What could we do? How could we bring them together? So our goal was, and continues to be, to try to reduce fragmentation and, if possible, integrate the various approaches into something that can be commonly accepted and maybe eventually grow into a de facto accepted approach to benchmarking, right?

There are new things surfacing. Besides SupermarQ, there are QPack scores that came out of a group in Europe. We've got QPack and QUARK. There are a lot of different projects emerging. The QED-C would like to see some sort of unification. Practically, can we achieve that?

That's yet to be seen. I personally would like to see less fragmentation and more collaboration and unification in the community, but it is very, very early, so we have to expect a lot of different approaches.

Yuval: Pranav, Matt mentioned cost. Is there a cost component to the benchmarks?

It would cost so many dollars to run it on IBM, and more or fewer dollars to run it on Honeywell, for instance.

Pranav: That's a great question. In the current benchmarks that we and others have done, cost is not part of the equation, for two reasons. One is that, practically speaking, it's often challenging to estimate cost.

Vendors offer different packages to different people, and they're not always transparent. The other piece is that no current quantum solution is, in some sense, cost-effective compared to what you could do with a well-tuned supercomputer running in some nice fridge, something like that. And the other thing is the asymptotics: at some point, not today, the solution quality you can get with quantum will easily outperform what you would get by just multiplying the number of CPU cores.

So we're a little more concerned with basic solution quality at this stage. That said, I will point to some interesting work that Igor Markov has done at the University of Michigan, I think, estimating the energy cost of solving certain problems with a quantum advantage. I think there's good theory work here, but it has yet to be brought into the benchmarking world. That's something we should look into as a community.

Yuval: But Matt, just to come back to you for a second, it would seem like users would really care about cost, right? Part of the budget is not just people, but cloud time or on-premise time.

Matt: Yeah, and I think there are probably two layers to it, and Pranav addressed what might be the harder of the two.

It wasn't exactly what I was thinking, but perhaps what I should have been thinking. Conversations with Pranav often lead me in this direction. What he's talking about is that, as part of the case for quantum computing, there may be some really valuable arguments to be made about the efficiency of quantum algorithms in certain use cases.

Of course, there will be some cases in which it's much less efficient and others in which it's much more efficient, and that, I think, is at the heart of the Markov work. I do think that's really important to pursue. But to Pranav's point, I haven't seen anything there that's quite convincing yet.

I haven't yet seen an example of a client onboarding quantum computing for this reason, though I think it's a great future avenue. I was referring more to within quantum computing: across different physical implementations, certain experiments can be much more costly if run on one piece of hardware versus another.

And there, I have seen folks looking for guidance and advice, not from BCG, it's not our sweet spot, but from others, on whether, in running a specific algorithm, they'll be at a major cost disadvantage if they do it on, say, ion traps versus superconducting qubits.

And I have seen some really large disparities in cost for specific algorithms across different physical implementations. So resource estimation work, for those already experimenting with quantum computing, is certainly valuable now and will only become more valuable as the algorithms and circuits folks try to run become more and more complex.

Yuval: And you could even envision comparing different cloud vendors if you've got the same computer available on both. Pranav, I wanted to get back to you on the quality of the benchmark. If I were doing an image classification benchmark, it's very easy for a human to say, "This is a cat, this is a dog," and therefore assess the quality of the image classification network.

But if I do a portfolio optimization benchmark on a quantum computer, how would I know whether the result is better or worse than something else? Maybe I'm just getting a much better result, you know? It sounds much more difficult than cat versus dog, or hot dog, not hot dog, if you go back to Silicon Valley.

Pranav: Fair, right. I think there are two pieces to this. The first is that there are certainly human-intuition-type problems where it's easy to construct a cost function mentally: this is a hot dog, this is not. But even though there isn't necessarily human intuition for a portfolio optimization problem, there is certainly a mathematical expression. If you're a financial firm solving for minimum volatility at a certain return, that's a very direct head-to-head comparison that can be made. Even though it's not the warm, fuzzy feeling of "this is a cat or a dog," if you're a financier, it's certainly a warm, fuzzy feeling to know you've got the lowest-volatility portfolio, that you did better on this quantum computer than on that quantum computer or that classical computer.

So the short answer there is that there's always a way to mathematically formulate the objective and compare against it. The other piece is that, in the regime of, say, up to 40 qubits, you can ask: if I ran this experiment on a perfect quantum computer, which you can simulate up to 30 or 40 qubits, what output would it give?

How many shots of each result would it give? Then you can do a variety of statistical tests of how the output from my noisy quantum computer compares to the idealized quantum computer. Granted, that doesn't scale; at 50-plus qubits, 100 qubits, you're toast. But at least in the current regime, it's a second approach we can take: does this match what we would get from a perfect, noiseless quantum computer?

In both cases, there's a mathematical approach that ultimately corresponds to business value, even if it's not a matter of human intuition like whether this is a cat or a dog.

Yuval: Tom, I wanted to ask you, other than joining QED-C, what would you like to see vendors, end users, or governments do to help move standards forward or make them easier to interpret?

Tom: Since a lot of these projects are open source, I think the best thing that can be done is to actually use them and contribute commentary, suggestions, or feedback on the value they provided. Actual utilization is very important, in my mind. I did want to add one quick comment to what Pranav was just talking about on cost.

You were mentioning cost as Pranav was talking there. From our perspective, we've done several rounds of work on our benchmark, and in the most recent round, we focused on something we call the time-versus-quality trade-off, right?

Because in a lot of problems, like machine learning or optimization, you wouldn't necessarily expect a quantum computer to provide an exact answer. It's how good an answer can you get? How close can I come to the optimal? And as Pranav said, there are a lot of existing classical formulations for what they call an optimality gap: how close your solution comes to optimal.

Then you can evaluate a solution based on: if I run it for 30 seconds, I get this quality of answer, but if I run it for 60, I get a better-quality answer. What's my trade-off there? At some point, that trade-off may become more valuable than what they can get from a classical solution.

That's what we would be looking for. But it's important not to view it as "you got the answer or you didn't." It's a time-versus-quality trade-off that's really important in quantum computing. I just wanted to mention that as a follow-up. But yes, utilization of the benchmarks and feedback would be most valuable.

Yuval: And Matt, same question to you.

What would you like to see vendors or others do to help you advise your clients on the best quantum computer for their particular application?

Matt: I think transparency around system performance is extremely important, and I do think the spirit of transparency is there.

But to me, the number one thing is making as much information available as possible, so that we can more readily insert the fundamental system pieces into our cross-implementation, cross-vendor benchmarking. Making sure there's a lot of transparency around the physical systems at this stage of the technology's development.

That will make it easier for us, and candidly for Tom and Pranav too. When they think about the benchmarking they're doing or the benchmark suites they set up, ensuring an apples-to-apples comparison is just much easier if the system specs are really transparent.

Yuval: As we get close to the end of our conversation, I had one more question for you, Tom.

What do you do when the solutions are not gate-based? You have companies like Pasqal or QuEra that do analog Hamiltonian simulation, and you have D-Wave, of course, that does annealers. How can you benchmark a solution if it's not based on a gate-based architecture?

Tom: Well, that's a very good question, and in fact a subject that has been at the forefront of discussion in our group meetings since the beginning.

We didn't want to limit ourselves to the gate model. So in the most recent work we did, we incorporated annealing as a form of solution. And going right back to what Pranav said earlier, it's about defining the problem you're solving not in terms of the specific circuit, but in terms of what problem you're actually solving.

What is the optimization problem? What is the machine learning problem? Here's a data set, and I'm going to derive some answer from that data set. That's the problem, and then it doesn't matter whether you use a gate model or an annealing machine, cold atoms or photonics. How quickly can you get a quality answer within some time?

Measuring that trade-off is what's really critical. So we have made our suite hospitable in that form. Like I said, it's a big, complicated problem that isn't going to be solved overnight.

Yuval: So we're now at the lightning round. Pranav to you, then Matt, and then Tom. If you could have dinner with one of the quantum greats, dead or alive, who would that person be?

Pranav: It's an easy answer, but I would probably go with Richard Feynman, in many ways the founding pioneer of our field. People often quote him on simulating quantum systems with quantum computers as the killer application of quantum computers, and there's actually a lot more depth to that statement than what's commonly quoted, so I'd like to have a conversation with him about it.

Maybe we can interest him in some benchmarking too.

Matt: For me, I think it would be Feynman as well. But among the living, maybe I'll say Aaronson. Both Feynman and Aaronson obviously have a role in the history of quantum computing, but I also think their thoughts on extensions of quantum mechanics into philosophy would make for great dinner conversation.

So that would be my choice.

Tom: Well, that is a really tough question, because it would be a great pleasure to sit down and have dinner with any one of them. Did you say only alive, or anyone?

Yuval: Dead or alive, but not both dead and alive.

Tom: Well, I have a particular interest in Pauli. I have a few questions I'd like to dig into.

So I might want to get together with him. But certainly any of the others would be a very engaging conversation.

Pranav: Yuval, how about you?

Yuval: I just ask the questions here.

But it is a good one. So, Pranav, Matt, and then Tom, how can people get in touch with you, and what kind of people are you most interested in hearing from?

Pranav: My email address is pranav.gokhale, G-O-K-H-A-L-E, @infleqtion.com. That's with a K in the middle. I'm interested in talking to anyone who would like to learn about our benchmarking methods, as well as help inform the next generation of benchmarks. And maybe stay tuned for what's coming next from us.

Matt: For me, it's langione.matt@bcg.com. I think really anyone can reach out. Certainly anyone from our traditional client base, whether vendors, users, investors, or governments, will find we have perspectives on their thorniest problems. But increasingly, we're seeing a lot of action from users, and I think that's probably the most exciting development in the technology, because with any deep tech, the early interest typically comes from the providers and investors first, and then it gravitates toward users.

So I think it's a good benchmark of the industry's maturity that we're seeing a lot of interest from users, and I would encourage users to keep reaching out.

Tom: And for me, I am at tlubinski, L-U-B-I-N-S-K-I, @quantumcircuits.com. I see benchmarking as a way to enable users to explore the potential of quantum computing.

So to me, it's a means to an end. What I'm particularly interested in is uncovering just how quantum computers can actually be taken advantage of in this day and age. What new types of programming and algorithmic approaches might we conceive of? Benchmarking is a good way to measure it, but I am very interested in exploring new ways to use quantum computers.

Yuval: And just to get back to your question, Pranav, for me, the answer would be Einstein. One reason is that he expressed so much hesitation and uncertainty about quantum effects, and I would love to see if the developments since his time have changed his mind. And interestingly, Einstein hasn't come up often, or actually ever, when I've asked this question before.

So that would be my answer. Well, gentlemen, thank you so much for joining me today. It's been a great pleasure.

Pranav: Likewise. Thanks so much.

Tom: All right. Thank you.