Zoran Krunic, Senior Manager of Data Science at Amgen Research and Development, is interviewed by Yuval Boger. Zoran and Yuval talk about leveraging quantum machine learning for clinical trial data, the challenges of implementing new technologies in regulated environments, combining quantum computing with generative AI, strategies for scaling quantum efforts in biopharma, and much more.
Transcript
Yuval: Hello, Zoran and thank you for joining me today.
Zoran: Hi, Yuval. Thank you for inviting me. It’s always a pleasure.
Yuval: So who are you and what do you do?
Zoran: So, my name is Zoran Krunic. I’m a senior manager of data science with Amgen Research and Development. Been in computer science for decades. That’s my background.
When I joined Amgen, one of the things that I was tasked with was to look at natural language processing and classical machine learning applications for clinical trials and electronic health records. And so that relatively quickly then transitioned into quantum computing, which was something that Amgen started fairly early with a pretty big collaboration with the IBM quantum research team. And so that became really the focal point for a number of years.
And now we are merging that with generative AI. So just like everybody else, we are looking at synergies there. The focus in biopharma, especially in clinical trials and clinical trial data, patient data, electronic health records, is how we can see the opportunities for quantum. How do we look at that tactically and strategically and kind of try to shape that up in some kind of coherent approach that both has short, mid and long term steps to be executed.
Yuval: Is it common that biopharma companies put someone on quantum? I mean, pharma always comes up as a quantum application in the future for material design or drug design. But do you see a lot of people in your peer companies that work on quantum?
Zoran: So not that much. That’s a very interesting question. I think if you look at quantum computing, it seems fairly early. And so in the journey of quantum that we anticipate is going to happen. So even if you contrast that with classical machine learning, it took a while before it became a common practice, I think, in these circles. I think there are a number of reasons.
I think it’s that it’s a highly regulated environment, that the stakes are high, you’re dealing with human health. So there are lots of aspects there that are quite challenging. And so bringing the new technology always comes up, comes with lots of strings attached, things that need to be checked, make sure that they are working correctly. So I think that’s a little bit slower, but more careful also adoption for a whole bunch of good reasons. And so I see when we started quantum, that was a big collaboration with IBM and there were quite a few people on it. And then my use case ended up being accepted as the one that we’re going to pursue for a few years. We published the paper and so forth and so we learned a lot.
But I think that there are two aspects there. And I’m looking from this perspective as a classical/quantum machine learning and quantum computing applied science, a practitioner, if you like. So just like in classical machine learning, we didn’t really invent new classical machine learning models. Usually you use what’s available on the internet and you apply it.
Why? Well, there are a couple of reasons. One is the setting is very complex and takes a lot of time and effort to understand how data works inside that medical/clinical setting. And so you have plenty of things to do developing those kind of models. This is a slightly different activity, I think in general is not present in other domains. I expect that to be the same situation with quantum. And so I think from that perspective, looking at the practical level is important, but I don’t think that that precludes or it’s any kind of contradiction or collision with strategic thinking. So strategy is important with quantum because it’s a long-term play and takes a long time to be adopted and to learn.
So I think those two are actually complementary. Because sometimes I think those two things get confused. “Oh, it’s too strategic, it’s not practical.” I don’t think so. I think practical needs strategy to not to lose focus and see where we are going and strategy needs to be practical. So it ends up being not just a research effort, but actually something that delivers value. So I think we are early on, that’s why we don’t see that much presence in these environments.
I think the other aspect is that we probably haven’t done a really good job in explaining quantum inside large organizations. I think that’s something that education, occasional sessions, finding people who are interested, I think that needs to be done far more and far better. Starting from simple elevator pitch to something that’s maybe an hour or two-hour session when you go deeper into possible use cases, starting from molecular and then to clinical trials. So I think there is a lot that we can do to make this better.
Yuval: And you actually don’t work on the molecular side, or at least the part that I’m aware of. What do you work on as it relates to quantum?
Zoran: So my focus was twofold. So we have the molecular part, which is where biopharma in some sense starts, in the most simplified picture. But then the biggest chunk of expense in developing new medication is actually clinical trials. Usually that’s about 70 to 80% while molecular is about 20%. Both are very important. I don’t mean to proportionate the relevance that way, but the major cost is in clinical trials.
And so I think that’s where we would like to see improvements. Now there are very interesting problems. One that I focused on was small data sets. That’s in classical machine learning. We are used to working with tens of thousands or hundreds of thousands or millions of records. But in the clinical setting, in phase one and phase two clinical trials, we have small data sets. You’re looking at 150, 200 patients, 300 patients. So it’s very tough for machine learning to do anything there. And usually you end up upsampling and then working that way. But I think because quantum machine learning has the ability to do better on small data sets with less parameterized models, we see an opportunity and opening to do that. That’s been really my focus over the last few years to see how we can take those small data sets. Because today, small data sets in quantum almost look like a deficiency.
You cannot put 10,000 records easily, right? Or 100,000 records. But actually that’s not a problem. We have lots of problems with 200 to 300 samples that are present in quantum computing today that are very relevant in clinical trials. So from that side, we don’t have a problem. We just need probably improvements in algorithms and quantum technology.
Yuval: And when you say quantum for these small data sets, are you running it on a quantum computer or is a simulator, a sufficiently strong simulator enough?
Zoran: So that’s a good question. I think it depends on the use case. We are barely scratching the surface in learning how to approach a problem. So if you look at classical machine learning, which is always, I think, a starting point, it’s always good to have your classical way of doing things done first. You understand the data, the context, and then you go into quantum rather than having to do all of that at the same time. Because then it seems like the project is taking two years, but really you are spending a year and a half on data and then understanding and working with clinicians. And I think that when you look at that side, it’s always the question of can we find in a model that let’s say has 200 features, for example, can we find the top 20 features because top 20 features is something we can map into the quantum, but not 100 features.
Now in many cases, I’m not going to say always, but in many cases, top 20 features, if you pick the right ones, will actually give you 90-95% of accuracy. I extensively use a Shapley approach, which is a common method to pick the top variables.There are others, so that depends on the situation.But if you get those, you’re going to be in a position that you can run simulations today.
And that’s, I think, a really good entry point because of the relatively high QPU cost, which often seems to be a barrier, “Oh, we’re not going to do that, we could take so much money, and we are not sure what we’re going to get.”
Well, start with simulations. I think you can go with 10, 15, 20-qubit simulations and get quite a bit done on a reasonably strong server and be able to figure out the algorithmic part, how you apply it, how the data is processed and all of that, and get a very good feel. And then you say, “Okay, now I want to scale this up to problems where I really have 50 or 100 features. Okay, then we need to have a budget for QPU time.” Because simulations that you can run on, let’s say, 32 CPU box with 100 gig of memory, that’s a couple of dollars per hour, it’s not really that much.
So I think that’s very, very good. And I think that’s actually often an overlooked approach to how you start quantum. I think that allows you to work this five-year strategy much easier.
Yuval: How close is it to production? Do you use quantum algorithms in clinical trials today? And if not, what’s your expectation of when you could start getting useful data from clinical trials using quantum algorithms?
Zoran: So we are not in production today. I think that’s not there yet. And I think that’s going to be probably subject to a couple of things.
One is, can we really see the reason that quantum will do better than classical computing? I think that’s something that’s been talked about a lot. And so I think that’s going to be the one. Now, could you run two models which are comparable in accuracy and perhaps find some way to learn from differences? Yeah, absolutely. But would that be a real production? It’s more of an evaluation testing mode. So I think that’s really where we are today.
How long? I think it’s going to depend on a couple of factors. One is that this strategy that we really need to have developed and I think to be preceded by more education. I think that’s the part where some of that work is going to be done and chart a two, three, four year strategy. Then you start working on the problems and maybe around them in some kind of semi-production mode. But I think that the real production is going to happen either in three to five years from now or if you start to see really good results that cannot be achieved classically, then I think that’s going to, whichever one happens first. I think it depends though on use cases quite a bit. I think that’s a little bit of catch-22 because today we really haven’t run that many use cases. Actually, it’s very rare to find very good clinical use cases that have been executed in quantum.
So you have relatively little to go with and see, okay, so where do I do well? Where do I do maybe more poorly? And then where do I focus? I think that’s the part where we need to start running more use cases, maybe with simulations, to be able to get a feel of where to push.
Yuval: Let me play devil’s advocate here for a second about strategy. You spoke about strategy and a five-year plan and so on. And I could say, why do you need a strategy? I mean, your manager basically just wants you to extract useful information from low-volume clinical trials. That manager may or may not care that you use quantum. So as long as you get the results, why do you need a strategy?
Zoran: Well, so I think the context and the planning is hard to downplay. I think, you know, experimentally, fine. I think if you’re working on something that’s your Friday afternoon project, right, then you can do all kinds of things there. But I think from the corporate environment perspective, I think it’s really important because working on a project that doesn’t necessarily have a chance to be accepted, it’s a relatively moot point. I wouldn’t say it’s pointless, but I think it’s not as attractive. And so usually in corporate environments, and I think it makes sense, there is a plan. There is always some kind of plan, some kind of calculation, and it should be there. But I think that is the part where if you just throw in another model, you’re not going to understand where machine learning is good, where classical machine learning goes, where quantum machine learning is good.
Understanding the data is really important. I spent a lot of time preceding that quantum project working on clinical projects with clinicians and doctors, a rheumatologist specialist, understanding how the data is created. Even statisticians, you will see there are books which explain why if you don’t understand the data and the context in which the data is created, you’re not going to really be able to apply the models correctly. And so blindly applying the models just in the pure kind of oversimplified practitioner mode, I think that just doesn’t lead to anything useful.
And I think it’s also that if you’re going to expand this, you really want to be able to present this to SVPs, to the C-suite, and I think that’s the level where strategy absolutely comes into play, or even the board of directors, depending on the size of the company.
Yuval: Is there a regulatory aspect to this? I mean, if you one day present something to the FDA and say, well, this is the result of my quantum machine learning model, are they going to completely flip out?
Zoran: Well, so that’s changing rapidly. I think if you look at five plus years ago, even classical machine learning was not easily accepted. And I think what we’ve seen over the last few years, the last time I checked that there were over 500 classical machine learning models that are accepted by the FDA, either for medical devices or for drug development are accepted. The natural language processing, also there I think, John Snow Labs or Hugging Face, one of those two in the NLP space, has partnered with FDA. So I think we are doing much better there.
Now I had that question asked by somebody a couple of years ago. So my point was, well, a quantum machine learning model is just a model, right? It runs on different technology, but at the practitioner level, you’re looking at it the same way you would look at XGBoost or SVM.
You’re looking at a couple of things. One is reproducibility, right? And that’s part of the science in general. It’s important in medical settings. Can I reproduce my experiment three times or an experiment from six months ago or your experiment? Can you do that in mine and so forth? And so that reproducibility has two sides then.
One is if you do quantum simulation, you do the noiseless quantum simulation, then you can reproduce it. If your code is correct, just like classical, you fix the random seed, your process should flow 10 times through and get 10 decimal digits identically at the end. If that’s not working, you’ve got a problem at the program level, right?
Now with QC time, with real QPU time, you’re going to see that there is some noise that’s going to give you some variability. You need to measure it and you need to talk more about it. A big part of my presentation in a couple of conferences is talking about the fact that I was seeing, for example, the 7-qubit runs on QPU, the standard deviation for everything else fixed just by running these 10 times was about 2-3%, 1-2%. So that’s not much. I was actually pleasantly surprised when I saw those numbers, but I think every platform is going to be different.
We know that the algorithmic part of simulations is very important and how you run this through QPU is very important, the noise reduction and all of that. Can we structure that in a way and present it to FDA to say, “Yes, I think that’s quite possible.I think that’s feasible.I think that should be done.”
And measuring that confidence range and reproducibility is important. And by the way, everything I say today is my opinion. It doesn’t represent Amgen. That’s an important disclaimer there.
Yuval: How long did it take you to get to this point? If you had a friend who’s starting a similar job at a sister company, what would you advise that friend to do to get to this level of expertise?
Zoran: So it depends on the environment. I think the way I like to think about quantum was, and I was lucky enough, things just happened to me that way. We have the classical machine learning problem solved first. Understand the data, understand the context, that’s not optional. You do that, you’re blindly applying. So that really needs to be done first. And then quantum becomes a delta. Okay, I’m going to swap the classical kernel with the quantum kernel. Everything else is the same. So first you reduce that barrier entry, which is not small with quantum. There’s a lot to learn. And so that allows you to do that way.
The other one, I did spend quite a bit of time looking at quantum circuits at Hadamard gates and CNOT gates and all of that, trying to learn that. I think I had enough of it at one point that I said, “Okay, so this is interesting. I see where this is going.” But that’s not probably going to be what we are going to be working at, practically applying this. I think this is almost like you’re writing assembler 20 years ago in programming. A few people do it, embedded devices, maybe, but usually you’re not going to do that. So I think having some knowledge of what’s inside the black box to crack it open, I think is good. So it’s not totally black box, but I think it’s more of understanding where quantum is going, what kind of options you have.
One challenge there, I think that’s interesting to mention is that there are lots of, there are quite a few quantum vendors today. And different software stacks. And that does present a challenge. In classical machine learning in Python, we often default to scikit-learn, for example, as a way and an environment where you pull the different functions and stuff like that. If you’re in an R programming environment, then you have comparable libraries that you can do that in R. I think those software stacks are going to be a little bit of a challenge because if you have five or 10 vendors, you’re trying to pick which one you’re going to go with. And then tomorrow you have to do it all over again with a different vendor. I think that’s a little bit of a problem and a turn off.
So I think maybe some consolidation that might be happening in quantum space, which we are starting to see good companies are getting big investments. Some unfortunately are not going to have that much success. That actually might be useful from, again, a practitioner standpoint from applied quantum because it’s going to simplify the situation to maybe what you have with the cloud today, where you really have three clouds. AWS, Azure, and GCP and you’re fine with that. Now platforms like Amazon Braket are good because you can front end some of these environments to the same API. If that’s the case, if it works well, I think that’s a good approach.
So I think those are the practical angles. And then another thing that happens there, and this is interesting, we’ve seen this in NLP, I think quantum in that way is kind of like NLP used to be maybe five years ago, is that, so there is a biopharma here. Or any kind of organization. Here is the quantum on the right side. And in the middle there is this gray zone that you need to make a connection, use case technology, everything else.
And I think startup companies and some kind of system integrators or companies that can glue this together, they are very, very important. It’s not easy to come into the biopharma environment. It’s complex, it’s regulated, there’s a lot going on. And so I think if somebody shares that knowledge, I mean, I spent most of my few years actually doing that type of glue, learning enough about quantum and knowing enough about the clinical setting and how to put those together.
Every use case is going to be a little bit different. I think if you’re going molecular, there may be one thing. If you are doing, let’s say, QML or clinical trials, trying to predict something about patients or you’re doing optimizations, which is also a very attractive area potentially, that’s a bit different.But I think figuring out how to bridge that space, it’s going to be important. I think one thing that we don’t have today in many organizations is it would be really useful to have one centralized quantum group inside the organization. That starts with maybe four or five people who are on that full time, dedicated, trying to map these technologies and then do the map to the local business or clinical parts of the organization and see, okay, where can we go? What are the good use cases to try?
And with enough budget start that way, eventually some of that will be embedded in groups because that business knowledge is crucial and you’re not going to get it if it’s centralized. But I think starting with a centralized unit is probably the most practical approach right now.
Yuval: Someone told me that physicists care about the derivatives, so they don’t ask you how are you, they ask you what’s new. So, what’s new? What have you learned in the last six to 12 months about this field that you didn’t know prior to that?
Zoran: Well, there is a lot going on. I think that’s a question that you can probably have a separate interview just on that. But I think there are a couple of things. One thing I learned was working with your team and the collaboration you’ve done recently. And so I think that’s very interesting. I think the different quantum technologies that underpin the approach that somebody would take at a practical level are going to play a role. And I think getting to know more about it is important.
The other aspect is AI. We had this grand entrance of GPT and LLM. So the last couple of years, year and a half really. And what you are seeing this year especially, and this is something I mentioned in Boston last year at Quantum Tech that I expected to start to converge. And actually it’s happening this year. We’ve seen now research coming out that you can, for example, optimize quantum circuits using LLMs, which is an insanely good idea. I think that’s something that’s absolutely amazing. So I think quantum teams are now looking to use AI as an assistant, gaining insight, optimizing the circuits. There are different things you can do. You can actually create very good training for different levels, for entry-level users, mid-experience and maybe very high-level users. Also for different users in terms of corporate hierarchy, are you going to C-suite or are you going somewhere in the director level perhaps to present. So this is something that quantum computing is starting to use. We are starting to see that topic.
I was just at a conference not so long ago where we talked about in IQT and we talked about some of that as well. So I think that’s a really, really new and exciting area. I think in 2025 we’re going to see far more of that. And so I think that has great potential. And I think should become part of that hypothetical quantum team that you’re going to have is that AI LLMs have to be somehow brought into it. Are you going to have two sub teams, one’s QC and one AI or something? I mean, it depends on the organization, but you really definitely want to bring that in and start working on that.
Yuval: And I assume you see that team also including computational chemists or machine learning scientists beyond just quantum people?
Zoran: Yes, I think that multidisciplinary aspect is becoming really interesting. Something that what used to be silent, molecular people do molecular part and this is clinical trials. Now you see that molecular teams are using AlphaFold and similar, AlphaFold is not LLM, but LLMs as well, you have NVIDIA’s BioNeMo platform, for example. So you are seeing that integration and that multidisciplinary aspect. And that’s another very interesting area.
Do you form the team from people from different backgrounds or do you expect people to actually learn more than one thing? That depends. But it’s really complicated things, but it’s also opening up new areas that you really haven’t had before. It’s quite exciting to see that.
Yuval: As we’re close to the end of our conversations, I wanted to ask you about competition versus cooperation. Do you see at this stage of the market that you’re more cooperating with your peers in companies that may compete for the same drugs or is it becoming to be very competitive?
Zoran: I think it’s too early for competition. I’ll say this because it’s just that the quantum is too big for any single organization. I think that’s been proven through in some way. I think we want to collaborate. Just look at, for example, clinical trial data. Clinical trials are similar, patients are similar across the globe. Data sets are quite similar, although each medication and what they call indication of therapeutic area and disease have slight differences. But certainly the ability to collaborate at that level is there.
The paper we are talking about that’s coming out soon with our collaboration is multiple companies working on that. So I think we should have and hopefully we’ll have more collaborations because there is enough of the common denominator and the benefit to everyone to do that. Competition usually comes later when they start to fine-tune things, but I think right now it’s time to collaborate more than to compete.
Yuval: Very good. Zoran, thank you so much for spending time with me today.
Zoran: Yuval, thank you so much for inviting me, my pleasure.