Dr. Karim Galil: Welcome to the Patientless Podcast. We discuss the Good, the Bad and the Ugly about Real World Data and AI in clinical research. This is your host, Karim Galil, cofounder and CEO of Mendel AI.I invite key thought leaders across the broad spectrum of believers and descenders of AI to share their experiences with actual AI and Real World Data initiatives.
Hi everyone. And welcome for another episode of Patientlesss podcast. Today's guest is Melissa Tucker. She started her career at McKinsey shortly after getting an MBA from Harvard, and then she went to the venture capital world at DFJ. Then, she decided to do the real thing and join Flatiron. She joined Flatiron six years ago. We're talking about Flatiron when it used to be a small startup and now, six years in and after a big acquisition, she leads product management as a VP for Product Management and Operations. Thank you so much Melisa up for taking the time for the podcast. And I'm very excited to have you, as a guest today.
Melisa Tucker, Flatiron: Thanks Karim, it’s great to be here.
Dr. Karim Galil: So obviously we all know about Flatiron, but for folks who maybe didn't hear about it before, which I really doubt, give us an idea what you guys do, what is Flatiron and what's the mission of the company.
Melisa Tucker, Flatiron: Yeah, absolutely. So Flatiron is a health technology company and we focus in oncology and our mission is to improve the lives of cancer patients by organizing the world's cancer data. So how we do that is, we have about half the company that builds tools and software for providers, community oncology, as well as academic oncologists. And through those tools, we improve their workflow and their ability to work with their patients. On the other half of the company, we. process and consolidate that data and spend a lot of time extracting that data and making it useful so that we can then create research datasets that are then used by cancer researchers, either in academia, life sciences companies, regulatory market access, and the goal is to create richer, larger data sets than have been possible traditionally so that we can improve decision making, accelerate research, and get effective therapeutics to patients faster.
Dr. Karim Galil: So which half of the company do you belong to: that provider facing half or the data side of the company?
Melisa Tucker, Flatiron: I lead the product management team on the data side of the company.
Dr. Karim Galil: So, you're working on the data, and you guys started that category of Real World Data, really? It became more of, I think when Flatiron started this like six years ago, before that it was mainly, the industry was very focused on I would say breadth rather than depth. So we had a lot of claims data, a lot of structured data. Can you walk me through the adoption curve? You joined Flatiron when it was 40, I think 40 people headcount. There you guys are still something. What's the adoption curve looking in the last six years in Real World Data?
Melisa Tucker, Flatiron: Yeah, absolutely. So, when I joined actually we had just raise our series B in acquired Altos, which was an oncology EMR company. So all of a sudden we went from about 20 sites in our network to 250. And overnight we had this, this data set that we thought was sort of at that critical scale to be able to do something useful with on the research side. And then, I mean, it was pretty simple. We just started opening charts and kind of looking at the data and seeing what was available there. And what we found was, and I think this is particularly true in oncology, even more so than in other therapeutic areas, almost all of the rich clinical data that you really needed to understand oncology was trapped in these unstructured notes.
So, the scan documents that come in, things that come over fax, sometimes are not even searchable or OCR-able, and just recognizing that a lot of the information, even about what type of cancer the patient had, or what stage of disease that was, and especially about genomics and understanding the specific makeup of the patient's tumor, which is really critical for understanding how best to treat that patient, that's all information that's just not readily available in structured form. And so what we found was we really need to be able to have a way to process this data in a scalable way, and to do so with high accuracy and reproducibility, and, and the ability to go back to source, right?
So if anyone ever asks us about a data point, we could actually go back and say, this is how we captured it. These are the, the, the documents that were reviewed and, and synthesized in order to come up with this data point. And so to do that we started essentially combining technology and people. So we built Patient Manager, which is our internal tool that is essentially a web interface for our team of abstracters who all have clinical training but are essentially opening the charts and sort of combing through them to come up with the information that we think is important to understanding that patient. And so we built Patient Manager to help those abstracters be more efficient, so that we can monitor them and understand the accuracy of what they're capturing. We can train them and, test them and compare what two different abstracters say. All of these kind of, data quality processes that are built in along the way. And then on the human side just finding the right people, hiring, scaling, figuring out how to manage this very large remote workforce that is helping us do these tasks over and over again. And so that was really the first probably year or so when I joined. And through that process, we were talking to a lot of life sciences companies, trying to get their feedback about what was the minimal data set that they needed to actually be able to do something valuable and how many patients they need that information on, and just trying to understand their use cases and try to build kind of an MVP that would meet those use cases. And it was tough going for, for probably the first 12 or 18 months, and then all of a sudden, and anything in that, during that timeframe, we were trying to find the right balance between depth and breadth, right? So we knew we needed to capture more depth than what other data sources had in the past, but we didn't quite have the right balance until, I remember this one meeting with a customer where, she just gave us this very blunt and sort of honest feedback that we didn't, that we didn't know kind of where we wanted to play relative to other data sets out there. And after that, we actually changed our whole strategy and started going a lot deeper and looking for the information that really only we could get, even if it meant a trade-off around patient numbers. And then all of a sudden, six months later, I feel I looked up and I realized, we had actually achieved product market fit. People were buying these data sets. We were way behind on hiring, relative to where we needed to be. And, and really it's been all been all about scaling since, since then.
Dr. Karim Galil: It's a very interesting point that you just raised actually, that sometimes it's not about how many patients you have. It's how much data you have about these patients. Do you see that being the main buying value or is it really situational, where depending on the customer, depending on the study, you have to make a decision? Or is it honestly throughout all the way, all my customers would like to see more depth rather than breadth?
Melisa Tucker, Flatiron: Yeah. We found that it's just very customer dependent and even within a single life sciences company, it actually depends on who is thinking about this? So the mistake we made initially was we said, we're going to be one data set that's going to help commercial teams figure out how they're doing relative to other therapies and understand market adoption. We're also gonna work with outcomes research teams who are interested in conducting studies and publishing. We're also gonna work with development teams and, and it just turns out it's impossible to build one dataset that does all that, because the way that each of those groups would trade off between depth and breadth, or the specific data points that they need, are all different. And so, kind of focusing in on the target customer and understanding their specific use cases, and then kind of initially finding, you can get to that product-market fit and then building out from there, I think is, is the way to go. And, it turned out that we were not the right data set for some of those customers, and we, once we said that was okay and we're not gonna go after them, I think that really helped us kind of focus in on what we needed to do and prioritize.
Dr. Karim Galil: I had a guest on one of the podcasts and he said there is actually a billing code for catching fire on a surf board. And there is no billing code for non small cell lung cancer. I didn't know. There's actually a billing code for it. What kind of data have you seen consistently missing from structured data, and you can only capture if you go deep into the notes and the scans and some of those data elements specifically for oncology?
Melisa Tucker, Flatiron: Yeah, so I mean, I'll start with a really simple one, which is date of death. This is something that every EHR probably has a field for it. It's not consistently populated because when you think about the workflow of an EMR, the point at which you find out a patient may have passed away, you may not be going back into their chart actively and updating it. So that's something that's very simple that we find is not routinely captured in structured data. And we also have this issue of patients may go to hospitals, or into hospice before they pass away. And so that information may just never make its way back to their treating oncologist. But then, from there it gets a lot more complicated. So, biomarkers, I mentioned earlier, genomic testing, this is with the trend toward precision medicine and developing treatments that can really target the specific mechanism of a patient's tumor. This is information that is really needed to kind of determine a patient's treatment journey and understand why are they getting one therapy over another, or why might they respond to one therapy over another. And it's really not readily available. It comes back as a scanned document, typically from whatever lab was conducting the testing. So that's one where we've spent a lot of time thinking about how do we capture this, depending on the use case and depending on the depth of that genomic information that's needed. And then progression and response are the main end points that are used to kind of understand a patient's outcome other than death in oncology. And so this is really the concept of, is the patient's tumor progressing or getting worse, or is it responding to therapy and shrinking essentially? And these also are just very tough, complex clinical concepts to understand. They're often not distilled from a single data point. You may be, a physician may be looking at scanned radiology scans with the patient's tumor. They may be looking at lab results. They may be conducting just a visual assessment of how the patients doing and are they able to get around in my office, and sort of integrating all of those perspectives into the therapy is working or not, and therefore I'm going to continue the patient on the therapy or switch them to something else. So a single data point, the patient progressed on this state, is actually really complex to capture and you may need to integrate a whole number of potentially conflicting pieces of evidence in order to come up with that assessment. So thinking about, how do we capture that in a way that's reproducible, that's scalable, that can work for different providers and different source systems has really a lot of work on our part.
Dr. Karim Galil: One of my mentors used to work for a very interesting biotech, I’m not gonna mention the name, but he shared with me a very interesting story that he did with Flatiron. Their drug had a very rough competition at one point, and they really didn't know what to do. Right? Their salespeople were getting killed out there, and then they came to you guys. They went to Flatiron. They were saying, listen, we have the problem. Can you guys help? Now I'm talking about a very commercial use case rather than an R&D case. And using the Flatiron data, they figured out, yeah, their competition may be cheaper and may have close or similar outcomes to their compound, but guess what? On a subset of patients, their drug was significantly better. And they were only able to do that using the Flatiron data. Now, all what they needed to do is go back to the salespeople and say, when you go to the oncologists, say, listen, if you have that kind of population, I have the best treatment out there in the market, and here's the data for it. Don't even take my word. And he was saying in less than six months, their sales numbers went like completely different track than what they expected. And I thought that’s, wow, this is a very impressive case here where commercial teams are actually making decisions informed with Real World Data. Is that the majority of your clients? Is a majority of your clients coming with those kinds of commercial use cases? Or is it 50/50 R&D v. commercial? And by the way, I thought this is a crazy impressive story: they didn't know that their drug performs well, like they are manufacturers and they didn't know that this is the sub population that they need to focus on.
Melisa Tucker, Flatiron: . Yeah. That's a really interesting story. I mean I would say our use cases are pretty split between commercial and outcomes research and development. It is the use case of understanding a subpopulation or a potential way to better target treatment, versus what you can study in a clinical trial, I think though is something that I'm very bullish on for RWD generally, because we always need, in my opinion, we're always in need of RCTs to understand whether the drug is working and whether it's safe, but there's only a certain number of people who can be studied a randomized trial. We know that they're biased toward certain demographics and certain people who participate more frequently in those trials. And there's always gonna be these rare sub cohorts, whether it's certain co-morbidities, or a particular biomarker, or other examples. And we did a study on male patients with breast cancer which is a tiny fraction of patients with breast cancer. They often can't be enrolled in clinical trials, but I think these are areas that are perfect for Real World Data because you're going to be capturing this information anyway. Because these patients are being treated in the real world and if you can aggregate across, they may be very rare at individual clinics or sites, but if you can aggregate across lots of clinics and sites and process the data and curate it in the same way so that it's actually analyzable, I think there's just so much that you can learn there that you could never even design a trial to understand or enroll a trial to understand. So I think this is an area where we've seen the most impact generally and whether that leads to either a label expansion in FDA, whether it leads to, now the company is going to go run a trial to better understand that population or talk to physicians about it better, or, be able to talk to patients about it better, I think those are all things that are I think very meaningful and sort of unlock potentially more effective therapy then we would have known about otherwise.
Dr. Karim Galil: How do you see the market? Are the sponsors or the pharma companies, they want to get the data and do their own work on it? Or they would like to see more vertical integration where, give me the data, offer me the professional services on top of it, maybe even give me some epidemiology work? Is it do it yourself kind of an approach, versus I actually would like to see the outcome, I don't want to know how it's being cooked in the backend?
Melisa Tucker, Flatiron: Yeah, it really depends on the customer. I think there's a huge variation in the industry between, sort of range in terms of, how much companies have invested in Real World Data data capabilities, how much they're equipped to do the analysis themselves. And then sort of able to do that. So we do have some customers, often smaller customers but not always, who come to us and sort of want an end to end solution. They want us to help them work on the protocol, write the analysis plan, give them the analysis results, and sometimes even support through a publication. And that is something that we have built capability to do, or sometimes partner with third parties as well to do. And then we have other customers who have dozens or even hundreds of people who know how to analyze Flatiron data, who sometimes whose job it is entirely to analyze Flatiron data. And so for them it's much more about, I want to be able to see the data and run with it in-house and someone internally is going to be able to collaborate better with understanding exactly what the development team needs, or how this fits into my regulatory submission or, how I want to publish on this. So, we've supported both. Our goal I think is really to try to teach people to fish. And so the more people who are developing this capability internally, that's something we're very supportive of. And we've tried to build customer support tools, ways for people to quickly get a response so that if they're using the data, they can talk to someone live. And then even branching out into data usability. So we're developing shared R-packages that different users at different companies can contribute to. So these are all things were kind of due to just increasing usability of Flatiron data and Real World Data data generally, so that we can get more people using it because, you know, initially the hurdle to learn the data and figure out how to how to use it can be pretty high. And so we want to shorten the time to insight, right? From when you get the data set initially to when you actually get something that's useful that can help you make a better decision or publish on it, to get that down so that it's not taking customers nine months as it sometimes did in the early days.
Dr. Karim Galil: One of the big debates in the industry is how big is the market of Real World Data. So you find IQVIA’s market research estimate $80 billion, while you see a lot of other market researchers are saying $2 billion to $4 billion. From 4 to 80 I mean this is not has not even, it's a huge scale, and I've seen companies saying that it's neither of the numbers actually. And it has to be categorized based on whether it's Real World E or Real World D, like if it were Real World Evidence or Real World Data. Anyways, what's your take on the market size? Obviously when you guys started, there was no market I believe at that time or not a significant market, so what's your take on that? What's the market sizing today, how big is that?
Melisa Tucker, Flatiron: Yeah I mean I think it really depends on which market we're talking about. I mean Real World Data is almost a platform, and when I think about what are we using the real world data for, I mean I think those are kind of the markets that are a little bit easier to think about sizing. So if we're talking about outcomes research or health economics, I think that's one market. If we're talking about Real World Data to support or supplement or enhance clinical trials, I think that's almost a totally different market, different user base, different budgets, and certainly different scale of budgets. So I think it really depends on which of those markets you're talking about and different companies play in different spaces, and to some extent maybe doing different things within those. So in terms of the sort of classic Real World Data market when we started, as you said, it was structured EHR data, it was claims. I would say today the market is a lot bigger than what it was when we got in it, and so when you think about the types of things that you can do with unstructured data, I think we've seen companies grow their budgets quite significantly over the past five years. In a way that's kind of commensurate with the value that we can deliver. And from here I think there's significant potential when I think about the roles that Real World Data can play in clinical trials to kind of further tap into those larger development budgets. We just launched perspective clinical genomic trial at a number of our sites in partnership with one of our sponsors, and there you're talking about budgets that are kind of a different order of magnitude from sort of the Real World Data budgets that we've been talking about in the past. So obviously different level of complexity as well. When you were talking about enrolling patients, consenting them, running genomic sequencing on them, and being able to capture additional information that's not routinely available in the chart. So I think that's essentially a different market entirely, and I think if I had to guess for those bigger numbers that you're talking about where IQVIA would come from.
Dr. Karim Galil: So is Real World Data/Real World Evidence a vitamin or a painkiller and why? Is it good to have or if you don't have it, you are at a disadvantage in 2020.
Melisa Tucker, Flatiron: I mean I think we're at the point where it's really kind of a must have in oncology right now and when I think about the penetration that I've seen, certainly other companies as well that are coming into play here, it's almost table stakes for a lot of companies to be able to understand even simple things as we talked about. And I think this is different than other diseases but even simple things in oncology like understanding who's getting your drug and how they're doing, I mean these are things that require really rich and broad data sets beyond what's available elsewhere. And I think there are other problems that Real World Data can solve, that I think are even more exciting and interesting ways that you can help accelerate drug discovery, make better and faster decisions on whether to move forward with a particular drug, or identifying patients who are eligible and could recruit to a clinical trial. I mean these are all things that I would say have even bigger impact and truly when I think about the painkiller analogy, these are things that if you could do that you absolutely would have to do that in order to stay competitive, and also just have a very real impact on patient's lives, right? And that's kind of ultimately what we're all here for at the end of the day, and I'm just a big believer in there's a lot that we've done so far but there's a lot more that that's still out there.
Dr. Karim Galil: So one of your biggest fans is David Shaywitz. He writes a lot about you guys in Forbes and his theory is that what Flatiron is doing is changing how people perceive how we can measure the effectiveness of a drug. One very interesting piece he wrote was “The Deeply Human Core Of Roche's $2.1 Billion Tech Acquisition -- And Why It Made It” and in this article he goes on why you need a very deep manual engine at the very core of tech company like Flatiron. You guys were started by ex-Googlers, people who know their way around technology. Why is technology not able yet to automate the generation of Real World Data and abstraction of data? Out of.. one question is like a lot of people who know AI is AI is all about labeled data, and you guys have been working for five years labeling data pretty much. So what's the technical challenges of healthcare that still requires a deep human core to make it happen to make it work?
Melisa Tucker, Flatiron: I think in healthcare it's so important and so hard to get it right. And so when we started, our focus has really been on we want to be the highest quality approach out there, and we want to have confidence in every data point that we're shipping because we know it's going to be used for some really important decisions and we want to make sure that we can get it right. I think the other part of this is in tech we haven't been good at kind of explaining the black box and what goes in it. And when you think about things like using Real World Data data in support of regulatory decisions I mean these are incredibly you know, fairly conservative organizations that have been doing things the same way for a long time, and unless you can explain to them exactly how a process works and how we got to it and what the biases are and how missing data affects it, they're not going to buy it. And so for us we really started with the philosophy of, we want to get it right and we want to make it explainable. And so that was why frankly for the first few years we almost had nothing to do with AI and ML, and we were focused on how do we build essentially a way to capture this data, for humans to capture this data, in a way that we can really stand behind. And to your point, along the way we've obviously seen the benefit of being able to label a lot of that data, and then and then I think finding opportunities where we think that certain tasks can be really well done or easily done by machines and sort of replacing them along the way, and so just kind of taking a pretty incremental approach to make sure that we can sort of stand behind how we're doing this. And, and every time we introduce ML or incorporate it into how we capture something there are certain metrics that we want to capture and we try to be really transparent about how something works and how well it works and just get acknowledged some of the limitations of it. I mean I think there's a lot more opportunity here and it's something that we're thinking about over the next three years as an area that we can kind of double down in, but to me there always going to be that human component just given how complex the, the source information is and how difficult it is to.. looking at even sometimes having two oncologists looking at the same chart you may get disagreement and you may have to talk through why, how they got to an answer, and so it becomes very challenging I think to automate some of that.
Dr. Karim Galil: You raised a great point which is the black box. I think one of the main themes of our podcast is we're always trying to educate our users about Real World Data/Real World Evidence, and also on the concept of AI. And I believe unless we untangle that black box, AI cannot really meet the promises in healthcare. As you said, no physician, nor regulatory agency, nor FDA, no nobody is going to be able to accept the data that cannot be tracked to the source code and can be explained, while the whole AI industry is all about black boxes. I'll give you an autopilot, but I'm not going to explain to you if your car crash, what happened actually for the car to crash. I think another big limitation of AI, I'm obviously a CEO of an AI company, but I still know the Good, the Bad, and the Ugly of AI. One of the limitations of AI that I see in healthcare is the approximation. AI is not about getting it 100%. You get AI folks to be really excited when the accuracy is 70%, 80%. They're like “Hey, I have this really good model!” And that’s great in the AI world, but it's not really something that physicians are trained on, right? You're trained on the highest standard of accuracy and it's very hard for you. I remember we were talking to a client and he said, “I can actually work on this model,” and the standard, it was an OCR thing and the standard was 70%, “I’ll get it to you at 92%,” and my AI guy was so excited! Then the guy goes what about the other 8%?
Melisa Tucker, Flatiron: Yes
Dr. Karim Galil: Just a very weird question it’s like what do you mean now, I just increased you 20%! I really agree with you, the black box is a big limitation in the world of healthcare, which brings me to this question what is your take on the concept of Patientless or in-silico research? Do you really see the future, one or the other, RCT or Real World Data, or are you going to see them coming together or or melting together to offer both safety, efficacy, but also effectiveness as a third endpoint?
Melisa Tucker, Flatiron: No I think you said it well, I mean I think it has to be both and we're seeing a real example of that play out right now with COVID. We have therapeutics being studied, vaccines being studied in these large scale randomized trials which I think is what we absolutely need in order to understand, right, like I'm not going to take a vaccine unless I know that it's been studied and in an RCT and that it's effective to a certain percentage above a certain threshold, and I understand the safety profile of that. And I think we're also seeing the value of Real World Data in COVID as well, at understanding in real time who's being infected, how are they doing in the real world, I think it's helping us improve the way that patients are being managed, just by kind of understanding that information kind of in the meantime while we wait for the results of the clinical trials, understanding where even the amount of people who are dying in a way that maybe undercoded or you know but by just kind of looking at the number of access to us, I think these are all really interesting things that can be studied through Real World Data but won't replace the need for clinical trials. I think the same is true, kind of generally in the industry to me, we’ll always need RCTs because even the best AI can't remove confounding or limitations, information that's not there. And so to have the ultimate degree of confidence in how a therapeutic or a drug is working I think you'll always need to have that truly kind of randomized, this is the best we can do to isolate the impact of a drug. But I think there's a lot that we can learn, we talked a little bit about about specific subpopulations and areas where we can pursue label expansions. We ran a study that I thought was really interesting where patients who had a certain preexisting condition, heart condition, that was contraindicated for a particular drug, the regulatory agency asks for the pharma company to study whether those patients were getting the drug and what the outcome was. And it turned out that we were able to cobble together close to a hundred patients across the network which gave us a pretty good sample size to understand whether these patients were actually having worse cardiac outcomes. And I think that's just an example where you wouldn't ever do an RCT, and so I think there are examples on both sides. And if you combine these two tools it will get us to a much richer and more complete view of patient health because obviously we need to know how, we need to be able to isolate the effect of a therapeutic in a very controlled setting, but we also need to know how it works in the real world and whether patients are taking it, and ultimately what the effect of this is in a broader population and I think that's sort of where the two can complement each other.
Dr. Karim Galil: So can you share with us what was the most interesting project that you have worked on, or the most interesting finding that you have seen in the last six years, if possible understanding confidentiality.
Melisa Tucker, Flatiron: See I talked about the heart condition study, I think that the one that we're probably most known for is studying effective CDK inhibitor in male breast cancer patients. These are patients that are not able to be studied in a clinical trial or weren’t enrolled in a clinical trial, and being able to understand whether those patients also benefit from the therapy, so that you could actually expand the label of the drug to incorporate those patients and actually give them a treatment option that wouldn't have been available to them, I think that those are the types of opportunities that can have huge patient impact and where I think Real World Data is especially well positioned to play. And so that's an example of a study that I think is really interesting.
Dr. Karim Galil: Sweet and best part of talking to you, I feel like I'm talking to a physician, honestly. You know a lot about Real World Data and oncology and all of that! What was the journey? What did you do? Walk me through the last transformative shifts to medicine.
Melisa Tucker, Flatiron: Yeah I mean I've always been really interested in healthcare, you know I started my career in consulting but working in healthcare, I think as with many people at Flatiron, considered the premed route and and ultimately decided not to do it. But I think that one of the things I love about Flatiron is, and I think this is true of any successful health-tech company, is how cross-functional our teams have to be and so when you're talking about being a product manager it's not just working with software engineers but it's also oncologists and data scientists and data abstractors who are all, actually like literally, on the team with you side by side. One of the things that makes people successful is just this kind of intellectual curiosity and being interested to learn and so we have software engineers who come in from Spotify, who’ve never done any healthcare and three weeks later they're talking about lines of therapy and understanding exactly how that interacts with the code that they’re writing. And I think that that's something that's really fun about health-tech. It's also really hard because, also as a PM, I can't just look at a dataset and know if it's good, right? I need to talk to the oncologist and the data scientist and understand the use case from the customer. And so it's really challenging but sort of the ability to make an impact in patient's lives and understanding the broader mission is really exciting. So I feel I've had a chance to to learn from all of these great oncologists who are practicing, a lot of our oncologists actually do practice in the clinic as well, usually one one day a week to stay current on treatments and standard of care, and so it's just a place where you're constantly learning and soaking up new things. I'm sure you’ve experienced this as well, given even how rapidly things are changing in the field.
Dr. Karim Galil: Well you guys certainly did affect a lot of patient lives in a very positive way, and you also kind of educated all of us in the market on why is it important to go outside the parameters. My last question, if you can zoom call any living person today, who would it be and why?
Melisa Tucker, Flatiron: Yeah! So I think I have to go outside of healthcare here for this and I would say Ruth Bader Ginsburg. I mean obviously she's a total badass, but is also someone who rose to the top of her profession despite pretty tough odds, had a really great partnership with her spouse, raised a family. I have a three and a half year old daughter, I think you do too, Karim, based on seeing her pop into our Zoom the other day. I just think it'd be really cool to have her meet someone who's done so much for shaping the world that we're in. And even if she's going to have different challenges, but I think a lot of them, will kind of build on shared lessons from thinking about how Ginsburg might have navigated the world 50 years ago.
So I just think there would be a lot of, a lot to learn there.
Dr. Karim Galil: That's a good choice. The first time I've talked to Melisa, I was very tense, because my daughter was pretty much jumping on the desk, and you couldn't see it in the camera then all of a sudden she comes in. Yeah, it's tough working from home, but again, thank you so much for coming in. There is a lot that I learned in today's podcast, and the passion that you bring in, and the energy just is very refreshing. So thank you so much for making the time for that and please if you have any questions, reach out to Melisa, she's an extremely, extremely helpful person. And also, please make sure that you like the podcast, share it, and add all of us on LinkedIn.
Melisa Tucker, Flatiron: Thank you so much Karim!
Dr. Karim Galil: Take care. Bye.
Delays in clinical trial enrollment and difficulties enrolling representative samples continue to vex sponsors, sites, and patient populations. Here we investigated use of an artificial intelligence-powered technology, Mendel.ai, as a means of overcoming bottlenecks and potential biases associated with standard patient prescreening processes in an oncology setting.
The application of Artificial Intelligence (AI) in healthcare has been revolutionary, especially with the recent advancements in transformer-based Large Language Models (LLMs). However, the task of understanding unstructured electronic medical records remains a challenge given the nature of the records (e.g., disorganization, inconsistency, and redundancy) and the inability of LLMs to derive reasoning paradigms that allow for comprehensive understanding of medical variables. In this work, we examine the power of coupling symbolic reasoning with language modeling toward improved understanding of unstructured clinical texts. We show that such a combination improves the extraction of several medical variables from unstructured records. In addition, we show that the state-of-the-art commercially-free LLMs enjoy retrieval capabilities comparable to those provided by their commercial counterparts. Finally, we elaborate on the need for LLM steering through the application of symbolic reasoning as the exclusive use of LLMs results in the lowest performance.
We’ve changed our look. Our goal remains the same: make medicine objective. The new site highlights the way our proprietary AI enables organizations to achieve quality and scale when structuring unstructured data. It comes down supercharging your clinical abstraction. We’ve validated that our human in the loop abstraction approach can support a machine that understands medical context like a physician. In our own experiments, the number of variables needing correction decreased by 40%. High quality abstraction = high quality data for cohort selection, real-world evidence, and registries.
The customer, a key player in the genomics space, had a strategic initiative to build a clinic genomic database to support their life sciences customers.
One clinical trial organization was using manual chart review and was looking to reduce the time it takes to find eligible patients.
From the Desk of the AI Team
Organizations that use patient data for internal or external research need to take steps to prevent the exposure of PHI to those who are not authorized to view it. They do this by redacting specific categories of identifiers from every patient document. Once the identifiers are masked, the risk profile of these datasets is significantly reduced. But how do you ensure that redaction engines are working to the highest accuracy?
The Mendel team is still buzzing from our week-long retreat in Cairo. The theme of the retreat was “coming together” and it was the first time the American and other remote employees were united with their Egyptian counterparts. Although there were many adventures–missing flights, seeing the pyramids, haggling at Khan el-Khalili–the highlight of the trip was collaborating together, as one global organization.
Competence via comprehension
Artificial intelligence (AI) is playing an increasingly important role in the healthcare industry. But to fully leverage the potential of AI, it must be equipped with clinical reasoning skills - the ability to truly comprehend clinical data, or in other words, to read it as a doctor would. When it comes to data processing tools, only a tool capable of clinical reasoning can effectively process unstructured clinical data.
Sailu Challapalli, our Chief Product Officer, spoke at a recent Harvard Business School Healthcare panel. The event brought together different healthcare and AI experts to discuss large language models and their impact.
Manually abstracting patient data at scale is an herculean task for humans alone. It is slow, expensive, difficult, and requires extreme precision and accuracy. Organizations have to choose between breadth and depth when it comes to making data useful for decision making. Because of these challenges, the Mendel team created Carbon. Carbon is an easy to use workspace that allows clinical abstraction teams to efficiently curate high quality clinical datasets at scale. The foundation of Carbon is Mendel’s AI. Carbon pulls directly from Mendel’s AI platform to give abstractors a headstart in identifying relevant data elements within a patient’s chart.
Within the real world evidence space, the generally accepted process for creating a regulatory grade data set is to have two human abstractors work with the same set of documents and bring in a third reviewer to adjudicate the differences. These datasets also serve a second purpose - as a reference standard against which the performance of human abstractors can be measured. Although this remains the industry standard, it is expensive, time consuming and difficult to scale.
From the Desk of the AI Team
AI projects have created tangible results for a wide range of industries. Despite the innovation, it is important to remember that AI is not a magic wand that will solve every problem in every industry with a single wave.