ibm watson at md anderson cancer center case study

More From Forbes

Md anderson benches ibm watson in setback for artificial intelligence in medicine.

• Share to Facebook
• Share to Twitter
• Share to Linkedin

Virginia "Ginni" Rometty, chief executive officer of International Business Machines Corp. (IBM) ... [+] Photographer: David Paul Morris/Bloomberg

It was one of those amazing “we’re living in the future” moments. In an October 2013 press release , IBM declared that MD Anderson, the cancer center that is part of the University of Texas, “is using the IBM Watson cognitive computing system for its mission to eradicate cancer.”

Well, now that future is past. The partnership between IBM and one of the world’s top cancer research institutions is falling apart. The project is on hold, MD Anderson confirms, and has been since late last year. MD Anderson is actively requesting bids from other contractors who might replace IBM in future efforts. And a scathing report from auditors at the University of Texas says the project cost MD Anderson more than $62 million and yet did not meet its goals. The report, however, states: "Results stated herein should not be interpreted as an opinion on the scientific basis or functional capabilities of the system in its current state."

“When it was appropriate to do so, the project was placed on hold,” an MD Anderson spokesperson says. “As a public institution, we decided to go out to the marketplace for competitive bids to see where the industry has progressed.”

Also on Forbes:

The disclosure comes at an uncomfortable moment for IBM. Tomorrow, the company’s chief executive, Ginni Rometty, will make a presentation to a giant health information technology conference detailing the progress Watson has made in healthcare, and announcing the launch of new products for managing medical images and making sure hospitals deliver value for the money, as well as new partnerships with healthcare systems. The end of the MD Anderson collaboration looks bad.  Even if the decision is as much a result of MD Anderson's mismanagement or red tape--which it may be--it is still a setback for a field without any big successes.

But IBM defended the MD Anderson product, known as the Oncology Expert Advisor or OEA. It says the OEA’s recommendations were accurate, agreeing with experts 90% of the time. “The OEA R&D project was a success, and likely could have been deployed had MD Anderson chosen to take it forward,” says an IBM spokesperson.

Watson, IBM’s language-based computing project, gripped the world’s imagination in 2011 when the supercomputer won an exhibition of the game show Jeopardy! against the show’s two highest-rated players. In March 2012, IBM signed a deal with Memorial Sloan Kettering Cancer Center in New York to develop a commercial product that would use the same technology to analyze the medical literature and help doctors choose treatments for cancer patients.

MD Anderson, Memorial’s longtime rival, entered the fray after this agreement was already in place. Lynda Chin, the former chair of the MD Anderson Department of Genomic Medicine and the wife of MD Anderson president Ronald DePinho, set up a collaboration with IBM to develop a separate project. Chin left MD Anderson for another job within the University of Texas system in 2015.

In a strange twist, MD Anderson would pay for the whole thing, eventually giving $39.2 million to IBM and $21.2 million to PricewaterhouseCoopers, which was hired to create a business plan around the product. According to the Washington Post , at least $50 million of the money came from Low Taek Jho, a flamboyant Malaysian financier whose business dealings are reportedly now under investigation by the U.S. Department of Justice .

Usually, companies pay research centers to do research on their products; in this case, MD Anderson paid for the privilege, although it would have apparently also owned the product. This was a “very unusual business arrangement,” says Vinay Prasad, an oncologist at Oregon Health & Science University.

According to the audit report, Chin went around normal procedures to pay for the expensive undertaking. The report notes "a consistent pattern of PwC fees set just below MD Anderson’s Board approval threshold," and its appendix seems to indicate this may have occurred with payments to IBM, too.* She also didn’t get approval from the information technology department.

It seems “very strange” that the IT department was bypassed, and “very unusual” that payments were not based on measurable deliverables, says John Halamka, the chief information officer at Beth Israel Deaconness Medical Center in Boston. He also notes that payments seem to have been made from donations that had not yet been received.

Despite all this drama, initial reports on the MD-Anderson/Watson collaboration were positive. In 2015 the Washington Post said MD Anderson doctors-in-training were amazed by the machine’s recommendations. “I was surprised,” one told the newspaper. “Even if you work all night, it would be impossible to be able to put this much information together like that.”

But inside the University of Texas, the project was apparently seen as one that missed deadlines and didn’t deliver. The audit notes that the focus of the project was changed several times, first focusing on one type of leukemia, then another, then lung cancer. The initial plan was to test out the product out in pilots at two other hospitals. That never happened.

MD Anderson changed the software it uses for managing electronic medical records, switching to a system made by Epic Systems of Madison, Wis. It has blamed this new system for a $405 million drop in its net income . According to the audit report, the Watson product doesn’t work with the new Epic system, and must be revamped in order to be re-tested. The information in the MD-Anderson/Watson product is also now out of date.

In September, IBM stopped supporting the product, according the audit, which was produced last November. The Cancer Letter and the Houston Chronicle reported on the audit last week. Forbes obtained a copy of a request for proposals confirming that MD Anderson is actively looking for a company to take on IBM’s role. In a statement, MD Anderson said that it was not excluding companies that had previously worked with it from job, implying that it might choose to go with IBM to reboot the project.

Meanwhile, IBM now sells a product it developed with Memorial Sloan Kettering. The goal, as with the MD Anderson product, is to help doctors select treatments. Without a computer, this is done with a so-called “tumor board,” a group of experts who meet weekly. IBM points to a dozen studies presented at academic meetings showing that Watson’s recommendations agree with those of tumor boards.

When IBM CEO Rometty makes her announcements tomorrow at HIMSS, the health-tech conference, the question for doctors and investors will be this: are they more like the Memorial Sloan Kettering effort, which seems to have resulted in a real product? Or are they like the mess that seems to have happened at MD Anderson?

*A previous version of this story said the payments below the board threshold were made to IBM.

• Editorial Standards
• Reprints & Permissions

• Search Menu
• Sign in through your institution
• Advance Articles
• Supplements
• Early Career Investigators
• Author Guidelines
• Detailed inclusive language guidelines
• Submission Site
• Why Publish with Us?
• Open Access
• Author Resource Centre
• Early Career Investigator Research Section
• Editorial Board
• Diversity, Equity, and Inclusion
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

• < Previous

M. D. Anderson Breaks With IBM Watson, Raising Questions About Artificial Intelligence in Oncology

• Article contents
• Figures & tables
• Supplementary Data

Charlie Schmidt, M. D. Anderson Breaks With IBM Watson, Raising Questions About Artificial Intelligence in Oncology, JNCI: Journal of the National Cancer Institute , Volume 109, Issue 5, May 2017, djx113, https://doi.org/10.1093/jnci/djx113

• Permissions Icon Permissions

In 2012, the University of Texas M. D. Anderson Cancer Center in Houston partnered with IBM to develop the artificial intelligence program, called IBM Watson, as a clinical decision tool in oncology. Five years and $62 million later, M. D. Anderson let its contract with IBM expire before anyone used Watson on actual patients.

Last February, a university audit of the project exposed many procurement problems, cost overruns, and delays. Although the audit took no position on Watson’s scientific basis or functional capabilities, it did describe challenges with assimilating Watson into the hospital setting.

Experts familiar with Watson’s applications in oncology describe problems with the system’s ability to digest written case reports, doctors’ notes, and other text-heavy information generated in medical care. That type of unstructured data differs from the structured data entered into drop-down boxes and other point-and click fields in an electronic medical record that Watson can more readily interpret.

“The M. D. Anderson experience is telling us that solving data quality problems in unstructured data is a much bigger challenge for artificial intelligence than was first anticipated,” said Amy Abernethy, M.D., chief medical officer at Flatiron Health, a New York–based health care technology company, and former director of cancer research at the Duke Cancer Institute in Durham, N.C.

“The M. D. Anderson experience is telling us that solving data quality problems in unstructured data is a much bigger challenge for artificial intelligence than was first anticipated.”

Watson’s selling point is that it helps doctors stay current with the volume of new findings published every day. IBM has been working on natural language processing capabilities that allow Watson to increasingly understand human speech. Oncologists have been trying to teach the computer system to think like a cancer doctor by training it with real and made-up cases. At M. D. Anderson, those training exercises started with a pilot focused on leukemia.

But medical diagnosis is more complicated than Jeopardy questions. Institutions use medical terms in different ways, and despite the best efforts of software engineers, Watson still can’t interpret medical language as well as humans can. At M. D. Anderson, for instance, Watson couldn’t reliably distinguish the acronym for acute lymphoblastic leukemia, ALL , from the shorthand for allergy, which is often also written ALL .

“To do that correctly, Watson would have to interpret those written characters in the appropriate context,” Abernethy said. “It would have to know that ALL in the context of high white blood cell counts or a bone marrow transplant means leukemia and not something else.” M. D. Anderson suspended its leukemia pilot midstream before switching to lung cancer “because project leaders thought that area would provide greater opportunity for a timely completion,” the UT audit stated.

Andrew Seidman, M.D., a medical oncologist at the Memorial Sloan Kettering Cancer Center in New York, has been trying to teach Watson how to treat breast cancer. He said that to bring its accumulated knowledge to bear on treating patients at the hospital, Watson needs to be better integrated with electronic medical records.

“The more efficient way to do that would be for it to extract medical attributes automatically,” Seidman said. “But many of the attributes we want Watson to digest aren’t found in categorical structured data—they’re buried in narrative form in doctors’ consultation notes.” And though natural language processing is central to Watson’s functionality, Seidman said that “it remains a work in progress.”

Andrew Norden, M.D., deputy chief health officer at IBM Watson Health, acknowledged the need to clarify abbreviations in unstructured medical notes and “other things Watson hasn’t seen before.” He added, “But this isn’t rocket science—the trick is getting the right digital content to Watson so that Watson can read it.” In M. D. Anderson’s case, a third party—the London-based consulting firm PricewaterhouseCoopers—was contracted to develop a bioinformatics tool to integrate Watson with the hospital’s record system.

According to M. D. Anderson’s audit, Watson’s treatment recommendations during the lung cancer pilot agreed with those of its human teachers nearly 90% of the time. “This is a very high level of accuracy,” Norden said.Abernethy , however, was more skeptical. “What does 90% accuracy really mean?” she asked. “Does that mean that for common, run-of-the-mill clinical scenarios the technology was wrong 10% of the time? Or does it mean that 10% of the time Watson couldn’t help on the more difficult cases for which treatment decisions may not be so clear-cut?”

Seidman said that in training the system, he’s trying to make Watson’s recommendations agree more often with practice standards at Sloan Kettering. But he said the hope is that Watson will eventually pick better treatments that he and his colleagues might not have considered without the technology. “What we’re really looking for are computer-assisted decisions that extend the time to disease progression and improve overall survival rates,” he said.

Harpreet Singh Buttar, a health care technology industry analyst with Frost & Sullivan, a San Antonio–based market research company, predicts that the M. D. Anderson debacle will affect IBM Watson Health primarily in the short term. Buttar said that in some ways the bigger problem for IBM Watson is the number of competitors entering the market. He recently reported that the health care artificial intelligence business now has more than 100 startups, and over half of them have raised funding from venture capital and other third-party investors since January 2015.

InAbernethy ’s view, all those companies will probably face problems similar to those that IBM Watson encountered at M. D. Anderson. “We would all like medical data to be as clean and fully curated as possible,” she said. “But practicing doctors have many reasons to produce unstructured documents, and we can’t take that away from them. We can’t ask them to structure everything; otherwise, we’re structuring data at the expense of good medical care.” Still, she added, “I do think that artificial intelligence systems will improve, as long as the unstructured data formats improve too.”

Email alerts

Citing articles via, looking for your next opportunity.

• Recommend to your Library

Affiliations

• Online ISSN 1460-2105
• Print ISSN 0027-8874
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• Health Tech
• Health Insurance
• Medical Devices
• Gene Therapy
• Neuroscience
• H5N1 Bird Flu
• Health Disparities
• Infectious Disease
• Mental Health
• Cardiovascular Disease
• Chronic Disease
• Alzheimer's
• Coercive Care
• The Obesity Revolution
• The War on Recovery
• Adam Feuerstein
• Matthew Herper
• Jennifer Adaeze Okwerekwu
• Ed Silverman
• CRISPR Tracker
• Breakthrough Device Tracker
• Generative AI Tracker
• Obesity Drug Tracker
• 2024 STAT Summit
• All Summits
• STATUS List
• STAT Madness
• STAT Brand Studio

Don't miss out

Subscribe to STAT+ today, for the best life sciences journalism in the industry

IBM pitched its Watson supercomputer as a revolution in cancer care. It’s nowhere close

By Casey Ross and Ike Swetlitz

Sept. 5, 2017

I t was an audacious undertaking, even for one of the most storied American companies: With a single machine, IBM would tackle humanity’s most vexing diseases and revolutionize medicine.

Breathlessly promoting its signature brand — Watson — IBM sought to capture the world’s imagination, and it quickly zeroed in on a high-profile target: cancer.

advertisement

But three years after IBM began selling Watson to recommend the best cancer treatments to doctors around the world, a STAT investigation has found that the supercomputer isn’t living up to the lofty expectations IBM created for it. It is still struggling with the basic step of learning about different forms of cancer. Only a few dozen hospitals have adopted the system, which is a long way from IBM’s goal of establishing dominance in a multibillion-dollar market. And at foreign hospitals, physicians complained its advice is biased toward American patients and methods of care.

STAT examined Watson for Oncology’s use, marketing, and performance in hospitals across the world, from South Korea to Slovakia to South Florida. Reporters interviewed dozens of doctors, IBM executives, artificial intelligence experts, and others familiar with the system’s underlying technology and rollout.

The interviews suggest that IBM, in its rush to bolster flagging revenue, unleashed a product without fully assessing the challenges of deploying it in hospitals globally. While it has emphatically marketed Watson for cancer care, IBM hasn’t published any scientific papers demonstrating how the technology affects physicians and patients. As a result, its flaws are getting exposed on the front lines of care by doctors and researchers who say that the system, while promising in some respects, remains undeveloped.

“Watson for Oncology is in their toddler stage, and we have to wait and actively engage, hopefully to help them grow healthy,” said Dr. Taewoo Kang, a South Korean cancer specialist who has used the product.

Related: A new advertising tack for hospitals: IBM’s Watson supercomputer is in the house

At its heart, Watson for Oncology uses the cloud-based supercomputer to digest massive amounts of data — from doctor’s notes to medical studies to clinical guidelines. But its treatment recommendations are not based on its own insights from these data. Instead, they are based exclusively on training by human overseers, who laboriously feed Watson information about how patients with specific characteristics should be treated.

IBM executives acknowledged Watson for Oncology , which has been in development for nearly six years, is in its infancy. But they said it is improving rapidly, noting that by year’s end, the system will offer guidance about treatment for 12 cancers that account for 80 percent of the world’s cases. They said it’s saving doctors time and ensuring that patients get top-quality care.

“We’re seeing stories come in where patients are saying, ‘It gave me peace of mind,’” Watson Health general manager Deborah DiSanzo said. “That makes us feel extraordinarily good that what we’re doing is going to make a difference for patients and their physicians.”

Sign up for Daily Recap

A roundup of STAT's top stories of the day.

But contrary to IBM’s depiction of Watson as a digital prodigy, the supercomputer’s abilities are limited.

Perhaps the most stunning overreach is in the company’s claim that Watson for Oncology, through artificial intelligence, can sift through reams of data to generate new insights and identify, as an IBM sales rep put it, “even new approaches” to cancer care. STAT found that the system doesn’t create new knowledge and is artificially intelligent only in the most rudimentary sense of the term.

While Watson became a household name by winning the TV game show “Jeopardy!”, its programming is akin to a different game-playing machine: the Mechanical Turk, a chess-playing robot of the 1700s, which dazzled audiences but hid a secret — a human operator shielded inside.

In the case of Watson for Oncology, those human operators are a couple dozen physicians at a single, though highly respected, U.S. hospital: Memorial Sloan Kettering Cancer Center in New York. Doctors there are empowered to input their own recommendations into Watson, even when the evidence supporting those recommendations is thin.

The actual capabilities of Watson for Oncology are not well-understood by the public, and even by some of the hospitals that use it. It’s taken nearly six years of painstaking work by data engineers and doctors to train Watson in just seven types of cancer, and keep the system updated with the latest knowledge.

“It’s been a struggle to update, I’ll be honest,” said Dr. Mark Kris, Memorial Sloan Kettering’s lead Watson trainer. He noted that treatment guidelines for every metastatic lung cancer patient worldwide recently changed in the course of one week after a research presentation at a cancer conference. “Changing the system of cognitive computing doesn’t turn around on a dime like that,” he said. “You have to put in the literature, you have to put in cases.”

Watson grew out of an effort to transform IBM from an old-guard hardware company to one that operates in the cloud and along the cutting edge of artificial intelligence. Despite its use in an array of industries — from banking to manufacturing — it has failed to end a streak of 21 consecutive quarters of declining revenue at IBM. In the most recent quarter, revenue even slid from the same period last year in IBM’s cognitive solutions division — which is built around Watson and is supposed to be the future of its business.

In response to STAT’s questions, IBM said Watson, in health care and otherwise, remains on an upward trajectory and “is already an important part” of its $20 billion analytics business. Health care is a crucial part of the Watson enterprise. IBM employs 7,000 people in its Watson health division and sees the industry as a $200 billion market over the next several years. Only financial services, at $300 billion, is considered a bigger opportunity by the company.

Related: How is artificial intelligence like a self-driving car? Cardiologists ponder the future

At stake in the supercomputer’s performance is not just the fortunes of a famed global company. In the world of medicine, Watson is also something of a digital canary — the most visible attempt to use artificial intelligence to identify the best ways to prevent and treat disease. The system’s larger goal, IBM executives say, is to democratize medical knowledge so that every patient, no matter the person’s geography or income level, will be able to access the best care.

But in cancer treatment, the pursuit of that utopian ideal has faltered.

STAT’s investigation focused on Watson for Oncology because that product is the furthest along in clinical care, though Watson sells separate packages to analyze genomic information and match patients to clinical trials. It’s also applying Watson to other tasks, including honing preventive medicine practices and reading medical images .

Doctors’ reliance on Watson for Oncology varies among hospitals. While institutions with fewer specialists lean more heavily on its recommendations, others relegate the system to a background role, like a paralegal whose main skill is researching existing knowledge.

Hospitals pay a per-patient fee for Watson for Oncology and other products enabled by the supercomputer. The amount depends on the number of products a hospital buys, and ranges between $200 and $1,000 per patient, according to DiSanzo. The system sometimes comes with consulting costs and is expensive to link with electronic medical records. At hospitals that don’t link it with their medical records, more time must be spent typing in patient information.

At Jupiter Medical Center in Florida, that task falls to nurse Jean Thompson, who spends about 90 minutes a week feeding data into the machine. Once she has completed that work, she clicks the “Ask Watson” button to get the supercomputer’s advice for treating patients.

On a recent morning, the results for a 73-year-old lung cancer patient were underwhelming: Watson recommended a chemotherapy regimen the oncologists had already flagged.

“It’s fine,” Dr. Sujal Shah, a medical oncologist, said of Watson’s treatment suggestion while discussing the case with colleagues.

He said later that the background information Watson provided, including medical journal articles, was helpful, giving him more confidence that using a specific chemotherapy was a sound idea. But the system did not directly help him make that decision, nor did it tell him anything he didn’t already know.

Related: Watson goes to Asia: Hospitals use supercomputer for cancer treatment

Jupiter is one of two U.S. hospitals that have adopted Watson for Oncology. The system has generated more business in India and Southeast Asia. Many doctors in those countries said Watson is saving time and helping more patients get quality care. But they also said its accuracy and overall value is limited by differing medical practices and economic circumstances.

Despite IBM’s marketing blitz, with years of high-profile Watson commercials featuring celebrities from Serena Williams to Bob Dylan to Jon Hamm, the company’s executives are not always gushing. In interviews with STAT, they acknowledged the system faces challenges and needs better integration with electronic medical records and more data on real patients to find patterns and suggest cutting-edge treatments.

“The goal as Watson gets smarter is for it to make some of those recommendations in a more automated way, to sort of suggest now may be the time and let us flip the switch” when a promising treatment option emerges, said Dr. Andrew Norden, a former IBM deputy health chief who left the company in early August. “As I describe it, you’re probably getting a sense it’s really hard and nuanced.”

Such nuance is absent from the careful narrative IBM has constructed to sell Watson.

I t is by design that there is not one independent, third-party study that examines whether Watson for Oncology can deliver. IBM has not exposed the product to critical review by outside scientists or conducted clinical trials to assess its effectiveness.

While it’s not unheard of for companies to avoid external vetting early on, IBM’s circumstances are unusual because Watson for Oncology is not in development — it has already been deployed around the world.

Yoon Sup Choi, a South Korean venture capitalist and researcher who wrote a book about artificial intelligence in health care, said IBM isn’t required by regulatory agencies to do a clinical trial in South Korea or America before selling the system to hospitals. And given that hospitals are already using the system, a clinical trial would be unlikely to improve business prospects.

“It’s too risky, right?” Choi said. “If the result of the clinical trial is not very good — [if] there’s a marginal clinical benefit from Watson — it’s really bad news to the whole IBM.”

Pilar Ossorio, a professor of law and bioethics at University of Wisconsin Law School, said Watson should be subject to tighter regulation because of its role in treating patients. “As an ethical matter, and as a scientific matter, you should have to prove that there’s safety and efficacy before you can just go do this,” she said.

Norden dismissed the suggestion IBM should have been required to conduct a clinical trial before commercializing Watson, noting that many practices in medicine are widely accepted even though they aren’t supported by a randomized controlled trial.

“Has there ever been a randomized trial of parachutes for paratroopers?” Norden asked. “And the answer is, of course not, because there is a very strong intuitive value proposition. … So I believe that bringing the best information to bear on medical decision making is a no-brainer.”

IBM said in its statement that it has collaborated with the research community and presented data on Watson at industry gatherings and in peer-reviewed journals. Some doctors said they didn’t need to see more research to know that the system is valuable. “Artificial intelligence will be adopted in all medical fields in the future,” said Dr. Uhn Lee, who runs the Watson program at Gachon University Gil Medical Center in South Korea. “If that trend, that change is inevitable, then why don’t we just start early?”

Related: Artificial intelligence is coming to medicine — don’t be afraid

So far, the only studies about Watson for Oncology are conference abstracts. The full results haven’t been published in peer-reviewed journals — and every study, save one, was either conducted by a paying customer or included IBM staff on the author list, or both. Most trumpet positive results, showing that Watson saves doctors time and has a high concordance rate with their treatment recommendations.

The “concordance” studies comprise the vast majority of the public research on Watson for Oncology. Doctors will ask Watson for its advice for treating a slew of patients, and then compare its recommendations to those of oncologists. In an unpublished study from Denmark, the rate of agreement was about 33 percent — so the hospital decided not to buy the system. In other countries, the rate can be as high as 96 percent for some cancers. But showing that Watson agrees with the doctors proves only that it is competent in applying existing methods of care, not that it can improve them.

IBM executives said they are pursuing studies to examine the impact on doctors and patients, although none has been completed to date.

Questions about Watson have begun spilling into public view, including in a recent Gizmodo story headlined “Why Everyone is Hating on IBM Watson — Including the People Who Helped Make It.” The most prominent failure occurred last February when MD Anderson Cancer Center, part of the University of Texas, cancelled its partnership with Watson.

The MD Anderson alliance was essentially the early face of Watson in health care. The Houston hospital was among IBM’s first partners, and it was using the system to create its own expert oncology adviser, similar to the one IBM was developing with Memorial Sloan Kettering. But the project disintegrated amid internal allegations of overspending, delays, and mismanagement. In all, MD Anderson spent more than three years and $60 million — much of it on outside consultants — before shelving the effort.

The hospital declined to answer questions. But the project leader, Dr. Lynda Chin, in her first media interview on the subject, told STAT about the challenges she faced. Chin left MD Anderson before the project collapsed; a subsequent audit flagged several violations of procurement rules under her leadership.

Chin said that Watson is a powerful technology, but that it is exceedingly difficult to make functional in health care. She and her team encountered numerous roadblocks, some of which still have not been fully addressed by IBM — at MD Anderson or elsewhere.

The cancer hospital’s first major challenge involved getting the machine to deal with the idiosyncrasies of medical records: the acronyms, human errors, shorthand phrases, and different styles of writing. “Teaching a machine to read a record is a lot harder than anyone thought,” she said. Her team spent countless hours on that problem, trying to get Watson to extract valuable information from medical records so that it could apply them to its recommendations.

Chin said her team also wrestled with deploying the system in clinical practice. Watson, even if guided by doctors, is as close as medicine has ever gotten to allowing a machine to help decide the treatments delivered to human beings. That carries with it thorny questions, such as how to test the safety of a digital treatment adviser, how to ensure its compliance with regulations, and how to incorporate it into the daily work of doctors and nurses.

“Importantly,” Chin said. “How do we create an environment that can ensure the most important tenet in medicine: Do no harm?”

Finally, the project ran into a bigger obstacle: Even if you can get Watson to understand patient variables and make competent treatment recommendations, how do you get it access to enough patient data, from enough different sources, to derive insights that could significantly advance the standard of care?

Chin said that was a showstopper. Watson did not have a connected network of institutions feeding data about specific cohorts of patients. “You may have 10,000 patients for lung cancer. That is still not a very big number when you think about it,” she said.

With data from many more patients, Chin said, you could see patterns — “subsets [of patients] that respond a certain way, subsets that don’t, subsets that have a certain toxicity. That pattern would help with better personalized and precision medicine. But we can’t get there without the ability to actually have a way of aggregating them.”

IBM told STAT that Chin’s work was separate from the effort to create Watson for Oncology, which was validated by cancer specialists at Memorial Sloan Kettering prior to its deployment. The company said that Watson for Oncology can extract and summarize substantial text from patient records, though the information must be verified by a clinician, and that it has made significant progress in obtaining more data to improve Watson’s performance. It pointed to partnerships with the health care publisher Elsevier and the analytics firm Doctor Evidence .

To date, more than 50 hospitals on five continents have agreements with IBM, or intermediary technology companies, to use Watson for Oncology to treat patients, and others are using the genomics and clinical trials products.

But the partnership with Memorial Sloan Kettering, and the product that grew out of it, resulted in complications that IBM has papered over with carefully parsed statements and misleading marketing.

I n its press releases, IBM celebrates Memorial Sloan Kettering’s role as the only trainer of Watson. After all, who better to educate the system than doctors at one of the world’s most renowned cancer hospitals?

But several doctors said Memorial Sloan Kettering’s training injects bias into the system, because the treatment recommendations it puts into Watson don’t always comport with the practices of doctors elsewhere in the world.

Given the same clinical scenario, doctors can — and often do — disagree about the best course of action, whether to recommend surgery or chemotherapy, or another treatment. Those discrepancies are especially wide for second- and third-line treatments given after an initial therapy fails, where evidence of benefits is slimmer and consensus more elusive.

Rather than acknowledge this dilemma, IBM executives, in marketing materials and interviews, have sought to downplay it. In an interview with STAT, DiSanzo, the head of Watson Health, rejected the idea that Memorial Sloan Kettering’s involvement creates any bias at all.

“The bias is taken out by the sheer amount of data we have,” she said, referring to patient cases and millions of articles and studies fed into Watson.

But that mischaracterizes how Watson for Oncology works. (IBM later claimed that DiSanzo was referring to Watson in general.)

The system is essentially Memorial Sloan Kettering in a portable box. Its treatment recommendations are based entirely on the training provided by doctors, who determine what information Watson needs to devise its guidance as well as what those recommendations should be.

When users ask Watson for advice, the system also searches published literature — some of which is curated by Memorial Sloan Kettering — to provide relevant studies and background information to support its recommendation. But the recommendation itself is derived from the training provided by the hospital’s doctors, not the outside literature.

Doctors at Memorial Sloan Kettering acknowledged their influence on Watson. “We are not at all hesitant about inserting our bias, because I think our bias is based on the next best thing to prospective randomized trials, which is having a vast amount of experience,” said Dr. Andrew Seidman, one of the hospital’s lead trainers of Watson. “So it’s a very unapologetic bias.”

Seidman said the hospital is careful to keep its training grounded in clinical evidence when the evidence exists, but it is not shy about giving its recommendations when it doesn’t. “We want cancer care to be democratized,” he said.  “We don’t want doctors who don’t have the thousands and thousands of patients’ experience on a more rare cancer to be handicapped. We want to share that knowledge base.”

At a recent training session of Watson on Manhattan’s Upper East Side, the tensions involved in programming the system were on full display. STAT sat in as Memorial Sloan Kettering doctors, led by Seidman, gathered with IBM engineers to train Watson to treat bladder cancer. Five IBM engineers sat on one side of the table. Across from them were three oncologists — one specializing in surgery, another in radiation, and a third in chemotherapy and targeted medicines.

Several minutes into the discussion, the question arose of which treatment to recommend for patients whose cancers persisted through six rounds of chemotherapy. The options in such cases tend to be as slim as the evidence supporting them. Should Watson recommend a radical surgery to remove the bladder? Dr. Tim Donahue, the surgical oncologist, noted that such surgery seldom cures patients and is not associated with improved survival in his experience.

Then what about another course of chemotherapy combined with radiation?

When Watson gives its recommendations, it puts the top recommendation in green, alternative options in orange, and not recommended options in red.

But in some clinical scenarios, it’s difficult to tell the colors apart.

“This is the hard part of this whole game,” Dr. Marisa Kollmeier, the radiation oncologist, said during the training. “There’s a lack of evidence. And you don’t know if something should be in green without evidence. We don’t have a randomized trial to support every decision.”

But the task in front of them required the doctors to press ahead. And they did, rifling through an array of clinical scenarios. In some cases, a large body of evidence backed up their answers. But many others fell into a gray area or were clouded by the inevitable uncertainty of patient preferences.

The meeting was one of many in a months-long process to bring Watson up to speed in bladder cancer. Subsequent sessions would involve feeding it data on real patient cases at Memorial Sloan Kettering, so doctors could reinforce Watson’s training with repetition.

That training does not teach Watson to base its recommendations on the outcomes of these patients, whether they lived, or died or survived longer than similar patients. Rather, Watson makes its recommendations based on the treatment preferences of Memorial Sloan Kettering physicians.

At some institutions using Watson, IBM’s lack of clarity on the cancer center’s role causes confusion. Some seem to think they are getting advice from doctors around the world.

“As we tell the patients, it’s like another consultation, but it’s a worldwide consultation,” said Dr. K. Adam Lee, medical director of thoracic oncology at Jupiter Medical Center, when STAT visited in June.

“Really worldwide,” added Kerri Ward, an oncology nurse at the hospital. “It pulls from 300 journals, just for oncology, the clinical database, so the national clinical database, journals, textbooks, and then Sloan Kettering is the one that’s feeding in the clinical [information] currently.”

Robert Garrett, the CEO of Hackensack Meridian Health, a group in New Jersey that is using a version of Watson for Oncology, said the information in Watson is “global.”

“If you’re a patient that has colon cancer, they have in their database, as I understand it, how colon cancer is treated around the world, by different clinicians, what’s been the most effective treatment for different phases of colon cancer,” Garrett said. “That’s what IBM Watson brings to the table.”

None of that accurately depicts how Watson for Oncology works.

S everal doctors who have examined Watson in other countries told STAT that Memorial Sloan Kettering’s role has given them pause. Researchers in Denmark and the Netherlands said hospitals in their countries have not signed on with Watson because it is too focused on the preferences of a few American doctors.

Martijn van Oijen, an epidemiologist and associate professor at Academic Medical Center in the Netherlands, said Memorial Sloan Kettering is packed with top specialists but doesn’t have a monopoly on cancer expertise. “The bad thing is, it’s a U.S.-based hospital with a different approach than some other hospitals in the world,” said van Oijen, who’s involved in a national initiative to evaluate technologies like Watson and is a strong believer in using artificial intelligence to help cancer doctors.

In Denmark, oncologists at one hospital said they have dropped the project altogether after finding that local doctors agreed with Watson in only about 33 percent of cases.

“We had a discussion with [IBM] that they had a very limited view on the international literature, basically, putting too much stress on American studies, and too little stress on big, international, European, and other-part-of-the-world studies,” said Dr. Leif Jensen, who directs the center at Rigshospitalet in Copenhagen that contains the oncology department.

In countries where doctors were trained in the United States, or they use similar treatment guidelines as the Memorial Sloan Kettering doctors, Watson for Oncology can be helpful. Taiwan uses the same guidelines as Americans, so Watson’s advice will be useful there, said Dr. Jeng-Fong Chiou, vice superintendent of the Taipei Cancer Center at Taipei Medical University, which started using Watson for Oncology with patients in July.

But he also said there are differences between American and Taiwanese patients — his patients often receive lower doses of drugs to minimize side effects — and that his oncologists will have to make adjustments from Watson’s recommendations.

The generally affluent population treated at Memorial Sloan Kettering doesn’t reflect the diversity of people around the world. The cases used to train Watson therefore don’t take into account the economic and social issues faced by patients in poorer countries, noted Ossorio, the University of Wisconsin law professor.

“What it’s going to be learning is race, gender, and class bias,” she said. “We’re baking those social stratifications in, and we’re making the biases even less apparent and even less easy for people to recognize.”

Sometimes, the recommendations Watson gives diverge sharply from what doctors would say for reasons that have nothing to do with science, such as medical insurance. In a poster presented at the Global Breast Cancer Conference 2017 in South Korea, researchers reported that the treatment Watson most often recommended for breast cancer patients simply wasn’t covered by the national insurance system.

IBM said it has convened an international group of advisers to gather input on Watson’s performance. It also said that the system can be customized to reflect variations in treatment practices, differences in drug availability and financial considerations, and that the company recently introduced tools reduce the time and cost of adapting Watson.

In a response to STAT’s questions, Memorial Sloan Kettering said international journals are part of the literature it provides to Watson, including the Lancet, the European Journal of Cancer, Annals of Oncology, and the BMJ. “As we do in all areas of cancer research, we will continue to observe and study how Watson for Oncology impacts care internationally, follow the evidence, and work with IBM to optimize the system,” the hospital said.

Some hospitals abroad are customizing the system for their patients, adding information about local treatments. Nan Chen, who manages the Watson for Oncology program at Bumrungrad International Hospital in Thailand, said his oncologists use Japanese guidelines, not American guidelines, for treating gastric cancer.

But he said doctors can find this localization redundant or unnecessary: They are not that interested in being told the same guidance they just taught Watson.

“Our doctors say, this treatment is our own treatment, we know that,” Chen said. “You don’t need to turn around and put those treatments in Watson, and let Watson tell us what kind of treatment that we are using here in the hospital.”

Chen said this modified system is incredibly beneficial, however — to a hospital in the capital of Mongolia that employs zero oncology specialists.

At UB Songdo Hospital, of which Chen’s company is a majority owner, doctors are following Watson’s suggestions nearly 100 percent of the time. Patients who otherwise would have been treated by generalists with little, if any, cancer training are now benefiting from top-level expertise.

“That is the kind of thing that IBM is dreaming about,” Chen said.

In South Korea, Dr. Taewoo Kang, a surgical oncologist at Pusan National University Hospital who specializes in breast cancer, pointed to another important problem that Watson needs to solve. Right now, it provides supporting evidence for the recommendations it makes, but doesn’t actually explain how it came to recommend that particular treatment for that particular patient.

Kang said that, sometimes, he will ask Watson for advice on a patient whose cancer has not spread to the lymph nodes, and Watson will recommend a type of chemotherapy drug called a taxane. But, he said, that therapy is normally used only if the cancer has spread to the lymph nodes. And, to support the recommendation, Watson will show a study demonstrating the effectiveness of the taxane for patients whose cancer did spread to their lymph nodes.

Kang is left confused as to why Watson recommended a drug that he does not normally use for patients like the one in front of him. And Watson can’t tell him why.

F or all the concerns, some doctors around the world who use Watson insist that artificial intelligence will one day revolutionize health care. They say that clinicians are realizing concrete benefits — saving doctors valuable time searching for studies, better educating patients, and undercutting hierarchies in the clinic that might interfere with evidence-based treatment.

In Taiwan, Chiou said Watson immediately provides the “best data” from the literature about a treatment — survival rates, for example — relieving doctors of the task of searching the literature to compare each possible treatment.

Watson’s information also empowers patients, said Lee, the doctor who runs the Watson program at Gil Medical Center in South Korea. Previously, doctors verbally explained different treatment options to patients. Now, physicians can give patients a comprehensive packet prepared by Watson, which includes potential treatment plans along with relevant scientific articles. Patients can do their own research about these treatments, and maybe even disagree with the doctor about the right course of action.

“This is one of the most important and significant changes,” Lee said.

Watson also holds senior doctors accountable to the data. At Gil Medical Center, patients sit in a room with five doctors and Watson itself, the interface displayed on a flat-screen television in the so-called “Watson center.” Lee said that Watson’s presence has a huge influence on the doctors’ decision-making process, leveling the hierarchy that traditionally prioritized the opinion of the senior doctor over junior colleagues.

Watson gives the junior physicians quick and easy access to data that might prove their elders wrong, displaying on the screen information such as the survival rate right alongside a recommended treatment. It would be humiliating for senior doctors to continue to push for a different treatment in light of this evidence, Lee said.

At Manipal Hospitals in India, Dr. S.P. Somashekhar said that while there are some regional disparities in Watson’s recommendations for patients with rectal and breast cancer, those cases are outliers: For the vast majority of patients, the program matched the recommendations given to patients by the hospital’s tumor board — a group of 20 physicians that typically study their cases for a week and spend an hour discussing them.

That means that in a handful of seconds, Watson did what it takes 20 doctors over a week to accomplish. “That is so precious and very highly valuable,” Somashekhar said. “Our physicians cannot discuss every case. For every case we discuss in the tumor board, there are five cases which we cannot discuss.”

While those benefits are significant, they fall short of breakthrough discoveries that could predict or eradicate disease.

IBM executives said that doesn’t mean Watson can’t accomplish those feats. Norden, the former deputy health officer for Watson for Oncology and Genomics, said the goal is to ultimately bring together streams of clinical trial data and real-world patient data, so that Watson could begin to pinpoint the best treatments on its own.

“My own belief is that over time we will be better at measuring and reporting outcomes, and that data will be increasingly influential,” he said. “Where cancer care is today, I don’t think that any computing system is ready to be let out into the world without a measure of expert human oversight.”

T he bigger question for IBM is not whether health care will see a revolution in artificial intelligence but who will drive it.

One former IBM employee says the company could become a victim of its own marketing success — the unrealistic expectations it set are obscuring real accomplishments.

“IBM ought to quit trying to cure cancer,” said Peter Greulich, a former IBM brand manager who has written several books about IBM’s history and modern challenges. “They turned the marketing engine loose without controlling how to build and construct a product.”

Greulich said IBM needs to invest more money in Watson and hire more people to make it successful. In the 1960s, he said, IBM spent about 11.5 times its annual earnings to develop its mainframe computer, a line of business that still accounts for much of its profitability today.

If it were to make an equivalent investment in Watson, it would need to spend $137 billion. “The only thing it’s spent that much money on is stock buybacks,” Greulich said.

IBM said it created the market for artificial intelligence and is pleased with the pace of Watson’s growth, noting that it and other new business units grew by more than $20 billion in the past three years. “It took Facebook and Amazon more than 13 years to grow $20 billion,” the company said in a statement.

Since Watson’s “Jeopardy!” demonstration in 2011, hundreds of companies have begun developing health care products using artificial intelligence. These include countless startups, but IBM also faces stiff competition from industry titans such as Amazon, Microsoft, Google, and the Optum division of UnitedHealth Group.

Google’s DeepMind, for example, recently displayed its own game-playing prowess, using its AlphaGo program to defeat a world champion in Go, a 3,000-year-old Chinese board game.

DeepMind is working with hospitals in London, where it is learning to detect eye disease and speed up the process of targeting treatments for head and neck cancers, although it has run into privacy concerns .

Meanwhile, Amazon has launched a health care lab, where it is exploring opportunities to mine data from electronic health records and potentially build a virtual doctor’s assistant.

A recent report by the financial firm Jefferies said IBM is quickly losing ground to competitors. “IBM appears outgunned in the war for AI talent and will likely see increasing competition,” the firm concluded.

While not specific to Watson’s health care products, the report said potential clients are backing away from the system because of significant consulting costs associated with its implementation. It also noted that Amazon has 10 times the job listings of IBM, which recently didn’t renew a small number of contractors that worked for the company following its acquisition of Truven, a company it bought for $2.6 billion last year to gain access to 100 million patient records.

In its statement, IBM said that the workers’ contracts ended and that it is continuing to hire aggressively in the Cambridge, Mass.-based Watson Health and other units, with more than 5,000 positions open in the U.S.

But the outlook for Watson for Oncology is challenging, say those who have worked closest with it. Kris, the lead trainer at Memorial Sloan Kettering, said the system has the potential to improve care and ensure more patients get expert treatment. But like a medical student, Watson is just learning to perform in the real world.

“Nobody wants to hear this,” Kris said. “All they want to hear is that Watson is the answer. And it always has the right answer, and you get it right away, and it will be cheaper. But like anything else, it’s kind of human.”

About the Authors

National Technology Correspondent

Casey Ross covers the use of artificial intelligence in medicine and its underlying questions of safety, fairness, and privacy.

Ike Swetlitz

Washington Correspondent

Ike is a Washington correspondent, reporting at the intersection of life science and national politics.

health tech

STAT encourages you to share your voice. We welcome your commentary, criticism, and expertise on our subscriber-only platform, STAT+ Connect

To submit a correction request, please visit our Contact Us page .

Recommended Stories

Hims takes a blow from Lilly’s vial news

STAT Plus: UnitedHealth pledged a hands-off approach after buying a Connecticut medical group. Then it upended how doctors practice

STAT Plus: Gene therapy pioneer Jim Wilson believes he can commercialize rare-disease drugs in a tough climate

STAT Plus: AI drug firm Recursion seeks to move from survival to industry domination

For IEEE Members

Ieee spectrum, follow ieee spectrum, support ieee spectrum, enjoy more free content and benefits by creating an account, saving articles to read later requires an ieee spectrum account, the institute content is only available for members, downloading full pdf issues is exclusive for ieee members, downloading this e-book is exclusive for ieee members, access to spectrum 's digital edition is exclusive for ieee members, following topics is a feature exclusive for ieee members, adding your response to an article requires an ieee spectrum account, create an account to access more content and features on ieee spectrum , including the ability to save articles to read later, download spectrum collections, and participate in conversations with readers and editors. for more exclusive content and features, consider joining ieee ., join the world’s largest professional organization devoted to engineering and applied sciences and get access to all of spectrum’s articles, archives, pdf downloads, and other benefits. learn more about ieee →, join the world’s largest professional organization devoted to engineering and applied sciences and get access to this e-book plus all of ieee spectrum’s articles, archives, pdf downloads, and other benefits. learn more about ieee →, access thousands of articles — completely free, create an account and get exclusive content and features: save articles, download collections, and talk to tech insiders — all free for full access and benefits, join ieee as a paying member., ibm watson makes a treatment plan for brain-cancer patient in 10 minutes; doctors take 160 hours, the ai doctor analyzed a patient's full genome, then suggested drugs and clinical trials.

Eliza Strickland is a Senior Editor at IEEE Spectrum covering AI and biomedical engineering.

In treating brain cancer, time is of the essence.

A new study , in which IBM Watson took just 10 minutes to analyze a brain-cancer patient’s genome and suggest a treatment plan, demonstrates the potential of artificially intelligent medicine to improve patient care. But although human experts took 160 hours to make a comparable plan, the study’s results weren’t a total victory of machine over humans.

The patient in question was a 76-year-old man who went to his doctor complaining of a headache and difficulty walking. A brain scan revealed a nasty glioblastoma tumor, which surgeons quickly operated on; the man then got three weeks of radiation therapy and started on a long course of chemotherapy. Despite the best care, he was dead within a year. While both Watson and the doctors analyzed the patient’s genome to suggest a treatment plan, by the time tissue samples from his surgery had been sequenced the patient had declined too far.

IBM has been outfitting Watson , its “cognitive computing” platform, to tackle multiple challenges in health care , including an effort to speed up drug discovery and several ways to help doctors with patient care. In this study, a collaboration with the New York Genome Center (NYGC), researchers employed a beta version of IBM Watson for Genomics .

IBM Watson’s key feature is its natural-language-processing abilities. This means Watson for Genomics can go through the 23 million journal articles currently in the medical literature, government listings of clinical trials, and other existing data sources without requiring someone to reformat the information and make it digestible. Other Watson initiatives have also given the system access to patients’ electronic health records, but those records weren’t included in this study.

Laxmi Parida , who leads the Watson for Genomics science team, explains that most cancer patients don’t have their entire genome (consisting of 3 billion units of DNA) scanned for mutations. Instead they typically do a “panel” test that looks only at a subset of genes that are known to play a role in cancer.

The new study , published in the journal Neurology Genetics , used the 76-year-old man’s case to answer two questions. First, the researchers wanted to know if scanning a patient’s whole genome, which is more expensive and time consuming than running a panel, provides information that is truly useful to doctors devising a treatment plan. “We were trying to answer the question, Is more really more?” says Parida. 

The answer to that question was a resounding yes. Both the NYGC clinicians and Watson identified mutations in genes that weren’t checked in the panel test but which nonetheless suggested potentially beneficial drugs and clinical trials. 

Secondly, the researchers wanted to compare the genomic analysis performed by IBM Watson to one done by NYGC’s team of medical experts, which included the treating oncologist, a neuro-oncologist, and bioinformaticians.

Both Watson and the expert team received the patient’s genome information and identified genes that showed mutations, went through the medical literature to see if those mutations had figured in other cancer cases, looked for reports of successful treatment with drugs, and checked for clinical trials that the patient might be eligible for. It took the humans “160 person hours” to formulate recommendations, while Watson got there in 10 minutes. 

However, while Watson’s solution was first, it might not have been best. The NYGC clinicians identified mutations in two genes that, when considered together, led the doctors to recommend the patient be enrolled in a clinical trial that targeted both with a combinatorial drug therapy. If the patient had still been healthy enough, he would have been enrolled in this trial as his best chance of survival. But Watson didn’t synthesize the information together this way, and therefore didn’t recommend that clinical trial. 

While it’s tempting to view the study as a competition between human and artificial intelligence , Robert Darnell , director of the NYGC and a lead researcher on the study, says he doesn’t see it that way. “NYGC provided clinical input from oncologists and biologists,” he writes in an email. “Watson provided annotation that made the analysis faster. Given that each team addressed different issues, this comparison is apples to oranges.”

IBM’s Parida notes that the cost of sequencing an entire genome has plummeted in recent years , opening up the possibility that whole-genome sequencing will soon be a routine part of cancer care. If IBM Watson, or AI systems like it, are given swift access to this data, there’s a chance they could provide treatment recommendations in time to save the lives of people like the brain-cancer patient in this study.

Darnell says he hopes IBM Watson will become a routine part of cancer care because the amount of data that clinicians are dealing with is already overwhelming. “In my view, having doctors cope with the avalanche of data that is here today, and will get bigger tomorrow, is not a viable option,” he says. “Time is a key variable for patients, and machine learning and natural-language-processing tools offer the possibility of adding something qualitatively different than what is currently available.”

This study was part of a collaboration between IBM and the NYGC announced in 2014 , which set out to study the genomics of a few dozen brain-cancer patients. Darnell says the team is now working on a paper about the outcomes for 30 patients enrolled as part of that larger study. 

It’s worth noting that not everyone is sold on the value of IBM Watson for health care: A recent Wall Street analyst report declared that the Watson effort is unlikely to pay off for shareholders. Even though it called Watson “one of the more mature cognitive computing platforms available today,” the report argued that Watson’s potential customers will balk at the cost and complications of integrating the AI into their existing systems. 

The report also called attention to a fiasco at the MD Anderson Cancer Center in Texas, in which an IBM Watson product for oncology was shelved —after the hospital had spent US $62 million on it. 

Eliza Strickland is a senior editor at IEEE Spectrum , where she covers AI, biomedical engineering, and other topics. She holds a master’s degree in journalism from Columbia University.

AI Inference Competition Heats Up

Ransomware-as-a-service is changing extortion efforts, essential skills for it professionals in the ai era, related stories, these technologists are trying to make covid-19 risk assessment more of a science, what role will at-home covid-19 tests play in an increasingly vaccinated world, quantum computing makes inroads towards pharma.

• Harvard Business School →
• Faculty & Research →
• August 2021
• Teaching Note
• HBS Case Collection

IBM Watson at MD Anderson Cancer Center

• Format: Print
• | Language: English
• | Pages: 10

About The Author

Shane M. Greenstein

More from the authors.

• Faculty Research

Navigating Software Vulnerabilities: Eighteen Years of Evidence from Medium and Large U.S. Organizations

• February 2024

Chime Solutions

Open devices and slices: evidence from wi-fi equipment.

• Navigating Software Vulnerabilities: Eighteen Years of Evidence from Medium and Large U.S. Organizations  By: Raviv Murciano-Goroff, Ran Zhuo and Shane Greenstein
• Chime Solutions  By: Shai Bernstein, William R. Kerr, Christopher Stanton, Raymond Kluender and Mel Martin
• Open Devices and Slices: Evidence from Wi-Fi Equipment  By: Do Yoon Kim, Roberto Fontana and Shane Greenstein

• Newsletters
• Account Activating this button will toggle the display of additional content Account Sign out

How IBM’s Watson Went From the Future of Health Care to Sold Off for Parts

Most likely, you’re familiar with Watson from the IBM computer system’s appearance on Jeopardy! in 2011, when it beat former champions Ken Jennings and Brad Rudder. Watson’s time on Jeopardy! was fun viewing, but it was also a very savvy public debut of a product that IBM wanted to sell: Watson Health.

Watson Health was supposed to change health care in a lot of important ways, by providing insight to oncologists about care for cancer patients, delivering insight to pharmaceutical companies about drug development, helping to match patients with clinical trials, and more. It sounded revolutionary, but it never really worked. Recently, Watson Health was, essentially, sold for parts: Francisco Partners, a private equity firm, bought some of Watson’s data and analytics products for what Bloomberg News said was more than $1 billion.

On Friday’s episode of What Next: TBD, I spoke with Casey Ross , technology correspondent for Stat News, who has been covering Watson Health for years, about how Watson went from being the future of health care to being sold for scraps. Our conversation has been edited and condensed for clarity.

Lizzie O’Leary: I look at the amount of money that went into pulling this together. Acquisition after acquisition. It was billions of dollars, and it sold for a billion in the end. Is there any way to read that as anything but a failure?

Casey Ross: Financially, certainly not. They spent way more money building this than they got back. Just the acquisitions alone cost them $5 billion. That it was sold so many years later, after so much in effort—7,000 employees at one point—means that this will as a total failure that they needed to just cut their losses and move on.

Why did IBM want to get into the health data business? What problem did they think Watson would help solve?

There’s a tremendous amount of information that is collected every day on the care of hundreds of millions of people. However, there is currently no way to connect that information, to link it to an individual across all the domains in which they get care, and then to develop a holistic picture of who they are, of what their diseases are, of what the best treatments are, and how to ensure that they get the best care at the lowest possible cost. There is no connectivity right now that can do that at scale. The people in the technology sector look at it and say, “This has to be fixed, and we’re going to fix it.”

Google, Microsoft, a lot of very big companies are extremely interested in health care. What is so attractive for these big tech companies about health care?

It’s one of the biggest parts of our economy. It’s a three trillion business that has legacy technology infrastructure that should be embarrassing. Tech companies are drawn to audacious challenges like this, and ones where they can make—if they’re successful—a ton of money.

That’s how things are today, but the same problems have been around since the advent of digitized data. In 2012, IBM closed a deal with Memorial Sloan Kettering, one of the preeminent cancer centers in the country, to train an AI to make treatment recommendations. What was the goal? What were they trying to do?

They were really trying to democratize the expertise of Memorial Sloan Kettering’s oncologists, to make that expertise available to patients all over the world and to develop this standardized engine for providing optimal treatment recommendations, customized to a patient, in front of a doctor, thousands of miles away. It was a beautiful notion. They were trying to say, “Well, let’s make it more objective. Let’s look at all of the data, and let’s tell every physician, for this patient in front of you, this is how they should be treated.”

So you get your biopsy results, and things don’t look good, but you’re not just getting the expertise or the biases of your particular oncologist. You’re getting the wealth of thousands of oncologists distilled into an algorithm?

Yes, you are getting all of that data, across so many different physicians, crunched down into a very digestible format and recommendation that could then lead to the best treatment for that patient.

Reading your reporting, it sounds like this was incredibly important to IBM. In 2015, Ginni Rometty, who was the CEO at the time, went on Charlie Rose . She said health care was “our moonshot.” How much of IBM’s hopes were hung on this thing?

The company made a huge bet that this could be the bridge to a different kind of future for IBM, which at the time was several years of quarterly revenue declines. They were trying to use Watson as a bridge to a different future where IBM wasn’t this old guard hardware company that everybody knew so well, but was operating on the cutting edge of artificial intelligence. Health care was the biggest, the buzziest use case. This was where they were going to really show the surpassing value of their technology.

To do that, IBM needed massive amounts of data on which to train Watson. It got that data through acquisitions, eventually spending some $5 billion buying a series of health data companies. What were those companies?

Truven, Phytel, Explorys and merge. Truven had the biggest insurance database in the nation with 300 million covered lives, Explorys provided a clinical data set of actual electronic health records kept by health systems representing about 50 million or so patients, Phytel added on top of that, and Merge had a huge imaging database. They had all this data and the idea was: Expose Watson to that, and it finds patterns that physicians and anyone else can’t possibly find when looking at that data, given all the variables in it.

Except that was not the reality. One of IBM’s high-profile partnerships with MD Anderson Cancer Center in Texas fell apart. A doctor involved said that there wasn’t enough data for the program to make good recommendations, and that Watson had trouble with the complexity of patient files. The partnership was later audited and shelved. What went wrong?

If you think about it, knowing what we know now or what we’ve learned through this, the notion that you’re going to take an artificial intelligence tool, expose it to data on patients who were cared for on the upper east side of Manhattan, and then use that information and the insights derived from it to treat patients in China, is ridiculous. You need to have representative data. The data from New York is just not going to generalize to different kinds of patients all the way across the world.

What was happening in a clinical setting? What was happening to patients?

Our window through the reporting was talking to physicians. We got concerns from them that the recommendations that it was giving were just not relevant. Maybe it would suggest a particular kind of treatment that wasn’t available in the locality in which it was making the recommendation, or the recommendation did not at all square with the treatment protocols that were in use at the local institution or, and more commonly so, especially in the U.S. and Europe, “you’re not telling me anything I don’t already know.” That was the big credibility gap for physicians. It was like, “Well duh. Yeah, I know that that’s the chemotherapy I should pursue. I know that this treatment follows that one.”

You got a hold of an internal IBM presentation from 2017 where a doctor at a hospital in Florida told the company this product was a piece of shit .

Seeing that written down in an internal document, which was circulated among IBM executives, was a shocking thing to see. It really underscored the extent of the gap between what IBM was saying in public and what was happening behind the scenes.

There were a lot of internal discussions, even a presentation, that indicated that the technology was not as far along as they’d hoped, that it wasn’t able to accomplish what they set out to accomplish in cancer care. There were probably a lot of people that believed, that truly did believe, that they would get there or that it was closer than maybe some people realized. I think the marketing got way ahead of the capabilities.

It’s very hard to listen to you and not think about Theranos , even though this is not a one-to-one parallel in any way. When you are trying to move by leaps and bounds with technology in the health care sector, it feels like a reminder that all things are not created equal, that making big leaps with people’s health is a much riskier proposition.

That underscores the central theme of this story: When you try to combine the bravado of the tech culture and the notion that you can achieve these huge audacious goals in a domain where you’re dealing with people’s lives and health and the most sacrosanct aspects of their existence and their bodies, you need to have evidence to back up that you can do what you say you can do.

Why did they continue on trying to rescue this product that they seemed to know internally was failing?

I think they had so much invested in it that it really was, for them, too big to fail. It had 7,000 employees. They’d invested so much time and energy on marketing in the success of the product that they really needed it to succeed.

Instead, they got a fail. But Watson’s fate certainly doesn’t mean that AI in health care is going away. Just recently, Microsoft and a large group of hospitals announced a coalition to develop AI solutions in health care. If you had to pin down a moral to the story, is it that AI in health care isn’t ready for prime time, or that IBM did it wrong?

I think it’s both of those. This will be a case study for business schools for decades. When you look at what IBM did and the strategy mistakes, the tactical errors that they made in pursuing this product, they made a lot of unforced errors here. It’s also true that the generation of technology that they had was nowhere near ready to accomplish the things that they set out to accomplish and promised that they could accomplish. I don’t think that the failure of Watson means that artificial intelligence isn’t ready to make significant improvements and changes in health care. I think it means the way that they approached it is a cautionary tale that lays out how not to do it.

Does the failure of Watson Health make you worry that it’s going to shut down other avenues for innovation? Will such a spectacular belly flop impede progress?

I don’t think so. There were so many mistakes that were made, that were learned from, that, if anything, it will facilitate faster learning and better decision making by other parties that are now poised to disrupt health care and make the progress that IBM failed to achieve. There’s a saying that pioneers often end up with arrows in their backs, and that’s what happened here. They’re an example, a spectacular example, of wrongheaded decision making and missteps that didn’t have to happen. By learning from that, I think advancement and progress and true benefits will be faster coming.

Future Tense is a partnership of Slate , New America , and Arizona State University that examines emerging technologies, public policy, and society.

Artificial Intelligence

Ibm's doctor watson: 10 years later, would the jeopardy champion revolutionize healthcare.

Posted November 25, 2021 | Reviewed by Lybi Ma

• Following IBM Watson's Jeopardy! victory, many regarded it as the future of medicine.
• Watson's Jeopardy! victory was not so clear cut as it appeared.
• Its introduction at MD Anderson Cancer Center was deemed a failure.
• Before we can predict what artificial intelligence might contribute, we need to understand what it is.

The year 2021 heralds a major 10-year anniversary for artificial intelligence that should not pass without acknowledgment. In 2011, IBM’s question-answering computer Watson roundly defeated the television game show Jeopardy! champions Ken Jennings and Brad Rutter with a three-day total of $77,000, as compared to Jennings’ $24,000 and Rutter’s $22,000, winning the first-place prize of $1 million.

Just a day after Watson’s Jeopardy! victory was televised, IBM began a marketing campaign to leverage its success. Within a few years, the company was taking a victory lap for Watson’s foray into cancer care, touting it as “a revolutionary approach to medicine and health care that is likely to have significant social, economic, and political consequences."

Champion Ken Jennings included in one of his Final Jeopardy! responses this assessment: “I, for one, welcome our new computer overlords.” Later, in an article in Slate, Jennings wrote:

Just as factory jobs were eliminated in the 20th century by new assembly-line robots, Brad and I were the first knowledge-industry workers put out of work by the new generation of "thinking" machines. "Quiz show contestant" may be the first job made redundant by Watson, but I'm sure it won't be the last.

In the intervening years, it has become clear that IBM overestimated the speed with which Watson would conquer the world. Perhaps Watson’s most prominent failure occurred in a 2013 partnership with MD Anderson Cancer Center, originally touted by the partners as leveraging the ability of cognitive systems to “‘understand’ the context of users’ questions, uncover answers from Big Data, and improve performance by continuously learning from experiences.”

But by 2017, MD Anderson “benched” Watson in what was heralded as a setback for artificial intelligence in medicine. Watson could not ingest medical information from patient records or the medical literature the way a physician would. It had difficulty comparing each new patient with the many patients who had come before. A “scathing” report by auditors at the University of Texas stated that the project had not achieved its goals and had cost approximately $62 million.

In fact, even Watson’s triumph in Jeopardy! is not as clear-cut as it seems. For one thing, Watson’s victory may have had more to do with its ability to “buzz in” to answer questions than its superior ability to answer questions. In many cases, contestants Jennings and Rutter also knew the answers but were “blocked out” by Watson’s superior timing.

Moreover, Watson was not perfect. On day two of the three-day contest, the Final Jeopardy category was U.S. Cities, and the answer was, “Its largest airport is named after a World War II hero and its second-largest airport is named after a WWII battle.” Watson answered, “What is Toronto?” That Watson could mistake Toronto for a U.S. city serves as a powerful reminder that even AI can make mistakes. A mistake in Jeopardy! is not a matter of life and death, but in medicine, it could be.

Some might argue that we have not given Watson sufficient time. Perhaps forecasts were excessively rosy, but with additional opportunities to work out the kinks, it will inevitably triumph in the end. Or perhaps there is something wrong with healthcare—namely, we simply don’t possess a clear enough understanding of patients, diseases, and physicians.

Yet other explanations are also possible. One would be that inputting and analyzing massive quantities of data is simply not enough to advance our understanding of patient care. Perhaps good patient care is not primarily a data-driven endeavor.

We should also think twice before we assume that real knowledge is an output solely attributable to following algorithms. If we liken a computer to a net, the mere fact that a net of a certain type catches some things and misses others in no way supports the conclusion that the only real or important things are the ones we find in the net.

It is quite possible that what physicians and other health professionals know and can do is not replicable in digital terms. A digital reproduction of a great painting may resemble such a painting a great deal, just as a digital audio recording may be nearly indistinguishable from a musical performance, but in the end, they are not the same.

The ultimate challenge in improving patient care is not to make patients, diseases, and physicians fit our models, but to make our models fit patients, diseases, and physicians. For one thing, we cannot refashion the world to fit our tools, and even more importantly, the world itself is a far more complex, rich, and beautiful reality than any of our models can capture.

Richard Gunderman, MD, Ph.D. , is Chancellor's Professor of Radiology, Pediatrics, Medical Education, Philosophy, Liberal Arts, Philanthropy, and Medical Humanities and Health Studies at Indiana University.

• Find a Therapist
• Find a Treatment Center
• Find a Psychiatrist
• Find a Support Group
• Find Online Therapy
• United States
• Brooklyn, NY
• Chicago, IL
• Houston, TX
• Los Angeles, CA
• New York, NY
• Portland, OR
• San Diego, CA
• San Francisco, CA
• Seattle, WA
• Washington, DC
• Asperger's
• Bipolar Disorder
• Chronic Pain
• Eating Disorders
• Passive Aggression
• Personality
• Goal Setting
• Positive Psychology
• Stopping Smoking
• Low Sexual Desire
• Relationships
• Child Development
• Self Tests NEW
• Therapy Center
• Diagnosis Dictionary
• Types of Therapy

Sticking up for yourself is no easy task. But there are concrete skills you can use to hone your assertiveness and advocate for yourself.

• Emotional Intelligence
• Gaslighting
• Affective Forecasting
• Neuroscience

IBM Watson Joins MD Anderson in Cancer Fight

A union of technology and human innovation made the first moon landing possible. In that same spirit, The University of Texas MD Anderson Cancer Center is leveraging technology to change the very foundation of cancer care.

MD Anderson’s Oncology Expert Advisor (OEA) powered by IBM Watson is the result of a year-long partnership between MD Anderson and IBM to develop a cognitive clinical decision support tool that can aggregate vast quantities of medical data and deliver relevant information for a particular patient, so that treating oncologists can make evidence-informed decisions for more effective care.

What Watson brings to this partnership is a powerful combination of programmatic computing, natural language processing, hypothesis generation and dynamic learning based on question-answer. It was this impressive computational power that earned Watson first place in the game show Jeopardy! in 2011. In the context of cancer care and research, the OEA is being trained to ingest complex structured and unstructured information from a variety of real-world sources —patient records, physician notes, laboratory results—and weigh these patient attributes against its ever expanding corpus of medical knowledge, oncology literature and treatment guidelines to propose appropriate evidence-based treatment options based on each patient’s unique disease profile.

The Oncology Expert Advisor also offers care pathway advisories, such as alerting a treating physician to a trend toward an adverse event so he or she can proactively manage a patient to minimize toxicity. In this way, OEA serves as a knowledge equalizer to provide practicing oncologists access to the most relevant and up-to-date medical information for decision making.

The Oncology Expert Advisor is only the beginning of what Lynda Chin, M.D., chair of Genomic Medicine and scientific director of the Institute for Applied Cancer Science at MD Anderson Cancer Center, describes as “rebuilding the ecosystem of cancer care.” The problem, as assessed by the Institute of Medicine (IOM), is that “cancer care is often not as patient- centered, accessible, coordinated or evidence-based as it could be.” Chin hopes that her team can harness big data, cognitive computing capabilities and mobile technologies to create a value-based, patient-centered oncology care ecosystem. In this new ecosystem, OEA is the tip of the spear that broadly drives the democratization of a superior standard of care.

“Democratization means we disseminate and share the knowledge and expertise so that patients who are not able to be treated by experts at a specialty center like MD Anderson can still access high quality care informed by the latest medical knowledge. OEA is akin to a knowledge equalizer that can enable practicing oncologists to make better care decisions based on a leveled knowledge base,” said Chin.

With a tool like OEA, Chin imagines that the latest and best care options practiced at a specialty center can be democratized, so that more patients in community settings receive compara- ble care. She hopes that access to expert cancer care will not be limited to the few tertiary care centers where cancer experts work. A majority of the newly diagnosed cancer patients are treated in community setting, explained Chin. “In most cases, these patients end up at tertiary care centers like MD Anderson only after they have failed multiple therapy options, at which point, the chance of a cure is very low. To achieve better patient outcomes, we have to be able to ensure that patients will receive a similarly superior standard of care at primary and secondary care settings, where more patients are at earlier stages of their diseases and their chances for a cure are much greater.”

The project seemed like a natural fit for IBM and MD Anderson to work on collaboratively, explained Steve Gold, vice president for the IBM Watson Group.

“This whole idea of an even broader vision of the eradication of cancer is truly inspirational. And in a very paral- leled way, IBM was on mission to define a whole new generation of computing,” said Gold. “The Watson capabilities provided an opportunity to really advance the way in which information could be explored and harnessed. It was that ability to pull together two really big, bold visions—and redefine the possibilities—that was the foundation for the relationship.”

“We began working with Lynda and her team at MD Anderson to explore ways in which Watson could be applied and how it could be used to reshape the way in which both research would be conducted as well as the way medicine could be practiced in a future patient-centered health care system, as she envisioned,” he added. “And that was one of the first ‘Aha!’ moments certainly, for IBM in the process, as we began to better appreciate the way health care operates. You realize this huge potential to leverage the insight and discoveries that were happening every day on the research side of the house, and the clinical side of the house, with really what was happening in clini- cal practice, at the patient level.”

In addition to the Oncology Expert Advisor, Chin is exploring other innovative technologies to leverage the power of information to revolution- ize cancer care. In particular, Chin is launching another initiative to create a secure and interactive mobile health platform to establish continuity and connectivity between patients and care providers. Such a mobile platform will enable remote and frequent monitoring of patient status, to improve the quality of care, deliver targeted health education, and facilitate personalized detection and prevention strategies to manage health and prevent disease.

As Chin sees it, technology holds tremendous potential to transform medicine. “It is unrealistic to expect any human mind to read, understand and retain the overwhelming amount of medical literature, much less to assimilate such in real-time to make evidence-based care decisions consis- tently for each patient. That is where technology comes in,” she said. “The question is not whether we should deliver the best possible care based on the latest and best medical evidence to all patients. The question is how. It is not good enough to mandate. We must enable.”

Recommended News

Five questions during COVID-19 with Doug Lawson, Ph.D., of St. Luke’s Health

Gilead sets price for COVID-19 drug remdesivir

Telemedicine has arrived, thanks to COVID-19

Product details

Time period

Geographical setting, featured company.

What Ever Happened to IBM’s Watson?

IBM’s artificial intelligence was supposed to transform industries and generate riches for the company. Neither has panned out. Now, IBM has settled on a humbler vision for Watson.

Credit... Video by Maria Chimishkyan

Supported by

• Share full article

By Steve Lohr

• Published July 16, 2021 Updated July 17, 2021

A decade ago, IBM’s public confidence was unmistakable. Its Watson supercomputer had just trounced Ken Jennings , the best human “Jeopardy!” player ever, showcasing the power of artificial intelligence. This was only the beginning of a technological revolution about to sweep through society, the company pledged.

“Already,” IBM declared in an advertisement the day after the Watson victory, “we are exploring ways to apply Watson skills to the rich, varied language of health care, finance, law and academia.”

But inside the company, the star scientist behind Watson had a warning: Beware what you promise.

David Ferrucci, the scientist, explained that Watson was engineered to identify word patterns and predict correct answers for the trivia game. It was not an all-purpose answer box ready to take on the commercial world, he said. It might well fail a second-grade reading comprehension test.

His explanation got a polite hearing from business colleagues, but little more.

“It wasn’t the marketing message,” recalled Mr. Ferrucci, who left IBM the following year.

It was, however, a prescient message.

IBM poured many millions of dollars in the next few years into promoting Watson as a benevolent digital assistant that would help hospitals and farms as well as offices and factories. The potential uses, IBM suggested, were boundless, from spotting new market opportunities to tackling cancer and climate change. An IBM report called it “the future of knowing.”

IBM’s television ads included playful chats Watson had with Serena Williams and Bob Dylan. Watson was featured on “60 Minutes.” For many people, Watson became synonymous with A.I.

We are having trouble retrieving the article content.

Please enable JavaScript in your browser settings.

Thank you for your patience while we verify access. If you are in Reader mode please exit and  log into  your Times account, or  subscribe  for all of The Times.

Thank you for your patience while we verify access.

Already a subscriber?  Log in .

Want all of The Times?  Subscribe .

Advertisement

Science-Based Medicine

Exploring issues and controversies in the relationship between science and medicine

IBM’s Watson versus cancer: Hype meets reality

Five years ago, IBM announced that its supercomputer Watson would revolutionize cancer treatment by using its artificial intelligence to digest and distill the thousands of oncology studies published every year plus patient-level data and expert recommendations into treatment recommendation. Last week, a report published by STAT News shows that, years later, IBM’s hubris and hype have crashed into reality.

Watson defeats two human champions. Unfortunately, it doesn’t seem to be doing as well for cancer.

For nearly as long as I can remember, I’ve been a fan of Jeopardy! Indeed, if I’m at home at 7:30 PM on a weeknight, Jeopardy! will usually be on the television. Given that, I remember what was basically a bit of stunt programming in 2011, when Jeopardy! producers had IBM’s artificial intelligence supercomputer Watson face off against two of the most winning champions in the history of the show, Ken Jennings and Brad Rutter. Watson won , leading Jenning’s to add to his Final Jeopardy answer, “I, for one, welcome our new computer overlords.”

Watson’s next challenge was similarly highly hyped: cancer. Since 2012, IBM has been collaborating with several cancer institutes to apply Watson’s talents to cancer treatment. For instance, Memorial Sloan-Kettering Cancer Center describes its Watson Oncology initiative thusly:

Watson Oncology is a cognitive computing system designed to support the broader oncology community of physicians as they consider treatment options with their patients. Memorial Sloan Kettering clinicians and analysts are partnering with IBM to train Watson Oncology to interpret cancer patients’ clinical information and identify individualized, evidence-based treatment options that leverage our specialists’ decades of experience and research. As Watson Oncology’s teacher, we are advancing our mission by creating a powerful resource that will help inform treatment decisions for those who may not have access to a specialty center like MSK. With Watson Oncology, we believe we can decrease the amount of time it takes for the latest research and evidence to influence clinical practice across the broader oncology community, help physicians synthesize available information, and improve patient care.

Not surprisingly, Watson’s entry into cancer care and interpretation of cancer genomics was, just like its appearance on Jeopardy! , highly hyped, with overwhelmingly positive press coverage and little in the way of skeptical examination of what, exactly, Watson could potentially do and whether it could actually improve patient outcomes. Overall, as Watson moved into the clinical realm, you’d be hard-pressed not to think that this was a momentous development that would change cancer care forever for the better. There were plenty of headlines like “ IBM to team up with UNC, Duke hospitals to fight cancer with big data ” and “ The future of health care could be elementary with Watson .” The future looked bright.

An article in STAT News last week by Casey Ross and Ike Swetlitz suggests otherwise, at least so far: “ IBM pitched its Watson supercomputer as a revolution in cancer care. It’s nowhere close .”

Watson: Hype versus reality

In the story, STAT looked at Watson for Oncology’s use, marketing, and actual performance in hospitals around the world, interviewing dozens of doctors, IBM executives, and artificial intelligence experts and concluded that IBM released a product without having fully assessed or understood the challenges in deploying it and without having published any papers demonstrating that the technology works as advertised, noting that, as a result, “its flaws are getting exposed on the front lines of care by doctors and researchers who say that the system, while promising in some respects, remains undeveloped.” From my perspective, that’s an understatement. Indeed, STAT observes:

Perhaps the most stunning overreach is in the company’s claim that Watson for Oncology, through artificial intelligence, can sift through reams of data to generate new insights and identify, as an IBM sales rep put it, “even new approaches” to cancer care. STAT found that the system doesn’t create new knowledge and is artificially intelligent only in the most rudimentary sense of the term. While Watson became a household name by winning the TV game show “Jeopardy!”, its programming is akin to a different game-playing machine: the Mechanical Turk, a chess-playing robot of the 1700s, which dazzled audiences but hid a secret — a human operator shielded inside. In the case of Watson for Oncology, those human operators are a couple dozen physicians at a single, though highly respected, U.S. hospital: Memorial Sloan Kettering Cancer Center in New York. Doctors there are empowered to input their own recommendations into Watson, even when the evidence supporting those recommendations is thin.

Another way of saying this is that Watson isn’t really an artificial intelligence when it comes to cancer, but rather a very powerful computer that is very good at coming up with treatment plans based on human-inputted algorithms that it’s taught. An example from a hospital in Florida is presented as an example:

On a recent morning, the results for a 73-year-old lung cancer patient were underwhelming: Watson recommended a chemotherapy regimen the oncologists had already flagged. “It’s fine,” Dr. Sujal Shah, a medical oncologist, said of Watson’s treatment suggestion while discussing the case with colleagues. He said later that the background information Watson provided, including medical journal articles, was helpful, giving him more confidence that using a specific chemotherapy was a sound idea. But the system did not directly help him make that decision, nor did it tell him anything he didn’t already know.

But it’s more than that. You might have noted in the MSKCC blurb I quoted above that MSKCC is described as “Watson’s teacher.” That is very literally true. Indeed, the STAT story refers to Watson as “essentially Memorial Sloan Kettering in a portable box,” noting that its treatment recommendations are “based entirely on the training provided by doctors, who determine what information Watson needs to devise its guidance as well as what those recommendations should be.” This reliance on a single institution introduces an incredible bias. MSKCC is, of course, one of the premiere cancer centers in the world, but it’s a tertiary care center. The patients seen there are not like the patients seen at most places—or, to some extent, even at my cancer center. They’re different, both in the mix of race and socioeconomic status. (MSKCC tends to attract more affluent patients.) Also, the usual differences between the patient mix in a tertiary care center and a typical hospital are more pronounced, because not only is MSKCC a tertiary care center, but it’s one of the premier cancer tertiary care centers in the world. There are more advanced and unusual cases, patients who have failed multiple lines of treatment and are looking for one last chance. The mix of patients, cancers, and other factors that doctors at MSKCC see might not be relevant to hospitals elsewhere in the world—or even in different parts of the US. As Pilar Ossorio, a professor of law and bioethics at University of Wisconsin Law School, points out in the article, from the cases used to train Watson, what Watson will learn is “race, gender, and class bias,” basically “baking those social stratifications in” and “making the biases even less apparent and even less easy for people to recognize.”

Bias is inevitable, particularly when it is only one institution’s physicians who are doing the teaching.

It’s also widely known in the oncology community that there is a “MSKCC way” of doing things that might not always agree with other centers. Yet IBM denies that reliance on a single institution to “teach” Watson injects bias, to the point where I literally laughed out loud (and was half tempted to insert an emoji indicating that) when I read a quote by Watson Health general manager Deborah DiSanzo, saying, “The bias is taken out by the sheer amount of data we have.” (She is referring to patient cases and millions of articles and studies fed into Watson.) I can’t help but also note that it isn’t just treatment guidelines that MSKCC is providing. It’s basically choosing all the medical literature whose results are inputted into Watson to help craft its recommendations. As I read the STAT article, as a clinician and scientist myself, I couldn’t help but marvel that IBM is either blissfully unaware that this is a self-reinforcing system, in which one institution’s doctors would tend to recommend the very literature that would support the treatment recommendations that they prefer.

And, MSKCC being MSKCC (i.e., a bit arrogant), the doctors “training” Watson don’t see the bias as a problem:

Doctors at Memorial Sloan Kettering acknowledged their influence on Watson. “We are not at all hesitant about inserting our bias, because I think our bias is based on the next best thing to prospective randomized trials, which is having a vast amount of experience,” said Dr. Andrew Seidman, one of the hospital’s lead trainers of Watson. “So it’s a very unapologetic bias.”

I laughed out loud at that quote, too. Having a “vast amount of experience” without having clinical trials upon which to base treatments can just as easily lead to continuing treatments that don’t work or hanging on to beliefs that are never challenged by evidence. I’m not saying that having experience is a bad thing. Far from it! However, if that experience is not tempered by humility, bad things can happen. It’s the lack of humility that I perceive here that troubles me. There are awesome cancer doctors elsewhere in the world, too, you know:

In Denmark, oncologists at one hospital said they have dropped the project altogether after finding that local doctors agreed with Watson in only about 33 percent of cases. “We had a discussion with [IBM] that they had a very limited view on the international literature, basically, putting too much stress on American studies, and too little stress on big, international, European, and other-part-of-the-world studies,” said Dr. Leif Jensen, who directs the center at Rigshospitalet in Copenhagen that contains the oncology department.

Sometimes, the recommendations Watson gives diverge sharply from what doctors would say for reasons that have nothing to do with science, such as medical insurance. In a poster presented at the Global Breast Cancer Conference 2017 in South Korea, researchers reported that the treatment Watson most often recommended for breast cancer patients simply wasn’t covered by the national insurance system.

None of this is surprising, given that Watson is trained by American doctors at one very prestigious American cancer center.

Then there’s a rather basic but fundamental problem with Watson, and that’s getting patient data entered into it. Hospitals wishing to use Watson must find a way either to interface their electronic health records with Watson or hire people to manually enter patient data into the system. Indeed, IBM representatives admitted that teaching a machine to read medical records is “a lot harder than anyone thought.” (Actually, this rather reminds me of Donald Trump saying, “Who knew health care could be so complicated?” in response to the difficulty Republicans had coming up with a replacement for the Affordable Care Act.) The answer: Basically anyone who knows anything about it. Anyone who’s ever tried to wrestle health care information out of a medical record, electronic or paper, into a form in a database that can be used to do retrospective or prospective studies knows how hard it is. Heck, just from my five year experience working on a statewide collaborative quality initiative in breast cancer, I know how hard it is, and what we were doing in our CQI was nowhere near as complex as what IBM is trying to do with Watson. For instance, we were looking at only one cancer (breast) and a subset of one state (25 institutions in Michigan), and we were not trying to derive new knowledge, but rather to look at aspects of care where the science and recommendations are clear and we could compare what our member institutions were doing to the best existing evidence-based guidelines.

What can Watson actually do?

IBM represents Watson as being able to look for patterns and derive treatment recommendations that human doctors might otherwise not be able to come up with because of our human shortcomings in reading and assessing the voluminous medical literature, but what Watson can actually do is really rather modest. That’s not to say it’s not valuable and won’t get better with time, but the problem is that it doesn’t come anywhere near the hype. I mentioned that there haven’t been any peer-reviewed studies on Watson in the medical literature yet, but that doesn’t mean there are no data yet. At the American Society of Clinical Oncology (ASCO) meeting this year, there were three abstracts presented reporting the results of studies using Watson in cancer care:

The first study carried out at the Manipal Comprehensive Cancer Centre in Bangalore, India, looked at Watson’s concordance with a multi-disciplinary tumour board used for lung, colon and rectal cancer cases. The AI achieved a concordance rate of 96% for lung, 81% for colon and 93% for rectal cancer. The second study compared Watson’s recommendations to those made by oncologists at Bumrungrad International Hospital in Bangkok, Thailand – this time across multiple cancer types. Its concordance rate was 83%. The third concordance study compared Watson’s decisions for high-risk colon cancer to a tumour board from Gachon University Gil Medical Centre in Incheon, South Korea. Its concordance rate in terms of colon cancer decisions was 73%, however, it was only 43% in gastric cancer. The company explained this was due to differences in treatment guidelines for the disease in South Korea, compared to where it was trained at Memorial Sloan Kettering.

This is mighty thin gruel after such grandiose claims for the technology. Sure, it’s a very good thing that Watson agrees with evidence-based guidelines a high percentage of the time. It’s not so great that its concordance with recommendations was so low for gastric cancer, but it is that lack of concordance that shows the weakness of a system so dominated by American oncologists and cancer surgeons. The reason that treatment recommendations in Asia differ so markedly from those in the US is because of differences in prevalence (which is much higher in Asia) and even biology .

Of course, it’s important that Watson be able to replicate evidence-based treatment recommendations for common cancers, but you don’t need a computer to do that, much less an AI. Where Watson was hyped by IBM was for its supposed ability to “think outside the box” (if you’ll excuse the term) and come up with recommendations that humans would not have thought of that would result in better outcomes for cancer patients. Even these modest results are being hyped in the form of embarrassing headlines. For instance, ASCO, touting the results of the three studies presented at its annual meeting and other results, wrote “ How Watson for Oncology Is Advancing Personalized Patient Care .” It read like a press release from IBM. Another article proclaimed that “ IBM’s Watson is really good at creating cancer treatment plans .” That’s nice. So are nearly all oncologists, and it’s even arguable that Watson is as good as a typical oncologist.

The M.D. Anderson experience

The M.D. Anderson Cancer Center was, along with MSKCC, one of the early adopters of Watson. Its experience with the project is another cautionary note that shows what can happen when not enough skepticism is applied to a project and how a project like Watson can turn into a massive boondoggle. This was revealed when the partnership between M.D. Anderson and IBM basically fell apart earlier this year :

According to a blistering audit by the University of Texas System, the cancer center grossly mismanaged its splashy program with IBM, which started back in 2012. The program aimed to teach Watson how to treat cancer patients and match them to clinical trials. Watson initially met goals and impressed center doctors, but the project hit the rocks as MD Anderson officials snubbed their own IT experts, mishandled about $62 million in funding, and failed to follow basic procedures for overseeing contracts and invoices, the audit concludes. IBM pulled support for the project back in September of last year. Watson is currently prohibited from being used on patients there, and the fate of MD Anderson’s partnership with IBM is in question. MD Anderson is now seeking bids from other contractors who might take IBM’s place.

As Matt Herper noted over at Forbes :

Usually, companies pay research centers to do research on their products; in this case, MD Anderson paid for the privilege, although it would have apparently also owned the product. This was a “very unusual business arrangement,” says Vinay Prasad, an oncologist at Oregon Health & Science University. According to the audit report, Chin went around normal procedures to pay for the expensive undertaking. The report notes “a consistent pattern of PwC fees set just below MD Anderson’s Board approval threshold,” and its appendix seems to indicate this may have occurred with payments to IBM, too.* She also didn’t get approval from the information technology department.

Yes, it was that bad .

Hype and hubris in AI: Beyond IBM

It’s very clear that AI will play an increasingly large role in medicine. The massive amount of genomic data being applied to “personalized medicine,” or, as it’s now more commonly called, “precision medicine,” basically demands it because no human can sift through the terabytes and petabytes of genomic data without assistance to find patterns that can be exploited in treatment. What I do have a problem with is hype, and IBM is clearly incredibly guilty of massively hyping its Watson product before it was ready for prime time, apparently not recognizing just how difficult it would be to train Watson to align company hype with scientific reality.

One way to think about it is to consider how machine learning works, how AI is trained to recognize patterns, come to conclusions, and make recommendations. In other words, how can a machine go beyond human-curated data and recommendations? It’s incredibly difficult:

To understand what’s slowing the progress, you have to understand how machine-learning systems like Watson are trained. Watson “learns” by continually rejiggering its internal processing routines in order to produce the highest possible percentage of correct answers on some set of problems, such as which radiological images reveal cancer. The correct answers have to be already known, so that the system can be told when it gets something right and when it gets something wrong. The more training problems the system can chew through, the better its hit rate gets. That’s relatively simple when it comes to training the system to identify malignancies in x-rays. But for potentially groundbreaking puzzles that go well beyond what humans already do, like detecting the relationships between gene variations and disease, Watson has a chicken-and-egg problem: how does it train on data that no experts have already sifted through and properly organized? “If you’re teaching a self-driving car, anyone can label a tree or a sign so the system can learn to recognize it,” says Thomas Fuchs, a computational pathologist at Memorial Sloan-Kettering, a cancer center in New York. “But in a specialized domain in medicine, you might need experts trained for decades to properly label the information you feed to the computer.”

That’s the bias introduced by relying on MSKCC physicians. It’s a bias that’s much worse than it needs to be because of how IBM relies on one institution and one relatively small group of physicians to train Watson, but, in fairness, it is an unavoidable bias at this stage in the development of an AI. The problem, as it all too often is, is arrogance. IBM appears to have vastly underestimated the challenge in moving beyond the training dataset (as it’s often called in studies like this), for which the answers are known in advance to the computer’s analysis, to the validation dataset (for which the answer is not known in advance).

None of this is to say that AI won’t eventually make a major contribution to the treatment of cancer and other diseases. Rather, it’s just to say that we’re nowhere near there yet. Moreover, IBM is no longer the only player in this game, as has been noted :

Since Watson’s “Jeopardy!” demonstration in 2011, hundreds of companies have begun developing health care products using artificial intelligence. These include countless startups, but IBM also faces stiff competition from industry titans such as Amazon, Microsoft, Google, and the Optum division of UnitedHealth Group. Google’s DeepMind, for example, recently displayed its own game-playing prowess, using its AlphaGo program to defeat a world champion in Go, a 3,000-year-old Chinese board game. DeepMind is working with hospitals in London, where it is learning to detect eye disease and speed up the process of targeting treatments for head and neck cancers, although it has run into privacy concerns. Meanwhile, Amazon has launched a health care lab, where it is exploring opportunities to mine data from electronic health records and potentially build a virtual doctor’s assistant. A recent report by the financial firm Jefferies said IBM is quickly losing ground to competitors. “IBM appears outgunned in the war for AI talent and will likely see increasing competition,” the firm concluded.

But the “cognitive computing” technologies under the Watson umbrella aren’t as unique as they once were. “In the data-science community the sense is that whatever Watson can do, you can probably get as freeware somewhere, or possibly build yourself with your own knowledge,” Claudia Perlich told Gizmodo, a professor and data scientist who worked at IBM Watson Research Center from 2004 to 2010 (at the same time Watson was being built), before becoming the chief scientist at Dstillery, a data-driven marketing firm (a field that IBM is also involved with). She believes a good data-science expert can create Watson-like platforms “with notably less financial commitment.”

None of this is also to say that IBM is alone in its hubris. It’s not. This hubris is shared by many tech companies, particularly those working on computing and AI. For instance, last year Microsoft was roundly (and properly) mocked for its claim that it was going to “solve cancer” in a decade based on this idea:

The company is working at treating the disease like a computer virus, that invades and corrupts the body’s cells. Once it is able to do so, it will be able to monitor for them and even potentially reprogramme them to be healthy again, experts working for Microsoft have said. The company has built a “biological computation” unit that says its ultimate aim is to make cells into living computers. As such, they could be programmed and reprogrammed to treat any diseases, such as cancer.

“The field of biology and the field of computation might seem like chalk and cheese,” Chris Bishop, head of Microsoft Research’s Cambridge-based lab, told Fast Company. “But the complex processes that happen in cells have some similarity to those that happen in a standard desktop computer.” As such, those complex processes can potentially be understood by a desktop computer, too. And those same computers could be used to understand how cells behave and to treat them.

Yes, there is a resemblance between cancer and computing in much the same way that counting on your fingers resembles a supercomputer. The hubris on display was unbelievable. My reaction was virtually identical to Derek Lowe’s , only more so. Indeed, he perfectly characterized the attitude of many in tech companies working on cancer as a “Gosh darn it fellows, do I have to do everything myself?” attitude. Yes, those of us in cancer research and who take care of cancer patients do tend to get a bit…testy…when someone waltzes onto the scene and proclaims to breathless headlines that he’s going to solve cancer in a decade because he has an insight that you stupid cancer biologists never thought of before: The cell is just a computer, and cancer is like a computer virus.

But I digress. I only mention Microsoft to demonstrate that IBM is not alone when it comes to tech companies and hubris about cancer. In any event, I made an analogy to Donald Trump earlier in this post. I was not surprised to find this article making a similar analogy :

“IBM Watson is the Donald Trump of the AI industry—outlandish claims that aren’t backed by credible data,” said Oren Etzioni, CEO of the Allen Institute for AI and former computer science professor. “Everyone—journalists included—know[s] that the emperor has no clothes, but most are reluctant to say so.” Etzioni, who helps research and develop new AI that is similar to some Watson APIs, said he respects the technology and people who work at Watson, “But their marketing and PR has run amok—to everyone’s detriment.” Former employees who worked on Watson Health agree and think the way that IBM overhypes Watson for Oncology is especially detrimental. One former IBM Watson Health researcher and UX designer told Gizmodo of a time they shadowed an oncologist at a cancer center that has partnered with IBM to train Watson for Oncology. The designer claims they spoke with patients who had heard of Watson and asked when it could be used to help them with their disease. “That was actually pretty heartbreaking for me as a designer because I had seen what Watson for Oncology really is and I was very painfully aware of its limitations,” the designer said. “It felt very bad and it felt like there was real hope that had been served by IBM marketing that could not be supported by the product I know.”

That’s part of the problem. Patients see the hype and believe it. They then want what IBM is offering, even if it is not ready for prime time. Watson Health general manager Deborah DiSanzo even said, “We’re seeing stories come in where patients are saying, ‘It gave me peace of mind,'” and concluded, “That makes us feel extraordinarily good that what we’re doing is going to make a difference for patients and their physicians.” Patient peace of mind is important, but not as important as actually producing a product that demonstrably improves patient outcomes.

Again, don’t get me wrong. AI is very likely to be quite important in years (more likely decades) to come in health care. Maybe one day it will lead to a real Tricorder, just like in the original Star Trek series. It’s just not there yet. I suspect that Watson will not be the last medical AI effort to fail to live up to its early grandiose claims.

Dr. Gorski's full information can be found here , along with information for patients. David H. Gorski, MD, PhD, FACS is a surgical oncologist at the Barbara Ann Karmanos Cancer Institute specializing in breast cancer surgery, where he also serves as the American College of Surgeons Committee on Cancer Liaison Physician as well as an Associate Professor of Surgery and member of the faculty of the Graduate Program in Cancer Biology at Wayne State University. If you are a potential patient and found this page through a Google search, please check out Dr. Gorski's biographical information, disclaimers regarding his writings, and notice to patients here .

• Posted in: Cancer , Computers & Internet , Science and Medicine , Science and the Media
• Tagged in: artificial intelligence , cancer , computers , IBM , M.D. Anderson Cancer Center , Memorial Sloan-Kettering Cancer Center , Watson

Posted by David Gorski

• Fierce Pharma
• Fierce Biotech
• Fierce Healthcare
• Fierce Life Sciences Events
• AI and Machine Learning
• Digital Health
• Special Reports
• Special Report
• Awards Gala
• Fierce Events
• Industry Events
• Whitepapers

IBM Watson Health touts recent studies showing AI improves how physicians treat cancer

The use of artificial intelligence technology in clinical decision making is still in an early phase. But recent studies indicate that AI has the potential to help improve the way clinicians treat cancer, according to IBM Watson Health.

One study —which focused on Manipal Hospitals in India and was presented over the weekend at the American Society of Clinical Oncologists (ASCO) annual meeting in Chicago—showed that physicians on a multidisciplinary tumor board changed their treatment decisions in 13.6% of cases based on information provided by Watson for Oncology, the tech giant's AI platform for cancer care.

Of those cases where the tumor board changed their treatment decisions and made a different recommendation, 55% of the time it was because Watson for Oncology provided more up-to-date, evidence-based information on newer treatments than what the physicians knew on their own, Nathan Levitan, M.D., chief medical officer for oncology and genomics at IBM Watson Health, told FierceHealthcare.

"We all know physicians face a nearly impossible task keeping up with all the emerging literature and these physicians found the information provided by Watson for Oncology so compelling they actually changed their treatment decisions," he said. 

RELATED:  IBM and MIT launch $240M AI research lab with a focus on healthcare applications

"The real power of leveraging AI technology is to manage a volume of information that the human brain can’t encompass at one time. Technology doesn’t tell a doctor what to do, it doesn’t make a diagnosis, it presents evidence-based treatment options to enable the doctor to be his or her best self in caring for that patient," Levitan said.

That study was based on a multidisciplinary tumor board 's blinded evaluation of 1,000 breast, lung and colorectal cancer patients. The study found that decisions on cancer treatment also changed due to the AI platform providing more personalized alternatives (30%) or new insights from genotypic and phenotypic data and evolving clinical experiences (15%).

That study was among 22 scientific studies IBM Watson Health presented at ASCO this year demonstrating progress in using AI to provide clinical decision support for cancer care.

The AI platform was trained by specialists at Memorial Sloan Kettering, according to IBM Watson.

The company has faced scrutiny as it has encountered challenges in its work to bring AI to oncology. A Stat report last July, based on internal IBM documents, indicated the Watson supercomputer often produced erroneous cancer treatment advice and that company medical specialists and customers identified “multiple examples of unsafe and incorrect treatment recommendations” as IBM was promoting the product to hospitals and physicians around the world.

With this latest study presented at the ASCO annual meeting, lead investigator SP Somashekhar, chairman of surgical oncology at Manipal Hospitals, said it builds on previous studies and suggests that AI decision support holds substantial promise to reduce the cognitive burden on oncologists, which is a significant problem impacting physician burnout today.

"We consider Watson for Oncology to be an important tool to support decision making, and this study suggests that AI could help reduce variability of care," Somashekhar said in a statement.

Ongoing progress in leveraging AI in clinical care

The studies Watson Health presented at ASCO this weekend demonstrate that the AI platform provides value in improving patient confidence in treatment plans and annotating genomic variants and identifying clinical interventions, the company said.

In another study showcased at the conference, Watson for Genomics was found to identify clinically actionable genomic variants that had not been identified in manual interpretation in a third of patients at a hospital in South Korea. "This helps suggest that the labor-intensive manual curation of such results could be augmented with tools like Watson for Genomics," the company said.

RELATED:  HIMSS19: IBM Watson teams with Brigham, Vanderbilt on $50M AI research initiative

In a third study, physicians from Beijing Chaoyang Integrative Medicine Emergency Medical Center’s oncology department reported patients had a better understanding of their disease and treatment options after Watson for Oncology was incorporated into the consultation process. This led to improved levels of patient engagement and stronger patient confidence in their care plans because patients had a comprehensive view of treatment options, Levitan said. 

IBM studies also suggest that machine learning can be used to automatically identify relevant clinical publications and may reduce the time clinicians spend finding pertinent evidence for their patients.

"As an oncologist, my email inbox is full of medical information and it is difficult to assimilate all of this at the point of care. Our data shows that this can be a very powerful tool to curate the literature and bring to it the physician the evidence that is most relevant to the decision at hand," Levitan said.

Medical researchers at IBM Watson also see the potential for the company's AI technology to help reduce variability in cancer care and improve cancer outcomes globally.

"With 18 million diagnoses globally each year, cancer is a devastating disease that has a heavy human toll, as well as a high health system cost," Levitan said. "Patients often face grueling and confusing treatment regimens, while oncologists sift through reams of medical literature and genomic data to identify the best care plan for each individual patient. All the while, researchers are hamstrung by trials that too often fail due to low patient recruitment."

Data, analytics and AI can be used to address these pressing health challenges, Levitan said.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• AACR Open Access
• PMC10690084

The New NCI Precision Medicine Trials

Lyndsay n. harris.

1 Division of Cancer Treatment and Diagnosis, NCI, Bethesda, Maryland.

Charles D. Blanke

2 SWOG Cancer Research Network, OHSU Knight Cancer Center, Portland, Oregon.

Harry P. Erba

3 Department of Medicine, Duke Cancer Center, Durham, North Carolina.

James M. Ford

4 Division of Oncology, Stanford University School of Medicine, Stanford, California.

Robert J. Gray

5 Department of Data Science, Dana-Farber Cancer Institute and Harvard T.H. Chan School of Public Health, Boston, Massachusetts.

Michael L. LeBlanc

6 SWOG Statistics and Data Management Center, Fred Hutchinson Cancer Center, Seattle, Washington.

Siwen Hu-Lieskovan

7 Department of Medicine, University of Utah School of Medicine, Salt Lake City, Utah.

Mark R. Litzow

8 Division of Hematology, Mayo Clinic, Rochester, Minnesota.

Selina M. Luger

9 Abramson Cancer Center, University of Pennsylvania, Philadelphia, Pennsylvania.

Funda Meric-Bernstam

10 Department of Investigational Cancer Therapeutics, The University of Texas MD Anderson Cancer Center, Houston, Texas.

Peter J. O'Dwyer

11 ECOG-ACRIN Cancer Research Group, University of Pennsylvania, Philadelphia, Pennsylvania.

Megan K.D. Othus

12 Biostatistics, Public Health Division, Fred Hutchinson Cancer Center, Seattle, Washington.

Katerina Politi

13 Section of Medical Oncology, Yale School of Medicine, New Haven, Connecticut.

Lois E. Shepherd

14 Canadian Cancer Trials Group, Queen's University, Kingston, Ontario, Canada.

Carmen J. Allegra

Helen x. chen, s. percy ivy, larissa a. korde, richard f. little, lisa m. mcshane, jeffrey a. moscow, david r. patton.

15 Clinical and Translational Research Branch, Center for Biomedical Informatics and Information Technology, NCI, Rockville, Maryland.

Magdalena Thurin

Laura m. yee, james h. doroshow.

Basket, umbrella, and platform trial designs (master protocols) have emerged over the last decade to study precision medicine approaches in oncology. First-generation trials like NCI-MATCH (Molecular Analysis for Therapy Choice) have proven the principle that studying targeted therapies on a large scale is feasible both from the laboratory and clinical perspectives. However, single-agent targeted therapies have shown limited ability to control metastatic disease, despite careful matching of drug to target. As such, newer approaches employing combinations of targeted therapy, or targeted therapy with standard therapies, need to be considered. The NCI has recently embarked on three second-generation precision medicine trials to address this need: ComboMATCH, iMATCH, and myeloMATCH. The design of these trials and necessary infrastructure are discussed in the following perspective.

Introduction

Availability of advanced molecular diagnostic tools and the development of numerous targeted agents ushered in the era of cancer precision medicine ( 1 ). Initial platform studies exploring precision medicine concepts, including the NCI-MATCH (Molecular Analysis for Therapy Choice) trial, have led to a proliferation of similar trials in the research community ( 2 ). The NCI has supported these efforts and has now developed the next generation of cancer precision medicine trials aimed at ongoing refinement of this approach. This article describes the rationale and collaborative structure of these trials ( Fig. 1 ).

The New Precision Medicine Initiatives of the National Cancer Institute.

NCI-MATCH and other precision medicine initiatives

The NCI-MATCH (Molecular Analysis for Therapy Choice) study was a precision medicine initiative (PMI) sponsored by NCI that recruited over 6,000 patients in a period of less than 2 years ( 3 ). The trial showed that it was feasible to screen and recruit patients using both central next-generation sequencing (NGS) and well-vetted community-based NGS. However, although 38% of screened patients had a potentially actionable mutation, there was a relatively low response rate to single agents. Resistance to therapy with single targeted agents has been attributed to multiple mechanisms, including resistance mutations and multi-genic or other adaptive responses, and suggests that approaches to overcome resistance are needed to achieve durable clinical benefit.

Since the original NCI-MATCH trial, the therapeutic landscape of cancer therapy has evolved substantially. The introduction of successful immunotherapies into cancer treatment has resulted in the need to understand and target underlying mechanisms of sensitivity and resistance to immune-targeted agents. In addition, the vision of precision medicine has evolved from the single target-agent paradigm to embrace more nuanced approaches to cancer treatment, in which therapies are developed based not only on driver mutations but also on observed resistance pathways. The next iteration of NCI's Precision Medicine Trials represents a multi-dimensional approach to cancer precision medicine. NCI has developed three new trials to address the areas of molecularly targeted treatment combinations, patient stratification or selection based on tumor immune characteristics, and tiered studies based on primary disease and minimal residual disease in acute myeloid leukemia (AML) and myelodysplastic syndrome (MDS; Fig. 1 ).

The Next Generation of PMI Trials

Most NCI-MATCH arms did not meet their clinical endpoints. This has reinforced the concept that inhibition of a single driver gene alone is not adequate to produce a clinical benefit in most malignancies. Cancers often recruit parallel or compensatory pathways to overcome inhibition of a single node, providing one plausible biological explanation for the NCI-MATCH clinical experience ( 4 ). ComboMATCH has been designed to test specific molecularly targeted combinations aimed at overcoming primary and adaptive resistance pathways.

Designing a precision medicine trial using drug combinations rather than single agents presents numerous complexities. In NCI-MATCH, the evidence threshold for testing a single drug against a single target was relatively straightforward; in ComboMATCH, the number of potential drug combinations is exponentially greater, and the number and complexity of targets is substantially enhanced. Therefore, a combination approved by the ComboMATCH Agents and Genes Working Group, made up of representation from each of the National Clinical Trial Network (NCTN) groups including the Children's Oncology Group, must demonstrate a combinatorial effect of the drug combination and a tumor response of regression or sustained stabilization in at least two relevant in vivo models to support its use, and a recommended phase II dose for the combination. Selected drug combinations without phase II dose determinations are diverted into phase I studies for later incorporation in ComboMATCH. In addition, appropriate statistical designs (typically randomized) and evidence thresholds for evaluating promising drug combinations were developed, and these are being applied uniformly to all prospective studies. Furthermore, many of the treatment arms limit the number of regimens and performance status in an effort to optimize the opportunity for treatment response.

Immunotherapy-MATCH

Immunotherapy-MATCH (iMATCH) will provide a central platform for tissue procurement and molecular testing, with the goal of enhancing immunotherapy trials through prospective patient enrichment or stratification as well as comprehensive exploratory analysis. Currently, most clinical studies of immunotherapy combinations are conducted in “all-comers” due to lack of patient selection markers, and the results are often negative and noninformative. We proposed that, while regimen-specific markers are unavailable, several clinical grade biomarkers developed for anti-PD-1/L1 monotherapy prospectively characterize elements of adaptive immunity for “biological” stratification. Examples include tumor mutational burden (TMB) as proxy for immunogenicity, and IFNγ signatures or tumor-infiltrating CD8 + lymphocyte staining as markers of tumor immunity ( 5, 6 ). It is hypothesized that composite biomarkers like TMB and tumor inflammation score (TIS) can be used to separate patients into subgroups with potentially different immune status (e.g., immune inflamed, immune excluded or immune desert), and each subgroup may have a set of immune evasion mechanisms that can be targeted with relevant combination strategies.

The iMATCH platform trial includes a central assay protocol for both integral and exploratory biomarkers. Patients will be stratified or enriched on the basis of TMB (High vs. Low), and TIS (High vs. Low). In addition, data from whole-exome sequencing (WES) and RNA sequencing (RNA-seq) will enable broader retrospective analysis for discovery of resistance mechanisms and to refine patient selection strategies in the context of specific tumor settings and treatment regimes. Treatment protocols will be developed by NCTN, with a focus on signal-seeking trials of immuno-oncology combinations. Agent/regimen selection will consider supporting data from preclinical models or clinical translational studies, as well as clinical experience and “credentials” of individual agents or the combination. We envision that most protocols under iMATCH will be histology-specific in either immunotherapy-naïve or -refractory setting, depending on feasibility and scientific hypothesis.

iMATCH addresses unique logistical, technical, and trial design challenges that set it apart from other NCI-supported precision medicine platforms in development. First, unlike in the NCI-MATCH trial, the integral markers in iMATCH will define biological subgroups not tied to specific molecular targets. Second, TMB and TIS measurements are continuous variables and require predefined cut-off points for prospective use, yet existing data are limited for identifying the optimal cutoff for all clinical settings under study (e.g., immunotherapy naïve vs. refractory setting). To address these challenges, a pilot trial is being conducted before full launch of iMATCH protocols, to resolve details of biomarker assessment as well as establish the feasibility and clinical acceptability of the turnaround times.

MyeloMATCH is the first NCI PMI for myeloid cancers funded by the NCI intended to treat patients with myeloid cancers from diagnosis throughout their treatment journey. The NCTN Leukemia committees have committed to the myeloMATCH platform as the means through which the AML and MDS trial portfolios for initially diagnosed patients will be established and conducted. The goal is to develop a suite of studies attractive to patients, investigators, and industry collaborators in myeloid malignancies. A myeloMATCH Master Screening and Reassessment Protocol (MM-MSRP) evaluates newly diagnosed patients and assigns them to a treatment protocol based on clinical and genomic features. In general, the investigations are planned as randomized phase II trials seeking large signals within clinical or molecular subgroups to inform subsequent more precise study of selected patients with AML/MDS.

The myeloMATCH MSRP and informatics platform will facilitate cross-treatment study interrogation of genomic features and response characteristics. This will enable hypothesis generating ad hoc studies that may help identify scientific opportunities to advance effective therapeutics in AML and MDS. The MSRP design provides a platform for myeloMATCH senior laboratory experts and statisticians with specialized expertise in biomarkers and genomics to assist and advise the clinical investigators in myeloMATCH. The myeloMATCH Senior Scientific Council then approves studies that are vetted by the NCTN Leukemia Steering Committee. In this manner, myeloMATCH brings a critical mass of AML and MDS experts from throughout the United States and Canada to devise clinical trials aimed at improving precision treatment for patients with progressively decreasing disease burden of myeloid cancers undergoing successive therapeutic approaches. This effort will also generate data to inform use of various flow cytometric and molecular assays for identification and targeting of residual disease. The overarching myeloMATCH goals are to assess the impact of genomically-selected treatments, from diagnosis (Tier 1) through consolidation (Tier 2), allogeneic transplant when indicated (Tier 3), and then effective targeting of measurable residual disease (Tier 4).

NCI Resources Deployed to Support PMIs

Each of these initiatives requires novel and extensive informatics and central laboratory support. Both iMATCH and myeloMATCH will require diagnostic testing with an expedited turnaround time for patient therapy assignment. Additional resources for biopsies, biobanking, and regulatory support will be provided ( Fig. 2 ).

NCI resources supporting the Precision Medicine Initiatives.

NCI-supported centralized biospecimen management, patient selection, and allocation

The molecular and immunologic diagnostic laboratory network.

These new PMIs will integrate lessons learned in NCI-MATCH and take advantage of novel technologies that provide more comprehensive molecular analysis, permitting a greater depth of understanding of tumor biology. The NCI has established the Molecular and Immunologic Diagnostic Laboratory Network (MDNet) to provide both real-time diagnostic services to support iMATCH and myeloMATCH and retrospective analyses such as WES, RNA-seq, and evaluation of cell-free DNA (cfDNA) in ComboMATCH and iMATCH. The goal is not only to improve outcomes with novel therapies but also to learn more from every patient about how molecular and other characteristics can be used to optimally select therapy.

ComboMATCH will initially use DNA sequencing performed by commercial and academic laboratories (known as the Designated Laboratory Network); generally, only one actionable mutation of interest (aMOI) will be used to select therapy although some arms do have exclusionary variants. MDNet will perform WES to assess molecular concordance with aMOIs detected by the Designated Laboratories. For iMATCH, MDNet will provide prospective TMB and TIS data as well as actionable mutations that will be used to define molecular subgroups for clinical testing of specific treatment arms. For myeloMATCH, MDNet assays will also be used to evaluate responses, and potentially reassign responding patients to the next myeloMATCH treatment protocol, proceeding roughly parallel to the standard approach for these diseases, including transplant and maintenance therapy. In addition, MRD will be assessed using both flow cytometry and double-strand NGS to improve methods for determination of this important treatment endpoint and for protocol assignment. These approaches will eventually amass sufficient data to allow for both longitudinal and cross-study assessments of the natural history of AML and MDS.

All three precision medicine studies will also include more detailed retrospective molecular characterization (WES and RNA-seq, cfDNA, protein studies) for further exploration. These investigations will be performed on the bone marrow biopsies for myeloMATCH, and from tumor biopsies for iMATCH and ComboMATCH.

The Precision Medicine Analysis and Coordination Center

To support the simultaneous development of multiple PMI trials, NCI recognized the need to leverage a unified informatics methodology. NCI established the Precision Medicine Analysis and Coordination Center (PMACC) as a data and bioinformatics support center to implement a high percentage of infrastructure commonality across initiatives. The primary goal of the PMACC is to facilitate the assignment of patients to the highest priority treatment without delay and with full traceability.

On the basis of lessons learned from NCI-MATCH and new, more sophisticated initiative designs, the PMACC is enhancing the existing NCTN clinical trial infrastructure. These enhancements include both modifications and standardized configurations to the patient registration system, clinical data management system, specimen tracking approach, terminology services, and integration layers. At the heart of ComboMATCH and myeloMATCH is the NCI-MATCHBox precision medicine platform. This platform houses the treatment assignment algorithms, coupled with a clinical verification team, and is fully integrated with the NCTN clinical trial infrastructure.

ComboMATCH will take advantage of capabilities developed to consume data from the Designated Laboratory Network under NCI-MATCH. These capabilities support the automated ingestion and harmonized annotation of molecular sequencing data. For myeloMATCH, the MATCHBox will consume assay data from MDNet to derive treatment assignments. All three of the new initiatives will leverage the existing PMACC Data Warehouse for analysis and reporting.

Regulatory support

NCI recognized that the PMI trials would have to be conducted under the Cancer Therapy Evaluation Program (CTEP) Investigational New Drug (IND) process to be feasible, as only NCI could provide the centralized infrastructure support needed for such initiatives. To facilitate these trials, NCI leadership allowed expeditious negotiation of collaborative research agreements with pharma partners for agents approved by the PMI trial governance committees. This decision greatly streamlined the ability of NCI to access agents required for all three PMI trials.

Challenges for This Next Generation of PMI Trials

Statistical design.

The three PMI trials (ComboMATCH, iMATCH, and myeloMATCH) differ substantially in the types of clinical and translational science questions they address. Accordingly, different statistical designs are needed to generate interpretable and convincing evidence of treatment efficacy and biomarker utility, including “intended use” and “fit for purpose” assays.

ComboMATCH is a platform trial comprising multiple subprotocols, each investigating a specific drug combination, that are clustered into cassettes run by different NCTN groups ( 7 ). Subprotocols may include multiple cohorts, differing by one or more molecular characteristic, tumor histology, and history of prior treatments and outcomes. Cohorts comprised of patients who never received either agent in the combination generally include a randomization between combination and single agents with a primary endpoint of progression-free survival unless background evidence strongly suggests limited efficacy of one or both single agents in similar cohorts. Nonrandomized single-arm two-stage designs with objective response endpoint are used to study drug combinations for cohorts of patients who have progressed on one or both agents, or when strong background evidence suggests limited single-agent activity. Appropriate, standardized designs for ComboMATCH have been facilitated by a statistical design working group which includes members from both the NCTN network groups and NCI.

The iMATCH trial provides prospective tumor characterization that can enable therapeutic trials in molecularly enriched or stratified patient populations. A phased approach and innovative statistical design were needed to address uncertainty about the turnaround times, cut-off points, feasibility, and predictive ability for candidate biomarkers and signatures. iMATCH has begun with a pilot study of nivolumab and cabozantinib in advanced refractory melanoma and head and neck squamous cell carcinoma which will assess the feasibility of prospective testing as well as efficacy in molecular subgroups based on TMB and TIS (TMB/TIS: High/High, High/Low, Low/High, and Low/Low). The study will have two stages with interim analyses and early stopping rules based on assay turnaround time, subgroup distribution and efficacy within subgroups, and the results from the first stage will determine whether and how stage 2 of the trial will proceed. Retrospective molecular analyses will be planned using WES, RNA-seq, NanoString PanCancer IO 360 panel and multiplex immunofluorescence to further explore optimal cutoffs for biological classification and to identify predictive markers for the specific regimen. The pilot trial will generate valuable evidence to aid in planning future treatment arms and biomarker signatures to be tested within the iMATCH trial platform.

MyeloMATCH aims to conduct studies to improve care for patients diagnosed with AML and MDS throughout the course of their disease. The MM-MSRP will be used to collect and coordinate genomic and other biomarker data to enroll eligible patients on trials at diagnosis and in subsequent phases of their treatment. MyeloMATCH subprotocols will primarily employ independent randomized phase II clinical trial designs powered to detect large signals for promising, new therapies. Patients’ eligibility to enroll on specific trial protocols will be based on phase of therapy (e.g., initial treatment, transplant, maintenance), age group, fitness, and genomic targets. Biospecimens collected under the MM-MSRP will also be leveraged to conduct prespecified and post hoc biomarker analyses using cutting-edge MRD technologies with the goal of clinically validating additional predictive biomarkers.

Complexity of cross-NCTN collaboration

The original NCI-MATCH trial was led by the ECOG-ACRIN network group with representation of other network groups as sub study principal Investigators. The current generation of PMI trials is intended to involve all the network groups equally. This requires an unparalleled collaborative effort, with each group having representation on the governance committees of each PMI and an active role in protocol authoring. This also requires collaboration for the NCTN operations offices and enrolling sites about data sharing and specimen tracking and allocation and requires software solutions for communication between PMACC and MDNet labs. The PMI collaboration has also motivated additional discussions between several of the network groups to formulate additional best practices and standards for data collection beyond the PMIs.

Each PMI trial also has its own governance structure to collaboratively review and approve subprotocols prior to submission to CTEP, NCI, and to manage the study development and implementation process.

For ComboMATCH, an effort was made to minimize the number of committees involved in protocol development. The Agents and Genes Working Group (C-AGWG), with representation from each of the NCTN groups, established the levels of evidence required for ComboMATCH studies, and C-AGWG review became the first step in study development. Concepts initially approved by C-AGWG are presented to the proposed collaborating pharmaceutical partners; if informally approved, then the study concept is presented to the Statistical Design Development Working Group, again composed of representatives from each of the NCTN groups, which critically evaluates the designs and their underlying assumptions for the cohorts within each subprotocol as described above. Subprotocols reaching this point are then presented to the ComboMATCH Steering Committee. Three other committees provided operational oversight for the ComboMATCH study development process: a Protocol Logistics Working Group, the PMACC, and the Molecular Biomarker and Specimen Management Committee.

For iMATCH, the pilot will be managed by the SWOG Cancer Research Network with other groups participating in the full-scale initiative.

For myeloMATCH, senior leadership comprises representatives from each of the NCTN Groups, including the Group Leukemia Committee Chairs. Individual NCTN investigators develop clinical trial concepts with guidance and input from leadership. Once a concept is endorsed by the senior myeloMATCH leadership, it is further developed within the Leukemia Committee of the NCTN group that will lead that specific treatment trial. Each protocol includes study champions from each NCTN group.

Each of the PMIs encourages participation of early and mid-career investigators to provide opportunities for collaboration and career development in clinical trials methodology. Young NCTN investigators receive mentoring throughout this process.

Summary and Conclusions

Our understanding of precision cancer medicine has changed dramatically in the decade between the planning of the original MATCH trial and the planning of its successor trials described in this article. To address the therapeutic challenges that initial precision medicine studies revealed, ComboMATCH, myeloMATCH, and iMATCH have been developed. Each involves multiple partners from the pharmaceutical industry, academia, NCI cooperative group networks, and NCI staff. This collaboration is necessary to assure that the carefully designed studies using well-characterized, analytically validated assays in appropriate patient populations will achieve the dual goals of optimizing individualized therapy for patients, while at the same time discovering approaches that have an impact on the treatment of future patients. The ability to conduct trials of this complexity is made possible by a strong commitment across the oncology community and will reap rewards for years to come.

Authors' Disclosures

C.D. Blanke reports grants from NCI during the conduct of the study. H.P. Erba reports grants and other support from AbbVie, Agios, Daiichi Sankyo, Glycomimetics, and Jazz; grants from ALX Oncology, Amgen, Forma, Forty Seven, Gilead, ImmunoGen, MacroGenics, Novartis, and PTC; and other support from Celgene/BMS, Incyte, Astellas, Genentech, Kura Oncology, Novartis, Syros, Takeda, and Trillium outside the submitted work. J.M. Ford reports grants from Genentech and Merus outside the submitted work. R.J. Gray reports grants from NCI during the conduct of the study. M.L. LeBlanc reports grants from NIH during the conduct of the study. S. Hu-Lieskovan reports personal fees from Regeneron, Ascendis, BMS, Merck, Amgen, Astellas, Genmab, Nektar, Novartis, Vaccinex, and Xencor during the conduct of the study. S.M. Luger reports personal fees from AbbVie, Amgen, and BMS and grants from Celgene and Hoffman-La Roche outside the submitted work. F. Meric-Bernstam reports personal fees from AbbVie, Aduro BioTech Inc., Alkermes, AstraZeneca, Daiichi Sankyo Co. Ltd., Calibr (a division of Scripps Research), DebioPharm, Ecor1 Capital, eFFECTOR Therapeutics, F. Hoffman-La Roche Ltd., GT Apeiron, Genentech Inc., Harbinger Health, IBM Watson, Infinity Pharmaceuticals, Jackson Laboratory, Kolon Life Science, Lengo Therapeutics, Menarini Group, OrigiMed, PACT Pharma, Parexel International, Pfizer Inc., Protai Bio Ltd, Samsung Bioepis, Seattle Genetics Inc., Tallac Therapeutics, Tyra Biosciences, Xencor, Zymeworks, Black Diamond, Biovica, Eisai, Fog Pharma, Immunomedics, Inflection Biosciences, Karyopharm Therapeutics, Loxo Oncology, Mersana Therapeutics, OnCusp Therapeutics, Puma Biotechnology Inc., Sanofi, Silverback Therapeutics, Spectrum Pharmaceuticals, Thera Technologies, and Zentalis; grants from Aileron Therapeutics, Inc., AstraZeneca, Bayer Healthcare Pharmaceutical, Calithera Biosciences Inc., Curis Inc., CytomX Therapeutics Inc., Daiichi Sankyo Co. Ltd., Debiopharm International, eFFECTOR Therapeutics, Genentech Inc., Guardant Health Inc., Klus Pharma, Takeda Pharmaceutical, Novartis, Puma Biotechnology Inc., and Taiho Pharmaceutical Co.; and other support from European Organization for Research and Treatment of Cancer (EORTC), European Society for Medical Oncology (ESMO), and Cholangiocarcinoma Foundation outside the submitted work. K. Politi reports grants from NCI via SWOG during the conduct of the study as well as grants and personal fees from AstraZeneca and Roche/Genentech, grants from Boehringer Ingelheim and D2G Oncology, and personal fees from Janssen and Halda Therapeutics outside the submitted work; in addition, K. Politi has a patent related to EGFR T790M mutation testing licensed and with royalties paid from MSKCC/MolecularMD. No disclosures were reported by the other authors.

The publication costs of this article were defrayed in part by the payment of publication fees. Therefore, and solely to indicate this fact, this article is hereby marked “advertisement” in accordance with 18 USC section 1734.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Alcohol consumption, smoking and risk of gastric cancer: case-control study from Moscow, Russia

Affiliation.

• 1 Department of Epidemiology and Prevention, Institute of Carcinogenesis, Russian Cancer Research Center, Russian Academy of Medical Sciences, Moscow.
• PMID: 10843447
• DOI: 10.1023/a:1008907924938

Objectives: To examine the risk of gastric cancer associated with alcohol consumption and smoking in men and women in Moscow, Russia.

Materials and methods: A case-control study which includes 448 cases and 610 controls was conducted. Cases consisted of patients with newly diagnosed histologically confirmed gastric cancer. Controls were patients admitted during the study period to the hospital with diagnoses other than cancer and/or gastrointestinal diseases. Information on demographic variables, smoking, alcohol consumption and diet was collected from all subjects. Venous blood was drawn from 361 cases and 441 controls. A serological test for Helicobacter pylori immunoglobulin G was performed.

Results: Alcohol consumption, particularly vodka consumption, was found to increase the risk of gastric cancer. In men the effect of hard liquor drinking was stronger for cancer of the cardia (OR = 3.4, CI = 1.2-10.2), while in women the effect was stronger for cancer of sites other than gastric cardia (OR = 1.5, CI = 1.0-2.3). Smoking increased the risk of developing gastric cancer in men, but not in women. In men a dose-response relationship between mean number of cigarettes smoked per day (p = 0.03), pack-years of cigarettes smoked (p = 0.01) and duration of smoking (p = 0.08) and the risk of cancer of gastric cardia was observed. Further statistical analysis revealed interactions between effect of smoking and alcohol consumption and between smoking and H. pylori infection status.

Conclusions: The findings further support the role of alcohol consumption and smoking in the etiology of gastric cancer.

PubMed Disclaimer

Similar articles

• Aspirin protects against gastric cancer: results of a case-control study from Moscow, Russia. Zaridze D, Borisova E, Maximovitch D, Chkhikvadze V. Zaridze D, et al. Int J Cancer. 1999 Aug 12;82(4):473-6. doi: 10.1002/(sici)1097-0215(19990812)82:4 3.0.co;2-k. Int J Cancer. 1999. PMID: 10404057
• Vodka consumption, tobacco smoking and risk of gastric cancer in Poland. Jedrychowski W, Boeing H, Wahrendorf J, Popiela T, Tobiasz-Adamczyk B, Kulig J. Jedrychowski W, et al. Int J Epidemiol. 1993 Aug;22(4):606-13. doi: 10.1093/ije/22.4.606. Int J Epidemiol. 1993. PMID: 8225732
• Meat intake and risk of stomach and esophageal adenocarcinoma within the European Prospective Investigation Into Cancer and Nutrition (EPIC). González CA, Jakszyn P, Pera G, Agudo A, Bingham S, Palli D, Ferrari P, Boeing H, del Giudice G, Plebani M, Carneiro F, Nesi G, Berrino F, Sacerdote C, Tumino R, Panico S, Berglund G, Simán H, Nyrén O, Hallmans G, Martinez C, Dorronsoro M, Barricarte A, Navarro C, Quirós JR, Allen N, Key TJ, Day NE, Linseisen J, Nagel G, Bergmann MM, Overvad K, Jensen MK, Tjonneland A, Olsen A, Bueno-de-Mesquita HB, Ocke M, Peeters PH, Numans ME, Clavel-Chapelon F, Boutron-Ruault MC, Trichopoulou A, Psaltopoulou T, Roukos D, Lund E, Hemon B, Kaaks R, Norat T, Riboli E. González CA, et al. J Natl Cancer Inst. 2006 Mar 1;98(5):345-54. doi: 10.1093/jnci/djj071. J Natl Cancer Inst. 2006. PMID: 16507831
• Gastric cancer: epidemiology and risk factors. de Martel C, Forman D, Plummer M. de Martel C, et al. Gastroenterol Clin North Am. 2013 Jun;42(2):219-40. doi: 10.1016/j.gtc.2013.01.003. Epub 2013 Mar 29. Gastroenterol Clin North Am. 2013. PMID: 23639638 Review.
• A meta-analysis on alcohol drinking and esophageal and gastric cardia adenocarcinoma risk. Tramacere I, Pelucchi C, Bagnardi V, Rota M, Scotti L, Islami F, Corrao G, Boffetta P, La Vecchia C, Negri E. Tramacere I, et al. Ann Oncol. 2012 Feb;23(2):287-97. doi: 10.1093/annonc/mdr136. Epub 2011 May 5. Ann Oncol. 2012. PMID: 21551004 Review.
• The association between dietary fiber intake and gastric cancer: a pooled analysis of 11 case-control studies. Collatuzzo G, Cortez Lainez J, Pelucchi C, Negri E, Bonzi R, Palli D, Ferraroni M, Zhang ZF, Yu GP, Lunet N, Morais S, López-Carrillo L, Zaridze D, Maximovitch D, Guevara M, Santos-Sanchez V, Vioque J, Garcia de la Hera M, Ward MH, Malekzadeh R, Pakseresht M, Hernández-Ramírez RU, Turati F, Rabkin CS, Liao LM, Sinha R, López-Cervantes M, Tsugane S, Hidaka A, Camargo MC, Curado MP, Zubair N, Kristjansson D, Shah S, La Vecchia C, Boffetta P. Collatuzzo G, et al. Eur J Nutr. 2024 Aug;63(5):1857-1865. doi: 10.1007/s00394-024-03388-w. Epub 2024 Apr 30. Eur J Nutr. 2024. PMID: 38687390 Free PMC article.
• Reproductive factors, hormonal interventions, and gastric cancer risk in the Stomach cancer Pooling (StoP) Project. Song M, Jayasekara H, Pelucchi C, Rabkin CS, Johnson KC, Hu J, Palli D, Ferraroni M, Liao LM, Bonzi R, Zaridze D, Maximovitch D, Aragonés N, Martin V, Castaño-Vinyals G, Guevara M, Tsugane S, Hamada GS, Hidaka A, Negri E, Ward MH, Sinha R, Lagiou A, Lagiou P, Boffetta P, Curado MP, Lunet N, Vioque J, Zhang ZF, La Vecchia C, Camargo MC. Song M, et al. Cancer Causes Control. 2024 Apr;35(4):727-737. doi: 10.1007/s10552-023-01829-1. Epub 2023 Dec 20. Cancer Causes Control. 2024. PMID: 38123742
• Systemic Diseases and Gastrointestinal Cancer Risk. Malkani N, Rashid MU. Malkani N, et al. J Cancer Allied Spec. 2023 Aug 13;9(2):473. doi: 10.37029/jcas.v9i2.473. eCollection 2023. J Cancer Allied Spec. 2023. PMID: 37575213 Free PMC article. Review.
• Leisure-time physical activity and gastric cancer risk: A pooled study within the Stomach cancer Pooling (StoP) Project. Mariani M, Pastorino R, Pires Marafon D, Johnson KC, Hu J, Molina de la Torre AJ, Fernández-Tardón G, Zaridze D, Maximovich D, Negri E, La Vecchia C, Zhang ZF, Kurtz RC, Pelucchi C, Rota M, Boccia S. Mariani M, et al. PLoS One. 2023 Jul 12;18(7):e0286958. doi: 10.1371/journal.pone.0286958. eCollection 2023. PLoS One. 2023. PMID: 37437057 Free PMC article.
• Challenges and Suggestions in the Management of Stomach and Colorectal Cancer in Uzbekistan: The Third Report of the Uzbekistan-Korea Oncology Consortium. Rim CH, Lee WJ, Akhmedov O, Sabirov U, Ten Y, Ziyayev Y, Tillyashaykhov M, Rim JS; consortium of the Republican Specialized Scientific Practical-Medical Center of Oncology and Radiology and South Korean Oncology Advisory Group. Rim CH, et al. Int J Environ Res Public Health. 2023 Apr 12;20(8):5477. doi: 10.3390/ijerph20085477. Int J Environ Res Public Health. 2023. PMID: 37107759 Free PMC article. Review.

Publication types

• Search in MeSH

Related information

• Cited in Books

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.
• MedlinePlus Health Information
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.