If Your Doctor Isn’t Using AI, Get Another Doctor

A GPT-powered tumor board and the paper that might change oncology forever

It’s been quite a delay between posts! I’ve been in the middle of a crazy renovation, work has gotten busier, the weather is getting lovely and we have summer stuff to plan. So, less time for the laptop, but I have been thinking about this all week. Enjoy!

So, last week, for tumor board, one of my co-workers (the illustrious Dr. Frank Vicini) and our breast surgeon came up with an idea. They wanted to pre-emptively run our tumor board cases through an LLM before discussing them. We gave it pretty basic information and then had it play the role of surgeon, med onc and radonc. After the group reviewed the case, we compared it to ChatGPT’s output. I won’t cut and paste the results, but they were very good - yet not perfect. We were using GPT 4o and nothing else on top of that. Overall, I’d give it a B+. I’d show you the output, as it is anonymized, I’d feel better if the patient and team okayed it. Just try it yourself.

Well, the Germans said “Hold my beer”

I just read what I feel to be the most important paper in oncology in many years.

“Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology” was published last week in Nature Cancer and I encourage you to read it.

But, in case you don’t, I had ChatGPT provide a summary..

---

Can an AI Actually Make Oncology Decisions? This One Kind of Does.

A team of researchers just built an autonomous AI agent—not just ChatGPT with a stethoscope, but a tool-using, data-crunching, image-reading machine—designed to help make real cancer treatment decisions.

Here’s how it works, in plain English:

---

🛠️ How They Built It

Instead of training one massive AI model to “know it all,” they gave GPT-4 a toolbox:

• Radiology tools (to interpret CT/MRI images)

• Histology tools (to detect mutations from biopsy slides)

• OncoKB (a mutation-to-drug lookup database)

• Google + PubMed search (to find up-to-date studies)

• Calculator + clinical guidelines (to measure tumor changes and find treatments)

Then they trained it to autonomously choose which tools to use, run them in sequence (e.g., find tumor on scan → measure its size → compare to last scan → decide if it's progressing), and synthesize a treatment plan.

---

📦 What Did They Test?

They gave it 20 complex, realistic cancer cases (mostly GI), including imaging, pathology, genomics, and clinical histories. No humans in the loop.

The agent:

• Picked and used tools with 87.5% success

• Made accurate treatment plans in 91% of cases

• Cited the right oncology guidelines 75% of the time
  (GPT-4 alone? Only 30% accuracy.)

---

🤖 Why Not Just Use a Bigger AI?

Because generalist models still struggle with:

• Reading small tumors on CT

• Interpreting rare mutations

• Knowing up-to-date niche guidelines

This modular approach is more like assembling Iron Man: use the best tool for each job, keep them updated individually, and let GPT-4 be the brain.

---

🚧 Limitations?

Yup. It’s a proof-of-concept. Only 20 test cases. Still needs local deployment (GPT-4 is cloud-based), and can’t yet read full 3D scans. But as a blueprint for AI-assisted oncology? This is the real deal.

---

Bottom line: The future of medical AI might not be a single all-knowing model—but a smart orchestrator using the right tools, at the right time, just like a good doctor does.

---

Okay, it’s me again. Holy f*&kb@lls. I am sorry for the strong language. They had the AI interpret pathology slides, review imaging, used Google and PubMed to find relevant papers, considered guidelines and got it right nearly 90% of the time. The naysayers (like each time AI does something incredible) will say that’s still not perfect and that expert physicians would be closer to 90%. But, what one doctor could review the path, images, look up all relevant data and make sure the recommendations were guideline concordant AND be right 9/10 times? There is probably someone out there that could, but I have never seen a clinician be able to review slides or a pathologist able to review imaging (trust me, when we have absences in tumor board we all try to sub in, but it’s a goat rodeo).

I’ve gone down a rabbit hole and discussed this with GPT 4o:

It took it a while to get to the main point and I had to ask point blank: “How will employment be affected?” And, if you read above, it definitely affect the physician labor force at some point in the future.

In every industry, we are hearing the same phrase “______ won’t be replaced by AI. ______ will be replaced by ______ who know how to deploy AI. I.e - lawyers won’t be replaced by AI, they will be replaced by lawyers who know how to deploy AI. Same for other white collar professionals and possible cognitive specialists in medicine. In a year or two, that 90% accuracy is going to go to 95% and then maybe in 3-4 years it will approach 99%. Why would a patient want a human doctor that is: 1) slower, 2) less accurate, and 3) more prone to cognitive biases? If my doctor right now isn’t using AI, then they are doing it wrong. I use it on nearly every new consult - I don’t want to miss anything and I also want it to quickly identify if they are eligible for trials (as my group enrolls many patients to NRG and other studies). In addition, I am continuing to know my limitations and I respect what AI can do for my patients. I recently had an unusual case of angiosarcoma and after not hearing back after emailing the specialists that originally saw them at a big cancer center in Houston, I discussed the case with many humans, but also went to OpenEvidence and ChatGPT 4o. I feel like we came up with a reasonable plan.

I am not an accelerationist but I’m not in the decel camp and I don’t have crazy p(doom). I welcome technology and I am always excited by the next new thing. But, this time it is different. We are a short ways away from being quite close to AGI and the key will be to learn it the best we can and to hope we don’t get swept up in the massive labor shifts that are inevitable. This paper shook me and I am continuing to think more about how I need to adjust and how my children will need to brace for what lies ahead. I am spending at least an hour with it a day to try to see what I can learn or what new thing I can do with it. When the kids can read, they will do the same. Here are some ways I currently utilize AI in my day to day life:

Case Reviews

I see about 4-5 consults a week right now and based on our numbers, when all my payors are in, I will hopefully be around 8 or so a week. I input nearly each case and make sure that the management aligns with what has happened thus far and hope that my recommendation is concordant with what they recommend. I can also ask if it knows of any studies the patient may be eligible for. Generally, it has been helpful and it has definitely picked up things I did not consider. Other times, it missed some basic things. This will improve over time.

Researching Rare Cases

All radoncs see rare cases occasionally. This has been a remarkable tool for finding papers, summaries and rational clinical decision making. I do have to constantly check if the references are real or hallucinations, but this improving. However, I would say 10-20% of the time there are made up references. You must be careful when doing this and always be skeptical if it comes up with the exact reference in a situation. It is very possible to be incorrect.

Writing

I use GPT often for writing. I do not have it write things for me. I have always noted in my posts if it is doing the work. However, I do have it help with strengthening my arguments - I.e. I have it play skeptic about all of my hypotheses and points I am making to help me “steelman” my argument. It has really helped, in my opinion. In addition, the grammar / syntax checking is good, but it has “a style” that is not my style. It loves — for whatever reason — em dashes (that is the connected dashes you are seeing). I don’t love those and I’ve asked about them. GPT admits that it is probably a “giveaway” that someone edited with an LLM, because most of us simply don’t write that way. I also have it help me make tables, graphs, charts that are easily viewable in the post.

Management

There is so much you can do if you are in leadership. I will do a post in the future about this, but I’ll just give an example: I asked my new group about referral patterns, consult numbers, etc. We didn’t have reports and such that were easily available. So, we asked one of the managers to get the raw data. I was able to easily see which clinic got how many consults from 2019 to 2024. I was also able to see which doctors referred the most patients over the years and over periods of time. I also could track shifts in specific doctors (increases / decreases in consults). I had it create some strategies to improve referral management. Let’s see how it works, but I would not have spent the time doing this manually and I’m not sure someone in house would have done as great a job with presenting the data so well.

Process Improvement/Workflow

This is a great area to look at. You can describe or upload your processes in the clinic and it can give feedback on areas for improvement—places where you can reduce errors or redundancy, as well as increase efficiency. I’ll write more about this later, as well. (Note the -em dash: that was the LLM’s suggestion; I’m just using it to point it out)

Making Ridiculous Pictures

Prompt was: “Show me dressed like Yung Gravy presenting at a conference.” I have hilarious pics of many radoncs, other friends, family members and such. I think of myself as a pretty creative guy, but this tool really allows the juices to flow. The kids love it. I did one my daughter wanted below, her prompt was “Show Calisi at a farm holding bunnies, with chicks in the grass and unicorns in the sky”

How adorable is that!

Good Night Stories

It gets challenging making up stories. So, I’ve occasionally given a prompt like “Create a bedtime story about my kids, Calisi (5, F) and Cailo (4, M). They live in Royal Oak, MI. She loves unicorns and bunnies. He loves being mischievous and ignoring his dad’s requests to listen. They should go on an adventure and save the day by the end. Reference Royal Oak places they may know.”

I’ve included their friends in some and then sent to their parents so they can read it the same evening. It is a hoot!

Recipes

I look in my fridge and pantry, tell it what I have and what I’m thinking (comfort food, healthy food, low carb, adherent to Mediterranean diet, to make it Indian-ish) and it whips up several options. Some dishes have been outstanding, some have been a bust, but it gives great inspiration and it is much easier than recipe websites that make you scroll through a bunch of crap before getting to the actual recipe. The prompt I used was: “I have chicken thighs, garlic, tomatoes, cous cous, asparagus and usual pantry staples. Can you come up with a few different recipe options? One comforting, one healthy, one keto-ish. If I need one or two items, that's okay to suggest.”

Work-outs

I’ve been on a workout kick for the last year. I am trying to gain muscle and bulk a bit. A recent prompt: “I can bench press 155 lbs x 10 times maximum. Using that, can you give me a push workout that I can complete at Planet Fitness in about 45 minutes.” They are pretty good and if I keep it in the same chat thread and input the weight and reps I did, it can continue to optimize the workout.

Travel

This is pretty great. I’ve used it quite a bit to help with the framework of a given trip. Here is one that I am working on for our January trip to Thailand. I then look up suggestions, cross check with Reddit and other sources and ask friends. It can also help with logistics - i.e. with kids, need to know where restaurants are in relation to the activity, etc. The hotel suggestions have been quite good, as well, especially if you say what you want.

Some of this is probably interesting, some of it is silly and some of it you think you may just not need. But, I have barely scratched the surface of what it can do. I find myself trying to use it for at least one hour a day. Whatever I’m trying to accomplish, I see if I can utilize AI to speed things up, improve the quality of what I’m doing, improve accuracy, find solutions to problems and overall, just experimenting.

I sound like a broken record, but we’re living in an exciting and scary world. The best way through will be to learn it the best you can and stay informed. I’ll do my part. If you have suggestions for how I can use it better, please do let me know! I also gave a short presentation to my practice that was well received. If you are interested, just email me and I will send it out.

Love you all,

Sim