Before You Use AI on Real World Data: a practical check for MedTech teams

I like what AI promises. I also know where it struggles.

Real world data can help teams see how a device performs in everyday care, yet it is messy, uneven and scattered. The most common mistake is to jump into algorithms before the basics are in place. This note is a simple, honest check you can run before you switch anything on. I will also point to where Milo helps, because a good platform should remove friction rather than add it.

Start with permission, not with models

Most real world datasets were collected for care or billing, not for research. That is fine if you confirm a lawful basis for secondary use. Do patients have consent that covers analysis? If not, does your jurisdiction allow secondary use with the right safeguards? Put your data use or data processing agreements in place at the start, not in week eight. Milo helps by tracking consent status, storing the agreements in context and applying de-identification during import. That way you are confident about the ground you are standing on.

Check that the source is stable and queryable

An AI pipeline is only as good as the system feeding it. Ask who owns the source, how often it updates, and how you will know when fields change. A hospital registry upgrade can break a link and no one notices for a month. In Milo, every import is time stamped and versioned, so you can see when a feed changed and what moved with it. That visibility sounds boring. It is also what prevents long weekends of repair.

Make sure the data is marked for the outcome you care about

If you plan to detect adverse events, those events must be recorded in a consistent way. If you plan to show performance, the registry must hold the outcome that proves it. Run a small labelling pilot with a few sites before you scale. It saves money and reputation later. Milo lets you run these pilots inside the study space, keep the notes on what worked, and promote only the fields that pass the test.

Balance matters more than size

It is easy to celebrate a large dataset and miss who is in it. Claims data may lean toward people with insurance. Sensor data may lean toward younger and more engaged users. Notes from one region may use language that does not travel. Check representation early. Compare basic demographics and care settings across sources. Milo surfaces simple balance checks on import so teams see skew before they build on it. Fixing a bias at the start is cheaper than explaining it to reviewers later.

Keep a clean trail of what changed and why

Analyses evolve. You add a site. You tighten a filter. You retire a draft method and try another. None of this is a problem if you can explain it. Keep a short note for each change and tie it to a date and a data slice. In Milo, every step is logged and linked to the evidence lake, so a reviewer can replay what you did without hunting across folders. This is not about box ticking. It is about trust.

Common traps to avoid

Do not rely on proxy endpoints without checking the clinical story

A billing code may confirm that a procedure happened, not that the patient improved or declined. Pair codes with clinical notes, labs or imaging when safety or benefit is on the line.

Do not ignore the denominator

Imagine ten alerts in a month. If twenty patients used the device, that is a warning. If ten thousand patients used it, that may be expected. Always know how many patients used the device, when they started, and how long they were followed. Milo helps by linking usage logs with registry entries and timestamps so rates are honest.

Watch for drift

A model that works well on one archive can stumble on another because scanners, populations or workflows differ. Schedule routine checks on fresh data. Milo makes these checks easier by keeping prior runs and new runs side by side with the same metrics, so teams compare like with like.

Be clear when AI insights drive a change

If an analysis leads you to adjust instructions for use or follow-up schedules, tell regulators early and show the evidence chain. Milo ties insights directly into risk registers, safety reports and PSUR templates so that change control is visible and complete.

Where Milo fits in the day to day

Milo is first a smart system for capture and monitoring. It brings eCRF, ePRO, imaging and device logs into one evidence lake. It supports small labelling pilots, checks for bias and completeness on import, and keeps a full audit trail. There is a secure sandbox to explore unstructured notes or images without touching the live dataset. When you are ready to report, outputs link straight into risk files and safety reports so nothing is lost in handover.

If you want to see your own data flow through this path, start in a sandbox. It is the fastest way to learn what works, what needs fixing and where AI genuinely adds value.

You can reach the team here and we will set it up with you: https://milo-healthcare.com/en/