How to bring hospitals out of the dark ages with screen capture

Anyone who has sat beside an ICU bed knows the bedside monitor: heart rate, respiratory rate, oxygen saturation, blood pressure, scrolling past in real time. You assume all of it is being recorded somewhere.

So where does that heart rate go once it leaves the screen?

Most of the time: nowhere. A nurse reads the numbers off the central station or the bedside monitor and types them into the electronic health record by hand, typically every four to eight hours on a general ward^[1][1]AHRQ PSNet, 2023. Nurses spend more than a third of their working time on documentation, nearly twice what they spend on direct patient care^[2][2]Hendrich et al., Permanente Journal, 2008. The monitor measures continuously; the chart gets a handful of values per shift.

Getting the data out, it turns out, is close to impossible. We spent a year finding that out the hard way. This post is about what we tried, what failed, and why the answer was taped to the back of the monitor the whole time.

A continuous heart rate curve over four hours; only two hand-charted values survive. — **Figure 1:** What the monitor sees, and what the hospital keeps. Everything in between fades out.

**Figure 1:** What the monitor sees, and what the hospital keeps. Everything in between fades out.

The bedside monitor sees every heartbeat, but the hospital keeps only the values that get charted, and deterioration hides in the gap between the two. Once those values are in the chart, everything in between is gone: ask what a patient’s heart rate was doing two hours ago, and the only answer is whatever someone happened to write down. A heart rate of 110 that has held steady for an hour and a heart rate of 110 that climbed from 75 in twenty minutes look identical. The monitor saw the difference, but the chart cannot show it.

The early warning signs are slow and subtle, and they fall between the charted values: a UK national enquiry found warning signs before 75% of in-hospital cardiac arrests and judged 38% of those arrests potentially avoidable^[3][3]NCEPOD, Time to Intervene?, 2012.

The gap has three costs. Nurses still spend hours of every shift entering data by hand. Clinicians miss slow deterioration, because a heart rate creeping up while blood pressure slides down over an hour stays invisible until the patient is already in trouble. And clinical AI has nothing to learn from and nowhere to run.

The state of clinical AI

Clinical AI is supposed to fix this. Every frontier lab has a team building models to detect the early onset of conditions like cardiac arrest, stroke, or sepsis far earlier and more precisely than any human could, simply by picking up subtle deviations in a patient’s physiological data.

Thousands of these models have been built, and in retrospective tests, many of them match or outperform clinicians^[4][4]Hannun et al., Nature Medicine, 2019. Yet almost none of them ever reach a patient. A review of 494 ICU AI studies found that only 2% had been tested in clinical practice, and exactly zero were in routine care^[5][5]van de Sande et al., Intensive Care Medicine, 2021.

Figure 2: Of 494 AI studies, only 10 reached clinical testing. Bars drawn to scale.

Why? Because without live, continuous data, the whole field is paralyzed.

First, the models have almost nothing to train on. The best dataset in clinical AI is MIMIC-IV, built by MIT and a Harvard teaching hospital. It gives researchers one vital sign every 15 minutes^[6][6]Johnson et al., Scientific Data, 2023 (MIMIC-IV). And most models don’t even get that. They train on hourly charting. Wearables don’t help either, because they don’t record clinical outcomes. From a smartwatch alone, an AI can’t tell the difference between a hot yoga class and a cardiac arrest.

Second, even if you build a perfect model, there is nowhere to deploy it. Because hospitals don’t have live data feeds or platforms for these models to run on, the most advanced clinical AI in the world usually just ends up sitting in a GitHub repo.

When a model does actually reach a patient, the results are incredible. Palantir deployed a sepsis prediction model at Tampa General that saved 886 lives and cut sepsis deaths by more than 68%^[7][7]TechSpot, citing The Times, June 2026. But that required a massive, custom integration built specifically for that one hospital. Everyone else is stuck.

We know this firsthand. Our co-founder Tim worked night shifts as a paramedic to pay his way through college. During one shift, he nearly killed a patient. He was sure the patient had a stroke, but a far more experienced colleague had a gut feeling it was something else, and made a different call. It was sepsis. That colleague is the only reason the patient survived.

Tim then spent four years building what was, at the time, the largest labeled clinical AI dataset in the world. He called every frontier lab working on clinical AI and asked for their data. Then he trained one of the most powerful sepsis models on top of it.

But when we pitched a leading US medical center to deploy our model, their Chief AI Officer told us they couldn’t even deploy their own stroke and cardiac arrest models, on which they had already spent millions. All they do is retrospective studies: going back through hand-charted records to work out who could have been saved.

That is how almost all clinical AI research works today.

A clusterf*** of legacy shi*

So how do we fix it?

The problem is that every hospital is a zoo of hundreds of devices, from dozens of manufacturers, spanning countless hardware generations. There is no single API. In fact, nearly all of those devices were built before AI even existed, with no API at all.

A grid of device types by vendor, each with a different interface and generation. — **Figure 3:** One ICU. Dozens of manufacturers. Countless device generations.

**Figure 3:** One ICU. Dozens of manufacturers. Countless device generations.

There MIGHT be a small incentive for manufacturers not to open up devices that carry high margins and offer next to nothing else to differentiate them. That might lead to a walled garden or two. Just a thought.

Because of these walled gardens, getting data out of hospital monitors means building custom, per-vendor software integrations. They are notoriously slow, expensive, and don’t scale beyond a single vendor. Integration runs $6,500 to $10,000 per bed, plus about 15% a year in maintenance^[8][8]West Health Institute, 2013. For a mid-sized hospital, that’s well over a million dollars just to get the data out.

Replacing the monitors isn’t realistic either. New equipment costs $15,000 to $35,000 per ICU bed, and the installed fleet stays in service for seven to ten years^[9][9]MedEquip Directory buyer guide, 2026.

So how do you bypass a million-dollar walled garden? We turned the monitors around and looked at the ports.

Turn the monitor around

Figure 4: Everything we tried is on this panel (illustration).

We spent a year trying to get the data out the hard way, evaluating every port on that back panel.

Diagnostic ports. These are built for diagnosing hardware faults, not extracting high-frequency data. They differ for every vendor and generation, manufacturers don’t provide documentation unless you pay their service technicians, and the data resolution varies wildly. It doesn’t scale.

Ethernet. The monitors already send everything to the central station over the network. Accurate, but the protocols are closed, undocumented, and encrypted. Reverse-engineering each one is the integration problem all over again.

Man-in-the-middle. We could try to intercept the encrypted packets, but that requires briefly logging every medical device off the hospital network to route traffic through our servers to catch the keys. Hospital IT politely declined to let a startup do that to live medical devices connected to real patients, as it would also kill the warranty.

Route	What we found	Verdict
Vendor API	Does not exist for live streaming on most installed devices	Dead end
Diagnostic ports	Per-device, per-generation protocols; resolution varies; compliance implications of attaching to a regulated device	Doesn’t scale
Ethernet	Closed, undocumented and encrypted protocols, different for every vendor and generation	Accurate, not scalable
Man-in-the-middle	Requires logging every device off the network and routing medical traffic through a separate server; would kill the warranty	Dead on arrival

Table 1: Routes we evaluated.

Every conventional route was dead on arrival. Which left one thing on the back panel we hadn’t taken seriously.

Maybe… just read the screen?

The screen is the one interface every monitor already standardizes, because it has to: it was built for humans to read. Better yet, the central monitoring station shows every patient on the ward at once, and it has an HDMI cable going right into it.

So we figured, if we just screen-capture the visual feed and train a custom AI to extract the physiological data locally, we could bypass the device networks entirely. We would get all the relevant clinical information in a way that is scalable, inexpensive, and highly reliable.

Central monitor, passive video tap, local vision model on the device, structured stream per bed. — **Figure 5:** From the central monitor’s video output to a structured, timestamped stream per bed.

**Figure 5:** From the central monitor’s video output to a structured, timestamped stream per bed.

Let’s get to work!

Does it work?

To test the thesis, we built a rig: a Raspberry Pi with a screen capture card.

A Raspberry Pi with a capture card on a desk next to a laptop. — **Figure 6:** The first rig.

We trained computer vision models to extract all the physiological data from the central screen and deployed them on the Pi. To keep from frying it, we took the monitor’s video stream via the capture card, cut it into single frames, and let the AI process them one by one. Amazingly, we could run all the compute directly on the edge.

Four steps: the raw frame, bed regions found, fields found and typed, and the structured output table. — **Figure 7:** One frame in, one structured row per bed out.

**Figure 7:** One frame in, one structured row per bed out.

After some mixed initial results, we scraped YouTube for instruction videos of central monitors to generate more training data. We fine-tuned the models on that footage and pointed them at some test stations. The data came out clean!

This approach has exactly one boundary: we capture only what is displayed on the screen. But it turns out that’s pretty much everything clinicians and AI models need to make decisions.

A central station with twelve beds; every value on screen is boxed and read by the model. — **Figure 8:** A clean central station screen, every value read.

**Figure 8:** A clean central station screen, every value read.

Reliably, at scale?

A proof of concept on YouTube isn’t scale. When we left the safety of YouTube and applied the models in harder clinical dojos, the first models fell apart on stations they hadn’t seen. Even for 2D image detection with CNNs, we hit all the classic computer vision edge cases that killed our performance:

Image quality (smearing, artifacts, compression)
Layouts that change randomly between vendors and wards
Alarm overlays physically covering the values, missing readings, and interrupted feeds
Matching the right patients to the right vital signs

A blurred feed where 130/83 was read as 130/88. — **Image quality.** A smeared, compressed feed: the blood pressure 130/83 read as 130/88.

**Image quality.** A smeared, compressed feed: the blood pressure 130/83 read as 130/88.

To fix this, we needed way more training data. But clinics, understandably, don’t exactly hand out thousands of screenshots of protected health information for startups to train AI on.

So we built our own ICU in our garage.

The garage with boxed hospital beds and wall panels on the floor. — **Figure 10:** The garage, mid-build.

Then we built 50,000 more (synthetic ones).

A base station and six variations: vendor layout, color scheme, font and digit style, bed count and grid, alarm overlays, image degradation. — **Figure 11:** Six of the dimensions we vary. Every synthetic station is a new combination.

**Figure 11:** Six of the dimensions we vary. Every synthetic station is a new combination.

We forced the models to learn on these generated screens. Here are the exact same failure cases from above, now read correctly:

The same blurred feed, 130/83 read correctly. — **Image quality.** The same feed: 130/83, read correctly.

**Image quality.** The same feed: 130/83, read correctly.

Three cases recorded as missing: a blank field, a value covered by an alarm, and a lost feed. — **Figure 13:** When a value isn’t readable, it’s recorded as missing, not guessed.

**Figure 13:** When a value isn’t readable, it’s recorded as missing, not guessed.

Then we dialed the difficulty up to max. We generated stations so glitched, compressed, and chaotic that it became hard for us to even read the images quickly.

The same reading progressively destroyed: clean, downscaled, heavy compression, motion blur, glare and moiré, all combined. — **Figure 14:** Sometimes it became hard for us to even read the images quickly.

**Figure 14:** Sometimes it became hard for us to even read the images quickly.

Meanwhile, as we threw heavier models at the problem and increased capture speed, the Raspberry Pi quickly hit its computational limits and overheated. We had to go into the software and gut it: we removed the OS except for core components, optimized two specific drivers, and aggressively balanced the power consumption. After that, it ran completely stable.

That kept the Pi alive, but it wasn’t enough for scale. Now, our Gen2 node is production hardware with a dedicated NPU to accelerate AI inference, more RAM, and upgraded cores and GPU. Our Pi proved that we could capture the data, but Gen2 gives us the compute to run actual clinical AI models directly on the edge.

System monitor on the Pi: CPU at 99.5%, 86.4 °C, throttled. — **Figure 15:** The Pi under load (left) and the Gen2 node (right).

**Figure 15:** The Pi under load (left) and the Gen2 node (right).

We had our Gen2 node.

The accidentally massive dataset

Once the pipeline was stable, we realized how much data we were actually pulling.

Because our node reads the screen, the math gets ridiculous fast. The central screen updates roughly every second. Most models today train on hourly charting, so that’s 3,600 times more data than what they learn from. Even against MIMIC-IV, the best dataset in the field, it’s 900 times more^[6][6]Johnson et al., Scientific Data, 2023 (MIMIC-IV).

Figure 16: Left, one charted data point an hour. Right, the live feed: every vital, every bed, every second.

MIMIC-IV compared with a projected year of 40 Sentis nodes: patient counts, and values per patient per hour. — **Figure 17:** Sentis vs. MIMIC-IV.

Historically, this kind of physiological data has been almost impossible to get. But because we deploy one node per 16-bed ward, our scale is totally different. With just 35 nodes running for a year, we’d have the highest-resolution clinical dataset in the world.

The Standard API for Health

Because our node only reads the screen, it strips away all the vendor-specific chaos. It doesn’t matter if it’s a Philips or a GE monitor, a brand-new device, or a ten-year-old brick. The data always comes out at the exact same resolution and in the exact same structure for every single vendor and generation. A node that works in Texas works anywhere in the world a central station has a video output.

We realized we hadn’t just built a data extractor. We had built the Standard API for Health.

Vendors won’t build this, because a vendor-neutral API is the last thing they want. And we’ve filed a patent covering the capture pipeline and the deployment platform.

This breaks open how clinical research works. Right now, physiological research is painfully small. If a researcher or a pharma company wants high-frequency data, they are usually restricted to a single ward at best, often just a couple of bed spaces. The most massive clinical studies today might span two sister hospitals, and only if someone spends millions of dollars on custom IT. It is incredibly hard to get good, continuous data for clinical trials and medication studies.

With one standard API, that flips. Each new site is one node. Every site speaks the same API, so a study that runs in one hospital can run in fifty, across several countries at once. You can run trials that reach thousands or millions of patients at the same time. It moves clinical research to a planetary scale.

The Terminal: where AI actually runs

But having the raw data stream is only half the battle. You need somewhere to actually put it, interact with it, and deploy against it.

So we built the Terminal: a next-generation central monitoring station.

For the nurses on the floor, it means two things. First, they can see every value the monitor has shown since the patient was admitted. Slow deterioration can’t hide between charting rounds anymore. Second, they don’t have to spend a third of their shift writing this sh*t down by hand. The Terminal is built to write the stream straight into the EHR, so nurses stop typing it in.

But the Terminal isn’t just a screen for humans. It’s the deployment infrastructure for AI.

Right now, if you build a new sepsis or stroke model, you have nowhere to put it. With the Terminal, you can train your models at scale on our high-resolution API, and then deploy them on the exact same infrastructure.

You can run a new model in shadow mode on the Terminal to validate it against live, high-resolution data. And once it’s proven, you can take it live on the same platform, at every site that runs a node. For the first time, clinical AI models can be trained and deployed on one platform, with real-world impact.

What now

We’re live in a hospital in Texas, deploying and iterating quickly, and an academic medical center is running a ground-truth validation study on everything we read.

Because the data is so clean, here is what users have asked us for so far:

One central screen with all patients plus their high-res time series
Data automatically written into the EHR, saving nurses a ton of time
An API for researchers and pharma to run studies across multiple sites
A platform to train and deploy AI models at scale, plus smart alarms

Reach out if you’re:

An AI researcher with a model that deserves real-life data
A physician who wants to see how their patients got here
An ICU head who wants trajectories, not snapshots
An AI lab that wants real clinical data

Questions? Reach out to tim@sentis.co.

Where this goes

Every frontier lab is building health AI: models predicting diseases by finding subtle patterns in the body. Today, they train on hand-charted records with a few values per shift.

We are building the highest-resolution record of the human body.

Our goal is to build the best health AI models in the world and bring them to every human.

We start in the ICU, as it gives us the highest-impact data. Every node we deploy adds to it.

Afterward, we go to the wrist. Wearables already record the same vital signs clinical models train on. But they don’t know what happened to the patient, so they can’t tell a stroke from a cardiac arrest or a yoga class. It’s impossible to train on that data, but it’s possible to deploy on it.

We want to predict every cardiac arrest, in every human, in the world.

The team

Sentis was founded by Tim Topper and Christin Seltmann. Tim is a former paramedic who nearly lost a patient to a misdiagnosis and has spent the four years since building what became the SentisNode architecture, with stints at Amazon, Columbia, and Stanford along the way. Christin is a third-time founder with one exit, a Harvard MBA, and a background at Siemens and McKinsey.

A big thanks to our backers:

Shri Ganeshram: MIT-trained serial founder and YC alum. FlightCar co-founder/CTO, second employee and SVP Engineering at Eaze, Forbes 30 Under 30; seed investor in Akido Labs.
Samuel Stanton: Co-founder of Coefficient Bio, acquired by Anthropic eight months after launch; ex-Genentech machine learning, now on Anthropic’s technical staff.
Hyde Patterson: Healthcare VC at Longitude Capital, which closed a $585M fund for biotech, medtech and health solutions; Harvard biology, ex-Bain, ex-Forethought growth lead; now CEO of a stealth startup.
Prof. Dr. Steven Hildemann: Global Chief Medical Officer of Merck KGaA’s Biopharma business 2014 to 2018, heading Global Drug Safety and medical governance; board-certified cardiologist and Professor of Medicine at the University of Freiburg.
Dr. Stefan Oschmann: CEO and Chair of Merck KGaA 2016 to 2021, previously CEO of Merck Biopharma and a long-time Merck & Co executive; now Chair of UCB and board member at Reckitt and Springer Nature; former EFPIA and IFPMA President.
Jason Haider: Founder and CEO of Xenco Medical since 2011, which commercialized the first fully disposable spinal implant system into hospitals and ambulatory surgery centers nationwide; World Economic Forum and MIT Sloan advisory boards.

References

McGrath S, Blike G, Gale B, Mossburg SE. Surveillance Monitoring to Improve Patient Safety in Acute Hospital Care Units. AHRQ PSNet, 2023. Link ↗
Hendrich A, Chow MP, Skierczynski BA, Lu Z. A 36-Hospital Time and Motion Study: How Do Medical-Surgical Nurses Spend Their Time? The Permanente Journal, 2008. Link ↗
Findlay GP, Shotton H, Kelly K, Mason M. Time to Intervene? A review of patients who underwent cardiopulmonary resuscitation as a result of an in-hospital cardiorespiratory arrest. NCEPOD, 2012. Link ↗
Hannun AY et al. Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nature Medicine, 2019. Link ↗
van de Sande D et al. Moving from bytes to bedside: a systematic review on the use of artificial intelligence in the intensive care unit. Intensive Care Medicine, 2021. Link ↗
Johnson AEW et al. MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data, 2023. Link ↗
TechSpot, June 10, 2026, citing The Times: Tampa General's Palantir-based sepsis system. Link ↗
West Health Institute. The Value of Medical Device Interoperability. 2013. Link ↗
MedEquip Directory. Patient Monitoring Systems Buyer Guide: ICU, Telemetry and Remote. 2026. Link ↗

Flying blind