Pulse Oximeter Accuracy: Bench vs Human Tests
A study ran 12 pulse oximeters on a bench tester and on people. Three of the six that passed the bench test had an error above 3% when tested on people.
Some pulse oximeter accuracy claims come from a bench tester: a box with an artificial finger that sends the oximeter a simulated signal. It's quicker and cheaper than testing on people.
A study from the University of California, San Francisco ran 12 oximeters both ways. The bench results didn't predict the results on people.
What's a bench tester?
The paper calls it a functional tester. It sends a made-up light signal that mimics a pulse at a set oxygen level, so an engineer can check the oximeter's electronics work.
The study used the Fluke ProSim 8, the one most often used in published oximeter studies. Its own manual says it isn't for calibrating medical equipment. The paper adds that the ISO standard for pulse oximeters says these testers can't be used to assess SpO2 accuracy.
The authors say they're often used for that anyway, including by aid organisations choosing which oximeters to buy after COVID-19.
What was tested?
- 12 oximeters: 11 fingertip models costing $20 to $299, and one handheld unit costing $5,999.
- On the bench. Two tests on the Fluke. The first stepped the simulated SpO2 down from 100% to 70%. The second held it at 90% and weakened the signal across 60 settings.
- On people. Healthy adults aged 18 to 45 breathed low-oxygen air in a lab, and each reading was compared with an arterial blood sample. Every oximeter was tested on at least 24 people, with at least a quarter in each of three skin tone groups.
Accuracy was scored as root mean square error, one figure for how far the readings sat from the reference. A pass was 3% or less, the FDA bar. For what that figure means, see pulse oximeter accuracy.
Did the bench test match the human test?
No.
- 6 of the 12 passed the first bench test.
- 3 of those 6 had an error above 3% on people: the Bodymed BDMOXMTRBLK (3.26%), Roscoe POX-ROS (4.3%) and Biolight M70 (5.08%).
- Those were the only three that missed the bar on people, so the first bench test passed all of them.
- The Biolight M70 had the largest error on people (5.08%) and the smallest of the 12 on the tester's default curve (0.96%).
- The Nonin CO-Pilot had the smallest error on people (1.39%) and failed the bench test (3.68%).
- At a simulated 70%, 8 of the 12 read more than 3 points too high.
The paper reports no statistically significant relationship between the two sets of scores, whichever curve was used. The abstract gives an R squared of 0.08, where 1 would be a perfect match.
The human-test figures are the latest in the lab's Open Oximetry database at publication, and the paper says the database is still being updated. Some differ from the lab's 2024 test of 11 fingertip models. The CONTEC model was at 4.3% there and is at 2.82% here, and the Zacurate 500C was at 3.4% and is at 2.51%. The paper doesn't say why.
What about weak signals?
The second bench test checked how each oximeter coped as the signal faded, as it can with poor circulation or a thicker finger.
- With a strong signal, every oximeter stayed within 3 points of the set value.
- With a weak signal, most either gave no reading or were more than 3 points out.
- On average an oximeter gave no reading in 20.2 of the 60 settings. The range was 8 to 36.
- 10 of the 12 passed this test. 2 of those 10 had missed the bar on people.
- Of the three that missed the bar on people, this test flagged one.
- These scores didn't line up with the scores on people either (R squared of 0.01).
Did the calibration curve matter?
Each oximeter turns its light measurements into an SpO2 value using a calibration curve, and the tester has to assume one. The Fluke had the maker's own curve for only three of the 12. The rest were run on its default curve.
Those three were run both ways:
- Masimo MightySat Rx: 1.64% on the default curve, 0.19% on its own.
- Nonin Onyx Vantage 9590: 2.61% and 0.42%.
- Nonin CO-Pilot: 3.68% and 3.49%.
None of the three changed from pass to fail or back.
What can't it show?
- It used one bench tester and 12 oximeters. Other testers and other models may behave differently.
- Nine of the 12 had no matching curve on the tester, so the authors say their bench figures may not reflect how they perform.
- The people were healthy adults, whose signals are stronger than the weakest ones simulated on the bench. The authors say that limits how well the two can be compared.
- No ring or wrist oximeters were tested.
Who paid for it? The oximeters were bought with foundation grants to the Open Oximetry Project. The lab also receives funding from pulse oximeter makers to test their devices. One author works for Medtronic, and another has helped develop simulators for companies including Fluke.
What can you do with this?
- A claim that an oximeter was "tested on a simulator" tells you its electronics respond. It doesn't tell you how close its readings are to your blood oxygen.
- Look for accuracy figures from tests on people against arterial blood. Here's what to check before buying an oximeter.
- Even a good result on healthy volunteers is an average. Read your own numbers as a trend over many nights, not as an exact value.
- Skin tone can affect readings too. See pulse oximeters and skin tone.
Not a medical device. Does not diagnose, treat or prevent any condition. Discuss results and changes with a clinician.
References
- Elmankabadi S, Dove J, Behnke E, Chou YC, Ortiz L, Leeb G, Auchus I, Chen D, Feiner J, Law TJ, Bickler PE, Hashi S, Zamora RV, Negussie F, et al. Comparing pulse oximeter performance using a common functional tester versus controlled desaturation studies on healthy participants. Journal of Clinical Monitoring and Computing. 2026;40(2):457-468. doi:10.1007/s10877-025-01381-0