Newdata out of Stanfordis lending credence to common wisdomthat fitness trackers suck at counting calories. Heart-rate monitoring, on the other hand, fared surprisingly well in the schools Medical Centerstudies.
The group of 60 volunteers was hand-selected to offer a broad range of subjects, factoring in things like gender, body mass index and skin color (optical heart-rate monitoring has a history of being a bit unintentionally racist). The 31 women and 29 men tested various combinations of the Apple Watch (first generation), Basis Peak, Fitbit Surge, Microsoft Band, Mio Alpha 2, PulseOn and Samsung Gear S2 while walking on treadmills and ridingstationary bikes.
Those results were compared to data from medical-grade equipment for measuring heart rate and carbon dioxide in the breath, a commonly accepted measure for gauging energy expenditure. Turns out all of the devices other than the Samsung did a pretty solid job on the heart-rate front. The rest had error rates of under five percent, which qualifies them as decent candidates for medical uses.
Theyre not going to replace chest straps in the lab any time soon, but the kind of day to day data they collect could be useful for doctors looking for a more complete picture of their patients health. At the very least, they could provide some interesting supplemental information.
Five percent is approaching something useful in a clinical setting, which suggests that doctors may be looking toward this data when evaluating their patients, graduate student Anna Shcherbina told TechCrunch. If a doctor has average data on the patient as theyre going about their day, this could provide a more complete picture.
Caloric information is a different story altogether. And its not surprising, really. Heart rate is a pretty direct reading. Calories burned differ hugely from person to person based on a variety of different factors. And the wrist isnt thelogical spot to start a search for caloric information. The system needs to do a fair amount of guesswork based on movements detected by the wearables built-in accelerometers.
The range of readings is pretty astounding. The best reading was off by an average of 27 percent and the worst by an appalling 93. To be of any real use for a medical setting, youre going to want a number under 10 percent. So theres a lot of work to be done here. Shcherbina says the Apple Watch and Fitbit Surge were the most accurate of the bunch; it was Finnish companyPulseOn really missed the mark.
The team unsurprisingly got some pushback from PulseOn when it didnt enter more precise information like a volunteers VO2 max level. In order to get a more accurate picture of the average user, the researchers didnt enter information that most of us dont have access to. I know I havent been on a treadmill with a VO2 measuring mask on my face any time recently. Perhaps Im overdue.
Since the study began in 2015, there are newer versions of a number of the devices here. The team admitsthat, in the case of the Apple Watch, there may be some discrepancies between active and total calories burned, an issue that may be reconciled as it continues testing on the new version.
The team purchased all devices independently for the integrity of the study but lead researcher professor Euan Ashleysays theyre open to communicating with the companies involved to make sure theres not some sort of proprietary secret sauce somehow being overlooked.
Both PulseOn and Apple have reached out with feedback, after the papers publication, and the team is open to future collaboration. No concrete plans as of now, Shcherbina explains, but we are definitely open to this idea if the companies are interested in working with us. Obviously maintaining our independence/impartiality is key, but closer communication might yield valuable insights about most effective ways to improve fitness algorithms for wearables.
Apple and Samsung declined to offer a statement, while Fitbit is clearly on the offensive here, sending along a big one. The wearable maker touches upon its extensive, ongoing research and development, but adds that the caloric estimates that trackers give are just that estimates:
Fitbit trackers show an estimated total number of calories burned based on users BMR (basal metabolic rate) and activity energy expenditure (AEE). Fitbit uses a scientifically validated estimate of BMR based on height, weight, age, and gender information that users provide when setting up their Fitbit account.
The company also argues that theres something to be said for the base-level motivation that simply wearing a tracker brings. Its a fair point. If the sort of gamification that these devices bring does get more people off their butts, thats probably a net positive and a big part of whats driving the categorys success. I know plenty of people (myself included) who find the simple act of wearing one of these devices on their wrist cause to get off their respective butts.
Shcherbina agrees with the point, but adds that theres a big downside for those who rely on such readings to make very precise decisions about exercise and food based on that, then it might be off by quite a bit and they may not get the results they want and become frustrated.
As more health providers look to trackers for data, the category is likely to come under increased scrutiny from regulators like the FDA. As it currently stands, however, that body doesnt oversee any sort of claims madefor these non-medical products.
Our position is that sunlight is better than regulation, says Ashley. We would encourage the companies to make available their validation studies in the public domain so the public can see for themselves which devices are most accurate.
At the very least, the study prompts some interesting questions about how companies collect and process the data that informs its health claims. The data collected may not be a perfect reflection of manufacturers intentions, butthe study is aplace to start the conversation.