Measuring or marketing?

There has been rapid development in atmospheric sensing technology over the past ten years, with a large number of sensors entering the market at very low cost. These sensors have, in turn, been incorporated into any number of consumer-level products such as²:

• Foobot
• uHoo
• Keepalert
• Netatmo
• Airthinx
• Air Quality Egg

But what evidence is there to suggest these devices actually measure what they purport to measure? There’s lots of marketing spiel³ on these websites, but a paucity of information on device accuracy. The primary purpose of buying something that measures indoor air quality should be that it does just that. That doesn’t mean something laboratory grade, but it does mean something that is sufficient to give some assurance that it’ll be fit for purpose. Marketing platitudes aren’t sufficient evidence.

It’s important to understand these consumer-level devices are basically integrations of sensors developed by others. There is very little the device developer can do to influence the accuracy of these underlying sensors, particularly as they can’t perform a site-specific calibration for every unit sold. These sensors are not made by Footbot, uHoo et al but rather are acquired from specialist sensor manufacturers.⁴ The packaging, web interface and smartphone integration can be very fancy - but these have nothing whatsoever to do with how well the device measures air quality.

Many of these devices claim to be able to measure carbon dioxide (CO₂), total volatile organic compounds (TVOCs) and particulate matter (PM). This post is concerned only with CO₂. While all three are difficult to measure, CO₂ is probably the “easiest.”

How can we test these gadgets?

This is challenging, and may be part of the reason why the suppliers seem so reticient to provide data on their system accuracy. The difficulty arises in not having access to laboratory-grade measurement equipment, or samples with known gas concentrations, to compare against. The only known CO₂ concentration to which we have ready access is the atmosphere, at a little more than 400 ppm. This is fine for a baseline calibration, but doesn’t really tell us much about the sensor sensitivity. So in the absence of this we’re limited to comparing devices against one another, never being quite sure which (if any) is correct. While this is far from perfect, it’s probably adequate for the purpose of indoor air quality monitoring anyway - what we want to know is about the general quantum, and how this changes over time.

Methodology

We’ve acquired a number of sensors (usually installed on breakout boards) that are commonly used in consumer-level air quality monitors to measure CO₂, VOC and PM. All the sensors are low cost (usually less than $100 in small volume, and in many cases less than $20). The sensors are all connected to a Particle Photon microcontroller. They were programmed and burnt-in in accordance with the manufacturers’ technical documentation (including, in many cases, being exposed to outdoor air), but not calibrated. The devices were then left to run for several days indoors in a home office with mechanical ventilation operating but windows closed. The sensors were left powered on and sampled for a single reading once a minute.

Sensors

There are two distinct detection principles used by CO₂ sensors: non-dispersive infrared (NDIR) and metal oxide semiconductor (MOS). The operating principle can be found online. What is relevant here is that NDIR sensors attempt to directly measure CO₂ concentration whereas MOS sensors instead detect other gases (usually VOCs, but also hydrogen and sometimes other gases), and then use these as a proxy for CO₂ concentrations. The relevance of this distinction will become important shortly. Further, the NDIR devices differ insofar as one (the Sensiron SCD30) has a dual wavelength sensor whereas the Senseair K30 has only a single wavelength sensor. This isn’t relevant for the present results, but the dual wavelength design reduces drift on the sensor over time. By comparison, single wavelength sensors need to be exposed to a known baseline CO₂ concentration (usually atmospheric concentrations) occasionally so they can recalibrate themselves. This may or may not be possible in an indoor environment.

Results

Two days worth of results are shown in Fig. 1. There are two distinct groups:

• the two NDIR devices (Senseair K30 and Sensiron SCD30) have fairly similar (and plausible) readings, and
• the two MOS devices measure much, much higher values and the trends seem far less plausible.

The NDIR sensors seem to respond similarly, which is encouraging. Furthermore, the differences between their readings is not huge (around 60 - 70 ppm) and remains fairly consistent. This variation is likely to be no practical significance for indoor air quality monitoring, and in any case can be readily calibrated out should the sensors be exposed to a known CO₂ concentration. It’s the MOS sensors which are a real worry - if we were to take these sensor values at face value we’d likely conclude we have poor air quality.⁵ But the NDIR sensors are suggesting much lower values, never above 930 ppm and averaging below 700 ppm. Given the number of building occupants and presence of mechanical ventilation the latter results is the much more credible result.

It is reiterated at this point that neither of the MOS sensor manufacturers claim they measure CO₂, rather their datasheets are clear that they are measuring wider gas concentrations and reporting a CO₂-equivalent value. For example, the SGP30 datasheet states that it is measuring hydrogen concentration and using this as a proxy for CO₂. It’s a shame the consumer-level devices using these sensors appear to have forgotten to mention this rather pertinent fact.

Figure 1: Temporal correlation for CO2 readings

To better understand whether the trends are consistent betweeen the sensors we can examine the correlation. The correlation in CO₂ readings between the NDIR sensors is excellent, with an R² of 0.997 and little apparent noise (Fig. 2). The iAQ-Core appears the better of the MOS sensors when correlated to the NDIR sensor values (R² of around 0.67). The SGP30 on the other hand seems to have very poor correlation with any of the other sensors.

Figure 2: Sensor correlation for CO2 readings

It could be argued the absolute CO₂ level isn’t so relevant if we were looking to use the measurements for demand control, say of a ventilation system. In such situations all we’re really after is for the sensor to respond appropriately - that is, to go up when CO₂ levels rise and down when they decrease. There is some evidence to suggest the MOS sensors may be reasonably effective at performing this role (Fig. 1); when the NDIR sensors suggest an increase in CO₂ in most cases so too do the MOS sensors. But the latter response seems to be a fair bit noisier, particularly from the SGP30. This points to the intended purpose of these MOS-based “CO₂-equivalent”" sensors, of providing a cheap means of demand control. For this purpose they may be sufficient. But using these sensors as a means of evaluating indoor air quality seems to push far beyond what these sensors are realistically capable of. And moreover, the cost difference between the MOS sensors (say $10-$40/unit) and NDIR sensors ($80-$120/unit) doesn’t seem so great when the difference in accuracy is taken into account.

What are MOS sensors doing?

It’s trivial to figure out what is going on with the iAQ-Core. For this sensor the CO₂-equivalent is being calculated internally as a simple linear relationship to the TVOC gas concentration. This is shown in Fig. 3 - the sensor is simply multiplying the total volatile organic compound (TVOC) estimate by 3.63 and subtracting a small constant (5.53).

Figure 3: Correlation between TVOC and CO~2: iAQ-Core

It’s a bit more complex with the SGP30. This device appears to be recalibrating itself at regular intervals, such that the relationship between TVOC and CO₂ is not as clear (Fig. 4). Nonetheless, and as we would expect, there is a strong correlation between TVOC and CO₂ readings from this sensor.

Figure 4: Correlation between TVOC and CO2: SGP30

Further work

This is a work in progress. While it seems pretty definitive that only NDIR sensors for CO₂ are fit for purpose, it’s not entirely clear what an appropriate solution is for indoor VOC and PM measurement. That’s next on the list.

There’s also the longer term issue of sensor drift, particularly for the single wavelength NDIR CO₂ sensors. This will require long term testing in a practical situation to see just how well the single wavelength devices can avoid drift when only ever exposed to indoor air.

Conclusion

Based on this analysis there are two conclusions we can reasonably draw with reference to measuring indoor air quality:

• it’s hard, and slick websites with nice marketing don’t miraculously improve sensor performance, and
• NDIR sensors appear far more accurate than MOS sensors for CO₂ measurement

Given the Foobot uses a MOS sensor (the iAQ-Core) I’d suggest this device isn’t fit for purpose for indoor air quality CO₂ monitoring. For the other consumer-level devices we just don’t know without testing them, and ideally pulling them apart and checking what sensor they are using internally. But based on the evidence so far, only NDIR devices seem sufficiently accurate. And among the two tested here the dual wavelength Sensirion SCD30 is likely to offer the best long-term stability in an indoor environment where it’s less likely to be exposed to atmospheric CO₂ concentrations regularly.