Towards mutualism: reconciling differences between field and EO data
Partner Post: Chris Anderson at Planet Labs PBC
Hello Cecil community, I’m Chris, Lead Scientist of the Forest Ecosystems team at Planet. It’s nice to meet you!
So I read this great book as a grad student, Writing Science by Joshua Schimel. I loved it because it reframes how scientists can construct and define their identities. Early career scientists are asked to do so many things these days: developing hypotheses, teaching, keeping up with new research, raising funding, writing code, mentoring, collecting data, performing statistical analyses, reviewing manuscripts, and writing papers. For example.
Schimel’s book challenged scientists to think of writing as more than something you do: writing is central to your identity. As a scientist, you are a writer. All the aforementioned tasks — the synthesis of existing knowledge with new data and analysis — most effectively advance a field when the results are clearly communicated. The strategies for communication can change based on the audience, from your peers in a reviewed scientific publication, to the prospective funders of a new proposal, or to a government agency that might implement new findings as policy recommendations. I really connected with this perspective. I love to think of writing as more than something I do; it is a part of who I am.
That’s because, like everyone else with a job, I have many things to do. My job is to lead an exceptionally talented team that builds satellite-based forest monitoring systems. I’ve had to develop many new skills as I grew into this role. My background in plant ecophysiology did not prepare me to learn about API design patterns, event-driven workflows, or really any form of software architecture. With some experience — it’s been about 10 years since I wrote my first line of code — I now find software architecture and software ecology fascinating.
Reaching further from my roots, I recently found myself reading a book about the evolution and marketing of technology products, Crossing the Chasm by Geoffrey Moore. This book seems to be de rigueur for anybody in corporate technology management. I didn’t expect to like it as much as I did. I thought it would be rather dry, with lots of acronyms and fluff. I was wrong! It’s full of interesting and thoughtful lessons on the growth, competition, and success of technology systems.
Moore articulated a useful distinction between continuous innovations and discontinuous innovations. A continuous innovation simply improves on what came before. You could upgrade from an HDTV to a 4K TV and keep using your peripherals and streaming services, for example. A discontinuous innovation, on the other hand, requires behavioral or technological changes to use the new system. Upgrading to a 3D TV would be discontinuous because you need special viewing systems and specialized content providers. And you may need to rearrange your living room. Discontinuous innovations must provide a compelling value proposition for users to adopt because they merit changing habits. And many folks are skeptical of change.
I bring this up because it relates to the last Cecil Newsletter by Tom Walker that discussed the challenges of evaluating nature datasets. Walker articulated a series of fundamental and persistent challenges to quantify and communicate the accuracy and uncertainty of nature datasets, particularly global ones. As developers of such data products, our team at Planet has had to address each of these challenges when building our Forest Carbon data products.
We recently published a manuscript detailing how we approach dataset inspection, intercomparison, and benchmarking. It describes the concepts behind our methods, it details the known and expected limitations of the product, and articulates a vision for the role that rigorous uncertainty quantification will play in enabling the use of satellite-derived carbon estimates in different policy contexts. I won’t recap it here, since the 15,000 words, 29 figures and 61 total pages of the manuscript was the most concise expression we could muster. I hope you find the opportunity to read it, particularly the Discussion section.
Instead of discussing the manuscript, I want to examine an implicit assumption we often make in benchmarking and evaluation. Here’s a trimmed-down summary from Tom’s post on the topic:
“The gold standard of dataset evaluation is to benchmark accuracy against an accepted source of truth… For nature data, this creates a challenge: the source of truth is usually field data that contains its own uncertainties. Field surveys [...] converting tree basal area measurements into biomass estimates carries inherent uncertainty. Even so, high quality field data remains the best available reference point for evaluating dataset performance.”
Well put. But let’s consider this point closely.
By implicitly framing satellite biomass datasets as a continuous innovation over field data — as a product that can only be understood in terms of its fidelity to field data — we adopt the perspective that satellite data is primarily an advance upon methods developed and refined in the field. What are the implications here? Champions of remote sensing might extrapolate from this premise to promise that, with further innovation, satellite data might one day surpass the quality of field measurements and make field data collection obsolete.
If you’ve spent any time in the field, you likely regard such a perspective with a great deal of skepticism. I too am a skeptic of this promise. That’s because it’s based on a false premise.
Field and satellite observations are fundamentally distinct modalities. They are different methods of estimation: they do not measure the same things, nor do they measure on the same spatial or temporal scales. Field and satellite measurements respectively represent bottom-up and top-down approaches to estimation, both literally and epistemologically. There are some things you can easily measure in the field but struggle to estimate from space, like understory stem counts or species turnover following a disturbance. Likewise, estimating annual deforestation rates across the tropics is much easier from space than it is from the field.
I believe we all know that these things are different, and that each presents a clear value proposition. Where we seem to encounter tension is in efforts to reconcile the bottom-up and top-down understandings afforded by these modalities.
One source of tension is the one-way nature of the relationship. Field data are typically required to calibrate remote sensing measurements. Yet I can’t think of an example where an ecological field measurement was calibrated by remote sensing observations (the closest might be the use of satellite-derived maps to inform sampling strategies). Some seem to interpret this relationship as parasitic; I interpret it as largely commensal; more on this later.
This one-way dynamic is somewhat inevitable. Satellites measure in units of energy (like at-sensor radiance in W/m2/sr/nm, or backscatter power in dB), not in ecological units (like abundance in species counts, or basal area in m2/ha). We must convert measurements from satellites to make sense of what they might mean in the field. This is hard.
I often describe my job as akin to a translator: converting data collected from space into calibrated estimates of forest structure, function, and change. This can only be credibly done based on a deep knowledge of how vegetation interacts with energy in the form of absorbed and reflected electromagnetic radiation. It’s a translation that needs to be performed at scale, running signal processing algorithms on trillions of pixels worth of data every year to represent global forest change.
Translators often describe their job as an imperfect art, where the unique contours of specific turns of phrase in their original language are often rendered dull or indistinct upon translation. So it goes when joining field and satellite data. Lost is the precision of the original measurement, as are the stories of the flooded camp sites, the lost permits, the things found in shoes, the quiet murmurs of nature transcribed into data. But what we gain from this translation is a new way to see our changing planet and quantify the consequences of change.
Plants interact with diurnal variations in electromagnetic radiation in profound and complex ways. By measuring these patterns continuously over time and space, satellite data offers a discontinuous innovation over what is measured in the field.
We should not think of remote sensing as a replacement to field data, but as a complement.
The key advantages of satellite data are the scales of measurement: the frequency, resolution, and volume of observations. Yet evaluating data quality with a sparse sample of tens to hundreds of field plots will almost always provide noisy and underdetermined results. The scale mismatches are simply too great. So how should we do things differently?
Acknowledging satellite data as a discontinuous innovation would mean developing different usage patterns for space-based and field-based estimates. I think such patterns can be quantitatively informed by the uncertainties that emerge when linking these data together.
When it comes to carbon, it will probably always be the case that satellite-based carbon estimations report higher uncertainty at fine scales than field measurements. There is plenty of uncertainty in field measurements to begin with, which are further compounded when modeling these data with proxies from remote sensing (see Réjou-Méchain et al. for more). But as we wrote in our Forest Carbon Diligence paper, the fact that satellite carbon estimates have higher uncertainty at fine scales does not mean high uncertainty persists across all scales. Spatial aggregation plays a key mechanistic role here.
“Aggregating pixels over larger areas — field plots, carbon projects, protected areas, or states, for example — systematically decreases uncertainty when estimating the standard error of the mean, or of the sum of total carbon stocks. What’s most critical is that uncertainty measurements are well-calibrated; that multiple sources of uncertainty are propagated to the prediction intervals; that the spatial distribution of errors is quantified; and that any comparison between projects or between time periods robustly tests for differences based on this aggregated uncertainty.”
This is where I both converge with — and diverge from — Tom’s earlier quote: “field data remains the best available reference point for evaluating dataset performance.” Indeed, field data are unequivocally the standard for how we define and quantify carbon stocks. However, what I believe should change is how we evaluate performance. Since satellite-based precision will always be lower than field precision, we should not expect satellite data to perfectly align with plot data at the pixel level. Instead, the statistical power required to estimate carbon stocks — and to detect significant changes in stocks — will almost always require aggregation to coarser scales to sufficiently minimize uncertainty.
Some models for elegantly aligning satellite and field data already exist. Neha Hunka et al., for example, recently used thousands of National Forest Inventory (NFI) plots in Mexico to recalibrate satellite observations. This recalibrated model then provided out-of-sample, NFI-like estimates at unmeasured locations with empirically tested uncertainties.
Their approach to aligning field and satellite data addresses some key points. First, satellite estimates become consistent with state- or country-level protocols, such as the allometries used to convert field data to biomass. Second, while field data alone can estimate summaries over large areas with the right sampling design, fusing field and remote sensing data in model-based inference ensures that the resulting estimates are informed both by the sampled plots and by unsampled areas for which remote sensing data are available. This enables estimation for small areas — like Improved Forest Management projects — where the sample size might not be large enough to use field data alone (a strength of model-based inference noted by Ståhl et al.).
This approach could also kickstart a positive feedback cycle. As Hunka et al. point out, “...due to various financial and logistical constraints, only 10,959 of the 26,220 proposed NFI locations were sampled in the 3rd NFI cycle between 2015 and 2019, resulting in spatially-irregular, non-random gaps in the probability design across the domain.” Assuming similar constraints will apply in the future, which locations should be prioritized for the next cycle?
This is where the positive feedback begins. Since their approach provides spatially explicit uncertainties, this kind of information could prioritize sampling in locations that have relatively high expected information gain. More field data will yield better models, which benefits remote sensing, and better models can guide sampling to provide more informative data in future field collection efforts.
This example also highlights that satellite carbon mapping merits the collection of thousands of additional field plots. We need much more field data, not less! Field data are incredibly valuable, and provide the foundation of remote sensing science. And their value increases further when integrated with satellite data, providing the kind of detailed insights into global stocktakes that are not possible with either dataset independently. For me, the future is in fusion.
I would love to see the community advance to where field and satellite data coexist in a sort of adversarial mutualism. Remote sensing missions need thousands of field measurements for calibration and validation, and these measurements should be continuously funded and made accessible to support multiple missions (like BIOMASS, NISAR, GEDI, and private constellations). These plots should reveal where satellite observations get it wrong, spurring research into potential improvements. More field data should then be collected in high uncertainty areas to test these improvements and identify future frontiers for R&D. This process of continuous evaluation and data collection could create a virtuous cycle of benefits for both communities — and would represent another discontinuous innovation over the status quo.
Let’s work together to make this a reality. As a satellite data provider, we want to advance the science and advance the impact of the great work done in this domain through collaboration. It’s one of the many reasons we’re excited to be a part of the growing Cecil community. We can go farther if we go together.
h/t to Max Joseph for his inputs on model-assisted design and sample prioritization, and to the Cecil team for their feedback during the writing process.
Recent updates at Cecil:
The IBAT Alliance’s core datasets: IUCN Red List of Threatened Species, World Database of Key Biodiversity Areas, and the World Database on Protected Areas
The Natural History Museum’s 1 km and 10 km Biodiversity Intactness Index
Impact Observatory’s 9-class land cover
University of Maryland (GLAD) global forest change
Lobelia’s surface soil moisture 30 m
Join our Slack Community to join the conversation with over 430 nature data enthusiasts
Sign up for a Cecil account and get started here.