Together with a doctoral student and a postdoctoral researcher, Jan Dirk Wegner aims to develop an indicator for green spaces in the Canton of Zurich and beyond. What’s innovative about it: the indicator is multimodal, meaning it integrates various types of data. This is made possible through deep learning. In this interview, Jan Dirk Wegner—Professor of Data Science for Sciences and Head of the EcoVision Lab at the Institute for Mathematical Modelling and Machine Learning (DM3L) at the University of Zurich—explains how this works and what he envisions for the final product.

Jan, why are you taking part in the Public Data Lab (PDL)?

Jan: I’m motivated by the interdisciplinary nature of the Public Data Lab: We look at a question—such as “What does well-being mean?”—from different perspectives. The question is simple but very important, and I find it exciting to reflect on it with highly capable colleagues. Although my approach—using deep learning, machine learning, and computer vision—is very technical, I can still contribute a great deal.

Could you briefly explain the terms Machine Learning, Deep Learning, and Computer Vision?

Machine Learning refers to algorithms that learn from data to identify patterns or regularities, classify data, or make predictions. It is the umbrella term for all traditional algorithms such as Random Forest, Support Vector Machines, and also includes modern approaches like Deep Learning. The key difference between traditional and modern algorithms is that traditional methods require expert knowledge to extract features, whereas Deep Learning automatically extracts these features. For example, if I want to examine the health of vegetation using a Random Forest model, I first need to calculate indicators such as the Normalized Difference Vegetation Index (NDVI) or the Enhanced Vegetation Index (EVI) and assess them. These indicators are then provided as features to the Random Forest algorithm. Deep Learning, on the other hand, works directly with raw data and identifies the most relevant features itself in order to derive an indicator. This is precisely what makes Deep Learning so successful.

Computer Vision encompasses methods for capturing, processing, analysing, classifying, and extracting information from images. These images stem from various sources such as video recordings, cameras, 3D scanners, laser sensors, and more.

“When we, as experts, struggle to define rules for something vague, machine learning works very well.”

Jan Dirk Wegner

What is your role in the PDL and your contribution to it?

I contribute my technical expertise in the fields of machine learning and deep learning. At the same time, I’m able to connect and communicate with colleagues from other disciplines. I also supervise one of the four flagship projects, in which we explore how nature and the environment contribute to well-being. In this context, we are developing various indicators and using machine learning to identify as many plant and animal species as possible, as well as their relationship to well-being. The entire project is based on monitoring systems that we have developed ourselves. For example, we have mapped vegetation using panoramic Street View images. This is where it gets interesting: When we, as experts, struggle to define rules for something vague—such as which plant species or green environments have a positive impact on us—machine learning works very well. It can combine a large number of indicators, identify correlations, interpolate between data points, and offer intriguing evaluations. These evaluations may not be directly usable, or perhaps shouldn’t be, but they do provide valuable food for thought.

You just mentioned the third flagship project, which focuses on developing multimodal indicators for green spaces. What kind of input data does the project use?

We begin by consulting with the Statistical Office to understand which data are currently used to assess well-being, whether it is measured directly or indirectly, what the spatial resolution is, and so on. After that, we will measure environmental parameters and attempt to establish a correlation between these parameters and well-being using deep learning. For the environmental parameters, we will initially work with image data, and possibly later with point clouds.
Following this, we will collaborate with the project partners to identify additional data and indicators that can be integrated into the model. I’m thinking of climate data, but population data could also be interesting. Perhaps there’s a happiness index we could use (laughs). It might also be interesting to incorporate work data from other flagship projects. This is precisely where the enormous potential of deep learning lies—in its multimodality: We can combine different types of data, such as images, text, sound, etc., relatively easily, and across long time series.

Your project aims to develop spatially explicit indicators. What does spatially explicit mean in this context?

Spatially explicit means that we calculate the indicators for each grid cell at a certain resolution, for example 100 x 100 metres. In the end, we will have a raster layer for each indicator in a GIS system. From this, we can then create various maps.

What is the end product of your flagship project?

The idea is that we will ultimately have a map with indicators that are important for well-being. In addition, there will be a second, higher-resolution map that represents well-being itself and shows which environmental indicators are decisive for the level of well-being in which locations. At the same time, the map should indicate how reliable the model is, or in other words, the degree of uncertainty. Ideally, the model should allow for interpretability—meaning it is not a black box, but can explain, based on the map, how it arrived at its conclusions.

What area does this map cover?

Certainly, the Canton of Zurich, since we are conducting the project for the canton. However, machine learning is particularly advantageous when scaling is involved. That’s why it’s important to me that we think beyond the canton from the outset, at a national scale, even if we don’t have the same data or the same resolution everywhere. For small areas like a city or the Canton of Zurich, machine learning wouldn’t necessarily be required. In such cases, much of the data can be interpreted manually. The added value of what we’re doing lies in the ability to detect highly complex patterns in the data that point to well-being, and in the ability to scale and automate the process.

What is your goal with the Public Data Lab and your project?

My goal is to develop an automated—or at least partially automated—method that can later be used by the Statistical Office or other institutions as a tool to measure and improve well-being over the long term.
Since it’s a research project, I don’t know how far we’ll get. However, my ambition is to conduct high-quality research and publish at a very technical level, while also bridging the gap between academic research and practical application. My aim is to create a prototype software that, thanks to higher resolution, allows for much better analysis than is currently possible. It should show where and how well-being could be improved, and also to what extent environmental factors play a role.

Where do you see challenges?

The biggest challenge—but also the most motivating and enjoyable one—is agreeing on a common language among us as project partners from different disciplines. We need to take the time to understand which questions are most important from a societal perspective and which ones we want to answer. If we don’t do that, we might develop something that results in a great academic paper but is irrelevant in practice. I see communication as the greatest challenge—not just within the lab, but beyond it. It’s important to meet regularly, exchange ideas, engage with one another, step out of our disciplinary routines, be patient, give each other space and time to conduct individual research, but also pursue a shared vision and present ourselves as a team.

Jan Dirk Wegner is Professor of Data Science for Sciences and heads the EcoVision Lab at the Institute for Mathematical Modelling and Machine Learning (DM³L) at the University of Zurich. His research focuses on the intersection of machine learning, computer vision, and remote sensing, aiming to address scientific questions in environmental sciences and geosciences to improve the well-being of humans, animals, and plants. Jan is a father of three children, enjoys playing the guitar, and is a passionate Werder-Bremen fan.