2-year Post-Doctoral Position: Multi-Modal Deep Learning for the Analysis of 3D Forestry Data
Keywords: 3D point clouds, multispectral imagery, deep learning, segmentation, classification, forest strata, morphological descriptors.
Workplace: LaSTIG lab. (IGN/UGE/EIVP) – Team STRUDEL – 73 avenue de Paris, 94165 Saint-Mandé, FRANCE
Supervision: Nesrine Chehata (ENSEGID – Bordeaux INP), Clément Mallet (LaSTIG – IGN), Loïc Landrieu (LaSTIG – IGN).
Motivation and background
This project aims to automatically derive forest biodiversity metrics from remote sensing multi-modal data acquisitions (images and 3D point clouds). These metrics are related to the complexity of vegetation structure and the spatial patterns of species. They are of utter importance for 1) better modeling the forest micro-climate, taking into account the spatial patterns of vegetation structure, and 2) studying the relationship between observed vegetation biodiversity and derived biodiversity metrics describing the 3D structure of the forests.
The characterization of the 3D structure of vegetation and, in particular, the different layers of vertical stratification (trees, shrubs, forest floor) is essential for modeling the relationship between micro-climatic and biodiversity variables (Huang et al., 2014 ; Davies and Asner, 2014). Currently, this characterization remains very coarse and imprecise: main approaches do not capture the full complexity of forest ecosystems.
The vertical stratification of forest canopies is traditionally conducted using the density pro- file analysis method (Holmgren and Persson, 2004), modelling vegetation profile by probability distribution functions, either in a unimodal fashion (Dean et al., 2009) or with a mixture of functions (Jaskierniak et al., 2011). Other solutions are based on standard segmentation algorithms of 3D LiDAR points to discriminate main vegetation layers (Morsdorf et al., 2010 ; Ferraz et al., 2016).
State-of-the-art methods are parametric and lack genericity. They only rely on local vertical point distribution and neglect the relationship between vegetation layer, individual trees, their species and their characteristics. Since many of these morphological and biological descriptors have latent semantics, which is not well-modeled by current methods, we propose to address the problem with a machine-learning, neural network-based approach.
Objectives
The post-doctoral work will focus on the automated analysis of 3D vegetation from a com- bination of 3D point clouds and multispectral multi-date geospatial imagery. More precisely, airborne LiDAR data with a high density of points (60 pts/m2) will be used jointly to multis- pectral Sentinel-2 time series and very high resolution UAV images.
The tasks expected of the finalized network are as follows :
— Semantic segmentation of the 3D point clouds into tree species ;
— Instance segmentation of individual trees and of horizontal strata ;
— Derivation of morphological indicator of individual trees, such as crown diameter, stem diameter, maximal height, thickness, and extent.
Methods
The problem is a multi-modal and multi-task problem, which combines high density point clouds, superspectral time-series, and high definition images. The interleaved goals are to segment and classify the trees, and produce morphological descriptors. The different tasks of this post-doctoral can be broken down as follow (by indicative chronological order) :
i) Format the ground truth data into a network-friendly structure ; ii) Develop a species-semantic segmentation network ; iii) Add an instance segmentation branch ; iv) Derive morphological features from this segmentation, add a morphological fidelity loss ; v) Combine information from aligned time-sequence of superspectral images, assess benefits ; vi) Combine information from aligned high definition aerial photographs, assess benefits ; vii) Produce an open-access, clean dataset for benchmarking and reproducibility.
(i) Data Formatting : several forest patches have been scanned and annotated by the French Forest Inventory. While extremely useful, the data is given at tree-level and needs to be re- formatted to be directly usable by a network. The information is sufficient : angle and distance from a the center of the point cloud, as well as species and morphological attributes. This easy first step will constitute a valuable data exploration phase from the perspective of a researcher.
(ii)-(iii) Species and Instance Segmentation : Over the last few years, a number of neural networks have been developed for processing 3D data. We propose to use the powerful torch- points3d 1 open-source framework implementing many state-of-the-art methods for various ap- plications, including semantic segmentation and instance segmentation (and in our case, panop- tic segmentation). This will allow us to quickly iterate and select the most relevant architecture for our problem.
(iv) Morphological Regression : The annotated morphological descriptors can help the learning phase by qualifying the accuracy of the instance segments. By implementing differen- tiable proxy for the sought descriptors, we will be able to use these annotations in an end-to-end fashion to improve the instance segmentation.
(v)-(vi) Multi-Modality : Multi-spectral and multi-temporal time series contains phenolo- gical information crucial to distinguish between plant species (Sainte-Fare-Garnot et al., 2020). We propose to align both Sentinel time-series and high-definition aerial imagery with our 3D data, and integrate this information to the point clouds. While the exact architecture of this fusion scheme is left to decide, our goal is for end-to-end joint learning.
(vii) Dataset : The Dataset provided is atypical and challenging by its multi-modal and multi- task nature. We will package it into a open-access benchmark, relevant in the communities of remote sensing, computer vision, and machine learning.
1. https://towardsdatascience.com/torch-points3d-a-unifying-framework-for-deep-learning-on- point-clouds-94115c0be4fb
Study Site and Data
The study site is situated in Southern Western France. A joint airborne multi-spectral and LiDAR acquisitions (Autumn 2019) were processed. A winter acquisition is scheduled in winter 2020-2021. The spatial resolution is 20 cm GSD and a LiDAR point density equals 60 pts/m2. Various scan angle acquisition were added. 31 sites with 1840 trees were observed and measured in field (height, crown diameter, stem diameter, density, specie,..).
Job requirements
— The candidate should have a PhD degree in informatics, remote sensing, computer vision or machine learning.
— Knowledge of deep learning methods and/or 3D point cloud processing is an asset.
— Mastery of Python is necessary, familiarity with PyTorch (preferred) or Tensorflow is a plus.
— Excellent motivation in developing research activities both at theoretical and application levels.
— Good spoken and written English. Knowledge of French would be useful, but not necessary.
— A prior knowledge and experience on forestry would be an asset.
Workplace description
The candidate will work at UMR LaSTIG, UGE/IGN/EIVP Paris (STRUDEL team) in col- laboration with EA G&E and UMR BIOGECO in Bordeaux.
The IGN (Institut National de l’Information Géographique et Forestière), in Paris is the French National Mapping Agency. Jointky with EIVP and Université Gustave Eiffel, the LaSTIG 2 lab gathers full-time researchers on computer vision, machine learning, GIS and visualization of geographic information data. EA G&E has expertise on 3D point cloud processing and UMR BIOGECO represents the end-user of biodiversity metrics.
Funding
The post-doc position offers full health, unemployment and retirement benefits and compe- titive salary. The candidate will be given the opportunity to develop his/her research skills by contributing to 3D-volumetric segmentation and semantic segmentation applied to geographic information sciences and forestry. The funding includes a two-year post-doctoral contract and an operating fund (for travel, consumables, etc.). Salary per month : around 2300 euros (excluding taxes and with no accommodation).
The position will start from October to December 2020.
Application
Applications (resume+motivation letter+list of publications in a single PDF) should be sent before 10th October 2020 to:
- Nesrine CHEHATA nesrine.chehata@ensegid.fr
- Loïc LANDRIEU loic.landrieu@ign.fr
- Clément MALLET clement.mallet@ign.fr
References
[Davies and Asner, 2014] Davies A.B. & Asner, G.P. Advances in animal ecology from 3D-Lidar ecosystem mapping. Trends in Ecology & Evolution 29, 681-691, 2014.
[Dean et al,. 2009] Dean T.J., Cao, Q.V., Roberts, S.D., & Evans, D.L., Measuring heights to crown base and crown median with Lidar in a mature, even-aged loblolly pine stand. Forest Ecology and Management, 257, 126-133, 2009.
[Ferraz et al., 2016] Ferraz A., Saatchi S., Mallet C. & Meyer V., Lidar Detection of Individual Tree Size in Tropical Forests. Remote Sensing of Environment, 183, pp.318-333, 2016.
[Holmgren and Persson, 2014] Holmgren J. & Persson, A., Identifying species of individual trees using airborne laser scanner. Remote Sensing of Environment, 90, 415-423, 2004.
[Huang et al., 2014] Huang Q., Swatantran, A., Dubayah, R. & Goetz, S.J. The Influence of Ve- getation Height Heterogeneity on Forest and Woodland Bird Species Richness across the United States. PLoS One 9(8), e103236, 2014.
[Jaskierniak et al., 2011] Jaskierniak D., Lane P.N.J., Robinson A. & Lucieer A., Extracting Lidar indices to characterise multilayered forest structure using mixture distribution functions. Remote Sensing of Environment, 115, 573-585, 2011.
[Morsdorf et al., 2010] Morsdorf F., Marell A., Koetz B., Cassagne N. & Pimont F., Discrimi- nation of vegetation strata in a multi-layered Mediterranean forest ecosystem using height and intensity information derived from airborne laser scanning, Remote Sensing of Environment, 2010.
[Sainte Fare Garnot et al., 2020] Garnot, V. S. F., Landrieu, L., Giordano, S., & Chehata, N. (2020). Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self- Attention. IEEE/CVF Conference on Computer Vision and Pattern Recognition.
