Keywords: 3D point clouds, multispectral imagery, deep learning, segmentation, classification, forest strata, morphological descriptors.

Workplace: LaSTIG lab. (IGN/UGE/EIVP) – Team STRUDEL – 73 avenue de Paris, 94165 Saint-Mandé, FRANCE

Supervision: Nesrine Chehata (ENSEGID – Bordeaux INP), Clément Mallet (LaSTIG – IGN), Loïc Landrieu (LaSTIG – IGN).

Motivation and background

This  project  aims to  automatically derive  forest biodiversity  metrics  from remote  sensing multi-modal data acquisitions (images and 3D point clouds). These metrics are related to the complexity of vegetation structure and the spatial patterns of species. They are of utter importance for 1) better modeling the forest micro-climate, taking into account the spatial patterns of vegetation structure, and 2) studying the relationship between observed vegetation biodiversity and derived biodiversity metrics describing the 3D structure of the forests.

The characterization of the 3D structure of vegetation and, in particular, the different layers of vertical  stratification  (trees,  shrubs, forest floor) is essential  for modeling the  relationship between micro-climatic and biodiversity variables (Huang et al., 2014 ; Davies and Asner, 2014). Currently,  this  characterization  remains very coarse and imprecise: main approaches  do not capture the full complexity of forest ecosystems.

The vertical stratification of forest canopies is traditionally conducted using the density pro- file analysis method (Holmgren and Persson, 2004), modelling vegetation profile by probability distribution functions, either in a unimodal fashion (Dean et al., 2009) or with a mixture of functions (Jaskierniak et al., 2011). Other solutions are based on standard segmentation algorithms of 3D LiDAR points to discriminate main vegetation layers (Morsdorf et al., 2010 ; Ferraz et al., 2016).

State-of-the-art  methods are parametric  and lack genericity.  They only rely on local vertical point distribution and neglect the relationship between vegetation layer, individual trees, their species and their characteristics. Since many of these morphological and biological descriptors have latent semantics, which is not well-modeled by current methods, we propose to address the problem with a machine-learning, neural network-based approach.

Objectives

The post-doctoral work will focus on the automated analysis of 3D vegetation from a com- bination of 3D point  clouds and multispectral  multi-date  geospatial imagery. More  precisely, airborne LiDAR  data with a high density of points (60 pts/m2) will be used jointly to multis- pectral Sentinel-2 time series and very high resolution UAV images.

The tasks expected of the finalized network are as follows :

— Semantic segmentation of the 3D point clouds into tree species ;

— Instance segmentation of individual trees and of horizontal strata ;

— Derivation of morphological indicator of individual trees, such as crown diameter, stem diameter, maximal height, thickness, and extent.

Methods

The  problem is a multi-modal and multi-task problem, which combines  high density point clouds, superspectral time-series, and high definition images. The interleaved goals are to segment and classify the trees, and produce morphological descriptors. The different tasks of this post-doctoral can be broken down as follow (by indicative chronological order) :

i)  Format the ground truth data into a network-friendly structure ; ii)  Develop a species-semantic segmentation network ; iii)  Add an instance segmentation branch ; iv)  Derive morphological  features from this segmentation, add a morphological fidelity loss ; v)  Combine information from aligned time-sequence of superspectral images, assess benefits ; vi)  Combine information from aligned high definition aerial photographs, assess benefits ; vii) Produce an open-access, clean dataset for benchmarking and reproducibility.

(i) Data  Formatting :   several forest patches have been scanned and annotated by the French Forest Inventory.  While extremely  useful, the  data is  given at tree-level  and needs to be re- formatted to be directly usable by a network. The information is sufficient : angle and distance from a the center of the point cloud, as well as species and morphological attributes. This easy first step will constitute a valuable data exploration phase from the perspective of a researcher.

(ii)-(iii) Species and Instance Segmentation :   Over the last few years, a number of neural networks have been developed for processing 3D data. We propose to use the powerful torch- points3d 1 open-source framework implementing many state-of-the-art methods for various ap- plications, including semantic segmentation and instance segmentation (and in our case, panop- tic segmentation). This will allow us to quickly iterate and select the most relevant architecture for our problem.

(iv)  Morphological  Regression  :   The  annotated morphological descriptors  can help the learning phase by qualifying the accuracy of the instance segments. By implementing differen- tiable proxy for the sought descriptors, we will be able to use these annotations in an end-to-end fashion to improve the instance segmentation.

(v)-(vi) Multi-Modality :   Multi-spectral and multi-temporal time series contains phenolo- gical information crucial to distinguish between plant species (Sainte-Fare-Garnot et al., 2020). We  propose  to  align both Sentinel  time-series  and high-definition  aerial imagery with  our 3D data, and integrate  this  information to  the  point  clouds. While  the  exact architecture  of this fusion scheme is left to decide, our goal is for end-to-end joint learning.

(vii) Dataset :   The Dataset provided is atypical and challenging by its multi-modal and multi- task nature. We will package it into a open-access benchmark, relevant in the communities of remote sensing, computer vision, and machine learning.

1. https://towardsdatascience.com/torch-points3d-a-unifying-framework-for-deep-learning-on- point-clouds-94115c0be4fb

Study  Site and Data

The study site is situated in Southern Western France. A joint airborne multi-spectral and LiDAR  acquisitions (Autumn 2019) were processed. A winter acquisition is scheduled in winter 2020-2021. The spatial resolution is 20 cm GSD and a LiDAR  point density equals 60 pts/m2. Various scan angle acquisition were added. 31 sites with 1840 trees were observed and measured in field (height, crown diameter, stem diameter, density, specie,..).

Job requirements

— The candidate should have a PhD degree in informatics, remote sensing, computer vision or machine learning.

— Knowledge of deep learning methods and/or 3D point cloud processing is an asset.

— Mastery of Python is necessary, familiarity with PyTorch (preferred) or Tensorflow is a plus.

— Excellent motivation in developing research activities both at theoretical and application levels.

— Good spoken and written English. Knowledge of French would be useful, but not necessary.

— A prior knowledge and experience on forestry would be an asset.

Workplace description

The candidate will work at UMR LaSTIG, UGE/IGN/EIVP Paris (STRUDEL team) in col- laboration with EA G&E and UMR BIOGECO in Bordeaux.

The IGN (Institut National de l’Information Géographique et Forestière), in Paris is the French National Mapping Agency. Jointky with EIVP and Université Gustave Eiffel, the LaSTIG 2 lab gathers  full-time  researchers  on computer  vision, machine learning,  GIS and visualization  of geographic information data. EA G&E has expertise on 3D point cloud processing and UMR BIOGECO represents the end-user of biodiversity metrics.

Funding

The post-doc position offers full health, unemployment and retirement benefits and compe- titive salary. The candidate will be given the opportunity to develop his/her research skills by contributing to 3D-volumetric segmentation and semantic segmentation applied to geographic information sciences and forestry. The funding includes a two-year  post-doctoral  contract and an operating fund (for travel,  consumables, etc.).  Salary per month  : around 2300 euros (excluding taxes and with no accommodation).

The position will start from October to December 2020.

Application

Applications (resume+motivation letter+list of publications in a single PDF) should be sent before 10th October 2020 to:

2. https://www.umr-lastig.fr/

References

[Davies and Asner, 2014] Davies A.B. & Asner, G.P. Advances in animal ecology from 3D-Lidar ecosystem mapping. Trends in Ecology & Evolution 29, 681-691, 2014.

[Dean et al,. 2009] Dean T.J., Cao, Q.V., Roberts, S.D., & Evans, D.L., Measuring heights to crown base and crown median with  Lidar in a mature, even-aged loblolly pine  stand.  Forest Ecology and Management, 257, 126-133, 2009.

[Ferraz et al., 2016] Ferraz A., Saatchi S., Mallet C. & Meyer V., Lidar Detection of Individual Tree Size in Tropical Forests. Remote Sensing of Environment, 183, pp.318-333, 2016.

[Holmgren and Persson, 2014] Holmgren J. & Persson, A., Identifying species of individual trees using airborne laser scanner. Remote Sensing of Environment, 90, 415-423, 2004.

[Huang et al., 2014] Huang Q., Swatantran, A., Dubayah, R. & Goetz, S.J. The Influence of Ve- getation Height Heterogeneity on Forest and Woodland Bird Species Richness across the United States. PLoS One 9(8), e103236, 2014.

[Jaskierniak et  al., 2011] Jaskierniak D., Lane P.N.J., Robinson A. & Lucieer A., Extracting Lidar indices to characterise multilayered forest structure using mixture distribution functions. Remote Sensing of Environment, 115, 573-585, 2011.

[Morsdorf et al., 2010] Morsdorf F., Marell A., Koetz B., Cassagne N. & Pimont F., Discrimi- nation of vegetation strata in a multi-layered Mediterranean forest ecosystem using height and intensity  information derived  from airborne laser  scanning,  Remote  Sensing of Environment, 2010.

[Sainte Fare Garnot et al., 2020] Garnot, V. S. F., Landrieu, L., Giordano, S., & Chehata, N. (2020). Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self- Attention. IEEE/CVF Conference on Computer Vision and Pattern Recognition.