Repository files navigation

This repository implements the complete processing and workflow required to reproduce the study about the impact ofradiometric uncertainty in Sentinel-2 Top-of-Atmosphere data on the retrieval of land surface metrics.

@article{graf_propagating_2023,title = {Propagating Sentinel-2 Top-of-Atmosphere Radiometric Uncertainty into Land Surface Phenology Metrics Using a Monte Carlo Framework},issn = {2151-1535},doi = {10.1109/JSTARS.2023.3297713},pages = {1--41},journaltitle = {{IEEE} Journal of Selected Topics in Applied Earth Observations and Remote Sensing},    journal = {IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing},author = {Graf, Lukas Valentin and Gorroño, Javier and Hueni, Andreas and Walter, Achim and Aasen, Helge},date = {2023},keywords = {Crops, Ecophysiological Parameters, Indexes, Land Surface Phenology, Measurement, Monte-Carlo Simulation, Radiative Transfer Modelling, Radiometry, Remote sensing, Sentinel-2, {TIMESAT}, Uncertainty, Vegetation mapping},    year = {2023}}

While the Python scripts are OS-independent, the shell-scripts only work in a *nix environment. They were developed and tested under Fedora 34 (64-bit) machine using Python 3.9.7.

See therequirements.txt file for a list of Python packages necessary to run the scripts. All dependencies should be installed beforehand into a clean virtual environment.

Furthermore, the Sentinel-2 Radiometric Uncertainty Toolbox and Sen2Cor are required.

For S2 data download, aCREODIAS account is required (free account). The API access token must be provided according to themodule doc string in the download module.

Forreading and writing raster data we use the E:earth_africa:dal Python package (Earth Observation Data Analysis Library). See for more information.

Installing E:earth_africa:dal will setup most of the required Python dependencies. It is recommended to install E:earth_africa:dal into a clean virtual environment to avoid any undesired side-effects.

All requirements besides E:earth_africa:dal dependencies are listed in therequirements.txt and can be installed into the virtual environment using

pip3 install -r requirements.txt

The installation of theSentinel-2 Uncertainty Toolbox is a bit cumbersome (requires Java and Python). The following steps worked for me:

1. Install theSNAP toolbox. Select a Python virtual environment when asked during the installation process. See alsohere how to connect SNAP with Python.
2. Installmaven if not yet installed. It is required to build the packages
3. Clone theSentinel-2 Uncertainty Toolbox from Github and follow the instructions in the README. It also links to a wiki page explaining how to integrate the toolbox into SNAP.
4. Most likely the toolbox will throw an error when trying to run it. This is becausejpy is missing but required by SNAP to connect Java and Python. To overcome the issue clone the Github repository and build it following the instructions in the README. Make sure that the environmental variables are set as specified in the Wiki. For Fedora I added the following lines of code into my~/.bashrc file:

export JAVA_HOME=$(dirname $(dirname $(readlink $(readlink $(which javac)))))export JDK_HOME=/=${JAVA_HOME}export PATH=$PATH:${JAVA_HOME}/binexport PATH=$PATH:${JAVA_HOME}/include

Don't forget to runsource ~/.bashrc after making the changes to apply them. Then continue with building jpy. Copy the wheel package (jpy-0.10.0.dev1-cp39-cp39-linux_x86_64.whl) into /home/$USER/.snap/snap-python/snappy/

The actual algorithm of the toolbox is coded in Python and takes its inputs from the metadata xml (MTD_MSIL1C.xml). The algorithm works bandwise and assign pixel uncertainties values between 0 and 250 (250 = 25% uncertainty or higher). Thus, a value of 12 corresponds to a uncertainty of 1.2%.

Since the core algorithm of the toolbox is written in Python it is necessary to get the Java-Python bridge of SNAP (snappy) running. To makesnappy working follow these steps (shown for Fedora):

• Create a new (i.e., clean) virtual python virtual environment:

virtualenv snap-pythonsource snap-python/bin/activate

• Change into the SNAP installation directory. In my case this was found in: /home/graflu/.snap/snap-python/snappy/ and run:

python setup.py install

to install snappy into the environment. Additionally install numpy:

pip install numpy

Additionally, you have to enable the Python-Java bridge by building the jpy package.To do so, clone the source from Github and build it:

git clone https://github.com/bcdev/jpy.gitcd jpypython setup.py build maven bdist_wheel

Then copy the created wheel packages and the shared object files (*.so) from the /build directory of jpy into the snappy directory (top-level).

IMPORTANT NOTE ON UPDATE OF PYTHON VERSION: If you have upgraded your Python version (e.g., from Python 3.8 to 3.9, make sure to alter the filenames in ~/snap/snap-python/build/jpyconfig.properties) so that the point towards the updated shared objects and wheel packages. This affects thejpy.jpyLib,jdl.jpyLib, andjpy.pythonLib entries in that file. Otherwise you will get Java errors because the old shared objects are either deleted or not working any more!

To check if snappy was installed correctly start Python in interactive mode:

(snap-python) [graflu@kp140-208 snappy]$ pythonPython 3.9.7 (default, Aug 30 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)] on linuxType "help", "copyright", "credits" or "license" for more information.>>> import snappyINFO: org.esa.snap.python.gpf.PyOperatorSpi: Python operator 'S2RutOp' registered (Python module: 's2_rut', class: 'S2RutOp', root: '/mnt/ides/Lukas/software/snap/s2tbx/modules/org-esa-snap-snap-rut.jar')INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.4 found on system. JNI driver will be used.INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.4 set to be used by SNAP.INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adaptersINFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.

Download Sen2Cor v2.9 from theofficial access point.

Thepath to the Sen2Cor executable must be provided in the shell script05_execute_sen2cor.sh or add the path to the binary directory to your $PATH.

The table below explains the file-system of this repository which will hold the Monte Carlo simulations and analysis results.static in the columncontent type means that the content is provided as is when cloning this repo, whereasdynamic means that the content is generated during execution of the workflow.

To re-generate the results presented in the paper, execute the Python modules and shell scripts in theprocessing sub-package in the order the scripts are named: I.e., start with thedownloader script (named01_download_data_creodias.py), continue with02_write_property_file.py and so on until you reachthe last script.

IMPORTANT

• You have to provide the path to the SNAP graph processing tool (gpt) executable in theS2RUT shell script
• You have to provide the path to the L2A_Bashrc file of the Sen2Cor installation in theshell script running Sen2Cor andhere

CAUTION:Some of the scripts requireseveral days to terminate and consume reasonable amounts of CPU time, RAM and disk storage (e.g., running Sen2Cor several hundred times). Therefore, we also do not provide a single shell script to execute everything at once. Instead we recommend careful testing beforehand of the scripts (e.g., by lowering the number of scenarios generated, starting with a single S2 scene, etc.)