Getting Started with GEDI L2A Version 2 Data in Python

This tutorial demonstrates how to work with the Elevation and Height Metrics (GEDI02_A.002) data product.

The Global Ecosystem Dynamics Investigation (GEDI) mission aims to characterize ecosystem structure and dynamics to enable radically improved quantification and understanding of the Earth's carbon cycle and biodiversity. The GEDI instrument produces high resolution laser ranging observations of the 3-dimensional structure of the Earth. GEDI is attached to the International Space Station and collects data globally between 51.6 N and 51.6 S latitudes at the highest resolution and densest sampling of any light detection and ranging (lidar) instrument in orbit to date. The Land Processes Distributed Active Archive Center (LP DAAC) distributes the GEDI Level 1 and Level 2 Version 1 and Version 2 products. The L1B and L2 GEDI products are archived and distributed in the HDF-EOS5 file format.

Use Case Example:

This tutorial was developed using an example use case for a project being completed by the National Park Service. The goal of the project is to use GEDI L2A Version 2 data to observe tree canopy height and profile over Redwood National Park in northern California.

This tutorial will show how to use Python to open GEDI L2A Version 2 files, visualize the sub-orbit of GEDI points (shots), subset to a region of interest, visualize GEDI canopy height, and export subsets of GEDI science dataset (SDS) layers as GeoJSON files that can be loaded into GIS and/or Remote Sensing software programs.

Data Used in the Example:

Topics Covered:

  1. Get Started
    1.1 Import Packages
    1.2 Set Up the Working Environment and Retrieve Files
  2. Import and Interpret Data
    2.1 Open a GEDI HDF5 File and Read File Metadata
    2.2 Read SDS Metadata and Subset by Beam
  3. Visualize a GEDI Sub-Orbit
    3.1 Subset by Layer and Create a Geodataframe
    3.2 Visualize a Geodataframe
  4. Work with GEDI L2A Data
    4.1 Import and Extract Specific Shots
    4.2 Plot Relative Height Metrics
    4.3 Combine RH Metrics and Waveforms
    4.4 Select Data from non-Default Algorithm
  5. Plot Transects
    5.1 Quality Filtering
    5.2 Plot Beam Transects 5.3 Subset Beam Transects
    5.4 Plot RH Metrics Transects
  6. Spatial Visualization
    6.1 Import, Subset, and Quality Filter all Beams
    6.2 Spatial Subsetting
    6.3 Visualize All Beams: Canopy Height and Elevation
  7. Export Subsets as GeoJSON Files

Before Starting this Tutorial:

Setup and Dependencies

It is recommended to use Conda, an environment manager to set up a compatible Python environment. Download Conda for your OS here: https://www.anaconda.com/download/. Once you have Conda installed, Follow the instructions below to successfully setup a Python environment on Linux, MacOS, or Windows.

This Python Jupyter Notebook tutorial has been tested using Python version 3.7. Conda was used to create the python environment.

If you do not have Jupyter Notebook installed, you may need to run:

conda install jupyter notebook

Having trouble getting a compatible Python environment set up? Contact LP DAAC User Services at: https://lpdaac.usgs.gov/lpdaac-contact-us/

If you prefer to not install Conda, the same setup and dependencies can be achieved by using another package manager such as pip.

Example Data:

This tutorial uses the GEDI L2A Version 2 observation from June 19, 2019 (orbit 02932, sub-orbit 02). Use the link below to download the file directly from the LP DAAC Data Pool:

You will need to have the file above downloaded into the same directory as this Jupyter Notebook in order to successfully run the code below.

Source Code used to Generate this Tutorial:

The repository containing all of the required files is located at: https://git.earthdata.nasa.gov/projects/LPDUR/repos/gedi-v2-tutorials/browse

NOTE: This tutorial was developed for GEDI L2A Version 2 files and should only be used for those products.

1. Get Started

1.1 Import Packages

Import the required packages and set the input/working directory to run this Jupyter Notebook locally.

1.2 Set Up the Working Environment and Retrieve Files

The input directory is defined as the current working directory. Note that you will need to have the Jupyter Notebook and example data (.h5 and .geojson) stored in this directory in order to execute the tutorial successfully.

NOTE: If you have downloaded the tutorial materials to a different directory than the Jupyter Notebook, `inDir` above needs to be changed. You will also need to add a line: `os.chdir(inDir)` and execute it below.

In this section, a Version 2 GEDI L2A .h5 file has been downloaded to the inDir defined above. You will need to download the file directly from the LP DAAC Data Pool in order to execute this tutorial.

Make sure to download the file into the inDir directory defined above.

2. Import and Interpret Data

In this section, begin working with the GEDI Level 2A Elevation and Height Metrics Data. Begin by checking out the example L2A file metadata.

2.1 Open a GEDI HDF5 File and Read File Metadata

Read the file using h5py.

The standard format for GEDI filenames is as follows:

GEDI02_A: Product Short Name
2019170155833: Julian Date and Time of Acquisition (YYYYDDDHHMMSS)
O02932: Orbit Number
02: Sub-Orbit Granule Number (1-4)
T02267: Track Number (Reference Ground Track)
02: Positioning and Pointing Determination System (PPDS) type (00 is predict, 01 rapid, 02 and higher is final)
003: PGE Version Number
01: Granule Production Version
V002: Product Version

Read in a GEDI HDF5 file using the h5py package.

The GEDI HDF5 file contains groups in which data and metadata are stored.

First, the METADATA group contains the file-level metadata.

This contains useful information such as the creation date, PGEVersion, and VersionID. Below, print the file-level metadata attributes.

2.2 Read SDS Metadata and Subset by Beam

The GEDI instrument consists of 3 lasers producing a total of 8 beam ground transects. The eight remaining groups contain data for each of the eight GEDI beam transects. For additional information, be sure to check out: https://gedi.umd.edu/instrument/specifications/.

One useful piece of metadata to retrieve from each beam transect is whether it is a full power beam or a coverage beam.

Below, pick one of the full power beams that will be used to retrieve GEDI L2A relative height metrics in Section 3.

Identify all the objects in the GEDI HDF5 file below.

Note: This step may take a while to complete.

3. Visualize a GEDI Orbit

In the section below, import GEDI L2A SDS layers into a GeoPandas GeoDataFrame for the beam specified above.

Use the lat_lowestmode and lon_lowestmode to create a shapely point for each GEDI shot location.

3.1 Subset by Layer and Create a Geodataframe

Read in the SDS and take a representative sample (every 100th shot) and append to lists, then use the lists to generate a pandas dataframe.

Above is a dataframe containing columns describing the beam, shot number, lat/lon location, and quality information about each shot.

Below, create an additional column called 'geometry' that contains a shapely point generated from each lat/lon location from the shot.

Next, convert to a Geopandas GeoDataFrame.

Pull out and plot an example shapely point below.

3.2 Visualize a GeoDataFrame

In this section, use the GeoDataFrame and the geoviews python package to spatially visualize the location of the GEDI shots on a basemap and import a GeoJSON file of the spatial region of interest for the use case example: Redwood National Park.

Import a GeoJSON of Redwood National Park as an additional GeoDataFrame. Note that you will need to have downloaded the GeoJSON from the bitbucket repo containing this tutorial and have it saved in the same directory as this Jupyter Notebook.

Defining the vdims below will allow you to hover over specific shots and view information about them.

Below, combine a plot of the Redwood National Park Boundary (combine two geoviews plots using *) with the point visual mapping function defined above in order to plot (1) the representative GEDI shots, (2) the region of interest, and (3) a basemap layer.

Above is a good illustration of the new GEDI Version 2 sub-orbit granules (remember that GEDI Version 1 files are stored as one ISS orbit). One of the benefits of using geoviews is the interactive nature of the output plots. Use the tools to the right of the map above to zoom in and find the shots intersecting Redwood National Park.

(HINT: find where the orbit intersects the west coast of the United States)

Below is a screenshot of the region of interest:

alt text

Side Note: Wondering what the 0's and 1's for quality_flag mean?

Above, 0 is poor quality and a quality_flag value of 1 indicates the laser shot meets criteria based on energy, sensitivity, amplitude, and real-time surface tracking quality. We will show an example of how to quality filter GEDI data in section 6.1.

After finding one of the shots within Redwood NP, find the index for that shot number so that we can find the correct RH metrics to visualize in Section 4.

Each GEDI shot has a unique shot identifier (shot number) that is available within each data group of the product. The shot number is important to retain in any data subsetting as it will allow the user to link any shot record back to the original orbit data, and to link any shot and its data between the L1 and L2 products. The standard format for GEDI Shots is as follows:

Shot: 29320600200465601

2932: Orbit Number
06: Beam Number
0: Reserved for future use
02: Sub-orbit Granule Number
004: Minor frame number
65601: Shot index

4. Work with GEDI L2A Data

Interpretation of RH Metrics

The GEDI L2A data product provides relative height (RH) metrics, which are “lidar perceived” metrics that have the following characteristics:

1. RH100 = elev_highestreturn - elev_lowestmode
2. The RH metrics are intended for vegetated surfaces. Results over bare/water surfaces are still valid but may present some confusing results.
3. The lower RH metrics (e.g., RH10) will often have negative values, particularly in low canopy cover conditions. This is because a relatively high fraction of the waveform energy is from the ground and below elev_lowestmode. For example, if the ground return contains 30% of the energy, then RH1 through 15 are likely to be below 0 since half of the ground energy from the ground return is below the center of the ground return, which is used to determine the mean ground elevation in the footprint (elev_lowestmode).

In this section, import and extract a specific GEDI L2A shot. From there, plot relative height (RH) metrics, and then combine with the L1B full waveform data. Note that we will continue to use BEAM0110 from Section 3.

4.1 Import and Extract Specific Shots

Notice that there are thousands of datasets available in the GEDI L2A product. In the code blocks below, you will subset to just a few of the datasets available.

Begin by subsetting just to the selected full power beam:

We will set the shot index used as an example from the GEDI L1B Tutorial to show how to subset a single shot of GEDI L2A data.

4.2 Plot Relative Height Metrics

In section 4.2, import the relative height metrics and begin exploring how to plot them.

New with Version 2:

The GEDI L2A product provides waveform processing results for multiple algorithm settings. Results for a default algorithm selection are provided in the root directory of the data product for each beam. In GEDI Version 1, the default for all shots was algorithm setting group 1.

In Version 2, the default algorithm is set to the best performing algorithm based on a number of characteristics for that specific shot. For information on selecting the non-default algorithm, see section 4.4 below. Below, bring in the selected_algorithm dataset to determine which algorithm (1-6) was set to the default for our shot.

Next, bring in other useful L2A datasets such as lat_lowestmode and lon_lowestmode.

Grab the location, relative height metrics, and algorithm for the shot defined above:

Put everything together to identify the shot that we want to extract:

In order to plot the RH metrics, you also need to import the elevation recorded at elev_lowestmode (the first elevation recorded) and elev_highestreturn or the last elevation recorded for that waveform.

Below, convert canopy height to canopy elevation by adding the ground elevation to each RH 1% interval.

Next, use the holoviews (hv) package to make a line plot showing the elevation at each percentile for the selected shot.

Congratulations! You have plotted your first relative cumulative RH profile.

The rh dataset stores the relative height metrics at 1% intervals, and thus each shot contains 101 values representing rh at 0-100%.

Next, add some additional context to the cumulative RH profile, including metrics such as the elev_lowestmode (ground elevation), the elev_highestreturn (highest reflecting surface height), and the relative height metrics at each quartile.

Based on the figure above, it looks like the densest portion of the tree canopy for this shot is at around 45-55 meters.

4.3 Combine RH Metrics and Waveforms

Note that you will need to have downloaded the waveform.csv file from the repo in order to execute the section below.

In this section, we will import the corresponding waveform for our selected shot.

Next, create a holoviews curve plot of the waveform. If you are interested in learning more about plotting L1B waveforms, be sure to check out the GEDI L1B Tutorial.

Next, combine the L2A Relative Height (RH) metrics to the corresponding L1B waveform.

Above, notice the amplitude spike where the ground return is defined, and the close grouping of RH50 and RH75 over the dense portion of the upper canopy detected for this shot.

4.4 Select Data from the Non-Default Algorithm

Recall from above that the GEDI L2A product provides waveform processing results for multiple algorithm settings. Results for the selected algorithm selection are provided in the root directory of the data product for each beam. New with Version 2 is that the selected algorithm is selected as identifying the lowest non-noise mode (see Table 5 of the GEDI L02 User Guide). Algorithm Setting Group 2 is the algorithm used for the data that we extracted in the sections above.

Elevation and height metrics outputs for all algorithm setting groups can be found in the geolocation subgroup of the L2A data product. For example:

- elev_lowestreturn_a<n> is the elevation of lowest return detected using algorithm setting group <n> 
- rh_a<n> are the relative height metrics at 1% intervals using algorithm <n> (in cm).

In some cases, the selection of an alternative algorithm setting will provide a better result. See the GEDI L02A User Guide for additional information.

Next, select the datasets for Algorithm Setting Group 3 (a3).

Repeat Section 4.2 for Algorithm Setting Group 3.

Notice above that the non-default algorithm RH metrics are stored in cm, so we convert to meters by dividing by 100.

Next, combine with the relative height metrics from the selected algorithm (Algorithm Setting Group 2):

Above we can see slight differences in the RH values between the two algorithms, particularly in the lower percentiles.

5. Plot Transects

Next, import a number of desired SDS layers for BEAM0110 (for the entire orbit) and create a pandas Dataframe to store the arrays.

Notice the unusual values listed above--those shots are flagged as poor quality and will be removed in Section 5.1.

Now that you have the desired SDS into a pandas dataframe, begin plotting the entire beam transect:

Congratulations! You have plotted your first GEDI sub-orbit beam transect. Notice above that things look a little messy--before we dive deeper into plotting full transects, let's quality filter the shots in the section below.

5.1 Quality Filtering

Now that you have the desired layers imported as a dataframe for the entire beam transect, let's perform quality filtering.

Below, remove any shots where the quality_flag is set to 0 by defining those shots as nan.

The syntax of the line below can be read as: in the dataframe, find the rows "where" the quality flag is not equal (ne) to 0. If a row (shot) does not meet the condition, set all values equal to nan for that row.

Below, quality filter even further by using the degrade_flag (Greater than zero if the shot occurs during a degrade period, zero otherwise) and the Sensitivity layer, using a threshold of 0.95.

From the GEDI Level 2 User Guide:

Use the sensitivity metric available in L2A and L2B to select "best" data. The L2A and L2B quality_flag datasets use a conservative sensitivity threshold of 0.9 over land (0.5 over ocean), but under some conditions (e.g. dense forest) the user may benefit from selecting a higher threshold.

Below, drop all of the shots that did not pass the quality filtering standards outlined above from the transectDF.

5.2 Plot Beam Transects

Next, plot the full remaining transect of high quality values using holoviews Scatter(). Combine the Tandem-X derived elevation, SRTM elevation (New to Version 2!), the GEDI-derived elevation, and the Canopy Top Elevation in a combined holoviews plot.

The plot still looks a bit messy this far zoomed out--feel free to pan, zoom, and explore different areas of the plot. The shot plotted in section 4 was at index 45730. If you zoom into the high-quality shots between 4.000e+5 and 5.000e+5, you will find the portion of the transect intersecting Redwood National Park, seen below:

alt text

5.3 Subset Beam Transects

Now, subset down to a smaller transect centered on the GEDI shot analyzed in the sections above.

Below, subset the transect using .loc.

5.4 Plot RH Metrics Transects

In order to get an idea of the length of the beam transect that you are plotting, you can plot the x-axis as distance, which is calculated below.

But how do you determine where the ground is? Next, create a similar plot but using the rh metrics from the L2A file. Notice that the rh metrics contain 100 values representing each percentile, so below we iterate through and append each value with the correct distance and elevation value to the list.

Now plot canopy height vs. shot index as a scatter plot.

Above, it looks like the transect is covering some tall trees! However, a quick google search confirms that Redwood trees can top out at over 100 m! The graph above is showing canopy height, but now add elevation to get a better idea of the overall three-dimensional structure of this transect across Redwood National Park.

Next, add in the ground elevation.

Above, you can get an idea about the terrain over the region of interest, particularly the classic "V" representing the river valley that is bisected by the transect. In terms of vegetation structure, this plot does a good job of showing not only which portions of the canopy are taller, but also where they are denser (darker shades of green).

At this point you have visualized the elevation, canopy, and vertical structure of specific footprints over Redwood National Park, and for a transect cutting through the national park. In section 6 you will look at mapping all of the high-quality shots from all eight GEDI beams for a given region of interest in order to gain knowledge on the spatial distribution and characteristics of the canopy over Redwood National Park.

6. Spatial Visualization

Section 6 combines many of the techniques learned above including how to import GEDI datasets, perform quality filtering, spatial subsetting, and visualization.

6.1 Import, Subset, and Quality Filter All Beams

Below, re-open the GEDI L2A observation--but this time, loop through and import data for all 8 of the GEDI beams.

Loop through each of the desired datasets (SDS) for each beam, append to lists, and transform into a pandas DataFrame.

Below, note that we are going to pull out the relative height metrics at the 25th, 98th, and 100th (canopy height) percentiles.

Note that the cell above may take a few seconds to complete.

6.2 Spatial Subsetting

Below, subset the pandas dataframe using a simple bounding box region of interest. If you are interested in spatially clipping GEDI shots to a GeoJSON region of interest, be sure to check out the GEDI-Subsetter python script available at: https://git.earthdata.nasa.gov/projects/LPDUR/repos/gedi-subsetter/browse.

Over 3.5 million shots are contained in this single GEDI orbit! Below subset down to only the shots falling within this small bounding box encompassing Redwood National Park. RedwoodNP our geopandas geodataframe can be called for the "envelope" or smallest bounding box encompassing the entire region of interest. Here, use that as the bounding box for subsetting the GEDI shots.

Filter by the bounding box, which is done similarly to filtering by quality in section 6.1 above.

NOTE: It may take up to a few minutes to run the cell above.

Notice you have drastically reduced the number of shots you are working with (which will greatly enhance your experience in plotting them below). But first, remove any poor quality shots that exist within the ROI.

Down to roughly 2000 shots, next create a Shapely Point out of each shot and insert it as the geometry column in the [soon to be geo]dataframe.

6.3 Visualize All Beams: Canopy Height and Elevation

Now, using the pointVisual function defined in section 3.2, plot the geopandas GeoDataFrame using geoviews.

Feel free to pan and zoom in to the GEDI shots in yellow.

Now let's plot the points in the geodataframe and add a colormap for Canopy Height (m) and Elevation (m).

Above and in the screenshot below, notice the higher canopy heights (shades of yellow) over the Redwood stands of the national park vs. other types of forests (pink-blue) vs. the low-lying (and consequently flat) profiles over lakes and rivers (purple).

alt text

Next, take a look at the GEDI-derived elevation over the shots. Notice below that the colormap is changed to 'terrain'.

Success! You have now learned how to start working with Version 2 GEDI L2A files in Python as well as some interesting strategies for visualizing those data in order to better understand your specific region of interest. Using this Jupyter Notebook as a workflow, you should now be able to switch to GEDI files over your specific region of interest and re-run the notebook. Good Luck!

7. Export Subsets as GeoJSON Files

In this section, export the GeoDataFrame as a .geojson file that can be easily opened in your favorite remote sensing and/or GIS software and will include an attribute table with all of the shots/values for each of the SDS layers in the dataframe.

Contact Information

Material written by Cole Krehbiel$^{1}$

    Contact: LPDAAC@usgs.gov
    Voice: +1-605-594-6116
    Organization: Land Processes Distributed Active Archive Center (LP DAAC)
    Website: https://lpdaac.usgs.gov/
    Date last modified: 04-20-2021
$^{1}$KBR Inc., contractor to the U.S. Geological Survey, Earth Resources Observation and Science (EROS) Center, Sioux Falls, South Dakota, 57198-001, USA. Work performed under USGS contract G15PD00467 for LP DAAC$^{2}$. $^{2}$LP DAAC Work performed under NASA contract NNG14HH33I.