Getting Started with the AρρEEARS API: Submitting and Downloading a Point Request

This tutorial demonstrates how to use Python to connect to the AρρEEARS API

The Application for Extracting and Exploring Analysis Ready Samples (AρρEEARS) offers a simple and efficient way to access and transform geospatial data from a variety of federal data archives in an easy-to-use web application interface. AρρEEARS enables users to subset geospatial data spatially, temporally, and by band/layer for point and area samples. AρρEEARS returns not only the requested data, but also the associated quality values, and offers interactive visualizations with summary statistics in the web interface. The AρρEEARS API offers users programmatic access to all features available in AρρEEARS, with the exception of visualizations. The API features are demonstrated in this notebook.


Example: Submit a point request with multiple points in U.S. National Parks for extracting vegetation and land surface temperature data

Connect to the AρρEEARS API, query the list of available products, submit a point sample request, download the request, become familiar with the AρρEEARS Quality API, and import the results into Python for visualization. AρρEEARS point requests allow users to subset their desired data using latitude/longitude geographic coordinate pairs (points) for a time period of interest, and for specific data layers within data products. AρρEEARS returns the valid data from the parameters defined within the sample request.

Data Used in the Example:


Topics Covered:

  1. Getting Started
    1a. Set Up the Working Environment
    1b. Login [Login]
  2. Query Available Products [Products API]
    2a. Search and Explore Available Products [List Products]
    2b. Search and Explore Available Layers [List Layers]
  3. Submit a Point Request [Tasks]
    3a. Compile a JSON [Task Object]
    3b. Submit a Task Request [Submit Task]
    3c. Retrieve Task Status [Retrieve Task]
  4. Download a Request [Bundle API]
    4a. Explore Files in Request Output [List Files]
    4b. Download Files in a Request (Automation) [Download File]
  5. Explore AρρEEARS Quality API [Quality API]
    5a. List Quality Layers [List Quality Layers]
    5b. Show Quality Values [List Quality Values]
    5c. Decode Quality Values [Decode Quality Values]
  6. BONUS: Import Request Output and Visualize
    6a. Import CSV
    6b. Plot Results (Line/Scatter Plots)

Dependencies:


AρρEEARS Information:

To access AρρEEARS, visit: https://appeears.earthdatacloud.nasa.gov/

For comprehensive documentation of the full functionality of the AρρEEARS API, please see the AρρEEARS API Documentation.

Throughout the tutorial, specific sections of the API documentation can be accessed by clicking on the bracketed [] links in the section headings.


Source Code used to Generate this Tutorial:


1. Getting Started


1a. Set Up the Working Environment

Import the required packages and set the input/working directory.

If you are missing any of the packages above, download them in order to use the full functionality of this tutorial.

If you plan to execute this tutorial on your own OS, `inDir` above needs to be changed.


1b. Login [Login]

To submit a request, you must first login to the AρρEEARS API. Use the getpass package to enter your NASA Earthdata login Username and Password. When prompted after executing the code block below, enter your username followed by your password.

Use the requests package to post your username and password. A successful login will provide you with a token to be used later in this tutorial to submit a request. For more information or if you are experiencing difficulties, please see the API Documentation.

Above, you should see a Bearer token. Notice that this token will expire approximately 48 hours after being acquired.


2. Query Available Products [Product API]

2a. Search and Explore Available Products [List Products]

The product API provides details about all of the products and layers available in AρρEEARS. Below, call the product API to list all of the products available in AρρEEARS.

Next, create a dictionary indexed by product name, making it easier to query a specific product.

The product service provides many useful details, including if a product is currently available in AρρEEARS, a description, and information on the spatial and temporal resolution.

Below, make a list of all product+version names, and search for products containing Leaf Area Index in their description.

Using the info above, start a list of desired products by using the highest temporal resolution LAI product, MCD15A3H.006.


2b. Search and Explore Available Layers [List Layers]

The product API allows you to call all of the layers available for a given product. Each product is referenced by its ProductAndVersion property. For a list of the layer names only, print the keys from the dictionary below.

Use the dictionary key 'LST_Day_1km' to see the information for that layer in the response.

AρρEEARS also allows subsetting data spectrally (by band). Create a tupled list with product name and specific layers desired.

Next, request the layers for the MCD15A3H.006 product.

Above, Lai_500m is the desired layer within the MCD15A3h.006 product.

Next, append Lai_500m to the tupled list of desired product/layers.

Below, take the tupled list (layers) and create a list of dictionaries to store each layer+product combination. This will make it easier to insert into the json file used to submit a request in Section 3.


3. Submit a Point Request [List Layers]

The Submit task API call provides a way to submit a new request to be processed. It can accept data via JSON, query string, or a combination of both. In the example below, compile a json and submit a request. Tasks in AρρEEARS correspond to each request associated with your user account. Therefore, each of the calls to this service requires an authentication token (see Section 1c.), which is stored in a header below.


3a. Compile a JSON [Task Object]

In this section, begin by setting up the information needed to compile an acceptable json for submitting an AρρEEARS point request. For detailed information on required json parameters, see the API Documentation.

For point requests, the coordinates property must also be inside the task object. Below, define a GeoJSON object containing the latitude and longitude (geographic projection) points desired. Optionally, set id and category properties to further identify your selected coordinates.

Finally, compile the JSON to be submitted as a point request, inserting additional parameters needed to complete the request.


3b. Submit a Task Request [Submit Task]

Below, post a call to the API task service, using the task json created above.


3c. Retrieve Task Status [Retrieve Task]

This API call will list all of the requests associated with your user account, automatically sorted by date descending with the most recent requests listed first.

The AρρEEARS API contains some helpful formatting resources. Below, limit the API response to 2 entries and set pretty to True to format the response as an organized json, making it easier to read. Additional information on AρρEEARS API pagination and formatting can be found in the API documentation.

Next, take the task id returned from the task_response that was generated when submitting your request, and use the AρρEEARS API status service to check the status of your request.

Above, if your request is still processing, you can find information on the status of how close it is to completing.

Below, call the task service for your request every 20 seconds to check the status of your request.


4. Download a Request [Bundle API]

Before downloading the request output, set up an output directory to store the output files, and examine the files contained in the request.


4a. Explore Files in Request Output [List Files]

The bundle API provides information about completed tasks. For any completed task, a bundle can be queried to return the files contained as a part of the task request. Below, call the bundle API and return all of the output files.


4b. Download Files in a Request (Automation) [Download File]

Now, use the contents of the bundle to select the file name and id and store as a dictionary.

Use the files dictionary and a for loop to automate downloading all of the output files into the output directory.


5. Explore AρρEEARS Quality API [Quality API]

The quality API provides quality details about all of the data products available in AρρEEARS. Below are examples of how to query the quality API for listing quality products, layers, and values. The final example (Section 5c.) demonstrates how AρρEEARS quality services can be leveraged to decode pertinent quality values for your data.

First, reset pagination to include offset which allows you to set the number of results to skip before starting to return entries. Next, make a call to list all of the data product layers and the associated quality product and layer information.


5a. List Quality Layers [List Quality Layers]

This API call will list all of the quality layer information for a product.


5b. Show Quality Values [List Quality Values]

This API call will list all of the values for a given quality layer.


5c. Decode Quality Values [Decode Quality Values]

This API call will decode the bits for a given quality value.


6. BONUS: Import Request Output and Visualize

Here, import the CSV file containing the results from your request, and create some basic visualizations using the matplotlib package.


6a. Import CSV

Use the Pandas package to import the CSV file containing the results from the AρρEEARS request.

Select the MOD11A2.061 LST Day column for the data from Grand Canyon National Park.

We will need to use the product service information to find the fill value and units for visualization purposes.


6b. Plot Results (Line/Scatter Plots)

Next, plot a time series of daytime LST for the selected point in Grand Canyon National Park for 2017.

Below, filter the LST data to exclude fill values, and plot as a time series with some additional formatting.

Next, add the LST Night values for Grand Canyon NP, and create a time series plot with both LST_Day_1km and LST_Night_1km.

Finally, bring in the daytime LST data from Zion National Park, and compare with daytime LST at Grand Canyon National Park, shown below in a scatterplot using matplotlib.

This example can provide a template to use for your own research workflows. Leveraging the AρρEEARS API for searching, extracting, and formatting analysis ready data, and importing it directly into Python means that you can keep your entire research workflow in a single software program, from start to finish.


Contact Information

Material written by LP DAAC$^{1}$

Contact: LPDAAC@usgs.gov

Voice: +1-605-594-6116

Organization: Land Processes Distributed Active Archive Center (LP DAAC)

Website: https://lpdaac.usgs.gov/

Date last modified: 04-28-2020

$^{1}$Innovate! Inc., contractor to the U.S. Geological Survey, Earth Resources Observation and Science (EROS) Center, Sioux Falls, South Dakota, 57198-001, USA. Work performed under USGS contract G15PD00467 for LP DAAC$^{2}$.

$^{2}$LP DAAC Work performed under NASA contract NNG14HH33I.