Tutorial WAPORWP Python notebooks on Google Colaboratory

Site: IHE DELFT OPEN COURSEWARE
Course: WaterPIP ToT2 – FAO WaPOR for Water Productivity - Part 1
Book: Tutorial WAPORWP Python notebooks on Google Colaboratory
Printed by: Guest user
Date: Tuesday, 26 October 2021, 11:32 AM

1. Introduction

WAPORWP is a standardized protocol for land and water productivity analyses using WaPOR database. It is aimed at guiding users to understand the different layers contained in the FAO Water Productivity Open-access portal (WaPOR) which can be used for land and water productivity analyses. 
WAPORWP provides python scripts which can be used to calculate land and water productivity and other performance indicators such as uniformity, efficiency (beneficial fraction), adequacy, relative water deficit as well as estimating productivity gaps. For each step, the protocol provides information about the assumptions used and provides links to reference materials. 
The protocol available on Github is for users who want to run analyses on their local machine (laptop or PC). However, users will need to install Python, Anaconda, and all dependencies of the Python scripts on their local machine.
For the learning purpose of this course, we will run WAPORWP on Google's cloud computer through Google Colaboratory (Colab) to skip installation steps. Colab allows anybody to write and execute arbitrary python code through the browser and is especially well suited to machine learning, data analysis, and education.

This tutorial will teach you the steps to
  • load WAPORWP protocol to Colab
  • connect Colab with Google Drive for data storage
  • run WAPORWP module 0 to bulk-download data to Google Drive

References
WAPORWP repository
WAPORWP wiki
'Water Productivity and Water Accounting using WaPOR' Open courseware

2. Set up WAPORWP on Google Colaboratory

Step 1: Download the WAPORWP zipfile from the course

Download WAPORWP for Colab zipfile

Step 2: Unzip the compressed WAPORWP folder 

If you don't have any program to unzip, you can download and install WinRAR for free.

unzip WAPORWP folder
Step 3: Log in Google Drive

Log in to https://drive.google.com/drive/my-drive with your Google account.

Step 4: Upload WAPORWP to Google Drive main folder 'My Drive'

Select the WAPORWP folder. Then, drag and drop it to Google Drive window to upload the whole unzipped folder. This step might take some time if the internet connection is poor.

Upload WAPORWP folder to Google Drive

Step 5: Connect Google Colaboratory app

Open the folder WAPORWP>Notebooks. In the folder Notebooks, select "Module_0_WaPOR_data_bulk_download.ipynb" and double-click to open. Select "Open with" and click "Connect more apps"

Open Module 0

Click on the search button (magnifying glass) and search for "colaboratory". Click on Colaboratory app and click Install. Click Continue and select your Google account to install the app.

Connect with colaboratory

After connecting the app with Drive, you can open jupyter notebook (.ipynb) file with Google Colaboratory

open with google colaboratory

3. Run WAPORWP Module 0

Open the notebook "Module_0_WaPOR_data_bulk_download.ipynb" in Google Colaboratory. In this notebook, there are 3 steps:
Step 1: Import libraries and mount to Google Drive

First, we need to mount the engine with google drive folder. Click the "Run" button of the code cell 


Click on the URL that appears, select your Google account, and allow the app to mount. Copy the authorization code and enter it in the notebook prompt.

Mount google drive

Run the next cell to install pyshp package

pip install pyshp

Run the next cell to import all libraries. When you import "WaPOR" module for the first time, you will be asked to enter WaPOR API token. Go to your WaPOR profile (My WaPOR>My Profile) and generate an API token. Copy and keep this token somewhere safe (e.g. a note file). Enter this token in the notebook prompt.

Import libraries, enter WaPOR API token

Step 2: Read geographical extent of the study area

Run the next two cells to read the extent of the study area from a shapefile. This will also plot the shapefile so that you can check how the shape looks like.

read extent and plot shapefile

In this example, we're using the shapefile of sugarcane field in Xinavane, Mozambique, which path is: "/content/drive/My Drive/WAPORWP/Data/1Boundary/Shapefile/Xinavane_1.shp".

When you do the analyses for your own study area, you will need to upload the shapefile of the new area to Google Drive and change this path name.

Step 3: Bulk-download WaPOR data for the study area extent

It will take about 3 hours to complete downloading all data for the Xinavane case to WAPORWP folder on Google Drive. However, the data for Xinavane has already been included in WAPORWP folder as an example. Therefore, for the purpose of following the tutorial you don't need to download data again. You can try to collect only Land Cover classification data to see how the script works.

If you run all the cells in step 3 and the script will automatically download raster data for these layers:
No.WaPOR DataSpatial resolutionTemporal resolutionTemporal coverage
1Actual Evapotranspiration & interception (AETI)100 mDekadal2015-2019
2Transpiration (T)100 mDekadal2015-2019
3Net Primary Production (NPP)100 mDekadal2015-2019
4Land cover classification (LCC)100 mAnnual2015-2019
5Precipitation (PCP)5 kmDekadal2015-2019
6Reference Evapotranspiration (RET)20 kmDekadal2015-2019

For example, the cell contains function used to download dekadal data of AETI for the geographical extent of the study area from 2015 to 2020 at level 2 (100m).


Download data

Tips:

Google Colab notebooks have an idle timeout of 90 minutes and absolute timeout of 12 hours. This means, if user does not interact with his Google Colab notebook for more than 90 minutes, its instance is automatically terminated. Also, maximum lifetime of a Colab instance is 12 hours. Therefore, to avoid timeout when you need to run the scripts for a long time on the cloud, you can do these following steps

In Google Chrome window, press Ctrl+Shift+I to open Developer tools. Under Console tab, copy and paste the following script and enter:

function ClickConnect(){ 
console.log("Working"); 
document.querySelector("#comments > span").click() 
}
setInterval(ClickConnect,5000)
Developer tools