ABSTRACT:

US States polygon shapefiles

ABSTRACT:

This shapefile contains flowlines of the NFIE Colorado Region.

ABSTRACT:

This shapefile contains flowlines of the NFIE Colorado Region.

ABSTRACT:

This gdal 2.1.2 binary was compiled on ubuntu 16.04 x64. This file link is referenced by a Dockerfile for dockerized Tethys on github, which does not allow to store file larger than 100mb.

ABSTRACT:

This resource contains a sqlite/spatialite geodatabase for assisting the workflow of subsetting NWM Ver1.1 netcdf.
1) grid cell polygon for land and forcing files
2) stream polyline (huc 8, 10, 12)
3) reservoir polygon

Download all the 5 split zip files into one folder and unzip the first one (nwm.zip.001) using 7z

ABSTRACT:

Subset National Water Model (NWM) Ver1.1 20170327 results for Bear River-Frontal Great Salt Lake HUC12 watershed (comid: 160102040504).
Python library used to prepare this data: https://pypi.python.org/pypi/subset_nwm_netcdf/

ABSTRACT:

This resource contains supporting files for subset_nwm_netcdf 1.1.3 (https://pypi.python.org/pypi/subset_nwm_netcdf/)
The split zip file nwm.zip.001 - 004 is the sqlite/spatialite geodatabase for stream, reservoir and watershed query.
xy_land_NAD1983.tif is for querying gird cell indices of NWM forcing and land files
xy_terrain_NAD1983.tif is for querying gird cell indices of NWM terrain files

ABSTRACT:

A subset of National Water Model (NWM) Ver1.1 20170404 results forTwoMileCreek watershed at Tuscaloosa, Alabama.
The watershed polygon is at https://www.hydroshare.org/resource/9d0e4cab63d74c0b8e6b6d83254c30de/

ABSTRACT:

TwoMileCreek watershed at Tuscaloosa, Alabama

ABSTRACT:

This resource contains supporting files for subset_nwm_netcdf 1.1.4-1.1.9 (https://pypi.python.org/pypi/subset_nwm_netcdf/)
The split zip file nwm.zip.001 - 004 is the sqlite/spatialite geodatabase for stream, reservoir and watershed query.

Download and unzip on Ubuntu
sudo apt-get install wget unzip
wget https://www.hydroshare.org/django_irods/download/23c05d3177654a9ab9dc9023d00d16ed/data/contents/nwm.zip.001
wget https://www.hydroshare.org/django_irods/download/23c05d3177654a9ab9dc9023d00d16ed/data/contents/nwm.zip.002
wget https://www.hydroshare.org/django_irods/download/23c05d3177654a9ab9dc9023d00d16ed/data/contents/nwm.zip.003
wget https://www.hydroshare.org/django_irods/download/23c05d3177654a9ab9dc9023d00d16ed/data/contents/nwm.zip.004
cat nwm.zip.* > nwm.zip
rm nwm.zip.*
unzip nwm.zip
rm nwm.zip

ABSTRACT:

A Subset of National Water Model (NWM) Ver1.1 20170419 results forTwoMileCreek watershed at Tuscaloosa, Alabama.
The watershed polygon is at https://www.hydroshare.org/resource/9d0e4cab63d74c0b8e6b6d83254c30de/

ABSTRACT:

bash script to download NWM v1.1 outputs

ABSTRACT:

Hand drawn Utah state border in geojson featurecollection format Projection: WGS84 (EPSG: 4326). This resource was created to test NWM viewer app.

ABSTRACT:

Hand drawn Alabama state border in geojson polygon format. This resource was created to test NWM viewer app.

ABSTRACT:

NWM v1.1 forcing_analysis_assimilation files of TwoMileCreek watershed region for 20170419

ABSTRACT:

This is a collection of step by step demonstrations on how to use HydroShare Apps.

ABSTRACT:

This is a step-by-step demonstration of how to Add Images, PDFs, and Videos to digital maps using the HydroShare GIS App using an example from this related HydroShare resource: Ames, D. (2016). Algae Growth in Utah Lake Time-lapse, HydroShare, http://www.hydroshare.org/resource/4c8ecb05a72647339df0df6e9a87718f

ABSTRACT:

This is a step-by-step demonstration of how to view and download forecasts from any stream in the National Hydrography Dataset with the National Water Model App.

ABSTRACT:

stream_sld

ABSTRACT:

Thunder Creek, Skagit River Basin, State of Washington, USA.

ABSTRACT:

The NWM Viewer app has 2 main features provided in Home Mode and Subset Mode respectively:
Home Mode: Retrieve and View NWM Time Series for a single stream reach, reservoir or grid cell.
Subset Mode: Subset NWM outputs (NetCDF files) with a watershed polygon to get 'shrunken' NetCDFs that only contain data for a specific area.

ABSTRACT:

The Data Rods Explorer (DRE) is a web client app that enables users to browse several NASA-hosted data sets. The interface enables visualization and download of NASA observation retrievals and land surface model (LSM) outputs by variable, space and time. The key variables are precipitation, wind, temperature, surface downward radiation flux, heat flux, humidity, soil moisture, groundwater, runoff, and evapotranspiration. These variables describe the main components of the water cycle over land masses.

ABSTRACT:

Rocky River HUC10 0303000305 at Raleigh NC

ABSTRACT:

This gdal 2.2.3 binary was compiled on ubuntu 16.04 x64. This file link is referenced by a Dockerfile for dockerized Tethys on github, which does not allow to store file larger than 100mb.

./configure --with-python

GDAL is now configured for x86_64-pc-linux-gnu

Installation directory: /root/gdal223build
C compiler: gcc -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++ compiler: g++ -std=gnu++11 -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++11 support: yes

LIBTOOL support: yes

LIBZ support: external
LIBLZMA support: no
cryptopp support: no
GRASS support: no
CFITSIO support: no
PCRaster support: internal
LIBPNG support: internal
DDS support: no
GTA support: no
LIBTIFF support: internal (BigTIFF=yes)
LIBGEOTIFF support: internal
LIBJPEG support: external
12 bit JPEG: no
12 bit JPEG-in-TIFF: no
LIBGIF support: internal
OGDI support: no
HDF4 support: no
HDF5 support: yes
Kea support: no
NetCDF support: yes
Kakadu support: no
JasPer support: no
OpenJPEG support: no
ECW support: no
MrSID support: no
MrSID/MG4 Lidar support: no
JP2Lura support: no
MSG support: no
GRIB support: yes
EPSILON support: no
WebP support: no
cURL support (wms/wcs/...):yes
PostgreSQL support: yes
MRF support: yes
MySQL support: no
Ingres support: no
Xerces-C support: no
NAS support: no
Expat support: yes
libxml2 support: yes
Google libkml support: no
ODBC support: no
PGeo support: no
FGDB support: no
MDB support: no
PCIDSK support: internal
OCI support: no
GEORASTER support: no
SDE support: no
Rasdaman support: no
DODS support: no
SQLite support: yes
PCRE support: no
SpatiaLite support: no
RasterLite2 support: no
Teigha (DWG and DGNv8): no
INFORMIX DataBlade support:no
GEOS support: yes
SFCGAL support: no
QHull support: internal
Poppler support: no
Podofo support: no
PDFium support: no
OpenCL support: no
Armadillo support: no
FreeXL support: no
SOSI support: no
MongoDB support: no

SWIG Bindings: python

Statically link PROJ.4: no
enable GNM building: yes
enable pthread support: yes
enable POSIX iconv support:yes
hide internal symbols: no

ABSTRACT:

This GDAL2.3-dev-bb4c395 binary was compiled on ubuntu 16.04 x64. This file link is referenced by a Dockerfile for dockerized Tethys on github, which does not allow to store file larger than 100mb.

./configure --with-python
GDAL is now configured for x86_64-pc-linux-gnu

Installation directory: /root/gdal_23_dev_bb4c395_binary
C compiler: gcc -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++ compiler: g++ -std=c++11 -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2
C++14 support: no

LIBTOOL support: yes

LIBZ support: external
LIBLZMA support: no
cryptopp support: no
crypto/openssl support: yes
GRASS support: no
CFITSIO support: no
PCRaster support: internal
LIBPNG support: internal
DDS support: no
GTA support: no
LIBTIFF support: internal (BigTIFF=yes)
LIBGEOTIFF support: internal
LIBJPEG support: external
12 bit JPEG: no
12 bit JPEG-in-TIFF: no
LIBGIF support: internal
JPEG-Lossless/CharLS: no
OGDI support: no
HDF4 support: no
HDF5 support: yes
Kea support: no
NetCDF support: yes
Kakadu support: no
JasPer support: no
OpenJPEG support: no
ECW support: no
MrSID support: no
MrSID/MG4 Lidar support: no
JP2Lura support: no
MSG support: no
GRIB support: yes
EPSILON support: no
WebP support: no
cURL support (wms/wcs/...):yes
PostgreSQL support: yes
MRF support: yes
MySQL support: no
Ingres support: no
Xerces-C support: no
NAS support: no
Expat support: yes
libxml2 support: yes
Google libkml support: no
ODBC support: no
PGeo support: no
FGDB support: no
MDB support: no
PCIDSK support: internal
OCI support: no
GEORASTER support: no
SDE support: no
Rasdaman support: no
DODS support: no
SQLite support: yes
PCRE support: no
SpatiaLite support: no
RasterLite2 support: no
Teigha (DWG and DGNv8): no
INFORMIX DataBlade support:no
GEOS support: yes
SFCGAL support: no
QHull support: internal
Poppler support: no
Podofo support: no
PDFium support: no
OpenCL support: no
Armadillo support: no
FreeXL support: no
SOSI support: no
MongoDB support: no

SWIG Bindings: python

Statically link PROJ.4: no
enable GNM building: yes
enable pthread support: yes
enable POSIX iconv support:yes
hide internal symbols: no

ABSTRACT:

This resource contains GIS data for National Water Model Viewer App in ArcGIS geodatabase format:
channels : line--2.7 million NHD+ stream reaches
reservoirs: point--1260 reservoirs locations
usgs_gauge: point--NHD+ USGS gauges locations
grid_land: polygon--grid cell polygons for land and forcing outputs

nwm_app_data.mxd : ArcMap 10 project file

ABSTRACT:

LittleWashita HUC10 1113030208 Polygon in OH

ABSTRACT:

Milburnie Lake - Neuse River HUC10 Watershed 0302020107 at Durham-Raleigh, NC

ABSTRACT:

croton_NY_domain_polygon

ABSTRACT:

Jupyter environment set in CyberGIS Center for interaction with HPC

ABSTRACT:

The US National Water Model (NWM) is a mesoscale hydrologic model that provides streamflow forecasts and other valuable hydrologic information for the continental United States. Since its release at the National Water Center (NWC) in 2016, the NWM has garnered broad attention and great interest across the hydrology science community. Several projects are underway with the goal of delivering this advanced modeling technique and its data to researchers and end users in the community.
As one of the flagship projects, the CUAHSI HydroShare project is working toward providing a complete solution for storing, managing and sharing NWM data. So far, it has set up the largest open-access data archive for the NWM outputs and has developed several different open-source tools and web applications assisting users with data access.
However, there is a growing demand in the hydrologic sciences community for the capability to run a local instance of NWM at regional watersheds to support research applications such as cross-model comparison, historical data analysis and etc.
In this paper, we present ongoing work to develop web applications on top of HydroShare for collecting NWM input data to support model execution at smaller scale watersheds.

ABSTRACT:

Place Holder

ABSTRACT:

This example test case includes a small region (15km by 16km) encompassing the West Branch of the Croton River, NY, USA (USGS stream gage 0137462010) during hurricane Irene, 2011-08-26 to 2011-09-02. The simulation begins with a restart from a spinup period from 2010-10-01 to 2011-08-26. There are 3 basic routing configurations included in the test case, National Water Model (NWM), Gridded, and NCAR Reach.

ABSTRACT:

czo community sql files

ABSTRACT:

watershed test

ABSTRACT:

Dev deploy for CyberGIS-Jupyter for Water

https://hsjp07.cigi.illinois.edu/hydroshare/login?next=/hub/spawn/{_HS_USR_NAME_}?next=/hub/user/{_HS_USR_NAME_}/hs-pull?id=${HS_RES_ID}%2526subfolder=Downloads

https://hsjp07.cigi.illinois.edu/hydroshare/login?next=/hub/spawn/{_HS_USR_NAME_}?next=/hub/user/{_HS_USR_NAME_}/hs-pull?start=${HS_FILE_PATH}%2526id=${HS_RES_ID}%2526subfolder=Downloads

ABSTRACT:

The CyberGIS-Jupyter for Water (CJW) platform aims to advance community hydrologic modelling, and support data-intensive, reproducible, and computationally scalable water science research by simplifying access to advanced cyberGIS and cyberinfrastructure capabilities through a friendly Jupyter Notebook environment. The current release has specific support for the Structure For Unifying Multiple Modeling Alternatives (SUMMA) model and the WRFHydro model.

You may open and view any notebook (*.ipynb file) with this app.

Please send comments and bug reports to help@cybergis.org.

ABSTRACT:

This resources contains 3 notebooks that walk you through running a SUMMA model on CyberGIS-Jupyter for Water platform.

summa_local.ipynb --- run a summa model on Jupyter notebook server (container) directly
summa_hpc.ipynb --- submit summa model as a job to XSEDE COMET High Performance Computing (HPC) cluster
summa_ensemble_hpc.ipynb --- submit an ensemble summa mode as a job to XSEDE COMET High Performance Computing (HPC) cluster

ABSTRACT:

User may want to set up a custom Python environment (kernel) and run notebooks with it on the CyberGIS-Jupyter for Water platform. This resource has 2 notebooks to demonstrate walk you through the steps.

install_custom_python_environment.ipynb --- install a custom Python env with user-defined libraries using conda, and set it as a Jupyter Kernel.
recover_custom_python_environment.ipynb --- after a container rebuild, user needs to reactivate a previously installed custom Python env.

ABSTRACT:

This Jupyter notebook illustrates the HAND workflow and its use in example flood emergency scenarios. The study area is Onion Creek (HUC10 code 1209020504).

This is also a demonstration of conducting geospatial anlysis with opensource toolkits (gdal) using an online Jupyter interface.

Environment required: CyberGIS-Jupyter for Water

ABSTRACT:

asd

ABSTRACT:

This example demonstrates how to use PostGIS capabilities in CyberGIS-Jupyter notebook environment. Modified from notebook by Weiye Chen (weiyec2@illinois.edu)

PostGIS is an extension to the PostgreSQL object-relational database system which allows GIS (Geographic Information Systems) objects to be stored in the database. PostGIS includes support for GiST-based R-Tree spatial indices, and functions for analysis and processing of GIS objects.

Resources for PostGIS:

Manual https://postgis.net/docs/
In this demo, we use PostGIS 3.0. Note that significant changes in APIs have been made to PostGIS compared to version 2.x. This demo assumes that you have basic knowledge of SQL.

ABSTRACT:

These are examples to test Data Processing Kernel in CyberGIS-Jupyter for water.
The 1_watershed_delineation folder is an example of a watershed delineation which is the basic step to analyze an interesting watershed. We used GRASS GIS 7.8 version and shell script to apply GRASS GIS library.
The 2_map_visualization folder is an example of an interactive map visualization which is the high-level visualization using PyViz tools as post-processing of environmental modeling. For this example, we used the following PyViz tools:
- geopandas (https://geopandas.org/), cartopy (https://scitools.org.uk/cartopy/), geoviews (https://geoviews.org/), and holoviews (https://holoviews.org/)

ABSTRACT:

!!! This is a fork from https://www.hydroshare.org/resource/5b964154ebf945848087bdc772cc921e/ with some minor modifications for CyberGIS-Jupyer for Water (CJW) platform !!!
----------------------------------------------------------------------------------------------------
The ability to test hypotheses about hydrology, geomorphology, and atmospheric processes is invaluable to research in the Earth and planetary sciences. To swiftly develop experiments using community resources is an extraordinary emerging opportunity to accelerate the rate of scientific advancement. Knowledge infrastructure is an intellectual framework to understand how people are creating, sharing, and distributing knowledge -- which has dramatically changed and is continually transformed by Internet technologies. We are actively designing a knowledge infrastructure system for earth surface investigations. In this paper, we illustrate how this infrastructure can be utilized to lower common barriers to reproducing modeling experiments. These barriers include: developing education and training materials for classroom use, publishing research that can be replicated by reviewers and readers, and advancing collaborative research by re-using earth surface models in new locations or in new applications. We outline six critical elements to this infrastructure, 1) design of workflows for ease of use by new users; 2) a community-supported collaborative web platform that supports publishing and privacy; 3) data storage that may be distributed to different locations; 4) a software environment; 5) a personalized cloud-based high performance computing (HPC) platform; and 6) a standardized modeling framework that is growing with open source contributions. Our methodology uses the following tools to meet the above functional requirements. Landlab is an open-source modeling toolkit for building, coupling, and exploring two-dimensional numerical models. The Consortium of Universities Allied for Hydrologic Science (CUAHSI) supports the development and maintenance of a JupyterHub server that provides the software environment for the system. Data storage and web access are provided by HydroShare, an online collaborative environment for sharing data and models. The knowledge infrastructure system accelerates knowledge development by providing a suite of modular and interoperable process components that can be combined to create an integrated model. Online collaboration functions provide multiple levels of sharing and privacy settings, open source license options, and DOI publishing, and cloud access to high-speed processing. This allows students, domain experts, collaborators, researcher, and sponsors to interactively execute and explore shared data and modeling resources. Our system is designed to support the user experiences on the continuum from fully developed modeling applications to prototyping new science tools. We have provided three computational narratives for readers to interact with hands-on, problem-based research demonstrations - these are publicly available Jupyter Notebooks available on HydroShare.

To interactively compute with these Notebooks, please see the ReadMe below.
To develop these Notebooks, go to Github: https://github.com/ChristinaB/pub_bandaragoda_etal_ems or https://zenodo.org/badge/latestdoi/187289993

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2020 Q2)

Dear HydroShare Users,

We are pleased to announce a new quarterly release of CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes new capabilities to support the geoanalytics suite of GRASS for model pre/post-processing, PostGIS database, and Landlab Earth surface modelling toolkit along with several enhancements to job submission middleware, system security as well as service infrastructure. Please refer to the following list for details and examples.

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

GRASS GIS for model pre/post-processing:
Learn how to consolidate the features of the GRASS geoanalytics suite to support pre/post-processing for SUMMA and RHESSYs models in CJW.
Example notebooks: https://www.hydroshare.org/resource/4cbcfdd6e7f943e2969dd52e780bc52d/

Manage geospatial data with PostGIS:
PostGIS is an extension to the PostgreSQL object-relational database system which allows geospatial data to be efficiently stored while providing various advanced functions for in-situ data analysis and processing.
Example notebooks: https://www.hydroshare.org/resource/bb779d4cce564dd6afcf463c8910786f/

Security and service infrastructure enhancements
Trusted group: Starting from this release, all users are required to join the “CyberGIS-Jupyter for Water” trusted group at https://www.hydroshare.org/group/157 in order to access the CJW platform, which is a preventive measure to protect the shared computing resources from being abused by malicious users. A complete user profile page is highly recommended to expedite the approval process.
User metric submission to XSEDE: CJW, as a science gateway, is now sending unique user usage metrics to XSEDE to comply with its requirements.

Landlab for enabling collaborative numerical modeling in Earth sciences using knowledge infrastructure
Example notebooks: https://www.hydroshare.org/resource/370c288b61b84794b847ef85c4dd4ffb/
https://www.hydroshare.org/resource/6add6bee06bb4050bfe23e1081627614/

Job submission enhancements
Refactored the structure of the cyberGIS job submission system
Data-driven implementation for avoiding excessive data transmission between HydroShare and CJW
Add the specification of input parameters into a JSON file to improve the flexibility and generality of model management
Enable HPC-SUMMA object that can directly call SUMMA
Example notebooks: https://www.hydroshare.org/resource/4a4a22a69f92497ead81cc48700ba8f8/

ABSTRACT:

This is an example of watershed delineation which is the basic step to analyze an interesting watershed. We used GRASS GIS 7.8 version and shell script to apply GRASS GIS library.

ABSTRACT:

These is an examples to test Data Processing Kernel in CyberGIS-Jupyter for water.
The 2_map_visualization folder is an example of an interactive map visualization which is the high-level visualization using PyViz tools as post-processing of environmental modeling. For this example, we used the following PyViz tools:
- geopandas (https://geopandas.org/), cartopy (https://scitools.org.uk/cartopy/), geoviews (https://geoviews.org/), and holoviews (https://holoviews.org/)

ABSTRACT:

sciunit container

ABSTRACT:

This HydroShare resource provides a Singularity image for Local Approach 4: Containerizing all software with Singularity for "Comparing containerization-based approaches for reproducible computational modeling of environmental systems" manuscript in Environmental Modeling and Software Journal.

For more detailed information, please see this GitHub
Git-3. Description of Approach-4 to show how to use a Singularity image (https://github.com/uva-hydroinformatics/SUMMA_Singularity_In_Rivanna.git)

ABSTRACT:

This a reproducible demonstration of the landslide modeling results from eSurf paper: Strauch et al. (2018)

ABSTRACT:

This example is to show the steps to run an ensemble SUMMA3.0 on HPC through the CyberGIS-Compute Service.

ABSTRACT:

This resource was created using CAMELS (https://ral.ucar.edu/solutions/products/camels) `TIME SERIES NLDAS forced model output` from 1980 to 2018.
The original NLDAS (North American Land Data Assimilation System) hourly forcing data was created by NOAA by 0.125 x 0.125 degree grid.
Through creating CAMELS datasets, hourly forcing data were reaggregated to 671 basins in the USA.
In this study, we merged all CAMELS forcing data into one NetCDF file to take advantage of OPeNDAP (http://hyrax.hydroshare.org/opendap/hyrax/) in HydroShare.
Currently, using SUMMA CAMELS notebooks (https://www.hydroshare.org/resource/ac54c804641b40e2b33c746336a7517e/), we can extract forcing data to simulate SUMMA in the particular basins in 671 basins of CAMELS datasets.

ABSTRACT:

[You can run this model with the notebook at https://www.hydroshare.org/resource/8fe974c108ca4c6eaaf9b060779329b0/ in CyberGIS-Jupyter for Water platform]

WRFHydro Test Case -- Croton River, NY

#Overview This test case includes prepared geospatial data and input files for a
sample domain (region of interest) and prepared forcing data. This domain is a small region (15km x 16km) encompassing the West Branch
of the Croton River, NY, USA (USGS stream gage 0137462010) during hurricane
Irene, 2011-08-26 to 2011-09-02. The simulation begins with a restart from a
spinup period from 2010-10-01 to 2011-08-26. The forcing data
prepared for this test case is North American Land Data Assimilation System
(NLDAS) hourly data. There are 3 basic routing
configurations included in the test case, National Water Model (NWM), Gridded,
and NCAR Reach. See the WRF-Hydro V5 Technical Description located at
https://ral.ucar.edu/projects/wrf_hydro for a more detailed description of model
physics options, configurations, and input files.

ABSTRACT:

The goal of this notebook is to show the steps to run an example National Water Model (WRFhydro) model on HPC resources through the CyberGIS-Compute Service. This notebook uses wrfhydropy, a Python wrapper for WRFHydro, in model preprocessing and postprocessing, and the resulting ready-to-run model is handed over to CyberGIS Computing Service for execution on a supported HPC resource (Virtual Roger/Keeling at UIUC or XSEDE COMET at SDSC). This example is adapted from the "ex_01_end_to_end.ipynb" notebook from wrfhydropy official github repo https://github.com/NCAR/wrf_hydro_py, and users are encouraged to refer to the tutorials there to get familair with wrfhydropy usages.

How to run:
1. Request to join the CyberGIS-Jupyter for Water group at https://www.hydroshare.org/group/157
2. Click the "Open with ..." button in the upper-right
3. Select "CyberGIS-Jupyter for Water"
4. Run through the notebook

ABSTRACT:

RHESSys East Coast version v7.2

ABSTRACT:

These are example application notebooks to simulate SUMMA using CAMELS datasets.
There are three steps: (STEP-1) Create SUMMA input, (STEP-2) Execute SUMMA, (STEP-3) Visualize SUMMA output
Based on this example, users can change the HRU ID and simulation periods to analyze 671 basins in CAMELS datasets.

(STEP-1) A_1_camels_make_input.ipynb
- The first notebook creates SUMMA input using Camels dataset using `summa_camels_hydroshare.zip` in this resource and OpenDAP(https://www.hydroshare.org/resource/a28685d2dd584fe5885fc368cb76ff2a/).
(STEP-2) B_1_camels_pysumma_default_prob.ipynb, B_2_camels_pysumma_lhs_prob.ipynb, B_3_camels_pysumma_config_prob.ipynb, and
B_4_camels_pysumma_lhs_config_prob.ipynb
- These four notebooks execute SUMMA considering four different parameters and parameterization combinations
(STEP-3) C_1_camels_analyze_output_default_prob.ipynb, C_2_camels_analyze_output_lhs_prob.ipynb, C_3_camels_analyze_output_config_prob.ipynb,
C_4_camels_analyze_output_lhs_config_prob.ipynb
- The final four notebooks visualize SUMMA output of B-1, B-2, B-3, and B-4 notebooks.

ABSTRACT:

This resource, configured for execution in connected JupyterHub compute platforms using the CyberGIS-Jupyter for Water (CJW) environment's supported High-Performance Computing (HPC) resource (XSEDE Comet) through CyberGIS-Compute Service, helps the modelers to reproduce and build on the results from the paper (Van Beusekom et al., 2021).

For this purpose, three different Jupyter notebooks are developed and included in this resource which explore the paper goal for one example CAMELS site and a pre-selected period of 18-month simulation to demonstrate the capabilities of the notebooks. The first notebook processes the raw input data from CAMELS dataset to be used as input for SUMMA model. The second notebook utilizes the CJW environment's supported HPC resource (XSEDE Comet) through CyberGIS-Compute Service to executes SUMMA model. This notebook uses the input data from first notebook using original and altered forcing, as per further described in the notebook. Finally, the third notebook utilizes the outputs from notebook 2 and visualizes the sensitivity of SUMMA model outputs using Kling-Gupta Efficiency (KGE). More information about each Jupyter notebook and a step-by-step instructions on how to run the notebooks can be found in the Readme.md fie included in this resource. Using these three notebooks, modelers can apply the methodology mentioned above to any (one to all) of the 671 CAMELS basins and simulation periods of their choice. As this resource uses HPC, it enables a high-speed running of simulations which makes it suitable for larger simulations (even as large as the entire 671 CAMELS sites and the whole 60-month simulation period used in the paper) practical and much faster than when no HPC is used.

ABSTRACT:

This notebook is created to support SUMMA general application workflows using CAMELS forcing, watershed attributes, and streamflow observation.
CAMELS datasets cover 671 basins across the USA, so users can apply SUMMA models in 671 basins.

ABSTRACT:

RHESSys (Regional Hydro-Ecological Simulation System) is a GIS-based, terrestrial ecohydrological modeling framework designed to simulate carbon, water and nutrient fluxes at the watershed scale. RHESSys models the temporal and spatial variability of ecosystem processes and interactions at a daily time step over multiple years by combining a set of physically-based process models and a methodology for partitioning and parameterizing the landscape. Detailed model algorithms are available in Tague and Band (2004).

This notebook demonstrates parallel job submissions of RHESSys ensemble simulations from CyberGIS-Jupyer for water to HPC (XSEDE), visualizes RHESSys output, and evaluate RHESSys efficiency with simulation runoff and observation streamflow

ABSTRACT:

This resource, configured for execution in connected JupyterHub compute platforms, helps the modelers to reproduce and build on the results from the paper (Van Beusekom et al., 2021). For this purpose, three different Jupyter notebooks are developed and included in this resource which explore the paper goal for one example CAMELS site and a pre-selected period of 18-month simulation to demonstrate the capabilities of the notebooks. The first notebook processes the raw input data from CAMELS dataset to be used as input for SUMMA model. The second notebook executes SUMMA model using the input data from first notebook using original and altered forcing, as per further described in the notebook. Finally, the third notebook utilizes the outputs from notebook 2 and visualizes the sensitivity of SUMMA model outputs using Kling-Gupta Efficiency (KGE). More information about each Jupyter notebook and a step-by-step instructions on how to run the notebooks can be found in the Readme.md fie included in this resource. Using these three notebooks, modelers can apply the methodology mentioned above to any (one to all) of the 671 CAMELS basins and simulation periods of their choice.

ABSTRACT:

CAMELS (Catchment Attributes and Meteorology for Large-sample Studies: https://ral.ucar.edu/solutions/products/camels) is a large-sample hydrometeorological dataset that provides catchment attributes and forcings for 671 small- to medium-sized basins across the CONUS.

This resource contains basin attributes and parameters in NetCDF files.

ABSTRACT:

CAMELS (Catchment Attributes and Meteorology for Large-sample Studies) is a large-sample hydrometeorological dataset that provides catchment attributes, forcings and GIS data for 671 small- to medium-sized basins across the CONUS (continental United States). HydroShare hosts a copy of CAMELS and exposes it through different public data access protocols (WMS, WFS and OPeNDAP) for easy visualization and subsetting of the dataset in community modeling research. This notebook demostrates how to set up SUMMA models with CAMELS dataset from HydroShare using various tools integrated in the CyberGIS-Jupyter for Water (CJW) environment and execution of ensemble model runs on a supported High-Performance Computing (HPC) resource (XSEDE Comet or UIUC Virtual Roger) through CyberGIS-Compute Service.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

RHESSys (Regional Hydro-Ecological Simulation System) is a GIS-based, terrestrial ecohydrologic modeling framework designed to simulate carbon, water and nutrient fluxes at the watershed scale. RHESSys models the temporal and spatial variability of ecosystem processes and interactions at a daily time step over multiple years by combining a set of physically based process models and a methodology for partitioning and parameterizing the landscape. Detailed model algorithms are available in Tague and Band (2004).

This notebook demonstrates how to configure an ensemble RHESSys simulation with pyRHESSys, submit it to a supported HPC resource (XSEDE COMET or UIUC Virtual Roger) for execution through CyberGIS Computing Service, visualize model outputs with various tooks integrated in the CyberGIS-Jupyter for Water (CJW).

The model used here is based off of a pre-built RHESSys model for the Coweeta Subbasin 18 (0.124 𝑘𝑚2 ), a subbasins in Coweeta watershed (16 𝑘𝑚2 ), from the Coweeta Long Term Ecological Research (LTER) Program.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as below.

1) Modeling CAMELS Basins with SUMMA: CAMELS (Catchment Attributes and Meteorology for Large-sample Studies: https://ral.ucar.edu/solutions/products/camels) is a large-sample hydrometeorological dataset that provides catchment attributes and forcings for 671 small- to medium-sized basins across the CONUS. In this release, CJW has included enhancements and new features that support the end-to-end workflow of CAMELS basin modeling with SUMMA. An example notebook is provided to walk users through several essential steps including basin data retrieval and subsetting, setup of single and ensemble models, computation job submission and execution, and model output visualization.

2) RHESSys support via CyberGIS Computing Service: CJW now supports running ensemble RHESSys models on HPC (High-Performance Computing) resources through the newly upgraded CyberGIS Computing Service. Also, the RHESSys Jupyter kernel has been updated to include the latest version of “pyRHESSys” (https://github.com/uva-hydroinformatics/pyRHESSys) and other new tools for model configuration, output analysis, and visualization. See the example notebook below for more details.

3) User testing of Kubernetes-based CJW instance: A newly deployed CJW instance powered by Kubernetes (Aka K8s: https://kubernetes.io/) is now available for user testing at https://go.illinois.edu/cjw-k8s. The adoption of this most sought-after and cutting-edge cloud application deployment technology is expected to significantly enhance the availability and scalability of CJW as we have observed increasing user demand and a surge in new user registrations. We welcome all users to join this testing process and would greatly appreciate your feedback. We anticipate the user testing on the new CJW instance will take 1-3 months, during which the current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final migration plan will be implemented before the next quarterly release of CJW.

Please refer to the following HydroShare resources for details and examples:
Modeling CAMELS Basins with SUMMA:
https://www.hydroshare.org/resource/17bc4f0031554944b8ec7558fd9ee3c2/

Run Ensemble RHESSys models on HPC through CyberGIS Computing Service:
https://www.hydroshare.org/resource/631914af4b8344e5a78e647255cf1d13/

Direct Access to Kubernetes-based CJW:
https://go.illinois.edu/cjw-k8s

Set up OpenWith for Kubernetes-based CJW:
https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/

See Release Notes on HydroShare
https://www.hydroshare.org/resource/54f3ec517ba44a83bb486e7d6c4edceb/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

ABSTRACT:

How to Fix the Side Effect caused by New SSL Cert on HydroShare

Revisions:
March 18, 2021; Zhiyu/Drew Li; zhiyul@illinois.edu

Symptoms:
Jupyter Hub fails in OAuth handshaking with HydroShare
“HTTP 599: server certificate verification failed. CAfile: none CRLfile: none”
hs_restclient fails to authenticate
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.hydroshare.org', port=443): Max retries exceeded with url: /hsapi/userInfo/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))

Cause:
HydroShare deployed a new SSL cert on March 17, 202. It is based on off a new CA, which is NOT included in the latest “ca-certificates” package (CA Bundle) on Ubuntu 18.04 and 20.04 as of this writing (other Linux distribution may also be affected).

Remedy:
Manually add this new CA into the CA Bundle on all clients that might need to talk to HydroShare.

Download the new CA cert:
Go to HydroShare keybase and download: star_hydroshare_org_124173627DigiCertCA.crt
Go to https://www.digicert.com/kb/digicert-root-certificates.htm, search for “GeoTrust TLS DV RSA Mixed SHA256 2020 CA-1” and download PEM format.

For Hub Dockerfile:

USER root
# get latest ca-bundle
RUN apt-get update && apt-get install -y ca-certificates
# load hydroshare new ca to image
COPY ./star_hydroshare_org_124173627DigiCertCA.crt /usr/local/share/ca-certificates/star_hydroshare_org_124173627DigiCertCA.crt
# update ca-bundle
RUN update-ca-certificates

For different conda envs in Dockerfile:

#Append new HydroShare CA to cacert.pem in Base conda env
RUN cat ./star_hydroshare_org_124173627DigiCertCA.crt >> /opt/conda/lib/python<VERSION>/site-packages/certifi/cacert.pem
# Append new HydroShare CA to user-created conda env
RUN cat ./star_hydroshare_org_124173627DigiCertCA.crt >> /opt/conda/envs/<ENV_NAME>/lib/python<VERSION>/site-packages/certifi/cacert.pem

References:
https://incognitjoe.github.io/adding-certs-to-requests.html
https://www.techrepublic.com/article/how-to-install-ca-certificates-in-ubuntu-server/

ABSTRACT:

The overall goal of this collection is to use the basic strategy and architecture presented by Choi et al. (2021) to make components of a modern and complex hydrologic modeling study (VB study; Van Beusekom et al., 2022) easier to reproduce. The design and implemention of the developed cyberinfrastructure to achieve this goal are fully explained by Maghami et al. (2023).

In VB study, hydrological outputs from the SUMMA model for the 671 CAMELS catchments across the contiguous United States (CONUS) and a 60-month actual simulation period are investigated to understand their dependence on input forcing behavior across CONUS. VB study layes out a simple methodology that can be applied to understand the relative importance of seven model forcings (precipitation rate, air temperature, longwave radiation, specific humidity, shortwave radiation, wind speed, and air pressure).

Choi et al. (2021) integrated three components through seamless data transfers for a reproducible research: (1) online data and model repositories; (2) computational environments leveraging containerization and self-documented computational notebooks; and (3) Application Programming Interfaces (APIs) that provide programmatic control of complex computational models.

Therefore, Maghami et al. (2023), integrated the following three components through seamless data transfers to make components of a modern and complex hydrologic study (VB study) easier to reproduce:
(1) HydroShare as online data and model repository;
(2) CyberGIS-Jupyter for Water for self-documented computational notebooks as computational environment (with and without HPC notebooks);
(3) pySUMMA as Application Programming Interfaces (APIs) that provide programmatic control of complex computational models.

This collection includes three resources:

1- First resource, provides the entire NLDAS forcing datasets used in the VB study.
2- Second resource provides an end-to-end workflow of CAMELS basin modeling with SUMMA for the paper simulations configured for execution in connected JupyterHub compute platforms. This resource is well-suited for a smaller scale exploration: it is preconfigured to explore one example CAMELS site and a period of 60-month actual simulation to demonstrate the capabilities of the notebooks. Users still can change the CAMELS site, the number of sites being explored or even the simulation period. To quickly assess the capabilities of the notebooks in this resource, we even recommend running an actual simulation period as short as 12 months.
3- Third resource, however, uses HPC (High-Performance Computing) through CyberGIS Computing Service. The HPC enables a high-speed running of simulations which makes it suitable for running larger simulations (even as large as the entire 671 CAMELS sites and the whole 60-month actual simulation period used in the VB study) practical and much faster than the second resource. This resource is preconfigured to explore four example CAMELS site and a period of 60-month actual simulation to only demonstrate the capabilities of the notebooks. Users still can change the CAMELS sites, the number of sites being explored or even the simulation period.

Greater details can be found in each resource.

ABSTRACT:

This HydroShare resource is an example to demonstrate the vPICO presentations in EGU General Assembly 2021 (https://meetingorganizer.copernicus.org/EGU21/session/40092#vPICO_presentations).
- Session: EOS5.3 session - The evolving open-science landscape in geosciences: open data, software, publications, and community initiatives
- Title: An Approach for Open and Reproducible Hydrological Modeling using Sciunit and HydroShare

Using this notebook, you can test how to create an immutable and interoperable Sciunit Container for open and reproducible hydrological modeling.

You can start using "NB_01_An_Approach_for_Open_and_Reproducible_Hydrological_Modeling_using_Sciunit_and_HydroShare.ipynb" notebook in "CyberGIS-Jupyter for water" after clicking "Open with...". in Right-Above.

ABSTRACT:

This HydroShare resource was created to share large extent spatial (LES) datasets in Maryland on GeoServer (https://geoserver.hydroshare.org/geoserver/web/wicket/bookmarkable/org.geoserver.web.demo.MapPreviewPage) and THREDDS (https://thredds.hydroshare.org/thredds/catalog/hydroshare/resources/catalog.html).

Users can access the uploaded LES datasets on HydroShare-GeoServer and THREDDS using this HS resource id. This resource was created using HS 2.

Then, through the RHESSys workflows, users can subset LES datasets using OWSLib and xarray.

ABSTRACT:

This HydroShare resource was created to share large extent spatial (LES) datasets in Virginia on GeoServer (https://geoserver.hydroshare.org/geoserver/web/wicket/bookmarkable/org.geoserver.web.demo.MapPreviewPage) and THREDDS (https://thredds.hydroshare.org/thredds/catalog/hydroshare/resources/catalog.html).

Users can access the uploaded LES datasets on HydroShare-GeoServer and THREDDS using this HS resource id. This resource was created using HS 2.

Then, through the RHESSys workflows, users can subset LES datasets using OWSLib and xarray.

ABSTRACT:

This HydroShare resource was created to share large extent spatial (LES) datasets in North Carolina on GeoServer (https://geoserver.hydroshare.org/geoserver/web/wicket/bookmarkable/org.geoserver.web.demo.MapPreviewPage) and THREDDS (https://thredds.hydroshare.org/thredds/catalog/hydroshare/resources/catalog.html).

Users can access the uploaded LES datasets on HydroShare-GeoServer and THREDDS using this HS resource id. This resource was created using HS 2.

Then, through the RHESSys workflows, users can subset LES datasets using OWSLib and xarray.

ABSTRACT:

This HydroShare resource provides the Jupyter Notebooks created for the study "An Approach for Creating Immutable and Interoperable End-to-End Hydrological Modeling Computational Workflows" led by researcher Young-Don Choi submitted to the 2021 EarthCube Annual meeting, Notebook Sessions.

To find out the instructions on how to run Jupyter Notebooks, please refer to the README file provided in this resource.

For the sake of completeness, the abstract for the study submitted to the EarthCube session is mentioned below:

"Reproducibility is a fundamental requirement to advance science. Creating reproducible hydrological models that include all required data, software, and workflows, however, is often burdensome and requires significant work. Computational hydrology is a rapidly advancing field with fast-evolving technologies to support increasingly complex computational hydrologic modeling. The growing model complexity in terms of variety of software and cyberinfrastructure capabilities makes achieving computational reproducibility extremely challenging. Through recent reproducibility research, there have been efforts to integrate three components: 1) (meta)data, 2) computational environments, and 3) workflows. However, each component is still separate, and researchers must interoperate between these three components. These separations make verifying end-to-end reproducibility challenging. Sciunit was developed to assist scientists, who are not programming experts, with encapsulating these three components into a container to enable reproducibility in an immutable form. However, there were still limitations to support interoperable computational environments and apply end-to-end solutions, which are an ultimate goal of reproducible hydrological modeling. Therefore, the objective of this research is to advance the existing Sciunit capabilities to not only support immutable, but also interoperable computational environments and apply an end-to-end modeling workflow using the Regional Hydro-Ecologic Simulation System (RHESSys) hydrologic model as an example. First, we create an end-to-end workflow for RHESSys using pyRHESSys on the CyberGIS-Jupyter for Water platform. Second, we encapsulate the aforementioned three components and create configurations that include lists of encapsulated dependencies using Sciunit. Third, we create two HydroShare resources, one for immutable reproducibility evaluation using Sciunit and the other for interoperable reproducibility evaluation using library configurations created by Sciunit. Finally, we evaluate the reproducibility of Sciunit in MyBinder, which is a different computational environment, using these two resources. This research presents a detailed example of a user-centric case study demonstrating the application of an open and interoperable containerization approach from a hydrologic modeler’s perspective."

ABSTRACT:

ATTENTION: All 3 model instances are now presnted in one resouce as 3 model instance aggregations. This resource is kept only for archiving purpose.

This HydroShare resource provides raw spatial input data for executing RHESSys workflows at Coweeta Subbasin18, North Carolina. Assessing the conventional data distribution approach, these spatial datasets were manually collected and shared at the file level through small files.

ABSTRACT:

ATTENTION: All 3 model instances are now presnted in one resouce as 3 model instance aggregations. This resource is kept only for archiving purpose.

This HydroShare resource provides raw spatial input data for executing RHESSys workflows at Scotts Level Branch, Maryland. Assessing the conventional data distribution approach, these spatial datasets were manually collected and shared at the file level through small files.

ABSTRACT:

ATTENTION: All 3 model instances are now presnted in one resouce as 3 model instance aggregations. This resource is kept only for archiving purpose.

This HydroShare resource provides raw spatial input data for executing RHESSys workflows at Spout Run, Virginia. Assessing the conventional data distribution approach, these spatial datasets were manually collected and shared at the file level through small files.

ABSTRACT:

We implemented automated workflows using Jupyter notebooks for each state. The GIS processing, crucial for merging, extracting, and projecting GeoTIFF data, was performed using ArcPy—a Python package for geographic data analysis, conversion, and management within ArcGIS (Toms, 2015). After generating state-scale LES (large extent spatial) datasets in GeoTIFF format, we utilized the xarray and rioxarray Python packages to convert GeoTIFF to NetCDF. Xarray is a Python package to work with multi-dimensional arrays and rioxarray is rasterio xarray extension. Rasterio is a Python library to read and write GeoTIFF and other raster formats. Xarray facilitated data manipulation and metadata addition in the NetCDF file, while rioxarray was used to save GeoTIFF as NetCDF. These procedures resulted in the creation of three HydroShare resources (HS 3, HS 4 and HS 5) for sharing state-scale LES datasets. Notably, due to licensing constraints with ArcGIS Pro, a commercial GIS software, the Jupyter notebook development was undertaken on a Windows OS.

ABSTRACT:

ATTENTION: All 9 workflows for RHESSys modeling are now condensed to one. This resource is kept only for archiving purpose.

This HydroShare resource offers Jupyter Notebooks for the RHESSys modeling workflow, employing the conventional approach at Coweeta Subbasin18, NC. For instructions on running the Jupyter Notebooks, please refer to the provided README file within this resource.

ABSTRACT:

ATTENTION: All 9 workflows for RHESSys modeling are now condensed to one. This resource is kept only for archiving purpose.

This HydroShare resource offers Jupyter Notebooks for the RHESSys modeling workflow, employing the GeoServer approach at Coweeta Subbasin18, NC. For instructions on running the Jupyter Notebooks, please refer to the provided README file within this resource.

ABSTRACT:

ATTENTION: All 9 workflows for RHESSys modeling are now condensed to one. This resource is kept only for archiving purpose.

This HydroShare resource offers Jupyter Notebooks for the RHESSys modeling workflow, employing the THREDDS approach at Coweeta Subbasin18, NC. For instructions on running the Jupyter Notebooks, please refer to the provided README file within this resource.

ABSTRACT:

ATTENTION: All 9 workflows for RHESSys modeling are now condensed to one. This resource is kept only for archiving purpose.

This HydroShare resource offers Jupyter Notebooks for the RHESSys modeling workflow, employing the conventional approach at Scotts Level Branch, MD. For instructions on running the Jupyter Notebooks, please refer to the provided README file within this resource.

ABSTRACT:

ATTENTION: All 9 workflows for RHESSys modeling are now condensed to one. This resource is kept only for archiving purpose.

This HydroShare resource offers Jupyter Notebooks for the RHESSys modeling workflow, employing the GeoServer approach at Scotts Level Branch, MD. For instructions on running the Jupyter Notebooks, please refer to the provided README file within this resource

ABSTRACT:

ATTENTION: All 9 workflows for RHESSys modeling are now condensed to one. This resource is kept only for archiving purpose.

This HydroShare resource offers Jupyter Notebooks for the RHESSys modeling workflow, employing the THREDDS approach at Scotts Level Branch, MD. For instructions on running the Jupyter Notebooks, please refer to the provided README file within this resource.

ABSTRACT:

ATTENTION: All 9 workflows for RHESSys modeling are now condensed to one. This resource is kept only for archiving purpose.

This HydroShare resource offers Jupyter Notebooks for the RHESSys modeling workflow, employing the conventional approach at Spout Run, VA. For instructions on running the Jupyter Notebooks, please refer to the provided README file within this resource.

ABSTRACT:

ATTENTION: All 9 workflows for RHESSys modeling are now condensed to one. This resource is kept only for archiving purpose.

This HydroShare resource offers Jupyter Notebooks for the RHESSys modeling workflow, employing the THREDDS approach at Spout Run, VA. For instructions on running the Jupyter Notebooks, please refer to the provided README file within this resource.

ABSTRACT:

This HydroShare resource aims to assess data consistency among two server-side methods (GeoServer and THREDDS Data Server) and the conventional data distribution approach (manually collecting and sharing at file-level). The evaluation spans three different-sized watersheds: Coweeta subbasin18, Scotts Level Branch, and Spout Run with 10, 30, and 60 m DEM resolutions, respectively. The workflow for resulting nine case studies, derived from the combination of three methods and three watersheds, are presented in one HydroShare resource (HS 7), yielding a total of nine RHESSys daily streamflow output files.

Within this resource, we include these nine output files and provide three Jupyter notebooks for conducting evaluations. Each notebook is dedicated to a specific watershed and focuses on the three methods, facilitating a comprehensive analysis of data consistency.

ABSTRACT:

This HydroShare resource was created to support the study presented in Choi et al. (2024), titled "Toward Reproducible and Interoperable Environmental Modeling: Integration of HydroShare with Server-side Methods for Exposing Large-Extent Spatial Datasets to Models." Ensuring the reproducibility of scientific studies is crucial for advancing research, with effective data management serving as a cornerstone for achieving this goal. In hydrologic and environmental modeling, spatial data is used as model input, and sharing this spatial data is a main step in the data management process. However, by focusing only on sharing data at the file level through small files rather than providing the ability to Find, Access, Interoperate with, and directly Reuse subsets of larger datasets, online data repositories have missed an opportunity to foster more reproducible science. This has led to challenges when accommodating large files that benefit from consistent data quality and seamless geographic extent.

To utilize the benefits of large datasets, the objective of the Choi et al. (2024) study was to create and test an approach for exposing large extent spatial (LES) datasets to support catchment-scale hydrologic modeling needs. GeoServer and THREDDS Data Server connected to HydroShare were used to provide seamless access to LES datasets. The approach was demonstrated using the Regional Hydro-Ecologic Simulation System (RHESSys) for three different-sized watersheds in the US. Data consistency was assessed across three different data acquisition approaches: the 'conventional' approach, which involved sharing data at the file level through small files, as well as GeoServer and THREDDS Data Server. This assessment was conducted using RHESSys to evaluate differences in model streamflow output. This approach provided an opportunity to serve datasets needed to create catchment models in a consistent way that could be accessed and processed to serve individual modeling needs. For full details on the methods and approach, please refer to Choi et al. (2024). This HydroShare resource is essential for accessing the data and workflows that were integral to the study.

This collection resource (HS 1) comprises 7 individual HydroShare resources (HS 2-8), each containing different datasets or workflows. These 7 HydroShare resources consist of the following: three resources for three state-scale LES datasets (HS 2-4), one resource with Jupyter notebooks for three different approaches and three different watersheds (HS 5), one resource for RHESSys model instances (i.e., input) of the conventional approach and observation data for all data access approaches in three different watersheds (HS 6), one resource with Jupyter notebooks for automated workflows to create LES datasets (HS 7), and finally one resource with Jupyter notebooks for the evaluation of data consistency (HS 8). More information on each resource is provided within it.

ABSTRACT:

This resource, configured for execution in connected JupyterHub compute platforms, helps the modelers to reproduce and build on the results from the VB study (Van Beusekom et al., 2022) as explained by Maghami et el. (2023). For this purpose, three different Jupyter notebooks are developed and included in this resource which explore the paper goal for one example CAMELS site and a pre-selected period of 60-month actual simulation to demonstrate the capabilities of the notebooks. For even a faster assesment of the capabilities of the notebooks, users are recommended to opt for a shorter simulation period (e.g., 12 months of actual simulation and six months of initialization) and one example CAMELS site. The first notebook processes the raw input data from CAMELS dataset to be used as input for SUMMA model. The second notebook executes SUMMA model using the input data from first notebook using original and altered forcing, as per further described in the notebook. Finally, the third notebook utilizes the outputs from notebook 2 and visualizes the sensitivity of SUMMA model outputs using Kling-Gupta Efficiency (KGE). More information about each Jupyter notebook and a step-by-step instructions on how to run the notebooks can be found in the Readme.md fie included in this resource. Using these three notebooks, modelers can apply the methodology mentioned above to any (one to all) of the 671 CAMELS basins and simulation periods of their choice.

ABSTRACT:

This resource, configured for execution in connected JupyterHub compute platforms using the CyberGIS-Jupyter for Water (CJW) environment's supported High-Performance Computing (HPC) resources (Expanse or Virtual ROGER) through CyberGIS-Compute Service, helps the modelers to reproduce and build on the results from the VB study (Van Beusekom et al., 2022) as explained by Maghami et el. (2023).

For this purpose, four different Jupyter notebooks are developed and included in this resource which explore the paper goal for four example CAMELS site and a pre-selected period of 60-month simulation to demonstrate the capabilities of the notebooks. The first notebook processes the raw input data from CAMELS dataset to be used as input for SUMMA model. The second notebook utilizes the CJW environment's supported HPC resource (Expanse or Virtual ROGER) through CyberGIS-Compute Service to executes SUMMA model. This notebook uses the input data from first notebook using original and altered forcing, as per further described in the notebook. The third notebook utilizes the outputs from notebook 2 and visualizes the sensitivity of SUMMA model outputs using Kling-Gupta Efficiency (KGE). The fourth notebook, only developed for the HPC environment (and only currently working with Expanse HPC), enables transferring large data from HPC to the scientific cloud service (i.e., CJW) using Globus service integrated by CyberGIS-Compute in a reliable, high-performance and fast way. More information about each Jupyter notebook and a step-by-step instructions on how to run the notebooks can be found in the Readme.md fie included in this resource. Using these four notebooks, modelers can apply the methodology mentioned above to any (one to all) of the 671 CAMELS basins and simulation periods of their choice. As this resource uses HPC, it enables a high-speed running of simulations which makes it suitable for larger simulations (even as large as the entire 671 CAMELS sites and the whole 60-month simulation period used in the paper) practical and much faster than when no HPC is used.

ABSTRACT:

ATTENTION: All 9 workflows for RHESSys modeling are now condensed to one. This resource is kept only for archiving purpose.

This HydroShare resource offers Jupyter Notebooks for the RHESSys modeling workflow, employing the GeoServer approach at Spout Run, VA. For instructions on running the Jupyter Notebooks, please refer to the provided README file within this resource.

ABSTRACT:

This resource holds a copy of the data used in WRFHydro Training Tutorial v5.2.0-rc1 (2020 Nov).
The data were originally published by NCAR WRFHydro Development Team at https://ral.ucar.edu/projects/wrf_hydro/training-materials

croton_NY_training_example_v5.2.tar.gz
https://drive.google.com/uc?id=1_Ivc02C1WWkZcuBaSl9YePrqQE5osiCp

geog_conus.tar.gz
https://drive.google.com/uc?id=1X71fdaSEJ5GWyNY2MDIy9cC6E7A0kihl

nldas_mfe_forcing.tar.gz
https://drive.google.com/uc?id=10Q-0eVakrVmFwZ27ftDDtsSHsg0YBQAT

ABSTRACT:

The HydroShare project is pleased to bring you this notebook that can set up a run-time environment on the CyberGIS-Jupyter for Water (CJW) platform for WRFHydro Hands-on Training v5.2.x (Nov 2020). In contrast to the Docker-based local setup, this HydroShare solution does not require installation or downloading of any software or data onto your local computer, and it enables you to access to more powerful computing resources in a clould-based CJW environment. All necessary materials required to complete this training are remotely accessible through a browser (Google Chrome recommended).

This notebook will retrieve the WRFHydro model codes and relevant data from different official repos on Github and Google Drive managed by the NCAR/UCAR WRFHydro Development Team, and put them in a similar directory structure as the Docker-based local setup.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

The HydroShare project is pleased to bring you this notebook that can set up a run-time environment on the CyberGIS-Jupyter for Water (CJW) platform for WRF&WRF-Hydro Coupled Testcase Online Lesson (v5.1.2). In contrast to the Docker-based local setup, this HydroShare solution does not require installation or downloading of any software or data onto your local computer, and it enables you to access to more powerful computing resources in a clould-based CJW environment. All necessary materials required to complete this training are remotely accessible through a browser (Google Chrome recommended).

This notebook retrieves the WRFHydro model codes and relevant data from different official repos on Github and Google Drive managed by the NCAR/UCAR WRFHydro Development Team, and puts them in certain directory structure (same as the Docker-based local setup) required by the training notebooks. Specifically, two new folders will be created (wrf-hydro-training, and WRF_WPS) alongside. The training notebooks are stored in wrf-hydro-training --> lessons as shown below.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

This is a collection that holds all the WRFHydro official training materials you can run on CyberGIS-Jupyter for Water without installing or downloading any software or data onto your local computer. This collection will expand as new training lessons be added.

ABSTRACT:

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as below.

1) Support for NCAR/UCAR WRFHydro Training Notebooks: Users are now able to set up the necessary environment to run WRFHydro hands-on training notebooks (https://ral.ucar.edu/projects/wrf_hydro/training-materials) through simple clicks on CJW. For details on how to run these training notebooks please review the following HydroShare resources: (a) WRFHydro Hands-on Training v5.2.x on CJW; and (b) WRF&WRFHydro Coupled Training v5.1.2 on CJW. This provides an alternative solution to the traditional Docker-based local setup (https://hub.docker.com/r/wrfhydro/training/) that makes it easy for users to complete this training as it does not require installation or downloading of any software or data onto the user’s local computer. Additionally, it enables users to access more powerful computing resources in a cloud-based Jupyter environment. All necessary materials required to complete the training are retrieved from official data sources managed by the NCAR/UCAR WRFHydro Development Team and accessible on CJW via a browser.

2) Transition from XSEDE Comet to Expanse: Comet HPC will be decommissioned on July 31, 2021 (https://portal.xsede.org/sdsc-comet), and all Comet allocations and resources awarded to HydroShare/CJW will be transferred to the Expanse HPC (https://portal.xsede.org/sdsc-expanse). As a result, the CyberGIS-Compute service will also drop the support on Comet and replace it with Expanse. Despite this change, we do not expect any action is required for the majority of users. The CyberGIS-Compute service and its SDK will redirect all jobs submitted to Comet to Expanse with a warning message showing up. All previously developed notebooks that use Comet will continue to run after this transition. Please contact us for solutions if you have models or notebooks that access Comet without going through the CyberGIS-Compute service.

3) Extended user testing of Kubernetes-based CJW instance: In the previous release, we announced a Kubernetes-based (Aka K8s: https://kubernetes.io/) CJW instance deployed at https://go.illinois.edu/cjw-k8s for user testing with a preliminary migration plan on deprecation of the current DockerSwarm-based CJW. Due to the growing complexity of K8s and more features being developed and added for a smooth user experience, we have decided to continue conducting extensive testing in this release. We encourage all users to join this testing process and would greatly appreciate your feedback. The current production CJW (http://go.illinois.edu/cybergis-jupyter-water) will continue to be available in parallel until a final migration plan is implemented.

Please refer to the following HydroShare resources for details and examples:
Run WRFHydro Hands-on Training v5.2.x on CJW
https://www.hydroshare.org/resource/d2c6618090f34ee898e005969b99cf90/
Run WRF&WRFHydro Coupled Training v5.1.2 on CJW
https://www.hydroshare.org/resource/c2389a2f05564da08ab218e59bdf1e81/
User testing on Kubernetes-based CJW:
https://go.illinois.edu/cjw-k8s
Set up OpenWith for Kubernetes-based CJW:
https://www.hydroshare.org/resource/e9686eadd4474b6587d83d9330d25854/
See Release Notes on HydroShare
https://www.hydroshare.org/resource/6a1ddb17155a4b27b885f442ad14e344/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

The CyberGIS-Jupyter for Water (CJW) platform aims to advance community hydrologic modelling, and support data-intensive, reproducible, and computationally scalable water science research by simplifying access to advanced cyberGIS and cyberinfrastructure capabilities through a friendly Jupyter Notebook environment. The current release has specific support for the Structure For Unifying Multiple Modeling Alternatives (SUMMA) model and the WRFHydro model.

You may open and view any notebook (*.ipynb file) with this app.

Please send comments and bug reports to help@cybergis.org.

ABSTRACT:

Dear CUAHSI community members,

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1)Community Commons for Easy Access to HydroShare Notebooks:
Reproducible computational notebooks have become increasingly important in hydrologic research and education. CJW now provides a friendly user interface for accessing HydroShare-related notebooks at https://cybergisxhub.cigi.illinois.edu/hydroshare/. Interested in learning how CJW enables computational reproducibility of such notebooks? Read on!

2)Enhanced Reproducibility of CJW:
Starting from this release, CJW has implemented a new solution in kernel and software management such that each notebook is now associated with a specific versioned kernel. This approach significantly enhances reproducibility in the CJW environment as each notebook is always opened with the original kernel it is tied to. We encourage users to test their notebooks and reach out to us if you experience any problems.

3)Transferring Large Model Output Datasets:
CJW has introduced a new Globus-based (https://www.globus.org/) data-transfer capability to enhance the performance and stability of transferring a large number of model outputs from high-performance computing (HPC) resources back to CJW, which is a common scenario in running hydrologic models.

4)Seamless Transition from XSEDE Comet to Expanse:
Comet was decommissioned on July 15, 2021 (https://portal.xsede.org/sdsc-comet), and all Comet allocations and resources awarded to HydroShare/CJW have been transferred to the Expanse (https://portal.xsede.org/sdsc-expanse). The CyberGIS-Compute service and its SDK now automatically redirect all jobs submitted to Comet to Expanse while prompting users with a simple warning message.

Please refer to the following HydroShare resources for details and examples:
Community Commons for Easy Access to HydroShare Notebooks
https://cybergisxhub.cigi.illinois.edu/hydroshare/
Enhanced Reproducibility of CJW
https://www.hydroshare.org/resource/df8244042e5445edb106d93e5a491d29/
Transferring Large Model Output Datasets
https://www.hydroshare.org/resource/5aa3d27ec5ef4b1b8dfa621ea284af14/
See Release Notes on HydroShare
https://www.hydroshare.org/resource/f3315ec8c1df4f4ab5d2274220de0351/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

This notebook demonstrates how to use Globus within CyberGIS-Compute to retrieve a large number of outputs generated by a model executed on HPC, which is often needed for postprocessing work performed on CJW. A new “data transfer” job type is provided for moving data from HPC back to the CJW Jupyter environment. Under the hood, this new job type utilizes the Globus service (https://www.globus.org/) to perform a point-to-point data transfer between HPC and CJW.

In this demo, we will first prepare a 60-member ensemble SUMMA mode and submit it to the XSEDE Expanse HPC for execution using the CyberGIS-Compute. When the model run is finished, we won't use the regular "download" function in the Compute SDK to retrieve the results. Instead, we submit another Globus job to the Compute, which will hand it off to the Globus scheduler and monitor the process (just like talking Slurm scheduler on HPC in the case of regular model submission). Please refer to the example notebook below for more details.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

Most of this notebook is going over advanced options and technical details behind our new design. There are however a few key things all users should know:

1 What do the different kernel names/versions mean?
2 Paths to some executables might have changed.
3 We have a new cjw command to manage kernel versions.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

Reproducible computational notebooks have become increasingly important in hydrologic research and education. CJW now provides a friendly user interface for accessing HydroShare-related notebooks at https://cybergisxhub.cigi.illinois.edu/hydroshare/.

ABSTRACT:

This resource provides a quick way to discover and navigate through different example notebooks developed in the CyberGIS-Jupyter for Water (CJW) webapp.
The collection is being updated periodically as new resources will be added and old ones gets deprecated.

How to launch a notebook with CJW:
0) Request to join the CyberGIS-Jupyter for Water Group on HydroShare (one-time effort): Click on the “Ask to join” button in the lower left corner, and a request will normally get approved very quick if your user profile is complete and up-to-date.
1) Click on a resource below to open up the landing page of selected resource;
2) Click on the OpenWith button in the upper-right corner and select "CyberGIS-Jupyter for Water";
3) Follow the instructions in the notebook;

ABSTRACT:

An example notebook walks you through how to setup customized kernels on CyberGIS-Jupyter for Water (CJW)

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

This resource illustrates how data and code can be combined together to support hydrologic analyses. It was developed June 2020 as part of a HydroLearn Hackathon.

ABSTRACT:

This notebook demonstrates how to configure ensemble runs for SUMMA 3.0 model and submit to a High-Performance Computing (HPC) resource for parallel execution though the CyberGIS-Compute V2 service. The content is intended for end users who want to use the SUMMA model in CJW environment. For developers who may want to contribute new models to CyberGIS-Compute, the implementation of SUMMA on Github (https://github.com/cybergis/cybergis-compute-v2-summa) can serve as an example.

Some steps in this notebook require user interaction. "Run cell by cell" is recommended. "Run All" may not work as expected.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

Dear CUAHSI community members,

We are pleased to announce a new quarterly release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

Transition to CyberGIS-Compute V2:
CyberGIS-Compute V2 is a new development phase of the CyberGIS-Compute framework that was initially released (denoted as V1) one year ago through CJW 2020-Q4. Compared with V1, V2 includes several major enhancements: 1) a new workflow for model contribution to facilitate adding new hydrologic models by community developers; 2) a GUI in the notebook environment to simplify and guide users through the job submission process; 3) transparent and bi-directional data transfers between CJW and high-performance computing (HPC) resources using Globus by default, and 4) detailed tracking of usage and metrics. It is worth noting that due to the upgraded architecture in V2, existing models implemented in V1 would need to be migrated. For a smooth transition and backward compatibility, services in V1 will remain available in parallel to those in V2, and all the old notebooks that use V1 remain functional.

SUMMA Model Migrated to CyberGIS-Compute V2:
We have migrated the SUMMA model to CyberGIS-Compute V2, and end users can now benefit from the new features mentioned above in SUMMA modeling work. Please refer to the example notebook below for details. In addition, the implementation of SUMMA in CyberGIS-Compute V2 is accessible on a Github repo (https://github.com/cybergis/cybergis-compute-v2-summa), which can serve as a real-world example to model developers who may want to contribute their models for sharing with the community. A “HelloWorld” implementation is also available serving as a model-agnostic example (https://github.com/cybergis/cybergis-compute-mpi-helloworld).

New Modules and Kernel Customization:
Upon user requests, two new easybuild-based modules have been added to the CJW toolbox and are now ready to use: NCL (https://www.ncl.ucar.edu/) for scientific data analysis and visualization (e.g., NetCDF, GRID, HDF); and CDO (https://code.mpimet.mpg.de/projects/cdo) for manipulation of climate and Numerical Weather Prediction (NWP) data. Furthermore, for advanced users who may want to customize the provided software environment and kernels, an example notebook (see below) is available for users to walk through the basics on how to install new libraries on top of existing environments or set up a Conda environment from scratch.

New UI Elements on CJW:
CJW has further customized the Jupyter Notebook user interface to include a virtual Announcement Board (in the header area) for timely communicating with users on upcoming events including downtimes and new releases, and a Bug Report button (at the upper-right corner) that opens an issue tracker page in a publicly accessible Github repo for quick bug reporting.

Please refer to the following resources for details and examples:
Run ensemble SUMMA 3.0 model with CyberGIS-Compute V2
https://www.hydroshare.org/resource/deac1b0b5b46415aaedb886b9dc16f45/

Customize Software Environment on CJW
https://www.hydroshare.org/resource/461a8a853d8e42a8ae170c68c4cfa8f1/

Implementation of SUMMA model using CyberGIS-Compute V2
https://github.com/cybergis/cybergis-compute-v2-summa

Implementation of HelloWorld model using CyberGIS-Compute V2
https://github.com/cybergis/cybergis-compute-mpi-helloworld

See Release Notes on HydroShare
https://www.hydroshare.org/resource/2086b241b16b453d827db847e8640475/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

This resource references the github repo (https://github.com/cybergis/cybergis-compute-v2-summa) implemented support for running ensemble SUMMA models on HPC resources via CyberGIS-Compute V2.

Model developers who may want to contribute other models to CyberGIS-Compute can use this repo as an example.

For end-users (mode users), please refer to the following resource for submitting an ensemble summa model
https://www.hydroshare.org/resource/deac1b0b5b46415aaedb886b9dc16f45/

ABSTRACT:

This resource references the github repo that implemented a MPI-based HelloWorld toy model for CyberGIS-Compute V2.
Model developers who may want to contribute their own models to CyberGIS-Compute can use this as an example.

For end-users, an example notebook is provided for running the toy model on a supported HPC resource:
https://github.com/cybergis/cybergis-compute-mpi-helloworld

ABSTRACT:

Compile latest RHESSys on CJW

ABSTRACT:

This resource holds a copy of the source code and example dataset released with WRFHydro v5.2.0 by NCAR/UCAR (retrieved from https://github.com/NCAR/wrf_hydro_nwm_public/releases/tag/v5.2.0)
It serves as a backup source for reproducibility as some HydroShare notebooks depend on this version and dataset.

ABSTRACT:

This notebook demonstrates how to prepare a WRFHydro model on CyberGIS-Jupyter for Water (CJW) for execution on a supported High-Performance Computing (HPC) resource via the CyberGIS-Compute service. First-time users are highly encouraged to go through the [NCAR WRFHydro Hands-on Training on CJW](https://www.hydroshare.org/resource/d2c6618090f34ee898e005969b99cf90/) to get familiar WRFHydro model basics including compilation of source code, preparation of forcing data and typical model configurations. This notebook will not cover those topics and assume users already have hands-on experience with local model runs.

CyberGIS-Compute is a CyberGIS-enabled web service sits between CJW and HPC resources. It acts as a middleman that takes user requests (eg. submission of a model) originated from CJW, carries out the actual job submission of model on the target HPC resource, monitors job status, and retrieves outputs when the model execution has completed. The functionality of CyberGIS-Compute is exposed as a series of REST APIs. A Python client, [CyberGIS-Compute SDK](https://github.com/cybergis/cybergis-compute-python-sdk), has been developed for use in the CJW environment that provides a simple GUI to guide users through the job submission process. Prior to job submission, model configuration and input data should be prepared and arranged in a certain way that meets specific requirements, which vary by models and their implementation in CyberGIS-Compute. We will walk through the requirements for WRFHydro below.

The general workflow for WRFHydro in CyberGIS-Compute works as follows:

1. User picks a Model_Version of WRFHydro to use;
2. User prepares configuration files and data for the model on CJW;
3. User submits configuration files and data to CyberGIS-Compute;
4. CyberGIS-Compute transfers configuration files and data to target HPC;
5. CyberGIS-Compute downloads the chosen Model_Version of WRFhydro codebase on HPC;
6. CyberGIS-Compute applies compile-time configuration files to the codebase, and compiles the source code on the fly;
7. CyberGIS-Compute applies run-time configuration files and data to the model;
8. CyberGIS-Compute submits the model job to HPC scheduler for model execution;
9. CyberGIS-Compute monitors job status;
10. CyberGIS-Compute transfers model outputs from HPC to CJW upon user request;
11. User performs post-processing work on CJW;

Some steps in this notebook require user interaction. "Run cell by cell" is recommended. "Run All" may not work as expected.

How to run the notebook:
1) Click on the OpenWith button in the upper-right corner;
2) Select "CyberGIS-Jupyter for Water";
3) Open the notebook and follow instructions;

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2022 Q1) [full-version]

Dear CUAHSI community members,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1) Integration of WRFHydro model with CyberGIS-Compute V2 to simplify access to High-Performance Computing (HPC) environments: A newly developed computation job template in CyberGIS-Compute enables users to configure a WRFHydro model and submit it to a HPC resource for execution. The client tool of the CyberGIS-Compute suite, CyberGIS-Compute SDK, walks users through job configuration, data transfer, job submission, and job status monitoring in a guided graphical interface. Since the overhead of HPC access is handled by CyberGIS-Compute, users can now focus on the modeling work. Currently, the implementation allows users to change almost every setting and configuration for a WRFHydro 5.x “offline run”. The whole process described above can be accomplished entirely within a notebook environment on CJW. Please refer to the example notebooks below for additional details.

2) Transition to JupyterLab: Starting with this release, CJW will launch the “next-generation notebook interface”, JupyterLab, as the default user environment. Although the new interface is different from the classic Notebook interface in many places, we anticipate this transition would be easy and smooth for most users. All existing notebooks should continue to run without modification, and the bug report and announcement UI elements have been migrated to the Lab interface. In addition, we have integrated the CUAHSI “HydroShare-on-Jupyter” extension - a handy tool that enables users to move data between CJW and HydroShare through a simple graphical user interface.

3) The “cjw” Command Line Interface (CLI): The “cjw” CLI is designed to help users manage different kernels on CJW for advanced use cases. For example, users can use this capability to set up personal kernels that will persist between sessions. For a quick start, open a terminal on CJW and try out the "cjw -h" command. Check out the documentation and examples below.

4) New Modules and Kernels: To support the latest RHESSys codebase, we have added Clang, a new C family compiler supplementing the existing GCC suite, to the CJW Easybuild-based toolbox. Accordingly, a new versioned RHESSys (2022-03) kernel has been created with Clang and other development tools pre-activated that are necessary for compilation of the RHESSys source code. Upon user requests, a new versioned WRFHydro (2022-03) kernel has been created to include the hvPlot toolset for advanced data visualization and updated versions of all the libraries from the previous WRFHydro (2021-09) kernel.

Please refer to the following resources for details and examples:
Run WRFHydro 5.x model on HPC with CyberGIS-Compute V2
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

Implementation of WRFHydro 5.x model using CyberGIS-Compute V2
https://www.hydroshare.org/resource/329ede31b88942c489aca3111b076446/

Customize Software Environment on CJW
https://www.hydroshare.org/resource/461a8a853d8e42a8ae170c68c4cfa8f1/

“cjw” Command Line Interface Documentation
https://cybergis.github.io/cybergisx-cli/cjw/

See Release Notes on HydroShare
https://www.hydroshare.org/resource/b0d094eef336427ab605066e166135d3/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

This resource references the github repo (https://github.com/cybergis/cybergis-compute-v2-wrfhydro) implemented support for running WRFHydro models on HPC resources via CyberGIS-Compute V2.

Model developers who may want to contribute other models to CyberGIS-Compute can use this repo as an example.

For end-users (mode users), please refer to the following resource for submitting an ensemble summa model
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

ABSTRACT:

03/30/2022
Dear CJW Users,

Part of the CyberGIS-Compute service has been recovered and is available for use.

What is operational:
CJW Jupyter notebook portal
Most CJW notebooks
CJW job submission to XSEDE Expanse HPC (including SUMMA and WRFHydro)

What is NOT operational (as of 03/30/2022 4PM CT):
CJW job submission to Virtual Roger (Keeling) HPC

We will keep you updated as we recover the services.

Sorry for any inconvenience.

Thanks
CyberGIS for Water Development Team
CUAHSI HydroShare Project

----------------old posts-----------------

03/20/2022
Dear CJW Users,

There was a rare but severe overheating incident that happened over the weekend of Mar 19-20, 2022 at a UIUC Data Center where many CyberGIS computing resources are hosted including Virtual Roger HPC (AKA Keeling) and the CyberGIS-Compute service.

What is still operational:
CJW Jupyter notebook portal
Most CJW notebooks (except for those use CyberGIS-Compute service)

What is NOT operational (as of 03/21/2022 12 PM CT):
CJW job submission to Virtual Roger (Keeling) HPC or XSEDE Expanse HPC
CJW notebooks that use CyberGIS-Compute

We will keep you updated once we get more info from our IT department.

Sorry for any inconvenience.

Thanks
CyberGIS for Water Development Team
CUAHSI HydroShare Project

ABSTRACT:

(This collection holds major CJW announcements with full-text of the most recent and important ones repeated in the Abstract section)
(For the latest features and example notebooks please refer to the links to Release Announcement in "Collection Content" down below.)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Updated on 07/13/2022
CJW 2022-Q2 release is live. Check it out at http://go.illinois.edu/cybergis-jupyter-water
For release notes: https://www.hydroshare.org/resource/34b04302d8b34cc6aab826f79b5e3802/

---------
5/18/2022 (Updated on 12PM CT)

Globus service interruption has been resolved on SDSC Expanse HPC. Job submission to Expanse is back online.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
03/2022
CyberGIS-Jupyter for Water Quarterly Release Announcement (2022 Q1) [full-version]

Dear CUAHSI community members,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1) Integration of WRFHydro model with CyberGIS-Compute V2 to simplify access to High-Performance Computing (HPC) environments: A newly developed computation job template in CyberGIS-Compute enables users to configure a WRFHydro model and submit it to a HPC resource for execution. The client tool of the CyberGIS-Compute suite, CyberGIS-Compute SDK, walks users through job configuration, data transfer, job submission, and job status monitoring in a guided graphical interface. Since the overhead of HPC access is handled by CyberGIS-Compute, users can now focus on the modeling work. Currently, the implementation allows users to change almost every setting and configuration for a WRFHydro 5.x “offline run”. The whole process described above can be accomplished entirely within a notebook environment on CJW. Please refer to the example notebooks below for additional details.

2) Transition to JupyterLab: Starting with this release, CJW will launch the “next-generation notebook interface”, JupyterLab, as the default user environment. Although the new interface is different from the classic Notebook interface in many places, we anticipate this transition would be easy and smooth for most users. All existing notebooks should continue to run without modification, and the bug report and announcement UI elements have been migrated to the Lab interface. In addition, we have integrated the CUAHSI “HydroShare-on-Jupyter” extension - a handy tool that enables users to move data between CJW and HydroShare through a simple graphical user interface.

3) The “cjw” Command Line Interface (CLI): The “cjw” CLI is designed to help users manage different kernels on CJW for advanced use cases. For example, users can use this capability to set up personal kernels that will persist between sessions. For a quick start, open a terminal on CJW and try out the "cjw -h" command. Check out the documentation and examples below.

4) New Modules and Kernels: To support the latest RHESSys codebase, we have added Clang, a new C family compiler supplementing the existing GCC suite, to the CJW Easybuild-based toolbox. Accordingly, a new versioned RHESSys (2022-03) kernel has been created with Clang and other development tools pre-activated that are necessary for compilation of the RHESSys source code. Upon user requests, a new versioned WRFHydro (2022-03) kernel has been created to include the hvPlot toolset for advanced data visualization and updated versions of all the libraries from the previous WRFHydro (2021-09) kernel.

Please refer to the following resources for details and examples:
Run WRFHydro 5.x model on HPC with CyberGIS-Compute V2
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

Implementation of WRFHydro 5.x model using CyberGIS-Compute V2
https://www.hydroshare.org/resource/329ede31b88942c489aca3111b076446/

Customize Software Environment on CJW
https://www.hydroshare.org/resource/461a8a853d8e42a8ae170c68c4cfa8f1/

“cjw” Command Line Interface Documentation
https://cybergis.github.io/cybergisx-cli/cjw/

See Release Notes on HydroShare
https://www.hydroshare.org/resource/b0d094eef336427ab605066e166135d3/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

(04/17/2022) -- CJW downtime on 04/19/2022 due to HydroShare maintenance

On Tuesday, April 19, 2022 at 9:00pm EDT HydroShare is scheduled for a downtime of approximately one hour for maintenance that will affect CJW access.
During the maintenance, you will NOT be able to login CJW as it depends on HydroShare OAuth login. If you have logged on CJW before HydroShare maintenance starts, you can continue to use CJW but you would NOT be able to perform any I/O operation from/to HydroShare, such as OpenWith.
CJW will be back to normal once HydroShare maintenance is completed.

CyberGIS-Hydro Team

ABSTRACT:

The CyberGIS-Jupyter for Water (CJW) platform aims to advance community hydrologic modelling, and support data-intensive, reproducible, and computationally scalable water science research by simplifying access to advanced cyberGIS and cyberinfrastructure capabilities through a friendly Jupyter Notebook environment. The current release has specific support for the Structure For Unifying Multiple Modeling Alternatives (SUMMA) model and the WRFHydro model.

You may open and view any notebook (*.ipynb file) with this app.

Please send comments and bug reports to help@cybergis.org.

ABSTRACT:

CJW has pre-installed a large collection of common libraries and tools to support your modeling work, but you may still want to install something specific to your work in some cases. CJW now allows you to directly use “!pip install XXX” in notebook cells to customize the existing kernels. CJW handles the additions or changes on a per-kernel basis, and they will not affect your other kernels. Please refer to the example notebook for more information.

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2022-Q2)

Dear CJW users,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

1) CJW moved to a new home. Jetstream-1, an NSF-funded high-performance cloud computing resource where CJW was hosted for the past 3 years, was permanently shut down on July 31, 2022. Its successor, Jetstream-2, which offers much more powerful capabilities, has become the new home of CJW. All existing CJW user data and notebooks have been migrated to Jetstream-2. We do not expect users to experience any change in usage due to this transition but to enjoy a faster and smoother Jupyter environment backed by the latest hardware and cloud technology. In exceptional cases, the previous CJW instance on Jetstream-1 could be accessible upon user request.

2) Improved user experience in CyberGIS-Compute job submission: Have you ever had a long-running job submitted to high-performance computing (HPC) resources but found your Jupyter session died after the browser was idle for too long? The latest CyberGIS-Compute SDK now allows you to reinstate job submission sessions for all previous jobs you submitted. Just switch to the new “Your Jobs” tab page in the user environment and “Restore” the jobs you are interested in. This also gives you a chance to re-download model outputs from previous jobs.

3) WRFHydro model integration supports merging model outputs: A new option “Merge_Output” is added to the WRFHydro workflow supported by CyberGIS-Compute. If enabled, single-timestep NetCDF files can be merged on the “Time” dimension after model execution. Currently supported output types include CHANOBS, LDASOUT, GWOUT, LAKEOUT, RTOUT, and LSMOUT. This optional merging step can reduce data transfer size and speed up post-processing work on CJW. The merged files are put into a separate folder called “Outputs_Merged” alongside the original model outputs. Users can choose to download either or both. Please refer to the example notebook for more information.

4) Enhanced support for user customization to CJW kernels: While CJW has pre-installed a large collection of common libraries and tools to support a suite of hydrologic analysis and modeling workflows, users may still want to install something specific to certain use cases. CJW now allows users to directly use “!pip install XXX” in notebook cells to customize existing kernels. CJW supports flexible additions or changes on a per-kernel basis, which does not affect other existing kernels. Please refer to this example notebook for more information.

5) Updates on CJW backend (kernel, plugin, and bugfix): A new general-purpose kernel, Python3-2022-06, is added, which incorporates a rich set of new geospatial packages. The ‘StickyLand” JupyterLab plugin is installed that allows users to create customizable dashboards and linear notebooks; A bug specific to Apple Safari browser in the OpenWith operation has been fixed.

Please refer to the following resources for details and examples:

Run WRFHydro model on HPC resources using CyberGIS-Compute V2 (updated 2022-07)
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

Customization to CJW Kernels with Pip
https://www.hydroshare.org/resource/d18886d2aedf4a2e8c6302165b8fe10f/

CyberGIS-Compute SDK new features
https://cybergis.github.io/cybergis-compute-python-sdk/release-notes.html

CJW 2022-Q2 Release Notes on HydroShare
https://www.hydroshare.org/resource/34b04302d8b34cc6aab826f79b5e3802/

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.

Best regards,
CyberGIS-Hydro team

ABSTRACT:

CyberGIS-Compute is a scalable middleware framework for enabling high-performance and data-intensive geospatial research and education on CyberGISX. This API can be used to send supported jobs to various supported HPC & computing resources.

ABSTRACT:

-------------- Updated on 07/12/2022. --------------

CJW maintenance is scheduled at 2-5PM CT, 07/13/2022 (Wed). We will finalize the migration of CJW from Jetstream-1 to Jetstream-2.

-------------- Updated on 07/01/2022. --------------
The NSF XSEDE Jetstream-1 (JS1) project has been the research cloud provider where CyberGIS-Jupyter for Water (CJW) service and data are hosted during the past 3 years. JS1 project is officially shutting down on 7/31/2022, and its successor, Jetstream-2 (JS2), will provide much more powerful computing resources backed by the latest hardware and cloud technology. The HydroShare project has recently been awarded an XSEDE allocation supplement to migrate CJW on JS1 (CJW-JS1) to JS2 platform. However, the migration is not an automated process since JS1 and JS2 are technically two separate clouds, just like AWS V.S. Google Could. The migration of CJW would require setting up a complete new CJW instance on JS2 with all data copied over from JS1.

As of writing (07/01/2022), we have set up CJW on JS2 (CJW-JS2) with user data copied over from CJW-JS1. The CJW-JS1 and CJW-JS2 will be accessible in parallel until CJW-JS1 is decommissioned on 7/13/2022. Since users may still produce new data on CJW-JS1 during this transition period, a nightly job is set up to copy new data (if any) on CJW-JS1 to CJW-JS2. Be aware that this is a one-way data sync process (JS1 --> JS2).

We encourage all users to start using CJW-JS2 ASAP. If you find any data is missing on CJW-JS2, please let us know.

CJW-JS1 to CJW-JS2 transition timeline

07/01/2022 - 07/12/2022
CJW-JS1 and CJW-JS2 are available for use in parallel (though we recommend you start using CJW-JS2)

CJW-JS1 entry point (accessible but NOT recommended):
Direct Access: https://go.illinois.edu/cybergis-jupyter-water (opens https://js-156-75.jetstream-cloud.org/)
OpenWith Access: choose CyberGIS-Jupyter for Water in the dropdown list on HydroShare

CJW-JS2 entry point:
Direct Access: https://go.illinois.edu/cybergis-jupyter-water-js2 (opens https://js2-155-137.jetstream-cloud.org/)
OpenWith Access: No Access

07/13/2022 --
CJW-JS1 will be shut down completely

CJW-JS2 entry point:
Direct Access: https://go.illinois.edu/cybergis-jupyter-water (opens https://js2-155-137.jetstream-cloud.org/)
OpenWith Access: choose CyberGIS-Jupyter for Water in the dropdown list on HydroShare

Please report any issue to help@cybergis.org

ABSTRACT:

CyberGIS-Jupyter for Water Quarterly Release Announcement (2022-Q3)

Dear CJW users,

We are pleased to announce a new release of the CyberGIS-Jupyter for Water (CJW) platform at https://go.illinois.edu/cybergis-jupyter-water. This release includes several new capabilities and features summarized as follows.

(1) Cern Virtual Machine File System (CVMFS): We have redesigned how we deliver software within CyberGIS-Jupyter. This new design drastically increases computational performance and reproducibility, and allows the platform to make the software environment available in a variety of settings. From an end-user perspective, there should be no change to your accessing and utilizing the CJW services.

(2) Improved user experience for CyberGIS-Compute: In previous versions, we introduced the capability for users to “Restore” their previously submitted jobs of interest. Based on user feedback, we’ve further refined the interface to support viewing and downloading outputs of all previously submitted jobs by simply navigating to the “Past Results” section. The result/output of any completed job can be accessed with a single click.

(3) Support for new High Performance Computing (HPC) backend in CyberGIS-Compute: Anvil is now available as a new HPC resource for CyberGIS-Compute. Supported by NSF, Anvil is a HPC system hosted at Purdue University that contains 1000 CPU nodes based on the third generation AMD EPYC "Milan" processor, delivering a peak performance of 5.3 petaflops. Allocations on Anvil are managed by NSF's ACCESS program (https://access-ci.org/). The large numbers of CPU nodes and cores (i.e., 128) enable superior computational performance for scalable codes, short queuing times, and fast execution for hydrologic models via CyberGIS-Compute. For more information on Anvil, refer to the documentation at: https://www.rcac.purdue.edu/anvil. The WRFHydro model is supported on Anvil via CyberGIS-Compute. Please refer to the example notebook below.

Please refer to the following resources for details and examples:

A Brief Overview Of Cern Virtual Machine File System (CVMFS)
http://www.hydroshare.org/resource/ab1555c0c8d34d3496997353ba8060d9

CyberGIS-Compute updates - 2022-Q3
http://www.hydroshare.org/resource/3b472641c3504161bb13a19d4c9fbc87

Submission of WRFHydro model to Anvil HPC
https://www.hydroshare.org/resource/cc28d769943046fdac0f9b0c0fc2afc6/

See Release Notes on HydroShare
http://www.hydroshare.org/resource/bf463f07e1244de4a17b3ea7b2d95916

Please let us know if you have any questions or run into any problems (help@cybergis.org). Any feedback would be greatly appreciated.
Best regards,
CyberGIS-Hydro team

ABSTRACT:

This computational notebook demonstrates how to generate a simple user source report on the CyberGIS-Jupyter for Water (CJW).

What you will learn:

Programmatically load CJW Group webpage in Python
Use BeatifulSoup library to extract html elements containing user info
Use HydroShare REST API client to retrieve user details
Load data into Pandas dataframe
Generate a simple report on "Users by Country" and "Institutions by Country"