Benchmark Datasets for Automated Quality Control of Aquatic Sensor Data

This resource contains benchmark datasets for testing automated quality control (QC) methods on aquatic sensor data, together with the Python workflow that produced them and the scripts that reproduce the figures and tables of the associated manuscript. The manuscript and the thesis that describes the workflow in detail are listed under "Related Resources" in this resource's metadata.

Summary

• Script.py compares raw and technician-corrected sensor data, extracts anomaly events, identifies their patterns, and assigns an anomaly type label (MNT, CAL, LDC, CS, PF, SLM, LWL, VIN) to each anomaly event and datapoint.
• batch_runner.py runs Script.py for every combination of variable, site and year.
• Initial_Settings.yaml sets the dataset to process and the thresholds used for labeling, so the analysis can be changed without editing the script.
• The benchmark covers 5 variables (specific conductance, water temperature, pH, dissolved oxygen, stage) at 5 Logan River Observatory (LRO) sites for 2020-2024 (125 datasets, 15-minute data).
• Manuscript_Reproducibility.zip/ reproduces Figures 6-21 and Tables 3-5 of the manuscript.

Labels are technician-QC-derived reference labels: a datapoint is anomalous when the manually corrected value differs from the raw value. They show where a technician made a correction, not independently verified sensor errors.

Quick start

1. Download the resource and unzip the input data

Download all files of this resource into one folder. Unzip InputDatasets.zip in that same folder, so that the folder InputDatasets/ (with Raw_data/, QC_data/ and FieldNote_data/ inside it) sits next to Script.py. Other zip files (Plots.zip and Results.zip) are not needed, you just need to unzip the InputDatasets.zip for running the workflow and regenerating results. The Plots.zip and Results.zip are just the results of the previous runs and you need unzip them in case if you need to access to the ready benchmark dataset. Also, the Manuscript_Reproducibility.zip file is only for reproducing the figures used in the manuscript.

2. Create the Python environment

Python 3.10 or 3.11 is required:

conda create -n aquatic-benchmark python=3.10 -y
conda activate aquatic-benchmark
pip install -r requirements.txt

3. Choose the dataset in Initial_Settings.yaml

Set Variable, site, year, Climate_site and Unit (see "Generating the benchmark datasets" below). For example, to process specific conductance at the Main Street site in 2024:

Variable: SpCond
site: LR_MainStreet_BA
year: 2024
Climate_site: LR_GC_C
Unit: uS/cm

4. Run the workflow

From the folder that contains Script.py, run one dataset:

python Script.py

or all 125 datasets (this updates Initial_Settings.yaml automatically for each run):

python batch_runner.py

The full batch takes about 30 minutes. Results are written to Results/<variable>_<site>_<year>/ and plots to Plots/<variable>_<site>_<year>/.

5. (Optional) Reproduce the manuscript figures and tables

python Manuscript_Reproducibility.zip/1_Dataset_Figures/make_figures_6_14.py
python Manuscript_Reproducibility.zip/1_Dataset_Figures/make_figure_15.py
python Manuscript_Reproducibility.zip/2_ML_Demonstration/isolation_forest.py
python Manuscript_Reproducibility.zip/2_ML_Demonstration/oc_svm.py
python Manuscript_Reproducibility.zip/2_ML_Demonstration/comparison_plots.py
python Manuscript_Reproducibility.zip/2_ML_Demonstration/event_level.py

See "Reproducing the manuscript figures and tables" below for which file produces which figure.

Version

This is the version of the workflow and benchmark data used for the manuscript published in Environmental Modelling & Software (see Related Resources for reference to the associated paper). Changes from prior versions are documented in the CHANGES.txt file included in this resource.

Repository structure

The files and folders in this resource are:

AquaticBenchmarkDataset/
|-- Script.py                                                 main workflow (one dataset per run)
|-- batch_runner.py                                           runs Script.py for all variable/site/year combinations
|-- Initial_Settings.yaml                                     settings for Script.py
|-- map_columns.py                                            optional: prepares your own input files (column names, file names)
|-- Column_Mapping.yaml                                       column mapping used by map_columns.py
|-- requirements.txt                                          Python packages
|-- CHANGES.txt                                               changes from the previous version
|-- README.md                                                 this file
|-- InputDatasets.zip/                                        distributed as InputDatasets
|   |-- Raw_data/RawAquatic/                                  RawAquatic_<site>_<year>.csv
|   |-- Raw_data/RawClimate/                                  DataSetClimate_<climate site>_<year>.csv
|   |-- QC_data/                                              QC_data_<variable>_<site>_<year>.csv (manually corrected data)
|   |-- FieldNote_data/                                       FieldNotes_<variable>_<site>_<year>.csv
|-- Results.zip/<variable>_<site>_<year>/
|   |-- File_1. Final labeling.csv
|   |-- File_2_Final_labeling_with_class_dictionary.csv
|   |-- File_3. List of the Events.csv
|-- Plots.zip/<variable>_<site>_<year>/                       interactive HTML plots of each dataset
|-- Manuscript_Reproducibility.zip/                           scripts for the manuscript figures and tables (see below)

Contents of each Results.zip/<variable>_<site>_<year>/ folder:

• File_1. Final labeling.csv: time series with all intermediate indicators.
• File_2_Final_labeling_with_class_dictionary.csv: raw value, corrected value, and class label (0 = normal, 1-8 = anomaly type).
• File_3. List of the Events.csv: one row per anomaly event (start, end, duration, pattern, label).

Script.py creates Results/ and Plots/ if they do not exist. The input files in InputDatasets/ (from InputDatasets.zip) must be present before running.

Setup

Python 3.10 or 3.11 is required (the pinned numpy and pandas versions do not support Python 3.12 or later). To create the environment:

conda create -n aquatic-benchmark python=3.10 -y
conda activate aquatic-benchmark
pip install -r requirements.txt

Generating the benchmark datasets

Run both scripts from this folder (the folder that contains Script.py); they use relative paths.

One dataset. Edit Initial_Settings.yaml, then run:

python Script.py

Main settings in Initial_Settings.yaml:

• Variable: variable to process: SpCond, WaterTemp_EXO, pH, ODO, or Stage.
• site: aquatic site code (e.g., LR_Mendon_AA).
• year: year to process (e.g., 2024).
• Climate_site: climate site for air temperature (LR_TG_C for LR_TG_BA, LR_GC_C for the other sites).
• Unit: unit of the variable, used in plots (e.g., m for stage).
• CrossVariable: variable checked to separate power failure (PF) from sensor/logger malfunction (SLM) (e.g., BattVolt).
• Time_Step_of_data: time step of the data (15T = 15 minutes).
• CalibrationList: keywords that identify calibration visits in the field notes (e.g., Cal, Calibration).
• ThrV_1 to ThrV_4, PerThr_3, n: thresholds for the low-water-level (LWL) indicators.

Your own data (column mapping). Script.py reads its input files from the folders and file names shown in "Repository structure" and uses a standard set of column names:

• raw aquatic file: LocalDateTime, BattVolt, EXOVolt, ODO, Stage, pH, SpCond, WaterTemp_EXO, TurbMed
• climate file: LocalDateTime, AirTemp_EE08_avg
• manually corrected file: LocalDateTime, <Variable>_QC1, QualifierCode
• field notes file: Event Number, Combined Time, Begin: End time, Method

The LRO files in InputDatasets/ already follow this convention. For data with other column names, list your four files and your column names in Column_Mapping.yaml, set Variable, site, Climate_site and year in Initial_Settings.yaml, and run:

python map_columns.py
python Script.py

map_columns.py renames the columns, adds the optional columns you do not have (EXOVolt, ODO, pH, TurbMed, QualifierCode) as empty columns, checks that the required columns are present (including water temperature, specific conductance and stage, which are used to assign the event types), and saves the four files with the standard names in InputDatasets/. Values are copied unchanged; only the header row is changed. Existing files are not replaced unless overwrite: true is set in Column_Mapping.yaml.

All 125 datasets. batch_runner.py updates Initial_Settings.yaml for each combination (backups are saved in .config_backups/), runs Script.py, and logs each run in batch_runs_log.csv:

python batch_runner.py

The variables, sites and years to run are listed at the top of batch_runner.py (VARIABLES, SITES, YEARS).

Reproducing the manuscript figures and tables

If you plan to reproduce the figures used in the paper, first you need to unzip theManuscript_Reproducibility.zip/, Results.zip/, and InputDatasets.zip/ files. The scripts in Manuscript_Reproducibility.zip/ only read Results.zip/; they do not change it. Each script finds the nearest Results.zip/ folder above its own location, so it can be run from any working directory. Generate Results.zip/ first if it is not present.

Dataset figures (about 1 minute):

python Manuscript_Reproducibility.zip/1_Dataset_Figures/make_figures_6_14.py
python Manuscript_Reproducibility.zip/1_Dataset_Figures/make_figure_15.py

ML demonstration (run in this order; oc_svm.py takes several minutes):

python Manuscript_Reproducibility.zip/2_ML_Demonstration/isolation_forest.py
python Manuscript_Reproducibility.zip/2_ML_Demonstration/oc_svm.py
python Manuscript_Reproducibility.zip/2_ML_Demonstration/comparison_plots.py
python Manuscript_Reproducibility.zip/2_ML_Demonstration/event_level.py

Which script produces which figure or table (paths are relative to Manuscript_Reproducibility.zip/):

• Figures 6-14: 1_Dataset_Figures/make_figures_6_14.py produces 1_Dataset_Figures/Figure_06_...png to Figure_14_...png.
• Figure 15: 1_Dataset_Figures/make_figure_15.py produces 1_Dataset_Figures/Figure_15_anomalous_datapoints_by_month.png.
• Figure 16: 2_ML_Demonstration/oc_svm.py produces 2_ML_Demonstration/outputs/oc_svm/ocsvm_confusion_matrix_TEST.png.
• Figure 17: 2_ML_Demonstration/oc_svm.py produces 2_ML_Demonstration/outputs/oc_svm/ocsvm_metrics_bars_TEST.png.
• Figure 18: 2_ML_Demonstration/isolation_forest.py produces 2_ML_Demonstration/outputs/isolation_forest/iforest_confusion_matrix_TEST.png.
• Figure 19: 2_ML_Demonstration/isolation_forest.py produces 2_ML_Demonstration/outputs/isolation_forest/iforest_metrics_bars_TEST.png.
• Figure 20: 2_ML_Demonstration/comparison_plots.py produces 2_ML_Demonstration/outputs/comparison_plots/hourly_anomalies_2024_test.png.
• Figure 21: 2_ML_Demonstration/comparison_plots.py produces 2_ML_Demonstration/outputs/comparison_plots/monthly_anomalies_2024_test.png.
• Table 3: 2_ML_Demonstration/oc_svm.py produces 2_ML_Demonstration/outputs/oc_svm/ocsvm_per_type_recall_TEST.csv.
• Table 4: 2_ML_Demonstration/isolation_forest.py produces 2_ML_Demonstration/outputs/isolation_forest/iforest_per_type_recall_TEST.csv.
• Metrics in Sections 4.3.1 and 4.3.2 (precision, recall, F1, accuracy, PR-AUC, ROC-AUC, confusion matrix, selected hyperparameters and threshold): 2_ML_Demonstration/outputs/oc_svm/ocsvm_metrics_TEST.csv and 2_ML_Demonstration/outputs/isolation_forest/iforest_metrics_TEST.csv.
• Table 5 (event-level evaluation in Section 4.3): 2_ML_Demonstration/event_level.py produces 2_ML_Demonstration/outputs/event_level/Table_5_event_level_TEST.csv (unrounded values in event_level_metrics_TEST.csv in the same folder).

The numbers behind Figures 6-15 are saved as CSV files in 1_Dataset_Figures/tables/.

ML demonstration: data and settings

• Data: Results.zip/SpCond_LR_MainStreet_BA_<year>/File_2_Final_labeling_with_class_dictionary.csv, 2020-2024 (specific conductance, Main Street site).
• Input: raw sensor values only (column SpCond_QC0 (uS/cm)); corrected values are not used.
• Labels: column class (0 = normal, 1-8 = anomaly type); binary target = class != 0.
• Split: train 2020-01-01 to 2022-12-31, validation 2023-01-01 to 2023-12-31, test 2024-01-01 to 2024-12-31 (calendar years, no shuffling).
• Fitting: models are fit on normal observations (class 0) only. Hyperparameters and the decision-threshold quantile are selected on the 2023 validation period by maximizing F1. The final model is refit on the normal observations of 2020-2023 and applied once to 2024.
• Scores: anomaly score = -decision_function(...), so a higher score means more anomalous. A datapoint is flagged as anomalous when its score is at or above the selected threshold.
• PR-AUC: sklearn.metrics.average_precision_score(y_true, score), anomaly = positive class. A random ranking gives a PR-AUC equal to the share of anomalous datapoints in the test period (about 0.75 in 2024). ROC-AUC: sklearn.metrics.roc_auc_score (random = 0.5).
• Exclusions: rows with any feature that cannot be computed are removed. These are (a) the first rows of the record, before the minimum rolling-window length is reached (the first usable row is 2020-01-01 23:30), and (b) rows whose raw value is missing (13 rows in 2020-2024, all labeled as anomalies) plus the one or two rows after each of them, whose difference features are undefined. Rows used (of 175,392): train 105,105 (111 excluded); validation 35,034 (6 excluded); test 35,130 of 35,136 (6 excluded). The number of rows in each split is printed to the console at the start of each run.

Tested environment

The Manuscript_Reproducibility.zip/ scripts were run in the environment above (Python 3.10.22, numpy 1.24.0, pandas 1.5.3, matplotlib 3.6.3, scikit-learn 1.1.3) and gave identical results with a newer environment (Python 3.13.11, numpy 2.4.2, pandas 2.3.3, matplotlib 3.10.8, scikit-learn 1.8.0).