Why?

Cleaning tide gauge data is often:

• manual
• poorly documented
• hard to reproduce
• difficult to review or share

Concept

The core idea of ioc_cleanup is declarative cleaning.

Instead of scripts or notebooks, all cleaning decisions are:

• Explicit
• Version controlled
• Human-readable
• Reviewable

Cleaning logic lives entirely in JSON files.

Why it matters

This methodology allows:

• Flagging:
  • bad or corrupt data (timestamp / data ranges)
  • sensor breakpoints
  • singular phenomena (e.g. tsunamis, meteo-tsunamis, seiches, or unidentified events)
• Reproducible cleaning
• Transparent and traceable decisions stored in plain JSON
• Peer review of cleaning decisions via GitHub
• Easy extension to any other datasets (e.g. GESLA, NDBC)
• Gradual growth in station coverage through community contributions

Transformations

Each station/sensor pair is described by a JSON file located in:

./transformations/

These files define the transformation from raw data → clean signal by declaring:

• valid time windows
• dropped timestamps
• dropped ranges
• breakpoints
• notes and metadata

More details in the JSON format

Dataset Details

Cleaned Stations

Download the database as a csv: cleaned_ioc_stations.csv

Coverage

Data availability

Ratio of data removed

Note

The above figures have been generated with the helper functions in scripts/ folder:

1. download_ioc.py to download IOC stations
2. generate_stations_csv.py to build the cleaned stations dataset from the transformations folder and the IOC station dataset (from step 1)
3. generate_maps.py to create maps and graphs for the online documentation
4. save_cleaning_scenarios.py to create the time series graphs used in the online documentation

Steps 3 and 4 require to have run step 1 for all cleaned IOC stations.