OceanArray Project Structure

This document provides an overview of the oceanarray codebase structure and organisation.

Project Structure Overview

oceanarray/
├── oceanarray/                    # Main Python package
│   ├── __init__.py                # Public API: process(), STAGES, parameters
│   ├── _version.py                # Package version (auto-generated by setuptools-scm)
│   ├── cli.py                     # [core] Command-line interface (oceanarray init/process/run/report/validate/…)
│   ├── utilities.py               # [core] Shared utility helpers used across the pipeline
│   ├── inspect.py                 # [core] Quick dataset inspection helpers for interactive use
│   ├── logger.py                  # [core] Logging configuration and helpers
│   ├── paths.py                   # [core] Filesystem path and filename conventions
│   │
│   ├── processors/                # [core] Processing stages + pipeline registry
│   │   ├── __init__.py            # Pipeline stage registry and public process() entry point
│   │   ├── stage1.py              # Stage 1: raw data → CF-NetCDF (*_stage1.nc)
│   │   ├── stage2.py              # Stage 2: clock correction + deployment trim (*_stage2.nc)
│   │   ├── stage3.py              # Stage 3: pressure interpolation + QARTOD QC (*_stage3.nc)
│   │   ├── pressure.py            # Pressure interpolation helpers for stage 3
│   │   ├── qc.py                  # QARTOD QC tests and CTD derivations for stage 3
│   │   ├── coordinate.py          # BEAM→ENU transforms and ADCP QC for stage 3
│   │   ├── caldip.py              # CalDip calibration-dip corrections (not yet implemented)
│   │   ├── stack.py               # MooringStacker: instruments → common time grid (*_stack.nc)
│   │   ├── grid.py                # MooringGridder: stack → regular pressure grid (*_grid.nc)
│   │   └── helpers.py             # Internal helpers for stack and grid operations
│   │
│   ├── analysis/                  # [analysis] Science utilities
│   │   ├── clock.py               # Clock-offset analysis functions
│   │   ├── hydrographic.py        # Hydrographic analysis utilities
│   │   ├── science.py             # QC and dataset processing functions
│   │   ├── spectral.py            # Spectral analysis of time series
│   │   ├── temporal.py            # Time-series analysis utilities
│   │   └── vector.py              # Vector rotation and progressive-vector utilities
│   │
│   ├── plotters/                  # [viz] Three-tier plotting package
│   │   ├── __init__.py            # Public API
│   │   ├── primitives.py          # Tier 1: data-agnostic plotting primitives
│   │   ├── current.py             # Tier 2: current/velocity domain plots
│   │   ├── diagnostic.py          # Tier 2: T-S, histogram, spectrum, QC diagnostic plots
│   │   ├── hydrography.py         # Tier 2: hydrographic section and isopycnal plots
│   │   ├── timeseries.py          # Tier 2: time-series and gridded-section plots
│   │   ├── spectrum.py            # Tier 2: spectral diagnostics
│   │   ├── ts.py                  # T-S diagram and thermohaline structure plots
│   │   ├── animation.py           # Animated plot functions
│   │   ├── helpers.py             # Shared colormap/style helpers
│   │   └── _cli_legacy.py         # Legacy CLI plots (pending `oceanarray plot <file>` redesign)
│   │
│   ├── reports/                   # [report] HTML report generation
│   │   ├── __init__.py            # Public entry-points (MooringReport, etc.)
│   │   ├── _mooring.py            # Mooring summary template + MooringReport orchestrator
│   │   ├── _instrument.py         # Per-instrument report ({mooring}_{serial}_report.html)
│   │   ├── _stack.py              # Stack report (multi-instrument timeseries, tilt panels)
│   │   ├── _grid.py               # Grid report (vertical section, spectra, N²)
│   │   ├── _array.py              # Array-level multi-mooring summary report
│   │   ├── _plots.py              # Tier 3: report-level figure wrappers (base64 PNGs)
│   │   ├── _html_helpers.py       # HTML/QC constants, base64 helpers, NC metadata readers
│   │   ├── _recovery_table.py     # Per-mooring cruise-report recovery table
│   │   └── _pdf.py                # combine_mooring_pdf: HTML pages → single A4 PDF (WeasyPrint)
│   │
│   ├── tools/                     # [core] Shared I/O infrastructure
│   │   ├── readers.py             # NetCDF and legacy-format instrument readers
│   │   ├── writers.py             # NetCDF write helpers (OceanSITES-compliant)
│   │   └── rapid_interp.py        # Physics-informed vertical interpolation for sparse profiles
│   │
│   ├── config/                    # Configuration
│   │   ├── parameters.py          # Package-level defaults (VARIABLES, QC ranges, style)
│   │   ├── validation.py          # Mooring YAML validation
│   │   ├── OS1_var_names.yaml     # OceanSITES variable name mappings
│   │   ├── OS1_sensor_attrs.yaml  # OceanSITES sensor attributes
│   │   ├── logging.yaml           # Logging configuration
│   │   └── legacy/                # Legacy RAPID/RODB configuration files
│   │
│   └── legacy/                    # Legacy RODB/RAPID processing (deprecated; excluded from lint)
│
├── tests/                         # pytest test suite
│   ├── unit/                      # seasenselib-free unit tests
│   ├── integration/               # Golden + pipeline tests (needs_seasenselib)
│   ├── fixtures/                  # Committed raw + processed dune2_1_2026 fixtures
│   └── legacy/                    # Tests for legacy RODB/RAPID processing
│
├── notebooks/                     # Demo notebooks
├── docs/                          # Sphinx documentation
│   └── source/
│       ├── methods/               # Method documentation (one page per processing step)
│       └── _static/               # Static files, code examples, CSS
│
├── CLAUDE.md                      # Claude Code guidance (local-only)
├── CITATION.cff                   # Citation metadata
├── CHANGELOG.md
├── pyproject.toml                 # Build system, deps, and project metadata
└── README.md

Processing Stages

The pipeline processes raw instrument data through sequential stages, each producing a CF-NetCDF output file. oceanarray.STAGES is the registry of stages, and oceanarray.process(mooring, stage=…, proc_dir=…, raw_dir=…) runs any subset.

Stage 1 — Standardisation (`processors/stage1.py`)

Input: raw instrument files (.cnv, .rsk, .dat, .hex, RDI raw)
Output: {proc_dir}/{mooring}/{mooring}_{serial}_stage1.nc
Faithful to the raw data — no QC, no trimming. Stores the transformation matrix and coordinate system so later stages can rotate correctly.

Stage 2 — Clock correction + trimming (`processors/stage2.py`)

Input: *_stage1.nc + mooring YAML (clock offsets, deployment window)
Output: *_stage2.nc
Applies linear clock-offset correction; trims to the deployment window.

Stage 3 — QC, rotation, derived variables (`processors/stage3.py`)

Input: *_stage2.nc
Output: *_stage3.nc
Gross-range and tilt QC flags, BEAM→ENU rotation (Aquadopp), magnetic declination correction, salinity, density.

Stack — multi-instrument coordination (`processors/stack.py`)

Input: *_stage3.nc files for all instruments on a mooring
Output: {mooring}_stack.nc — (N_LEVELS, time) Dataset
Aligns instruments onto a common time axis; HAB-ordered deepest-first (index 0).

Grid — vertical gridding (`processors/grid.py`)

Input: *_stack.nc
Output: {mooring}_grid.nc — (N_LEVELS, time) on uniform pressure levels
Simple 1-D linear interpolation at each time step (preliminary; no objective mapping).

Data Flow

Raw files
   │
   ▼ oceanarray process --stage 1
*_stage1.nc   (faithful copy, CF-NetCDF)
   │
   ▼ oceanarray process --stage 2
*_stage2.nc   (clock-corrected, trimmed)
   │
   ▼ oceanarray process --stage 3
*_stage3.nc   (QC flagged, ENU velocities, salinity/density)
   │
   ▼ oceanarray process --stage stack
{mooring}_stack.nc   (N_LEVELS × time)
   │
   ├──▶ oceanarray report  →  HTML reports
   │
   ▼ oceanarray process --stage grid
{mooring}_grid.nc   (uniform pressure × time)
   │
   └──▶ oceanarray report  →  grid HTML report

The full pipeline runs in one command: oceanarray process MOORING --stage 1 2 3 stack grid (or oceanarray run MOORING).

Plotters Package Architecture

Three-tier architecture (see .claude/plotters_update-20260718.md for the rules):

Tier 1 (plotters/primitives.py): low-level axes primitives, no domain knowledge
Tier 2 (plotters/current.py, timeseries.py, hydrography.py, etc.): domain functions that know about oceanographic variables
Tier 3 (reports/_plots.py): report wrappers that call Tier-2 functions and return base64 PNG strings for embedding in HTML

Report Package

Report types, each in its own module:

Report	Module	Output file
Mooring summary	`reports/_mooring.py`	`{mooring}_report.html`
Per-instrument	`reports/_instrument.py`	`{mooring}_{serial}_report.html`
Stack	`reports/_stack.py`	`{mooring}_stack_report.html`
Grid	`reports/_grid.py`	`{mooring}_grid_report.html`
Array	`reports/_array.py`	`{array}_array_report.html`

All figures are generated by reports/_plots.py (Tier 3) and embedded as base64 PNGs.

Key Design Principles

Data provenance: never silently substitute defaults; store all processing parameters in NetCDF global attributes so treatment can be reconstructed from the file.
CF-compliant: CF conventions for metadata and variable naming throughout.
xarray-based: xr.Dataset is the primary data structure in all stages.
Discrete colorbars: all figures use _nice_colorbar_bounds + BoundaryNorm; continuous colorbars are not used.
Configurable: YAML-driven configuration for QC ranges, clock offsets, deployment windows, and instrument metadata.

Legacy Modules

legacy/ contains the RODB/RAPID-format processing path, kept for backward compatibility with older datasets. New projects use the stage 1–3 pipeline above.