caldip API

core

Functional calibration-dip processing using xarray and numpy.

Primary functions:

• find_bottle_stops() -> list of dict: detect bottle stops from CTD pressure data.

• stats_for_time_period() -> dict: statistics for a dataset within a time period.

• stats() -> pandas.DataFrame: statistics for each bottle stop and instrument.

caldip.core.find_bottle_stops(ctd_data: Dataset, threshold_dbar_per_min: float | None = None, min_duration_seconds: float | None = None) → list[dict]

Find bottle stops in CTD data based on pressure rate of change.

Looks for periods where pressure change rate is < threshold_dbar_per_min (typically < 15 dbar/min for bottle stops vs 30-60 dbar/min for normal ops).

Parameters:

• ctd_data (xr.Dataset) – CTD dataset with pressure and time variables

• threshold_dbar_per_min (float, optional) – Maximum pressure change rate for bottle stops (dbar/min). Defaults to caldip.config.parameters.BOTTLE_STOP_THRESHOLD_DBAR_PER_MIN.

• min_duration_seconds (float, optional) – Minimum duration for valid bottle stops (seconds). Defaults to caldip.config.parameters.BOTTLE_STOP_MIN_DURATION_SECONDS.

Returns:

List of bottle stop dictionaries with keys: - start_idx, end_idx: indices in the data - start_time, end_time: timestamps - pressure: mean pressure during stop - duration_seconds: duration in seconds

Return type:

List[Dict]

caldip.core.resolve_quality_thresholds(config: dict, temp_threshold: float | None = None, cond_threshold: float | None = None, press_threshold: float | None = None) → dict[str, float]

Return the per-variable quality-flag thresholds for a cast.

Precedence per variable: an explicit argument, then the cast YAML’s quality_flags mapping, then the built-in default in caldip.config.parameters. Shared by stats() and the netCDF writer so the flag variable’s threshold attribute matches the value used to set the flag.

Parameters:

• config (dict) – Cast configuration; may carry a quality_flags mapping.

• temp_threshold (float or None) – Explicit overrides; None falls through to YAML then default.

• cond_threshold (float or None) – Explicit overrides; None falls through to YAML then default.

• press_threshold (float or None) – Explicit overrides; None falls through to YAML then default.

Returns:

Mapping {"temp": float, "cond": float, "press": float} in degree_C / mS cm-1 / dbar.

Return type:

dict

caldip.core.stats(instrument_data: dict, reference_data: dict, config: dict, threshold_dbar_per_min: float | None = None, min_duration_seconds: float | None = None, temp_threshold: float | None = None, cond_threshold: float | None = None, press_threshold: float | None = None) → DataFrame

Calculate statistics for each bottle stop and each instrument (any type).

Returns DataFrame with one row per bottle stop per instrument.

Quality flag thresholds (temp_threshold, cond_threshold, press_threshold) determine when a difference is flagged as “reads high/low” vs “OK”. They can also be set per-cast in the YAML under quality_flags: {temp_threshold: 0.005, ...}; explicit arguments take precedence over YAML values, which in turn override the built-in defaults (±0.005 °C, ±0.02 mS/cm, ±5 dbar).

caldip.core.stats_for_time_period(data: Dataset, start_time: Timestamp, end_time: Timestamp, variables: list[str]) → dict

Calculate statistics for any dataset within a specified time period. Normally this time period will be 3 minutes long ending 30 seconds before the end of a bottle stop, but this function can be used for any time period and any variables.

Parameters:

• data (xr.Dataset) – Dataset to analyze

• start_time (pd.Timestamp) – Start of time period

• end_time (pd.Timestamp) – End of time period

• variables (List[str]) – List of variable names to calculate statistics for

Returns:

Statistics dictionary with means, stds, and sample count for each variable

Return type:

Dict

readers

Data loading functions for caldip processing.

Public API

find_config_file(path) -> Path

Locate a .caldip.yaml config file given a file or directory path.

load_config(path) -> Dict

Parse a .caldip.yaml configuration file.

load_instruments_from_config(config, data_dir) -> Dict[str, Dict]

Load all instruments listed in a config. Priority per instrument: _use.nc → _raw.nc (creates _use.nc if absent) → source file (normalizes, applies clock offset, saves both _raw.nc and _use.nc).

load_reference_data(config, data_dir) -> Dict[str, Dict]

Load CTD reference data; reads pre-processed .nc if present.

resolve_data_dir(config_file, config, override) -> Path

Resolve the data directory from config or an explicit override.

Internal helpers

load_instrument_data() — dispatch to format-specific loaders _normalize_instrument_vars() — rename raw variables to canonical names _normalize_ctd_vars() — rename CTD variables; selects primary/secondary sensor _wild_edit_ctd() — apply SeaBird wild-edit spike removal _resample_1hz() — resample CTD to 1 Hz medians

caldip.readers.CRUISE_CONFIG_NAME = 'caldip.cruise.yaml'

Fixed name of the cruise-level YAML, distinct from the per-cast *.caldip.yaml.

caldip.readers.discover_cast_configs(cal_dip_dir: str | Path) → list

Return the per-cast *.caldip.yaml configs discovered under a directory.

The cruise sweep discovers casts from the directory rather than a hand-kept list, so it cannot drift from what is on disk. The cruise YAML itself (caldip.cruise.yaml) does not match *.caldip.yaml and is not returned.

Parameters:

cal_dip_dir (str or pathlib.Path) – The cal_dip directory holding one subdirectory per cast.

Returns:

The per-cast config paths, sorted.

Return type:

list of pathlib.Path

caldip.readers.find_config_file(path: str | Path) → Path | None

Find caldip configuration file in directory or use provided file.

Parameters:

path (str or Path) – Directory path or direct path to .yaml config file

Returns:

Path to config file, or None if not found

Return type:

Path or None

caldip.readers.find_cruise_config(start: str | Path) → Path | None

Return the nearest caldip.cruise.yaml at or above start, or None.

Parameters:

start (str or pathlib.Path) – A directory (or file) to search from, climbing toward the filesystem root.

Returns:

The nearest cruise YAML, or None if none is found.

Return type:

pathlib.Path or None

caldip.readers.load_config(yaml_file: str | Path) → dict

Load caldip configuration from YAML file.

Each instrument’s instrument: field is normalised in place to an oceanarray class name (see resolve_instrument_class()) and its serial to the shared join key, so every downstream consumer and the instrument_type written to the netCDF use the single controlled vocabulary. Blank instrument:/serial: fields (an unfinished scaffold stub) are left untouched to be filled in later; a serial that two instruments share after normalisation is rejected. cruise/ship/year are inherited from the nearest caldip.cruise.yaml when one is present.

caldip.readers.load_cruise_config(path: str | Path) → dict

Parse a cruise-level YAML (cruise/ship/year + cal_dip dir).

Parameters:

path (str or pathlib.Path) – Path to a caldip.cruise.yaml file.

Returns:

The parsed cruise configuration (empty dict if the file is empty).

Return type:

dict

caldip.readers.load_ctd_data(file_path: str | Path) → Dataset

Load CTD 911 data from SeaBird hex/cnv file.

Parameters:

file_path (str or Path) – Path to CTD data file (.hex or .cnv format)

Returns:

CTD data with standardized variable names and metadata

Return type:

xarray.Dataset

caldip.readers.load_instrument_data(file_path: str | Path, file_type: str, **kwargs: object) → Dataset

Load instrument data using the appropriate loader based on file_type.

Parameters:

• file_path (str or Path) – Path to the data file

• file_type (str) – Seasenselib format key (e.g. ‘sbe-cnv’, ‘sbe-ascii’, ‘sbe-hex’, ‘rbr-rsk’, ‘nortek-csv’). ‘sbe-asc’ is accepted as a deprecated alias for ‘sbe-ascii’.

• **kwargs – Additional arguments passed to the specific loader

Returns:

Dataset with standardized variable names and metadata

Return type:

xr.Dataset

Raises:

• ValueError – If file_type is not supported

• FileNotFoundError – If file does not exist

caldip.readers.load_instruments_from_config(config: dict, data_dir: str | Path | None = None) → dict[str, dict]

Load all instruments specified in a caldip configuration.

Parameters:

• config (dict) – Caldip configuration dictionary

• data_dir (str or Path, optional) – Base directory for data files. If None, uses config[‘directory’]

Returns:

Instrument serial numbers as keys; each value is a dict with data (xr.Dataset), config (the instrument’s YAML config dict), type (str, the file type) and file (str, the full path).

Return type:

dict

caldip.readers.load_microcat_data(file_path: str | Path) → Dataset

Load microCAT (SBE37) data from SeaBird hex/asc/cnv file.

Deprecated: load_instrument_data() now routes sbe-cnv/sbe-hex/sbe-asc through seasenselib directly. This function is retained for direct use and testing only.

Parameters:

file_path (str or Path) – Path to microCAT data file (.hex, .asc, or .cnv format)

Returns:

MicroCAT data with standardized variable names and metadata

Return type:

xarray.Dataset

caldip.readers.load_nortek_csv_data(file_path: str | Path, header_file: str | None = None) → Dataset

Load Nortek CSV data exported from AquaPro software.

Deprecated: load_instrument_data() now routes nortek-csv through seasenselib directly. This function is retained for direct use and testing only.

Parameters:

• file_path (str or Path) – Path to the CSV data file (e.g., “Average Velocity DF3.csv”)

• header_file (str, optional) – Path to Units.csv file for metadata (optional)

Returns:

Dataset with Nortek CSV data

Return type:

xr.Dataset

caldip.readers.load_reference_data(config: dict, data_dir: str | Path | None = None) → dict[str, dict]

Load CTD reference data from config.

If a pre-processed NetCDF cache ({ctd_file}.nc) exists, it is loaded directly. The cached file must have been built with the same ctd_sensor value as the current config; if the stored ctd_sensor attribute disagrees with the requested value, a ValueError is raised so the user knows to delete the cache and re-run caldip ctd.

Parameters:

• config (dict) – Caldip configuration dictionary. The ctd_sensor key (integer, 1 = primary, 2 = secondary) selects the CTD sensor pair; the deprecated ctd_sensors key is accepted with a warning.

• data_dir (str or Path, optional) – Base directory for data files. If None, uses config['directory'].

Returns:

Dictionary with CTD data keyed by CTD file stem: {ctd_name: {'data': xr.Dataset, 'file': str}}.

Return type:

dict

Raises:

ValueError – If a cached NetCDF exists but was built with a different ctd_sensor than requested.

caldip.readers.normalize_serial(value: str | int) → str

Normalise an instrument serial to its join-key form.

A trailing marker asterisk is stripped, and leading zeros are stripped from an all-digit serial, so 013874 and 9920* become "13874" and "9920" while an all-zero serial collapses to "0". A non-numeric serial keeps its leading zeros (they are not padding). The serial is the join key shared with oceanarray, which normalises the same way.

Parameters:

value (str or int) – The raw serial field from a cruise YAML or a filename.

Returns:

The normalised serial.

Return type:

str

Raises:

ValueError – If the serial is None or blank; a serial cannot be defaulted.

caldip.readers.read_ctdcast_reference(ds: Dataset, ctd_sensor: int, config: dict | None = None) → tuple[Dataset, dict]

Map a ctdcast stage netCDF to caldip’s CTD reference, with provenance.

The dual-sensor ctdcast variables (ctd_temperature_1/_2, conductivity_1/_2, or the single-sensor forms) are mapped to caldip’s canonical temperature / conductivity / pressure for the requested ctd_sensor, honouring each compared variable’s QARTOD _qc companion (a fail flag masks that sample). Provenance is copied from the file’s global attributes and the SENSOR_* catalog; cruise is taken verbatim.

Parameters:

• ds (xarray.Dataset) – A ctdcast per-cast stage dataset (see _is_ctdcast_nc()).

• ctd_sensor (int) – Which CTD sensor (1 or 2) to use as the reference.

• config (dict, optional) – The cast configuration; used only to warn when its cruise disagrees case-insensitively with the file’s cruise.

Returns:

(dataset, provenance) — the canonical-named CTD reference resampled to 1 Hz, and a dict of provenance attributes with UNK where unsourced.

Return type:

tuple

caldip.readers.resolve_data_dir(config_file: Path, config: dict, override: str | None = None) → Path

Return the data directory for a cast, with optional CLI override.

Priority: explicit override > config ‘directory’ key > parent of config file. For config files sitting inside a cast directory (name starts with ‘cast’), the config file’s parent is used directly.

caldip.readers.resolve_instrument_class(instrument: str | None, file_type: str | None = None) → str

Resolve a cruise-YAML instrument value to an oceanarray class name.

A value that matches caldip.config.parameters.INSTRUMENT_CLASSES case-insensitively is returned in its canonical (lowercase) form; a documented legacy alias is mapped to its class with a deprecation warning (aliases are removed at v1.0.0); a real class caldip compares nothing for (empty INSTRUMENT_CLASS_VARIABLES, e.g. seapoint) is refused with a distinct message; anything else raises.

Parameters:

• instrument (str or None) – The instrument: field from the cruise YAML.

• file_type (str or None, optional) – The instrument’s file_type; disambiguates rbr (rbr-matlab-legacy -> tr1050, rbr-rsk -> rbrsolo).

Returns:

A class name from caldip.config.parameters.INSTRUMENT_CLASSES.

Return type:

str

Raises:

ValueError – If the value is a real class caldip does not compare, or is neither a known class nor a documented alias.

caldip.readers.sbe37_xmlcon_reader(xmlcon_file: str | Path) → dict

Parse an SBE37 xmlcon file for sensor configuration and calibration coefficients.

Deprecated: retained for direct use and testing only.

Parameters:

xmlcon_file (Union[str, Path]) – Path to .xmlcon file

Returns:

Dictionary containing sensor configurations and coefficient objects

Return type:

Dict

scaffold

Scaffold utilities for initialising new caldip cast directories.

Public API: - generate_stub_yaml() — scan a cast directory and write a starter .caldip.yaml

caldip.scaffold.generate_stub_yaml(directory: str, print_only: bool = False) → dict

Generate a stub YAML configuration for a caldip directory.

Parameters:

• directory (str) – Path to caldip directory (e.g., ‘data/proc_calib/cruise123/cal_dip/castM3’)

• print_only (bool) – If True, print to stdout instead of writing file

Returns:

Configuration dictionary

Return type:

Dict

tools

Shared, general-purpose utilities used across the caldip package.

Primary functions:

• to_xarray() -> xarray.Dataset: convert seabirdscientific objects to xarray Datasets.

• trim_to_deployment() -> tuple: trim instrument and reference data to deployment/recovery times.

• summary_stats() -> pandas.DataFrame: extract summary statistics from detailed bottle-stop statistics.

Used by caldip.readers, caldip.core, and the CLI entry points in caldip.cli.

caldip.tools.summary_stats(detailed_stats_df: DataFrame, config: dict) → DataFrame

Extract summary statistics from detailed bottle stop statistics.

Uses the deepest bottle stop data for each instrument.

Parameters:

• detailed_stats_df (pandas.DataFrame) – Per-bottle-stop detailed statistics.

• config (dict) – Configuration dictionary (unused; retained for call-site compatibility).

Returns:

One summary row per instrument, using its deepest bottle stop.

Return type:

pandas.DataFrame

caldip.tools.to_xarray(instrument_data: object) → Dataset

Convert seabirdscientific InstrumentData object to xarray Dataset.

Parameters:

instrument_data (seabirdscientific.InstrumentData) – The InstrumentData object from seabirdscientific.

Returns:

Dataset with measurements as data variables and time coordinate.

Return type:

xarray.Dataset

caldip.tools.trim_to_deployment(instruments: dict[str, dict], reference_data: dict[str, dict], config: dict) → tuple

Trim instrument and reference data to deployment/recovery times.

Parameters:

• instruments (dict) – Instrument data dictionary

• reference_data (dict) – Reference data dictionary

• config (dict) – Configuration with deployment_time and recovery_time

Returns:

(trimmed_instruments, trimmed_reference_data)

Return type:

tuple

sbe_hex_reader

Readers for SBE37 xmlcon and hex files, including calibration-coefficient parsing.

caldip.sbe_hex_reader.parse_hex_header_sensors(hex_file: str | Path) → dict

Parse SBE37 hex file header to extract enabled sensors and calibration coefficients.

Parameters:

hex_file (Union[str, Path]) – Path to .hex file

Returns:

Dictionary with enabled_sensors list and calibration_coefficients

Return type:

Dict

caldip.sbe_hex_reader.sbe37_hex_reader(hex_file: str | Path) → Dataset

Read SBE37 hex file using seabirdscientific library.

Parameters:

hex_file (Union[str, Path]) – Path to .hex file

Returns:

Dataset containing temperature, conductivity, pressure, and/or oxygen data

Return type:

xr.Dataset

caldip.sbe_hex_reader.sbe37_xmlcon_reader(xmlcon_file: str | Path) → dict

Parse an SBE37 xmlcon file for sensor configuration and calibration coefficients.

Deprecated: retained for direct use and testing only.

Parameters:

xmlcon_file (Union[str, Path]) – Path to .xmlcon file

Returns:

Dictionary containing sensor configurations and coefficient objects

Return type:

Dict

report

Build per-cruise HTML calibration reports from caldip results.

build_report reads a directory of caldip stats output CSVs (and the saved {cast}_plot.html figures beside them) and writes an index page plus one page per cast. It adds no data model: everything comes from files already on disk, so a report is reproducible from an archive of results without the raw instrument data.

The report is a self-contained folder (an index, a casts/ subfolder, and one shared plotly.min.js), not a set of self-contained files: the interactive figures are Plotly and share the sibling bundle. This is the deliberate trade for hover/zoom on bottle stops, but it means there is no PDF path — a headless renderer such as WeasyPrint will not execute the Plotly script, so a cast page has no figure in print. If PDF output is ever wanted, a static PNG must be produced alongside the interactive figure (the shared design system’s print_css and <figure><img> path already support that shape).

caldip.report.build_report(results_dir: str | Path, out_dir: str | Path, *, cruise_name: str | None = None) → Path

Build a per-cruise HTML report from a directory of caldip results.

Parameters:

• results_dir (str or pathlib.Path) – Directory holding {cast}_detailed_statistics.csv (and, ideally, {cast}_summary_statistics.csv and {cast}_plot.html) for each cast.

• out_dir (str or pathlib.Path) – Directory to write the report into; created if absent. The index is written to out_dir/index.html and cast pages to out_dir/casts/{cast}.html.

• cruise_name (str or None, optional) – Cruise label for the index heading. If omitted, it is inferred only when the results directory is a cal_dip/ folder (from its parent); otherwise it is set to "UNK" with a warning rather than guessing.

Returns:

Path to the written index page.

Return type:

pathlib.Path

Raises:

FileNotFoundError – If no *_detailed_statistics.csv files are found in results_dir.

report.index

Build the per-cruise index page: one row per cast, linking to its page.

caldip.report.index.build_index_html(summaries: list[CastSummary], *, cruise_name: str) → str

Build the cruise index HTML from per-cast summaries.

The per-cast cruise (read from each {cast}_caldip.nc) is always shown as a column: it is the cruise recorded with the cast, which need not match the report heading when a directory mixes casts from more than one cruise.

Parameters:

• summaries (list of CastSummary) – One entry per cast, in display order.

• cruise_name (str) – Cruise label shown in the heading and title.

Returns:

A complete HTML document for the index page.

Return type:

str

report.cast

Build a per-cast page: bottle stops, summary, embedded figure, per-stop detail.

caldip.report.cast.build_cast_page_html(summary: CastSummary, *, fallback_href: str | None, plotly_src: str, inventory_href: str | None = None) → str

Build the HTML page for a single cast.

Parameters:

• summary (CastSummary) – The cast’s resolved paths and counts.

• fallback_href (str or None) – Relative href from this page to the saved plot file, used only if the figure cannot be embedded. None if no saved plot exists.

• plotly_src (str) – Relative href from this page to the shared plotly.min.js.

• inventory_href (str or None, optional) – Relative href to this cast’s netCDF inventory page, linked in the nav. None when the cast has no netCDF.

Returns:

A complete HTML document for the cast page.

Return type:

str

report.inventory

Build a per-file netCDF inventory page: dimensions, variables, attributes.

A viewable “what was generated” view of a {cast}_caldip.nc file — the counterpart to ncdump -h, styled with the shared report design system. The inventory is read as plain data by read_nc_meta() and rendered by build_inventory_html(); write_inventory() writes the HTML file. Used by the caldip inspect command and linkable from the per-cruise report.

caldip.report.inventory.build_inventory_html(nc_path: Path) → str

Build the netCDF inventory HTML for a file.

Parameters:

nc_path (pathlib.Path) – The netCDF file to inventory.

Returns:

A complete self-contained HTML document.

Return type:

str

caldip.report.inventory.read_nc_meta(nc_path: Path) → dict[str, Any]

Read a netCDF file into a plain-data inventory.

Parameters:

nc_path (pathlib.Path) – The netCDF file to inventory.

Returns:

filename, filesize (human-readable), dims, coords and data_vars (each a _var_meta() row), and global_attrs in the file’s own order. On a read error, {"filename", "error"} so the page can report it rather than fail.

Return type:

dict

caldip.report.inventory.write_inventory(nc_path: Path, out_path: Path) → Path

Write the netCDF inventory HTML for nc_path to out_path.

Parameters:

• nc_path (pathlib.Path) – The netCDF file to inventory.

• out_path (pathlib.Path) – Destination HTML path; parent directories are created.

Returns:

The written out_path.

Return type:

pathlib.Path