Data files (netCDF)

Layout

Each instrument has one root that ctdcast owns — ctd_root and ladcp_root in the config’s data: block. Inside it, one directory per processing stage, and the compiled product at the top:

<ctd_root>/
    stage1/  mixsed2_017_stage1.nc     raw CNV converted
    stage2/  mixsed2_017_stage2.nc     + soak / back-on-deck flags
    stage3/  mixsed2_017_stage3.nc     + QC and calibration
    profiles.nc                        compiled product

<ladcp_root>/
    stage1/  ladcp_017_stage1.nc       LADCP has one stage
    ladcp_profiles.nc

Stages are non-destructive: each reads its predecessor and writes a new file, so a stage can be re-run without redoing the ones before it, and the full lineage stays on disk. See ctdcast processing framework for what each stage does.

Stage 1 is faithful; stage 2 is curated. Stage 1 is a faithful translation — every column in the CNV reaches the stage-1 file, dropping nothing. SeaBird-computed quantities (density, depth, sound velocity, timeJ/timeS, the per-scan flag, oxygen saturation) are kept under an sbe_ prefix so they cannot be mistaken for ctdcast’s own (ctdcast writes sigma0 via TEOS-10, not SBE density); an unrecognised channel is kept under its source name and warned about once per cast (so a rig with an unmodelled channel warns on every cast until a VARIABLES entry is added). Stage 2 then removes the sbe_ set by default — the drop is deliberate and recorded (a history line and a dropped_channels attribute), driven by the config trim.drop_sbe: list rather than a flag so the output is reproducible from the config alone. Stage-2 output is therefore not a strict superset of stage 1: the stage-1 file is the faithful record, and every drop is written down (a history line and the dropped_channels attribute) so a removed channel is always traceable to the stage-1 file that still holds it.

The stage appears in the directory and the filename. The redundancy is deliberate: a file copied out of stage2/ still says what it is, whereas provenance living only in the path is lost the first time someone moves a file. For that reason parse_stage reads the suffix from the filename and does not trust the parent directory.

Note

A directory of unsuffixed per-cast files, written before the stage layout existed, is still read — as stage 1. Point ctd_root at it and nothing needs regenerating.

Best-available selection

Anything reading per-cast files — build_profiles, the report — takes the highest stage present for each cast: stage 3 if it exists, else stage 2, else stage 1. So a cruise part-way through processing compiles honestly rather than failing, and casts may legitimately sit at different stages.

Anything writing is stricter: a stage reads only its immediate predecessor (stage 3 from stage 2, never from stage 1). A missing predecessor skips that cast with a warning. The asymmetry is deliberate — mixsed2_017_stage3.nc must mean one thing, and if stage 3 could silently consume stage 1 the same filename would sometimes mean “QC’d, soak flagged” and sometimes “QC’d, not soak flagged”.

Cast identity is the (number, suffix) pair, so 017 and 017b are distinct events rather than one cast listed twice.

Per-cast files

Each file covers one CTD cast. The required dimension and variables are:

Name	Description
`time` (dim)	Time coordinate (1-D, one value per scan).
`pressure`	Sea pressure in dbar.
`ctd_temperature` / `ctd_temperature_1` / `ctd_temperature_2`	In-situ temperature in °C (ITS-90). Plain name for single-sensor instruments; `_1`/`_2` suffix for dual-sensor rigs.
`ctd_salinity` / `ctd_salinity_1` / `ctd_salinity_2`	Practical salinity (PSU).
`ctd_oxygen` / `ctd_oxygen_1` / `ctd_oxygen_2`	Dissolved oxygen in µmol kg⁻¹.
`ctd_fluor`	Fluorescence in mg m⁻³ (chlorophyll-a equivalent; 1 mg m⁻³ = 1 µg L⁻¹).
`ctd_turbidity`	Turbidity in NTU.
`ctd_altimeter`	Altimeter distance to seafloor in m.
`conductivity_1` / `conductivity_2`	Electrical conductivity in mS cm⁻¹ (no CCHDO equivalent; keeps `_1`/`_2` suffix always).
`transmissometer`	Beam transmittance in % (WET Labs C-Star; no CCHDO equivalent).
`par`	Photosynthetically active radiation in µmol photons m⁻² s⁻¹ (Biospherical/Licor/Chelsea).
`spar`	Surface PAR in µmol photons m⁻² s⁻¹ (deck-mounted reference sensor).
`volt{N}_raw`	Raw voltage (V) for sensors whose conversion is not implemented (e.g. pH) or whose calibration coefficients are absent. `N` is the zero-based voltage channel index.

Global attributes: cruise identity (cruise, platform_*, expocode), raw_filename, raw_metadata, a stamped history, and the upstream correction ledger (sbe_*, correction_*, time_coordinate_source, time_clock_offset_seconds). Each is described in the table below.

File identity and lineage

Every file ctdcast writes carries a tracking_id (a UUID4, ACDD/OceanSITES convention) and, from stage 2 on, a source_tracking_id naming the file it was made from — so the chain back to the raw CNV is readable from any single output.

tracking_id answers which instance of this file this is, not whether the content changed. It is not a content hash: a fresh id is written on every write, so a re-run of stage 3 produces a new id even if nothing else moved. Stability would be the bug — a stable id could not detect a rewrite.
source_tracking_id (stage 2, stage 3) holds the tracking_id of the stage file that was read. In profiles.nc it is a per-N_PROF variable, naming the source cast file per profile (the compiled file has its own tracking_id global attribute).
Stage 1 has no upstream netCDF, so it records the raw source filename in a distinct attribute instead — source_cnv for CTD, source_mat for LADCP.
processing_stage (1/2/3) records the stage in the file itself, not only in the filename, so a file copied out of its stage directory still declares what it is — the form a cross-package consumer (caldip) reads.
cast_id records the cast identity (the canonical zero-padded form, e.g. 011 or 011b) in the file for the same reason; it is stamped at stage 1 and carried forward.
date_created is set once (first write) and preserved across re-runs; date_modified moves on every write. So a stage-3 rewrite records when it was rewritten without resetting the creation time.

To read the best-available ctdcast file per cast from another package, call ctdcast.select_best_available(root) (stage 3, else stage 2, else stage 1) rather than reimplementing the precedence — see Python API.

Processing state

Two OceanSITES fields say what state a file is in, using that standard’s own vocabulary.

data_mode (reference table 4) is a global attribute on every file: P provisional (the default — some processing may have been done, but the file is not the product of record), D delayed-mode (all calibrations and QC applied), M mixed. D is declared, never inferred: it comes only from cruise_info.data_mode: D in the config, because no code can know that every calibration a cruise needed was applied. A quick-look (ctdcast draft) file is always P. In profiles.nc the global value is M only when the compiled casts genuinely differ, and a per-N_PROF source_data_mode variable then says which profile is in which mode; casts merely compiled from different stages do not make the file M (stages 1–3 are all provisional — source_stage records the stage difference). data_mode_meaning always accompanies it.

processing_level (reference table 3) is a per-variable attribute recording what has been done to that variable, using table 3’s strings verbatim:

Instrument data that has been converted to geophysical values — stamped at stage 1 on every measured channel (temperature, conductivity, pressure, oxygen, fluorescence, turbidity, altimeter), identified by its link to a sensor in the CNV <Sensors> block. Computed channels (ctd_salinity_*, density, …) do not carry it: they were derived from already-converted inputs, never instrument data. Absence is not “raw” — table 3 has an explicit raw value; absence means not stated.
Ranges applied, bad data flagged — stage-2 soak/deck flags and stage-3 gross-range and spike tests.
Post-recovery calibrations have been applied — a stage-3 conductivity calibration; it propagates to re-derived salinity, whose values now embody the calibration.

When several apply to one variable they are joined with "; " in the order applied (comma is unusable — Ranges applied, bad data flagged contains one), so the attribute doubles as the procedure sequence. The list is an ordered set: re-running a stage never duplicates a value. The rule behind the split is that a value-changing procedure (calibration, flagging) propagates to variables derived from the changed inputs, while an origin claim (conversion) does not. Spelling is lowercase processing_level throughout, matching the <PARAM>: template; do not “correct” a file to a capitalised variant from a stray manual example.

The compiled profiles.nc carries the per-variable processing_level too, so the archive is interpretable without the stage files: for each variable it is the value the compiled casts agree on, or — where they differ — an explicit “mixed across casts” marker, never a union of their sentences (a union would claim a procedure on a cast that never had it). The per-profile treatment stays reachable through source_stage / source_data_mode / source_tracking_id.

Profiles file (`<ctd_root>/profiles.nc`)

Compiled on a 1 dbar pressure grid, dimensions N_PROF × pressure:

Name	Description
`cast_number`	Integer cast number.
`cast_suffix`	Letter suffix for a repeated cast (`"b"`), empty otherwise. Identity is the `(cast_number, cast_suffix)` pair.
`source_stage`	Which processing stage this profile was compiled from — `int8` with `flag_values`/`flag_meanings`, the same idiom as the QARTOD flags: `1` converted, `2` soak flagged, `3` QC and calibration, and `0` unknown. `0` means the source was an unsuffixed file read through the flat-layout shim, which does not state its own stage — recorded as unknown rather than assumed to be 1.
`cast_direction`	`"down"` or `"up"` (`cast_type` is a deprecated alias for the same values).
`latitude`	Latitude in decimal degrees north.
`longitude`	Longitude in decimal degrees east.
`time_start`	Start time of the cast (datetime64).
`time_end`	End time of the cast (datetime64).
`ctd_temperature` / `ctd_temperature_1`	In-situ temperature on the 1 dbar grid.
`ctd_salinity` / `ctd_salinity_1`	Practical salinity on the 1 dbar grid.
`ctd_oxygen` / `ctd_oxygen_1`	Dissolved oxygen in µmol kg⁻¹ on the 1 dbar grid.

Each cast contributes two profiles — downcast and upcast — so N_PROF is twice the cast count. The LADCP product has one profile per cast, so the two files’ N_PROF axes do not align; join on (cast_number, cast_suffix) rather than by index.

The pressure coordinate is the bin centre, so a binned value sits at the mean depth of the samples it averages rather than at the bin’s shallow edge.

Samples flagged QARTOD suspect (3) or fail (4) — from the stage-2 soak / back-on-deck trim and the stage-3 gross-range and spike tests — are dropped before binning, so flagged data does not enter the bin means. Each science variable records qc_input_samples (finite input samples) and qc_excluded_samples (dropped); the netCDF inventory page reports these per variable as a percentage of pre-binning samples, so the figure is not confounded by binning’s own reduction in point count. Note the soak/deck trim flags the same scans on every variable, so a variable can be excluded here without its own gross-range or spike test firing.

Where each attribute is written

A fact is attached at the earliest stage at which it is true, so a per-cast stage file is self-describing and the compiled product mostly inherits rather than originates. The test is not “is it knowable at stage 1” but “can it still change after the ship docks” — a value written into 200 frozen stage files and then edited in config is a stale copy in 200 places.

Class	Examples	Where, and why
Identity	`cruise`, `platform_*`, `expocode`	Written at stage 1 on every per-cast file, and lifted unchanged into the compiled products. Fixed the moment a cast is taken, and a file copied out of its directory must still say which cruise and which ship.
Derived from data	`geospatial_`, `time_coverage_`, vertical bounds	Computed at every level. Not moved early: per-cast bounds describe that cast, compiled bounds describe the cruise. Same function, different scope.
Authored, and revisable	contributors, `license`, `embargo`, `acknowledgement`, `title`	Compile time only. ORCIDs get corrected and embargo dates shift for years afterwards; per-cast copies would be plausible and wrong.
About the product	`pressure_spacing_dbar`, `source`, `id`	Compile time only. They describe the gridded artefact, not the measurement, so there is nothing earlier to originate them.
Upstream provenance	`sbe_processing_order`, `correction_*`, `time_coordinate_source`, `sbe_acquisition`, `sbe_processing`	Written at stage 1, read from the CNV header. Describes what was done to the cast before ctdcast — see the ledger below.
Stage-local	`history`	Each stage appends. The model the rest of this table follows.

Lifting is strict

When the compiled product takes identity from the per-cast files, disagreement on a cruise-defining attribute is an error, not a merge: a compiled product describes one cruise, so two values of cruise or expocode mean either a cast from another cruise in the directory or two legs sharing one root. (Legs depart on different dates, so they have different EXPOCODEs — compile each into its own root.)

The rest of the identity layer — the platform_* block — describes the ship, and casts disagreeing there is ordinary registry drift: a platform_vocabulary URI edited between two stage-1 runs, a vessel renamed mid-programme. That says nothing about whether these casts are one cruise, so it warns rather than failing: cruise_info’s value is used where it states one, and the attribute is omitted where it does not, rather than picking one cast’s answer arbitrarily.

An attribute no per-cast file states falls back to cruise_info with a warning, which is the path for files written before identity was recorded at stage 1.

Where the two sources disagree about identity, the files win — a stage file records the cruise the cast was actually taken on, and a config can be edited years later. The compiled product’s title is built from the same lifted value, so a file cannot be titled for one cruise and attributed to another. Re-run stage 1 if it is the per-cast files that are wrong.

For everything outside identity, config remains the source of truth: writing identity at stage 1 makes the per-cast file portable, not the authority on what the cruise is called in the report.

The correction ledger

A CNV usually arrives already processed — by the deck unit at acquisition and by SBE Data Processing afterwards. Stage 1 reads the header and records what it finds, so a stage file states not only what ctdcast did to it but what had already been done. See ctdcast processing framework for why this matters; this section is the attribute reference.

Attribute	Meaning
`sbe_acquisition`	The `` block, verbatim*, minus the `<Sensors>` XML (which is kept separately as `raw_metadata`). Archival ground truth: no interpretation, so it cannot be wrong, and a key a future SBE release adds is preserved even though this version’s parser has never heard of it.
`sbe_processing`	The `#` module block, verbatim, same rationale.
`sbe_processing_order`	The corrections in the order they were applied, e.g. `align(deck) datcnv wildedit filter celltm Derive binavg`. Recorded explicitly rather than left to be re-derived: the deck-unit alignment carries no timestamp of its own, and some module writers emit none either, so file order is the only reliable ordering key.
`correction_align`	The deck-unit advance, per channel, e.g. `SBE 11plus V 5.0 deck unit: primary conductivity +0.073 s, secondary conductivity +0.043 s, voltage 0 +0.000 s`. Per channel because older deck units do not always set the two conductivity channels alike, and a channel advanced less than the physical lag carries a residual the other does not. Written whenever the deck unit stated an advance at all — including one set to `+0.000` on every channel, since “present and set to zero” is a different fact from “no deck unit”. In that case `align(deck)` is absent from `sbe_processing_order`, because nothing was applied.
`correction_celltm`, `correction_wildedit`, `correction_binavg`, …	One attribute per module in the chain, naming the agent and the parameters it used. There is no curated subset: deciding which modules “count” as corrections would mean predicting them, and real headers carry modules that were not predicted. A module that ran more than once — which Sea-Bird explicitly sanctions for Wild Edit — is suffixed `correction_wildedit`, `correction_wildedit_2`, in file order, so no run’s parameters are lost. Absence means “not recorded”, never “not done” — a file whose chain ctdcast could not parse has the verbatim blocks and no `correction_*` entries at all.
`time_coordinate_source`	Which clock the `time` coordinate is anchored to, and to which moment: `System UTC, first data scan`, `NMEA time, header`, and so on. SBE records this in a bracket on its `start_time` line and it varies between cruises, so a file that does not state it leaves the reader unable to tell GPS time from a possibly-drifting acquisition clock.
`time_clock_offset_seconds`	`NMEA UTC` minus `System UTC`, where the header carries both. Several seconds is normal and the sign varies. It is reported, not applied — a single cast cannot distinguish a drifting clock from a stale NMEA sentence; that takes the whole cruise.

The unit of record is the correction, not the module — which matters because the most consequential one, the deck-unit conductivity alignment, is not a module at all and lives in a different part of the header. Every module in the chain then contributes a correction of its own, so in practice the ledger is one entry per module plus one for the deck unit.

Parameters keep the SBE channel names the header uses (t090C, c0S/m) rather than being translated to ctdcast’s canonical names: the file states what its source stated, and translation happens where the mapping is needed.

Sensor provenance

profiles.nc also records which physical sensor produced each measurement, using three families of variables. The capitalisation and the _channel_ infix are meaningful — keep them distinct:

SENSOR_<TYPE>_<SERIAL> — upper-case, dimensionless

One variable per distinct physical device used anywhere in the cruise, e.g. SENSOR_TEMPERATURE_5806 or SENSOR_FLUOROMETER_FLNTURTD_3219. It holds no data; all provenance is in its attributes (sensor_model, sensor_serial_number, sensor_calibration_date, sensor_maker, the L05/L22/L35 vocabulary URIs, model_source, and its sensor_role and sensor_channel). A frequency sensor (temperature, conductivity, pressure) also carries sensor_calibration_slope and sensor_calibration_offset — the datcnv drift/span correction already baked into the data before ctdcast read it; a value away from the identity (slope 1, offset 0) means a correction was applied. Every entry (frequency and voltage alike) also carries sensor_config_xml — the whole <sensor> block from the CNV <Sensors> header, verbatim: every base calibration coefficient and its type context, so the raw→physical conversion is reconstructable from the compiled file alone, not only from the stage-1 raw_metadata. The serial identifies the device, so a cell used as both primary and secondary of one type is a single entry; sensor_shared_with cross-links one device serving two roles (e.g. a combined FLNTU as both fluorometer and turbidity).

Each measured data variable carries a single sensor attribute naming the SENSOR_* entry that produced it (e.g. ctd_temperature_1’s sensor = "SENSOR_TEMPERATURE_5806"). That one link is all a variable holds; its role and channel live on the entry, so a device with no stored variable (pH, a transmissometer) still records both. This is what lets the compile aggregate the catalog without re-reading the header.

sensor_<role> — lower-case, dimension N_PROF

Per profile, a string naming the SENSOR_* variable that filled each role — e.g. sensor_temperature_1 may be "SENSOR_TEMPERATURE_5806" on early casts and "SENSOR_TEMPERATURE_4823" after a swap. This answers “which sensor’s calibration applies to this cast?”; diffing it down the casts gives the sensor-change log.

sensor_channel_<role> — lower-case, dimension N_PROF

Per profile, the integer raw acquisition channel that role’s sensor was wired into (-1 where unused). A change here while sensor_<role> holds constant is a re-cabling, not a hardware swap.

Roles use ctdcast’s canonical names: temperature_1/_2, conductivity_1/_2, oxygen_1/_2, pressure, fluorometer, turbidity, transmissometer, ph, altimeter. The universal SensorID → model table ships in ctdcast/config/sbe_sensors.yaml; per-cruise refinements come from the sensors: block in config.yaml (see above). The SBE sensors report page presents all of this as configuration, inventory and rewiring tables.

Where it is built, and two kinds of provenance

The catalog is resolved once, at stage 1 — the per-cast netCDF files carry their own SENSOR_* entries — and the compile simply aggregates them; nothing re-reads the SBE header downstream. This splits each entry’s attributes into two provenances that mean different things when two casts disagree:

Header-native (sensor_calibration_date, sensor_calibration_slope, sensor_calibration_offset) are fixed for a serial. A mismatch across casts cannot be a real recalibration at sea, so it is flagged as a parsing or data error.
Config-resolved (sensor_model, sensor_maker, the vocabulary URIs) come from the SensorID registry and the sensors: overrides. A mismatch just means the casts were stamped under different config versions — expected, and fixed by re-running stage 1 (or a future enrich step) to restamp, not by treating it as a data error.

Because provenance lives on the file, the per-cast page’s Sensors and calibration state table reads the catalog directly — one row per device, joined to the variable it produced and showing its model, calibration date and any applied slope/offset — rather than re-parsing the SBE header. A file that predates the catalog falls back to the header parse.

A per-cast file that carries no SENSOR_* catalog predates this provenance. The build_profiles library call still compiles it, with a warning and a sensor_catalog attribute recording the gap; ctdcast process --stage profiles and ctdcast run instead refuse, because re-running stage 1 to build the catalog is the actionable fix before a product is shipped.