.. _data_files: =================== Data files (netCDF) =================== Layout ~~~~~~ Each instrument has one **root** that ctdcast owns — ``ctd_root`` and ``ladcp_root`` in the config's ``data:`` block. Inside it, one directory per processing stage, and the compiled product at the top: .. code-block:: text / stage1/ mixsed2_017_stage1.nc raw CNV converted stage2/ mixsed2_017_stage2.nc + soak / back-on-deck flags stage3/ mixsed2_017_stage3.nc + QC and calibration profiles.nc compiled product / stage1/ ladcp_017_stage1.nc LADCP has one stage ladcp_profiles.nc Stages are **non-destructive**: each reads its predecessor and writes a new file, so a stage can be re-run without redoing the ones before it, and the full lineage stays on disk. See :doc:`processing_framework` for what each stage does. **Stage 1 is faithful; stage 2 is curated.** Stage 1 is a faithful translation — *every* column in the CNV reaches the stage-1 file, dropping nothing. SeaBird-computed quantities (density, depth, sound velocity, ``timeJ``/``timeS``, the per-scan flag, oxygen saturation) are kept under an ``sbe_`` prefix so they cannot be mistaken for ctdcast's own (ctdcast writes ``sigma0`` via TEOS-10, not SBE density); an unrecognised channel is kept under its source name and warned about once per cast (so a rig with an unmodelled channel warns on every cast until a ``VARIABLES`` entry is added). Stage 2 then removes the ``sbe_`` set by default — the drop is deliberate and recorded (a ``history`` line and a ``dropped_channels`` attribute), driven by the config ``trim.drop_sbe:`` list rather than a flag so the output is reproducible from the config alone. Stage-2 output is therefore not a strict superset of stage 1: the stage-1 file is the faithful record, and every drop is written down (a ``history`` line and the ``dropped_channels`` attribute) so a removed channel is always traceable to the stage-1 file that still holds it. The stage appears in the **directory and the filename**. The redundancy is deliberate: a file copied out of ``stage2/`` still says what it is, whereas provenance living only in the path is lost the first time someone moves a file. For that reason ``parse_stage`` reads the suffix from the filename and does not trust the parent directory. .. note:: A directory of **unsuffixed** per-cast files, written before the stage layout existed, is still read — as stage 1. Point ``ctd_root`` at it and nothing needs regenerating. Best-available selection ~~~~~~~~~~~~~~~~~~~~~~~~ Anything *reading* per-cast files — ``build_profiles``, the report — takes the highest stage present for each cast: stage 3 if it exists, else stage 2, else stage 1. So a cruise part-way through processing compiles honestly rather than failing, and casts may legitimately sit at different stages. Anything *writing* is stricter: a stage reads **only** its immediate predecessor (stage 3 from stage 2, never from stage 1). A missing predecessor skips that cast with a warning. The asymmetry is deliberate — ``mixsed2_017_stage3.nc`` must mean one thing, and if stage 3 could silently consume stage 1 the same filename would sometimes mean "QC'd, soak flagged" and sometimes "QC'd, not soak flagged". Cast identity is the ``(number, suffix)`` pair, so ``017`` and ``017b`` are distinct events rather than one cast listed twice. Per-cast files ~~~~~~~~~~~~~~ Each file covers one CTD cast. The required dimension and variables are: .. list-table:: :header-rows: 1 :widths: 20 80 * - Name - Description * - ``time`` (dim) - Time coordinate (1-D, one value per scan). * - ``pressure`` - Sea pressure in dbar. * - ``ctd_temperature`` / ``ctd_temperature_1`` / ``ctd_temperature_2`` - In-situ temperature in °C (ITS-90). Plain name for single-sensor instruments; ``_1``/``_2`` suffix for dual-sensor rigs. * - ``ctd_salinity`` / ``ctd_salinity_1`` / ``ctd_salinity_2`` - Practical salinity (PSU). * - ``ctd_oxygen`` / ``ctd_oxygen_1`` / ``ctd_oxygen_2`` - Dissolved oxygen in µmol kg⁻¹. * - ``ctd_fluor`` - Fluorescence in mg m⁻³ (chlorophyll-a equivalent; 1 mg m⁻³ = 1 µg L⁻¹). * - ``ctd_turbidity`` - Turbidity in NTU. * - ``ctd_altimeter`` - Altimeter distance to seafloor in m. * - ``conductivity_1`` / ``conductivity_2`` - Electrical conductivity in mS cm⁻¹ (no CCHDO equivalent; keeps ``_1``/``_2`` suffix always). * - ``transmissometer`` - Beam transmittance in % (WET Labs C-Star; no CCHDO equivalent). * - ``par`` - Photosynthetically active radiation in µmol photons m⁻² s⁻¹ (Biospherical/Licor/Chelsea). * - ``spar`` - Surface PAR in µmol photons m⁻² s⁻¹ (deck-mounted reference sensor). * - ``volt{N}_raw`` - Raw voltage (V) for sensors whose conversion is not implemented (e.g. pH) or whose calibration coefficients are absent. ``N`` is the zero-based voltage channel index. Global attributes: cruise identity (``cruise``, ``platform_*``, ``expocode``), ``raw_filename``, ``raw_metadata``, a stamped ``history``, and the upstream correction ledger (``sbe_*``, ``correction_*``, ``time_coordinate_source``, ``time_clock_offset_seconds``). Each is described in the table below. File identity and lineage ~~~~~~~~~~~~~~~~~~~~~~~~~~ Every file ctdcast writes carries a ``tracking_id`` (a UUID4, ACDD/OceanSITES convention) and, from stage 2 on, a ``source_tracking_id`` naming the file it was made from — so the chain back to the raw CNV is readable from any single output. - ``tracking_id`` answers *which instance of this file this is*, **not** *whether the content changed*. It is **not a content hash**: a fresh id is written on **every** write, so a re-run of stage 3 produces a new id even if nothing else moved. Stability would be the bug — a stable id could not detect a rewrite. - ``source_tracking_id`` (stage 2, stage 3) holds the ``tracking_id`` of the stage file that was read. In ``profiles.nc`` it is a per-``N_PROF`` variable, naming the source cast file per profile (the compiled file has its own ``tracking_id`` global attribute). - Stage 1 has no upstream netCDF, so it records the raw source filename in a distinct attribute instead — ``source_cnv`` for CTD, ``source_mat`` for LADCP. - ``processing_stage`` (``1``/``2``/``3``) records the stage in the file itself, not only in the filename, so a file copied out of its stage directory still declares what it is — the form a cross-package consumer (caldip) reads. - ``cast_id`` records the cast identity (the canonical zero-padded form, e.g. ``011`` or ``011b``) in the file for the same reason; it is stamped at stage 1 and carried forward. - ``date_created`` is set once (first write) and preserved across re-runs; ``date_modified`` moves on every write. So a stage-3 rewrite records *when* it was rewritten without resetting the creation time. To read the best-available ctdcast file per cast from another package, call ``ctdcast.select_best_available(root)`` (stage 3, else stage 2, else stage 1) rather than reimplementing the precedence — see :doc:`api`. Processing state ~~~~~~~~~~~~~~~~~ Two OceanSITES fields say what state a file is in, using that standard's own vocabulary. ``data_mode`` (reference table 4) is a **global** attribute on every file: ``P`` provisional (the default — some processing may have been done, but the file is not the product of record), ``D`` delayed-mode (all calibrations and QC applied), ``M`` mixed. ``D`` is **declared, never inferred**: it comes only from ``cruise_info.data_mode: D`` in the config, because no code can know that *every* calibration a cruise needed was applied. A quick-look (``ctdcast draft``) file is always ``P``. In ``profiles.nc`` the global value is ``M`` only when the compiled casts genuinely differ, and a per-``N_PROF`` ``source_data_mode`` variable then says which profile is in which mode; casts merely compiled from different *stages* do not make the file ``M`` (stages 1–3 are all provisional — ``source_stage`` records the stage difference). ``data_mode_meaning`` always accompanies it. ``processing_level`` (reference table 3) is a **per-variable** attribute recording what has been done to that variable, using table 3's strings verbatim: - ``Instrument data that has been converted to geophysical values`` — stamped at stage 1 on every **measured** channel (temperature, conductivity, pressure, oxygen, fluorescence, turbidity, altimeter), identified by its link to a sensor in the CNV ```` block. **Computed** channels (``ctd_salinity_*``, density, …) do not carry it: they were derived from already-converted inputs, never instrument data. Absence is not "raw" — table 3 has an explicit raw value; absence means *not stated*. - ``Ranges applied, bad data flagged`` — stage-2 soak/deck flags and stage-3 gross-range and spike tests. - ``Post-recovery calibrations have been applied`` — a stage-3 conductivity calibration; it **propagates** to re-derived salinity, whose values now embody the calibration. When several apply to one variable they are joined with ``"; "`` in the order applied (comma is unusable — ``Ranges applied, bad data flagged`` contains one), so the attribute doubles as the procedure sequence. The list is an ordered **set**: re-running a stage never duplicates a value. The rule behind the split is that a value-changing procedure (calibration, flagging) propagates to variables derived from the changed inputs, while an origin claim (conversion) does not. Spelling is lowercase ``processing_level`` throughout, matching the ``:`` template; do not "correct" a file to a capitalised variant from a stray manual example. The compiled ``profiles.nc`` carries the per-variable ``processing_level`` too, so the archive is interpretable without the stage files: for each variable it is the value the compiled casts agree on, or — where they differ — an explicit "mixed across casts" marker, never a union of their sentences (a union would claim a procedure on a cast that never had it). The per-profile treatment stays reachable through ``source_stage`` / ``source_data_mode`` / ``source_tracking_id``. Profiles file (``/profiles.nc``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Compiled on a 1 dbar pressure grid, dimensions ``N_PROF × pressure``: .. list-table:: :header-rows: 1 :widths: 20 80 * - Name - Description * - ``cast_number`` - Integer cast number. * - ``cast_suffix`` - Letter suffix for a repeated cast (``"b"``), empty otherwise. Identity is the ``(cast_number, cast_suffix)`` pair. * - ``source_stage`` - Which processing stage this profile was compiled from — ``int8`` with ``flag_values``/``flag_meanings``, the same idiom as the QARTOD flags: ``1`` converted, ``2`` soak flagged, ``3`` QC and calibration, and ``0`` *unknown*. ``0`` means the source was an unsuffixed file read through the flat-layout shim, which does not state its own stage — recorded as unknown rather than assumed to be 1. * - ``cast_direction`` - ``"down"`` or ``"up"`` (``cast_type`` is a deprecated alias for the same values). * - ``latitude`` - Latitude in decimal degrees north. * - ``longitude`` - Longitude in decimal degrees east. * - ``time_start`` - Start time of the cast (datetime64). * - ``time_end`` - End time of the cast (datetime64). * - ``ctd_temperature`` / ``ctd_temperature_1`` - In-situ temperature on the 1 dbar grid. * - ``ctd_salinity`` / ``ctd_salinity_1`` - Practical salinity on the 1 dbar grid. * - ``ctd_oxygen`` / ``ctd_oxygen_1`` - Dissolved oxygen in µmol kg⁻¹ on the 1 dbar grid. Each cast contributes **two** profiles — downcast and upcast — so ``N_PROF`` is twice the cast count. The LADCP product has one profile per cast, so the two files' ``N_PROF`` axes do not align; join on ``(cast_number, cast_suffix)`` rather than by index. The pressure coordinate is the **bin centre**, so a binned value sits at the mean depth of the samples it averages rather than at the bin's shallow edge. Samples flagged QARTOD suspect (``3``) or fail (``4``) — from the stage-2 soak / back-on-deck trim and the stage-3 gross-range and spike tests — are dropped before binning, so flagged data does not enter the bin means. Each science variable records ``qc_input_samples`` (finite input samples) and ``qc_excluded_samples`` (dropped); the netCDF inventory page reports these per variable as a percentage of pre-binning samples, so the figure is not confounded by binning's own reduction in point count. Note the soak/deck trim flags the same scans on *every* variable, so a variable can be excluded here without its own gross-range or spike test firing. Where each attribute is written ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A fact is attached at the earliest stage at which it is *true*, so a per-cast stage file is self-describing and the compiled product mostly **inherits** rather than **originates**. The test is not "is it knowable at stage 1" but "can it still change after the ship docks" — a value written into 200 frozen stage files and then edited in config is a stale copy in 200 places. .. list-table:: :header-rows: 1 :widths: 22 30 48 * - Class - Examples - Where, and why * - **Identity** - ``cruise``, ``platform_*``, ``expocode`` - Written at **stage 1** on every per-cast file, and lifted unchanged into the compiled products. Fixed the moment a cast is taken, and a file copied out of its directory must still say which cruise and which ship. * - **Derived from data** - ``geospatial_*``, ``time_coverage_*``, vertical bounds - **Computed at every level.** Not moved early: per-cast bounds describe that cast, compiled bounds describe the cruise. Same function, different scope. * - **Authored, and revisable** - contributors, ``license``, ``embargo``, ``acknowledgement``, ``title`` - **Compile time only.** ORCIDs get corrected and embargo dates shift for years afterwards; per-cast copies would be plausible and wrong. * - **About the product** - ``pressure_spacing_dbar``, ``source``, ``id`` - **Compile time only.** They describe the gridded artefact, not the measurement, so there is nothing earlier to originate them. * - **Upstream provenance** - ``sbe_processing_order``, ``correction_*``, ``time_coordinate_source``, ``sbe_acquisition``, ``sbe_processing`` - Written at **stage 1**, read from the CNV header. Describes what was done to the cast *before* ctdcast — see the ledger below. * - **Stage-local** - ``history`` - **Each stage appends.** The model the rest of this table follows. Lifting is strict ^^^^^^^^^^^^^^^^^ When the compiled product takes identity from the per-cast files, disagreement on a **cruise-defining** attribute is an **error**, not a merge: a compiled product describes one cruise, so two values of ``cruise`` or ``expocode`` mean either a cast from another cruise in the directory or two legs sharing one root. (Legs depart on different dates, so they have different EXPOCODEs — compile each into its own root.) The rest of the identity layer — the ``platform_*`` block — describes the *ship*, and casts disagreeing there is ordinary registry drift: a ``platform_vocabulary`` URI edited between two stage-1 runs, a vessel renamed mid-programme. That says nothing about whether these casts are one cruise, so it **warns** rather than failing: ``cruise_info``'s value is used where it states one, and the attribute is omitted where it does not, rather than picking one cast's answer arbitrarily. An attribute no per-cast file states falls back to ``cruise_info`` with a warning, which is the path for files written before identity was recorded at stage 1. Where the two sources disagree about identity, **the files win** — a stage file records the cruise the cast was actually taken on, and a config can be edited years later. The compiled product's ``title`` is built from the same lifted value, so a file cannot be titled for one cruise and attributed to another. Re-run stage 1 if it is the per-cast files that are wrong. For everything outside identity, config remains the source of truth: writing identity at stage 1 makes the per-cast file *portable*, not the authority on what the cruise is called in the report. The correction ledger ~~~~~~~~~~~~~~~~~~~~~ A CNV usually arrives already processed — by the deck unit at acquisition and by SBE Data Processing afterwards. Stage 1 reads the header and records what it finds, so a stage file states not only what ctdcast did to it but what had already been done. See :ref:`processing_framework` for why this matters; this section is the attribute reference. .. list-table:: :header-rows: 1 :widths: 30 70 * - Attribute - Meaning * - ``sbe_acquisition`` - The ``*`` block, **verbatim**, minus the ```` XML (which is kept separately as ``raw_metadata``). Archival ground truth: no interpretation, so it cannot be wrong, and a key a future SBE release adds is preserved even though this version's parser has never heard of it. * - ``sbe_processing`` - The ``#`` module block, verbatim, same rationale. * - ``sbe_processing_order`` - The corrections in the order they were applied, e.g. ``align(deck) datcnv wildedit filter celltm Derive binavg``. Recorded explicitly rather than left to be re-derived: the deck-unit alignment carries no timestamp of its own, and some module writers emit none either, so file order is the only reliable ordering key. * - ``correction_align`` - The deck-unit advance, per channel, e.g. ``SBE 11plus V 5.0 deck unit: primary conductivity +0.073 s, secondary conductivity +0.043 s, voltage 0 +0.000 s``. Per channel because older deck units do not always set the two conductivity channels alike, and a channel advanced less than the physical lag carries a residual the other does not. Written whenever the deck unit stated an advance at all — including one set to ``+0.000`` on every channel, since "present and set to zero" is a different fact from "no deck unit". In that case ``align(deck)`` is **absent** from ``sbe_processing_order``, because nothing was applied. * - ``correction_celltm``, ``correction_wildedit``, ``correction_binavg``, … - **One attribute per module in the chain**, naming the agent and the parameters it used. There is no curated subset: deciding which modules "count" as corrections would mean predicting them, and real headers carry modules that were not predicted. A module that ran **more than once** — which Sea-Bird explicitly sanctions for Wild Edit — is suffixed ``correction_wildedit``, ``correction_wildedit_2``, in file order, so no run's parameters are lost. **Absence means "not recorded", never "not done"** — a file whose chain ctdcast could not parse has the verbatim blocks and no ``correction_*`` entries at all. * - ``time_coordinate_source`` - Which clock the ``time`` coordinate is anchored to, and to which moment: ``System UTC, first data scan``, ``NMEA time, header``, and so on. SBE records this in a bracket on its ``start_time`` line and it **varies between cruises**, so a file that does not state it leaves the reader unable to tell GPS time from a possibly-drifting acquisition clock. * - ``time_clock_offset_seconds`` - ``NMEA UTC`` minus ``System UTC``, where the header carries both. Several seconds is normal and the sign varies. It is *reported*, not applied — a single cast cannot distinguish a drifting clock from a stale NMEA sentence; that takes the whole cruise. The unit of record is the **correction**, not the module — which matters because the most consequential one, the deck-unit conductivity alignment, is not a module at all and lives in a different part of the header. Every module in the chain then contributes a correction of its own, so in practice the ledger is one entry per module plus one for the deck unit. Parameters keep the SBE channel names the header uses (``t090C``, ``c0S/m``) rather than being translated to ctdcast's canonical names: the file states what its source stated, and translation happens where the mapping is needed. Sensor provenance ~~~~~~~~~~~~~~~~~ ``profiles.nc`` also records which physical sensor produced each measurement, using three families of variables. The capitalisation and the ``_channel_`` infix are meaningful — keep them distinct: ``SENSOR__`` — upper-case, dimensionless One variable per **distinct physical device** used anywhere in the cruise, e.g. ``SENSOR_TEMPERATURE_5806`` or ``SENSOR_FLUOROMETER_FLNTURTD_3219``. It holds no data; all provenance is in its attributes (``sensor_model``, ``sensor_serial_number``, ``sensor_calibration_date``, ``sensor_maker``, the L05/L22/L35 vocabulary URIs, ``model_source``, and its ``sensor_role`` and ``sensor_channel``). A **frequency** sensor (temperature, conductivity, pressure) also carries ``sensor_calibration_slope`` and ``sensor_calibration_offset`` — the ``datcnv`` drift/span correction already baked into the data before ctdcast read it; a value away from the identity (slope 1, offset 0) means a correction was applied. Every entry (frequency and voltage alike) also carries ``sensor_config_xml`` — the whole ```` block from the CNV ```` header, verbatim: every base calibration coefficient and its type context, so the raw→physical conversion is reconstructable from the compiled file alone, not only from the stage-1 ``raw_metadata``. The serial identifies the device, so a cell used as both primary and secondary of one type is a single entry; ``sensor_shared_with`` cross-links one device serving two roles (e.g. a combined FLNTU as both fluorometer and turbidity). Each measured **data variable** carries a single ``sensor`` attribute naming the ``SENSOR_*`` entry that produced it (e.g. ``ctd_temperature_1``'s ``sensor = "SENSOR_TEMPERATURE_5806"``). That one link is all a variable holds; its role and channel live on the entry, so a device with no stored variable (pH, a transmissometer) still records both. This is what lets the compile aggregate the catalog without re-reading the header. ``sensor_`` — lower-case, dimension ``N_PROF`` Per profile, a **string** naming the ``SENSOR_*`` variable that filled each role — e.g. ``sensor_temperature_1`` may be ``"SENSOR_TEMPERATURE_5806"`` on early casts and ``"SENSOR_TEMPERATURE_4823"`` after a swap. This answers "which sensor's calibration applies to this cast?"; diffing it down the casts gives the sensor-change log. ``sensor_channel_`` — lower-case, dimension ``N_PROF`` Per profile, the **integer** raw acquisition channel that role's sensor was wired into (``-1`` where unused). A change here while ``sensor_`` holds constant is a re-cabling, not a hardware swap. Roles use ctdcast's canonical names: ``temperature_1``/``_2``, ``conductivity_1``/``_2``, ``oxygen_1``/``_2``, ``pressure``, ``fluorometer``, ``turbidity``, ``transmissometer``, ``ph``, ``altimeter``. The universal SensorID → model table ships in ``ctdcast/config/sbe_sensors.yaml``; per-cruise refinements come from the ``sensors:`` block in ``config.yaml`` (see above). The ``SBE sensors`` report page presents all of this as configuration, inventory and rewiring tables. Where it is built, and two kinds of provenance ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The catalog is resolved **once, at stage 1** — the per-cast netCDF files carry their own ``SENSOR_*`` entries — and the compile simply aggregates them; nothing re-reads the SBE header downstream. This splits each entry's attributes into two provenances that mean different things when two casts disagree: - **Header-native** (``sensor_calibration_date``, ``sensor_calibration_slope``, ``sensor_calibration_offset``) are fixed for a serial. A mismatch across casts cannot be a real recalibration at sea, so it is flagged as a parsing or data error. - **Config-resolved** (``sensor_model``, ``sensor_maker``, the vocabulary URIs) come from the SensorID registry and the ``sensors:`` overrides. A mismatch just means the casts were stamped under different config versions — expected, and fixed by re-running stage 1 (or a future ``enrich`` step) to restamp, not by treating it as a data error. Because provenance lives on the file, the per-cast page's **Sensors and calibration state** table reads the catalog directly — one row per device, joined to the variable it produced and showing its model, calibration date and any applied slope/offset — rather than re-parsing the SBE header. A file that predates the catalog falls back to the header parse. A per-cast file that carries **no** ``SENSOR_*`` catalog predates this provenance. The ``build_profiles`` library call still compiles it, with a warning and a ``sensor_catalog`` attribute recording the gap; ``ctdcast process --stage profiles`` and ``ctdcast run`` instead refuse, because re-running stage 1 to build the catalog is the actionable fix before a product is shipped.