Provenance roadmap ================== ctdcast is the first of three packages in one chain: ctdcast processes CTD casts to per-stage netCDF; `caldip `_ compares mooring instruments against a ctdcast CTD file at calibration-dip bottle stops; `oceanarray `_ applies the resulting offsets to mooring instruments. This page records how provenance travels along that chain, what is implemented, and what is planned. Status labels: **done** — in the released or merged code; **in progress** — being built; **planned** — designed and agreed, not yet built. The principle ------------- Two rules shape every decision below. **A file must be interpretable on its own.** Given one output file, a reader should be able to recover which input produced it, what was done to it, and with which parameters — without a directory listing, a configuration file, or a conversation. **Each fact is recorded once, by the package that first observes it, and passed through unchanged.** ctdcast observes the CTD's sensors and identity; caldip observes which of them it used as reference and the comparison itself; oceanarray consumes and records what it applied. A package that re-derives a fact another already recorded is duplicating, not validating — and shared facts keep the same attribute names the whole way down, so no consumer needs a translation layer. File identity and lineage — done -------------------------------- Every file ctdcast writes carries a ``tracking_id`` (UUID4, ACDD/OceanSITES convention), fresh on every write, and from stage 2 onward a ``source_tracking_id`` naming the file it was made from. Stage 1 records ``source_cnv`` instead, since it has no upstream netCDF; the LADCP stage-1 file records ``source_mat``. ``profiles.nc`` carries ``source_tracking_id`` per profile alongside ``source_file``. ``tracking_id`` answers *which instance of this file this is* — it is deliberately **not** a content hash, and it changes even when a re-run produces identical values. Comparing ``tracking_id`` tells you the file was rewritten; comparing the recorded processing parameters tells you whether anything actually moved. A cross-stage test asserts that the sensor catalog, the raw instrument header, the cruise and cast identity and the lineage attributes all survive stage 1 → 2 → 3. It runs on the canonical Linux job, where the CNV reader is installed; it is skipped on the other test-matrix cells, so an operating-system-specific fault in the conversion path is not covered by it. Cross-package discovery — done ------------------------------ ``ctdcast.select_best_available`` is the supported entry point for other packages: it returns, per cast, the best-available stage file and which stage it came from. It exists so downstream packages do not reimplement the stage-3-else-2-else-1 precedence, which would drift the first time a stage is added. Processing state — done ----------------------- Two OceanSITES fields, rather than bespoke ones. ``data_mode`` (reference table 4) says whether a file is the product of record. It is on every file — the per-cast stage files as well as the compiled products — so a consumer that reads a stage file through ``select_best_available`` (caldip does) sees the mode without opening ``profiles.nc``. The default is ``P`` provisional; ``D`` delayed-mode is a claim that all calibrations and quality control have been applied, so it is **declared** in the config (``cruise_info.data_mode: D``), never inferred — and a ``D`` declared with no calibration block warns and names the cast. Quick-look output (``ctdcast draft``) is ``P`` regardless of configuration. ``processing_level`` (reference table 3) says what has been done, **per variable**, using that table's wording. The stage-1 conversion value is stamped on every measured channel — the ones that map to a sensor in the CNV ```` block — and on no computed channel, since salinity, density and the rest were derived from already-converted inputs and were never instrument data. Later procedures append their own values: range-flagging at stages 2 and 3, and calibration at stage 3, which propagates to re-derived salinity. Several values can apply to one variable at once, so they are joined with ``"; "`` in the order applied, as an ordered set — re-running a stage never duplicates a value — which makes the attribute double as the per-variable sequence. No value in that table describes a clock correction, so the time coordinate does not claim one; the shift is recorded in ``history``, in the clock-offset variable and in the retained original time coordinate. Note that ``processing_level`` names a *procedure that was applied*, not a guarantee about the data. "Ranges applied, bad data flagged" means a range test ran and its failures were flagged; the thresholds it used are recorded on the flag variables, which is what makes the scope of the claim checkable. Mixed processing states — done ------------------------------ Compiling casts that sit at different stages is permitted: re-running two casts to check a change should not mean re-processing a cruise. Two kinds of heterogeneity are kept distinct, because stage is not data mode: - **Mixed stages.** Casts compiled from different stages warn loudly, naming every cast and its stage; the per-profile ``source_stage`` already carries the fact. Stages 1, 2 and 3 are all provisional, so this does **not** make the file ``M`` — an all-provisional compile is ``data_mode = "P"``. - **Mixed data modes.** ``data_mode = "M"`` appears only when the casts genuinely differ (a declared ``D`` compiled alongside ``P`` ones). ``M`` then triggers the manual's obligation to say which data is in which mode, satisfied by a per-``N_PROF`` ``source_data_mode`` variable read from each cast's own attribute. Sensor calibration coefficients — planned ----------------------------------------- Stage-1 files preserve the instrument header verbatim, so the base calibration coefficients are present at stage 1 but do not currently reach the compiled product. Carrying each sensor's complete configuration block onto its catalog entry makes the raw-to-physical conversion reconstructable from the compiled file alone. Known limits ------------ **Stage 3 can apply only a conductivity slope.** There is no conductivity offset, no temperature correction and no pressure correction. A downstream package's temperature reference therefore cannot be corrected by ctdcast as it stands, whichever stage it reads. Any check that enumerates ctdcast's correction fields is complete only until a second correction is added. **Calibration-dip integration is not yet implemented in either direction.** caldip's machine-readable output is moving to one netCDF per cast, with the statistics CSV retained as a derived export for cruise reports; oceanarray then applies those corrections at the front of its stage 3. Until that lands, corrections determined from calibration dips are not applied automatically anywhere in the chain.