# Outputs

Running caldip on a cast produces an **interactive plot** and **per-cast statistics**. The statistics are written as a **netCDF file** (`{cast}_caldip.nc`, the machine-readable output) with a **CSV export** derived from it for reading and for the report tables. A third command, `caldip report`, gathers those per-cast outputs for a whole cruise into a browsable HTML report (see below and [Running caldip](running.md)).

---

## Interactive plot

Generated by `caldip plot`, the plot is a self-contained HTML file that can be opened in any browser — no internet connection required. This makes it suitable for use at sea.

```bash
caldip plot data/proc_calib/msm142_2026/cal_dip/castM4/castM4.caldip.yaml \
    --output castM4 -o outputs/
```

The plot shows time series of pressure, temperature, and conductivity for all instruments alongside the CTD reference, with bottle stop periods highlighted.

### Zooming in on a bottle stop

The three panels (pressure, temperature, conductivity) are synchronised on the time axis — zooming one zooms all. To inspect instrument agreement within a single bottle stop:

1. Find the bottle stop in the **pressure panel** — it appears as a flat segment flanked by blue (start) and red (end) vertical lines.
2. Click and drag across that flat segment in the pressure panel to zoom in.
3. Check that instrument temperatures (and conductivities, if available) track the CTD reference (black) within that window.
4. Double-click anywhere to zoom back out.

Starting with the pressure panel confirms the CTD was genuinely stationary before you assess the sensor offsets in the other panels. It also helps identify if you have clock offset problems that need to be fixed before calculating comparison stats.

**<a href="_static/castM4_plot.html" target="_blank">Open example plot (castM4, MSM142)</a>**

---

## Statistics

Generated by `caldip stats`:

```bash
caldip stats data/proc_calib/msm142_2026/cal_dip/castM4/castM4.caldip.yaml \
    --ctd-sensor 2 -o outputs/
```

This writes, per cast: `{cast}_caldip.nc` (the machine-readable statistics), `{cast}_detailed_statistics.csv` and `{cast}_summary_statistics.csv` (CSV exports), and a `{cast}_timing.txt` timing report with bottle stop start/end times and pressures.

### Machine-readable statistics (`{cast}_caldip.nc`)

The netCDF is the primary statistics output and the file downstream tools (e.g. oceanarray) read. It is a two-dimensional **`(instrument, stop)` grid** — cal-dip data is rectangular, since every instrument is on the same wire for the whole cast and so sees every bottle stop. Each variable declares which axis it varies along; an instrument that missed a stop is a `NaN` cell with a `no_data` flag. **Stop order is not meaningful — select stops by value (e.g. deepest = `argmax(ctd_press)`), never by position.** Every physical variable carries `units`; every difference states its sign convention in a `comment`; absent values are `NaN` (`_FillValue`), never `0`.

| Variable | Varies along | Type | Notes |
|----------|--------------|------|-------|
| `serial` (coord) | `instrument` | string | Instrument serial number |
| `instrument_type`, `label` | `instrument` | string | Instrument identity |
| `bl_press` (coord) | `stop` | int16 | Nominal integer stop label — the measured pressure is `ctd_press` |
| `time` (coord), `time_start`, `time_end` | `stop` | datetime | Comparison-window mid-time and bounds |
| `ctd_temp`, `ctd_cond`, `ctd_press` | `stop` | float64 | CTD reference means over the comparison window |
| `temp_diff`, `cond_diff`, `press_diff` | `(instrument, stop)` | float64 | Offset `instrument − CTD`; `comment` = `"instrument minus CTD; corrected = measured - diff"` |
| `temp_std`, `cond_std`, `press_std` | `(instrument, stop)` | float64 | Within-window standard deviation |
| `inst_temp`, `inst_cond`, `inst_press` | `(instrument, stop)` | float64 | Instrument means over the comparison window |
| `N` | `(instrument, stop)` | int32 | Instrument samples in the comparison window |
| `temp_flag`, `cond_flag`, `press_flag` | `(instrument, stop)` | int8 | CF/QARTOD flag: `1` ok, `2` no_data, `3` flagged, `4` missing, `9` unknown; each carries the `flagging_threshold` used |

Cast-level facts are **global attributes**, so a per-cast value is recorded once rather than repeated on every row: identity and provenance (`cast_id`, `cruise`, `schema_version`, `caldip_version`, `tracking_id`, `date_created`/`date_modified`), processing state (`data_mode` = `P` on the raw-CNV path, `data_mode_meaning`, `qc_flags_honoured`, `qc_masked_flag_values`, `preferred_pair`, `input_mode`), lineage (`source_tracking_id`, `source_instrument_files`, `config_digest`), and the CTD reference block (`ctd_path`, `ctd_sensor_used`, `ctd_conductivity_slope`, `ctd_{temp,cond}_sensor_serial`, `ctd_{temp,cond,press}_processing_level`, …). `config_digest` is a hash over the result-affecting config subset (instrument list and clock offsets); `preferred_pair` is the reference's declared sensor pair (`undeclared` until ctdcast writes it). Fields caldip cannot yet source are written as `UNK` with a warning, never guessed. `caldip report --check` and the `caldip inspect` inventory read these back to report whether each cast's CTD reference has finished.

To view a netCDF's contents in a browser — dimensions, variables (with types and units) and every global attribute — `caldip inspect` writes a styled inventory page beside it (a viewable counterpart to `ncdump -h`):

```bash
caldip inspect outputs/castM4_caldip.nc      # writes outputs/castM4_caldip_inventory.html
```

### Summary statistics (`{cast}_summary_statistics.csv`)

One row per instrument, taken from the **deepest bottle stop** of the cast.
(This is a single-stop snapshot, not an average across all stops — use the detailed CSV to see all stops.)

| Column | Description |
|--------|-------------|
| `serial` | Instrument serial number |
| `instrument_type` | Instrument class from the controlled vocabulary (`microcat`, `tr1050`, `rbrsolo`, etc.) |
| `label` | Human-readable model label |
| `n_samples` | Number of instrument data points in the 2-minute comparison window |
| `bl_press` | Pressure of the deepest bottle stop (dbar) |
| `temp_diff_mean` | Temperature offset vs CTD at the deepest stop (°C) |
| `temp_diff_std` | Standard deviation of temperature within the comparison window (°C) |
| `cond_diff_mean` | Conductivity offset vs CTD at the deepest stop (mS/cm) |
| `cond_diff_std` | Standard deviation of conductivity within the comparison window (mS/cm) |
| `press_diff_mean` | Pressure offset vs CTD at the deepest stop (dbar) |
| `press_diff_std` | Standard deviation of pressure within the comparison window (dbar) |

RBR thermistors (temperature-only) leave the conductivity and pressure columns empty.

**<a href="_static/castM4_summary_statistics.csv">Download example summary CSV (castM4, MSM142)</a>**

### Detailed statistics (`{cast}_detailed_statistics.csv`)

A CSV **export of the netCDF**, for reading and for the report tables — not a separate source of truth. One row per instrument per bottle stop. The `*_status` prose is rendered from the netCDF's difference and flag threshold at export time (so it never disagrees with the flag), and absent values are **empty cells** (the netCDF holds `NaN`). Values are rounded per variable for reading — temperature and conductivity to 4 decimal places, pressure to 0.1 dbar, and each standard deviation one place finer than its value; the netCDF itself keeps full precision. Columns:

| Column | Description |
|--------|-------------|
| `serial` | Instrument serial number |
| `instrument_type` | Instrument family |
| `bl_press` | Bottle stop pressure (dbar), rounded |
| `temp_diff` | Temperature offset: instrument − CTD (°C) |
| `temp_std` | Standard deviation of instrument temperature in comparison window (°C) |
| `cond_diff` | Conductivity offset: instrument − CTD (mS/cm); empty for thermistors |
| `cond_std` | Standard deviation of instrument conductivity (mS/cm) |
| `press_diff` | Pressure offset: instrument − CTD (dbar); empty if no pressure sensor |
| `press_std` | Standard deviation of instrument pressure (dbar) |
| `temp_status` | Quality flag string, e.g. `T OK`, `T reads high by 0.012` |
| `cond_status` | Quality flag string for conductivity |
| `press_status` | Quality flag string for pressure |
| `date` | Date of the comparison period (UTC) |
| `time_start` | Start time of the 2-minute comparison window |
| `time_end` | End time of the 2-minute comparison window |
| `ctd_temp` | CTD reference temperature during comparison window (°C) |
| `ctd_cond` | CTD reference conductivity during comparison window (mS/cm) |
| `inst_temp` | Instrument mean temperature during comparison window (°C) |
| `inst_cond` | Instrument mean conductivity during comparison window (mS/cm) |
| `inst_press` | Instrument mean pressure during comparison window (dbar) |
| `N` | Number of instrument data points in the comparison window |
| `label` | Human-readable instrument label |
| `ctd_sensor_used` | CTD sensor number used for the reference (1 or 2) |
| `cast_id` | Cast identity, repeated so a stray CSV names itself |
| `schema_version` | Output schema version |
| `tracking_id` | The netCDF's `tracking_id`, so the CSV traces back to the run that made it |

### Quality flag thresholds

The `temp_status`, `cond_status`, and `press_status` prose (and the netCDF `temp_flag` / `cond_flag` / `press_flag` variables) compare the instrument offset against these thresholds:

| Variable | Default threshold | Flag triggered when |
|----------|-------------------|---------------------|
| Temperature | ±0.005 °C | `\|instrument − CTD\| > 0.005 °C` |
| Conductivity | ±0.02 mS/cm | `\|instrument − CTD\| > 0.02 mS/cm` |
| Pressure | ±5 dbar | `\|instrument − CTD\| > 5 dbar` |

These thresholds are appropriate for SBE37 MicroCATs. For RBR thermistors or other instruments with wider specifications, you can override them in the YAML:

```yaml
quality_flags:
  temp_threshold: 0.01   # °C
  cond_threshold: 0.05   # mS/cm
  press_threshold: 10.0  # dbar
```

---

## Output file location

By default, output files are written to the **parent of the cast directory** (i.e., the `cal_dip/` level), not to the YAML directory. The command prints the exact paths when it runs. Use `-o` (or `--output-dir`) to specify a different location:

```bash
caldip stats castM4/castM4.caldip.yaml -o results/
```

---

## Batch processing

`scripts/generate_all_caldip_plots.sh` runs both commands across all casts in a cruise:

```bash
bash scripts/generate_all_caldip_plots.sh
```

The script uses `caldip plot` and `caldip stats`. Edit the `CTD_SENSOR` variable at the top to switch between primary (1) and secondary (2) sensors for the whole cruise in one step.

---

## Per-cruise report

Once a cruise's casts have their netCDF, CSVs and plots, `caldip report <dir>` builds a self-contained HTML report from them — an index of casts and one page per cast with the interactive figure, a bottle-stops table, the deepest-stop summary and the per-stop detail. It takes the usability flags and the cruise name from each `{cast}_caldip.nc` (a cast without its netCDF is reported with zero flags and an `UNK` cruise, with a warning). It reads only the files already on disk; see [Running caldip](running.md) for details and options.

```bash
caldip report data/proc_calib/msm142_2026/cal_dip/
```