seagliderOG1 demo

The purpose of this notebook is to demonstrate the functionality of seagliderOG1 to convert from Seaglider basestation files to OG1 format.

OG1 format is a newly agreed format (since June 2024) for glider data sets from various platforms (e.g., Seaglider, Slocum, Seaexplorer). It lives on github here: (https://github.com/OceanGlidersCommunity/OG-format-user-manual).
OG1 manual: https://oceangliderscommunity.github.io/OG-format-user-manual/OG_Format.html

The test case is to convert sg015 data from the Labrador Sea in September 2004.

The demo is organised to show

Conversion of a single dive cycle (single p*.nc file)
Conversion for a folder of local dive-cycle files (full mission of p*.nc files)
Download from remote server + conversion (directory with full mission of p*.nc files)

Options are provided to only load e.g. 10 files, but note that OG1 format expects a full mission.

[1]:

import pathlib
import sys

script_dir = pathlib.Path().parent.absolute()
parent_dir = script_dir.parents[0]
sys.path.append(str(parent_dir))
sys.path.append(str(parent_dir) + '/seagliderOG1')
print(parent_dir)
print(sys.path)
### silence future warnings
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

import xarray as xr
import os
from seagliderOG1 import readers, writers, plotters
from seagliderOG1 import convertOG1, vocabularies

/home/runner/work/seagliderOG1/seagliderOG1
['/home/runner/micromamba/envs/TEST/lib/python313.zip', '/home/runner/micromamba/envs/TEST/lib/python3.13', '/home/runner/micromamba/envs/TEST/lib/python3.13/lib-dynload', '', '/home/runner/micromamba/envs/TEST/lib/python3.13/site-packages', '/home/runner/work/seagliderOG1/seagliderOG1', '/home/runner/work/seagliderOG1/seagliderOG1/seagliderOG1']

[2]:

# Specify the path for writing datafiles
data_path = os.path.join(parent_dir, 'data')

Convert to OG1 format

Process:

For one basestation dataset, split the dataset by dimension (split_ds)
Transform into OG1 format: dataset with dims sg_data_point
- Change the dimension to N_MEASUREMENTS
- Rename variables according to vocabularies.standard_names
- Assign variable attributes according to vocabularies.vocab_attrs. (Note: This could go wrong since it makes assumptions about the input variables. May need additional handling.)
Add missing mandatory variables:
- From split_ds[(gps_info,)], add the LATITUDE_GPS, LONGITUDE_GPS and TIME_GPS (Note: presently TIME_GPS is stripped before saving, but TIME values contain TIME_GPS)
- Create PROFILE_NUMBER and PHASE
- Calculate DEPTH_Z which is positive up
Update attributes for the file.
- Combines creator and contributor from original attributes into contributor
- Adds contributing_institutions based on institution
- Reformats time in time_coverage_* and start_time–> start_date
- Adds date_modified
- Renames comments–>history, site–>summary
- Adds title, platform, platform_vocabulary, featureType, Conventions, rtqc_method* according to OceanGliders format
- Retains naming_authority, institution, project, geospatial_* as OG attributes
- Retains extra attributes: license, keywords, keywords_vocabulary, file_version, acknowledgement, date_created, disclaimer

Future behaviour to be added:

Retain the variables starting with sg_cal and check whether they vary over the mission (shouldn’t)
Add sensors, using information in the split_ds with no dimensions
- Need (from sg_cal_constants: sg_cal plus volmax, vbd_cnts_per_cc, therm_expan, t_*, mass, hd_*, ctcor, cpcor, c_*, abs_compress, a, Tcor, Soc, Pcor, Foffset)
- Maybe also reviewed, magnetic_variation (which will change with position), log_D_FLARE, flight_avg_speed_north and flight_avg_speed_east also with _gsm, depth_avg_curr_north and depth_avg_curr_east also with _gsm, wlbb2f - means sensor sg_cal_mission_title sg_cal_id_str calibcomm_oxygen calibcomm sbe41 means ?? hdm_qc glider

Convert a single (sample) dataset

[7]:

# Loads one dataset (p0150500_20050213.nc)
ds = readers.load_sample_dataset()

ds_OG1, var_list = convertOG1.convert_to_OG1(ds)

# Check the results - uncomment the following lines to either generate a plot or show the variables.
plotters.plot_profile_depth(ds_OG1)

Processing datasets:   0%|          | 0/1 [00:00<?, ?dataset/s]Variable 'eng_depth' not in OG1 vocabulary.
Variable 'eng_aa4330_Temp' not in OG1 vocabulary.
Variable 'longitude_gsm' not in OG1 vocabulary.
Variable 'speed_gsm' not in OG1 vocabulary.
Variable 'vert_speed_gsm' not in OG1 vocabulary.
Variable 'eng_aa4330_TCPhase' not in OG1 vocabulary.
Variable 'eng_sbect_tempFreq' not in OG1 vocabulary.
Variable 'eng_sbect_condFreq' not in OG1 vocabulary.
Variable 'eng_rec' not in OG1 vocabulary.
Variable 'north_displacement_gsm' not in OG1 vocabulary.
Variable 'horz_speed_gsm' not in OG1 vocabulary.
No conversion information found for micromoles/kg to micromoles/kg
Variable 'eng_aa4330_O2' not in OG1 vocabulary.
Variable 'eng_aa4330_AirSat' not in OG1 vocabulary.
Variable 'latitude_gsm' not in OG1 vocabulary.
Variable 'aanderaa4330_instrument_dissolved_oxygen' not in OG1 vocabulary.
Variable 'eng_elaps_t_0000' not in OG1 vocabulary.
Variable 'buoyancy' not in OG1 vocabulary.
Variable 'east_displacement_gsm' not in OG1 vocabulary.
Variable 'sound_velocity' not in OG1 vocabulary.
Variable 'density_insitu' not in OG1 vocabulary.
Variable 'density' not in OG1 vocabulary.
Variable 'eng_aa4330_CalPhase' not in OG1 vocabulary.
Variable 'eng_GC_phase' not in OG1 vocabulary.
No conversion information found for cm s-1 to cm s-1
Variable 'conservative_temperature' not in OG1 vocabulary.
Variable 'glide_angle_gsm' not in OG1 vocabulary.
Variable 'eng_elaps_t' not in OG1 vocabulary.
Variable 'absolute_salinity' not in OG1 vocabulary.
No conversion information found for micromoles/kg to micromoles/kg
Variable 'ctd_pressure' not in OG1 vocabulary.
Variable 'depth' not in OG1 vocabulary.
Processing datasets: 100%|██████████| 1/1 [00:00<00:00,  1.49dataset/s]

[8]:

### print the list of inital variables of the dataset
var_list

[8]:

['gc_st_secs',
 'eng_aa4330_Temp',
 'log_D_FLARE',
 'sg_cal_therm_expan',
 'depth_avg_curr_east_gsm',
 'log_RAFOS_CORR_THRESH',
 'flight_avg_speed_north_gsm',
 'eng_head',
 'log_SIM_PITCH',
 'log_MISSION',
 'log_SENSORS',
 'log_SPEED_FACTOR',
 'sg_cal_mass_comp',
 'temperature_raw_qc',
 'log_FG_AHR_24Vo',
 'latlong_qc',
 'log_HD_C',
 'log_COMM_SEQ',
 'glide_angle_gsm',
 'log_PHONE_SUPPLY',
 'log_PITCH_MIN',
 'gc_end_secs',
 'sg_cal_optode_TempCoef2',
 'sg_cal_optode_FoilCoefA5',
 'log_TCM_TEMP',
 'gc_gcphase',
 'sg_cal_optode_FoilCoefA12',
 'glide_angle',
 'log_D_NO_BLEED',
 'log_ICE_FREEZE_MARGIN',
 'log_D_SURF',
 'sg_cal_optode_TempCoef1',
 'eng_rollCtl',
 'log_COMPASS2_DEVICE',
 'sg_cal_id_str',
 'log_N_NOSURFACE',
 'sg_cal_abs_compress',
 'speed_gsm',
 'sg_cal_optode_FoilCoefB12',
 'log_CAPMAXSIZE',
 'log__XMS_NAKs',
 'log_SEABIRD_C_H',
 'temperature',
 'temperature_qc',
 'log_PITCH_TIMEOUT',
 'log_TCM_PITCH_OFFSET',
 'log_TT8_MAMPS',
 'log_PITCH_ADJ_GAIN',
 'sg_cal_optode_FoilCoefA8',
 'north_displacement_gsm',
 'log_N_GPS',
 'log_CFSIZE',
 'sg_cal_optode_FoilCoefB11',
 'gc_roll_secs',
 'log_DATA_FILE_SIZE',
 'log_SM_GC',
 'log_STROBE',
 'gc_pitch_secs',
 'sg_cal_optode_FoilCoefA0',
 'eng_rollAng',
 'log_ESCAPE_HEADING_DELTA',
 'log_LOGGERDEVICE4',
 'sg_cal_temp_ref',
 'sg_cal_sg_configuration',
 'log_INTERNAL_PRESSURE',
 'vert_speed',
 'log_APOGEE_PITCH',
 'log_RAFOS_PEAK_OFFSET',
 'log_FG_AHR_10Vo',
 'log_C_PITCH',
 'eng_elaps_t',
 'log_GPS_DEVICE',
 'ctd_time',
 'vert_speed_gsm',
 'temperature_raw',
 'GPS2_qc',
 'log_FG_AHR_24V',
 'gc_roll_errors',
 'eng_vbdCC',
 'depth_avg_curr_error',
 'depth_avg_curr_north',
 'sg_cal_optode_FoilCoefB2',
 'gc_state_state',
 'log_FERRY_MAX',
 'surface_curr_east',
 'sg_cal_optode_FoilCoefB13',
 'sg_cal_optode_TempCoef3',
 'log_NAV_MODE',
 'sg_cal_optode_TempCoef5',
 'sg_cal_optode_FoilCoefB8',
 'log_SENSOR_MAMPS',
 'log_ALTIM_BOTTOM_TURN_MARGIN',
 'log_T_RSLEEP',
 'sg_cal_optode_PhaseCoef3',
 'eng_aa4330_AirSat',
 'log_T_WATCHDOG',
 'log_T_NO_W',
 'log_SEABIRD_C_G',
 'sg_cal_optode_PhaseCoef2',
 'log_DEEPGLIDER',
 'sg_cal_c_g',
 'log_R_STBD_OVSHOOT',
 'log_INT_PRESSURE_SLOPE',
 'log_T_BOOST',
 'log_SURFACE_URGENCY_FORCE',
 'sg_cal_optode_FoilCoefB7',
 'log_ROLL_ADJ_DBAND',
 'speed',
 'aanderaa4330_results_time',
 'log_TGT_AUTO_DEFAULT',
 'log_PHONE_DEVICE',
 'sg_cal_optode_FoilCoefB6',
 'eng_sbect_tempFreq',
 'log_KALMAN_Y',
 'sigma_t',
 'log_SM_CCo',
 'log_VBD_TIMEOUT',
 'log__SM_DEPTHo',
 'gc_pitch_i',
 'log_MINV_10V',
 'log_T_TURN',
 'log_VBD_PUMP_AD_RATE_SURFACE',
 'log_ALTIM_PULSE',
 'log_ESCAPE_HEADING',
 'log_MINV_24V',
 'flight_avg_speed_east',
 'sg_cal_optode_FoilCoefB9',
 'log_SEABIRD_T_I',
 'flight_avg_speed_north',
 'log_CALL_TRIES',
 'log_ROLL_MAX',
 'latitude_gsm',
 'log_AH0_24V',
 'log_ERRORS',
 'log_CAPUPLOAD',
 'log_SPEED_LIMITS',
 'hdm_qc',
 'log_TGT_RADIUS',
 'eng_sbect_condFreq',
 'conductivity_raw',
 'salinity_raw_qc',
 'GPS1_qc',
 'log_VBD_BLEED_AD_RATE',
 'log_XPDR_VALID',
 'reviewed',
 'sound_velocity',
 'log_24V_AH',
 'eng_pitchAng',
 'log_IRIDIUM_FIX',
 'dissolved_oxygen_sat',
 'log_D_BOOST',
 'depth_avg_curr_qc',
 'log_SEABIRD_C_J',
 'log_KALMAN_X',
 'gc_pitch_ad',
 'log_T_GPS',
 'log_GPS',
 'log_DEVICE4',
 'log_ROLL_MAXERRORS',
 'log_COMPASS_USE',
 'surface_curr_error',
 'sg_cal_optode_FoilCoefB10',
 'sg_cal_c_i',
 'sg_cal_optode_FoilCoefB4',
 'log_SENSOR_SECS',
 'sg_cal_c_h',
 'log_D_SAFE',
 'log_KALMAN_CONTROL',
 'log_RHO',
 'log_FILEMGR',
 'latitude',
 'gc_vbd_retries',
 'log_SEABIRD_T_G',
 'log_PITCH_GAIN',
 'log_MEM',
 'conservative_temperature',
 'log_ALTIM_PING_DELTA',
 'gc_pitch_retries',
 'sg_cal_t_g',
 'log_T_TURN_SAMPINT',
 'buoyancy',
 'log_ALTIM_FREQUENCY',
 'absolute_salinity',
 'salinity',
 'sg_cal_mass',
 'sg_cal_optode_FoilCoefB5',
 'gc_vbd_errors',
 'time',
 'log_USE_ICE',
 'log_TGT_DEFAULT_LON',
 'log_PITCH_AD_RATE',
 'log_C_ROLL_DIVE',
 'log_RELAUNCH',
 'log_T_MISSION',
 'log_TGT_LATLONG',
 'log_SIM_W',
 'log_DEVICES',
 'log_ALTIM_PING_DEPTH',
 'log__CALLS',
 'log_T_LOITER',
 'log_gps_lon',
 'start_of_climb_time',
 'log_AD7714Ch0Gain',
 'log_SURFACE_URGENCY_TRY',
 'log_VBD_CNV',
 'log_HD_A',
 'sg_cal_rho0',
 'eng_aa4330_CalPhase',
 'log_N_NOCOMM',
 'log_MASS',
 'log_PITCH_VBD_SHIFT',
 'magnetic_variation',
 'sg_cal_optode_FoilCoefA6',
 'sg_cal_pitchbias',
 'sg_cal_hd_b',
 'log_SURFACE_URGENCY',
 'sg_cal_optode_FoilCoefA13',
 'log_TCM_ROLL_OFFSET',
 'gc_state_eop_code',
 'log_HEADING',
 'log_COMPASS_DEVICE',
 'log_D_OFFGRID',
 'log_gps_time',
 'sg_cal_optode_FoilCoefA10',
 'log_ALTIM_SENSITIVITY',
 'log_DEVICE5',
 'log_ALTIM_TOP_PING_RANGE',
 'log_C_ROLL_CLIMB',
 'eng_elaps_t_0000',
 'log_PITCH_MAX',
 'log_DIVE',
 'gc_state_secs',
 'log_N_FILEKB',
 'log_R_PORT_OVSHOOT',
 'longitude_gsm',
 'depth_avg_curr_north_gsm',
 'ctd_depth',
 'log_ROLL_TIMEOUT',
 'sg_cal_mission_title',
 'log_GPS1',
 'log_ROLL_ADJ_GAIN',
 'conductivity_qc',
 'gc_vbd_secs',
 'log_D_PITCH',
 'conductivity_raw_qc',
 'sg_cal_optode_TempCoef0',
 'log_T_ABORT',
 'log_INT_PRESSURE_YINT',
 'gc_vbd_ctl',
 'log_gps_lat',
 'log_PITCH_MAXERRORS',
 'sg_cal_optode_SVU_enabled',
 'east_displacement',
 'log_ROLL_AD_RATE',
 'sg_cal_cpcor',
 'log_CAP_FILE_SIZE',
 'CTD_qc',
 'gc_roll_ad',
 'eng_rec',
 'theta',
 'sg_cal_hd_s',
 'log_HD_B',
 'log_FIX_MISSING_TIMEOUT',
 'sg_cal_optode_FoilCoefA9',
 'sg_cal_optode_PhaseCoef1',
 'log_MHEAD_RNG_PITCHd_Wd',
 'aanderaa4330_instrument_dissolved_oxygen',
 'log_T_GPS_CHARGE',
 'log_USE_BATHY',
 'log_ALTIM_TOP_PING',
 'log_UNCOM_BLEED',
 'salinity_raw',
 'log_D_GRID',
 'log_RAFOS_DEVICE',
 'log_TGT_DEFAULT_LAT',
 'gc_pitch_errors',
 'log_ALTIM_BOTTOM_PING_RANGE',
 'pressure',
 'log_CALL_WAIT',
 'sigma_theta',
 'sg_cal_ctcor',
 'log__XMS_TOUTs',
 'log_SEABIRD_T_H',
 'log_VBD_MAX',
 'log_P_OVSHOOT',
 'gc_roll_i',
 'salinity_qc',
 'eng_depth',
 'horz_speed',
 'log_SM_CC',
 'log_HEAPDBG',
 'log__SM_ANGLEo',
 'sg_cal_hd_c',
 'surface_curr_north',
 'log_DEVICE_SECS',
 'log_D_ABORT',
 'log_XPDR_INHIBIT',
 'log_VBD_MIN',
 'log_LOGGERDEVICE3',
 'depth_avg_curr_east',
 'log_ROLL_DEG',
 'log_DEVICE6',
 'horz_speed_gsm',
 'sg_cal_vbd_cnts_per_cc',
 'log_CALL_NDIVES',
 'ctd_pressure',
 'log_XPDR_PINGS',
 'eng_pitchCtl',
 'east_displacement_gsm',
 'log_MOTHERBOARD',
 'aanderaa4330_qc',
 'flight_avg_speed_east_gsm',
 'gc_ob_vertv',
 'sg_cal_optode_FoilCoefB3',
 'log_T_DIVE',
 'log_UPLOAD_DIVES_MAX',
 'log_C_VBD',
 'longitude',
 'eng_aa4330_TCPhase',
 'aanderaa4330_dissolved_oxygen_qc',
 'log_XPDR_DEVICE',
 'log_DEVICE_MAMPS',
 'sg_cal_optode_PhaseCoef0',
 'log_T_GPS_ALMANAC',
 'log_LOGGERDEVICE1',
 'log_ALTIM_TOP_TURN_MARGIN',
 'eng_aa4330_O2',
 'density',
 'log_VBD_PUMP_AD_RATE_APOGEE',
 'gc_pitch_ctl',
 'log_DEVICE2',
 'trajectory',
 'sg_cal_optode_FoilCoefA11',
 'speed_qc',
 'log_TGT_NAME',
 'sg_cal_optode_FoilCoefA4',
 'sg_cal_t_i',
 'conductivity',
 'log_KERMIT',
 'log_ROLL_CNV',
 'GPSE_qc',
 'north_displacement',
 'log_KALMAN_USE',
 'sg_cal_calibcomm',
 'sg_cal_t_j',
 'log_DEVICE3',
 'log_DEEPGLIDERMB',
 'log_COURSE_BIAS',
 'log_LOGGERS',
 'log_AH0_10V',
 'log_ROLL_MIN',
 'sg_cal_calibcomm_optode',
 'gc_roll_retries',
 'log_HEAD_ERRBAND',
 'log_DEVICE1',
 'log_SEABIRD_C_I',
 'aa4330',
 'surface_curr_qc',
 'log_ALTIM_TOP_MIN_OBSTACLE',
 'glider',
 'gc_vbd_i',
 'log_LOGGERDEVICE2',
 'gc_depth',
 'log_ID',
 'log_MAX_BUOY',
 'log_PRESSURE_YINT',
 'density_insitu',
 'log_D_CALL',
 'eng_GC_phase',
 'log_GLIDE_SLOPE',
 'avg_latitude',
 'sg_cal_t_h',
 'log_GPS2',
 'log_PITCH_DBAND',
 'log_SEABIRD_T_J',
 'sg_cal_c_j',
 'sg_cal_optode_FoilCoefA3',
 'log_CF8_MAXERRORS',
 'aanderaa4330_dissolved_oxygen',
 'log_PITCH_CNV',
 'gc_data_pts',
 'sg_cal_hd_a',
 'depth',
 'sg_cal_optode_FoilCoefA2',
 'sg_cal_optode_FoilCoefA1',
 'sg_cal_volmax',
 'gc_vbd_ad',
 'log_RAFOS_HIT_WINDOW',
 'log_FG_AHR_10V',
 'log_D_TGT',
 'sg_cal_optode_FoilCoefA7',
 'log_10V_AH',
 'sg_cal_optode_FoilCoefB0',
 'log_D_FINISH',
 'sg_cal_optode_FoilCoefB1',
 'sg_cal_optode_TempCoef4',
 'log_PRESSURE_SLOPE',
 'log_VBD_MAXERRORS',
 'log_HUMID',
 'sbe41',
 'log_VBD_DBAND',
 'log_PITCH_ADJ_DBAND']

[9]:

# Print to screen a table of attributes
plotters.show_contents(ds_OG1,'attrs')

information is based on xarray Dataset

[9]:

	Attribute	Value	DType
0	title	OceanGliders trajectory file	str
1	id	sg033_20100906T112156_delayed	str
2	platform	sub-surface gliders	str
3	platform_vocabulary	https://vocab.nerc.ac.uk/collection/L06/curren...	str
4	naming_authority	edu.washington.apl	str
5	institution	School of Oceanography\nUniversity of Washingt...	str
6	geospatial_lat_min	18.516025675252838	ndarray
7	geospatial_lat_max	18.524446699350996	ndarray
8	geospatial_lon_min	-65.98221600599415	ndarray
9	geospatial_lon_max	-65.97450004341222	ndarray
10	geospatial_vertical_min	0.0	ndarray
11	geospatial_vertical_max	511.71017477249814	ndarray
12	time_coverage_start	20100906T111037	str
13	time_coverage_end	20100906T142637	str
14	site	Test deployments of DeepGliders in the Puerto ...	str
15	project	Rapid-Mocha San Juan 3 September 2010	str
16	contributor_name	Charlie Eriksen	str
17	contributor_role	PI	str
18	contributor_role_vocabulary	http://vocab.nerc.ac.uk/search_nvs/W08	str
19	contributor_email	charlie@ocean.washington.edu	str
20	contributing_institutions	University of Washington - School of Oceanography	str
21	contributing_institutions_vocabulary	https://edmo.seadatanet.org/report/1434	str
22	contributing_institutions_role	PI	str
23	contributing_institutions_role_vocabulary	http://vocab.nerc.ac.uk/collection/W08/current/	str
24	uri	569b1f80-1985-11e7-8a51-0026bb609360	str
25	rtqc_method	No QC applied	str
26	rtqc_method_doi	n/a	str
27	comment	Processing start:\nINFO: QC.py(418): Changed (...	str
28	start_date	20100906T112156	str
29	date_created	20170404T220433	str
30	featureType	trajectoryProfile	str
31	Conventions	CF-1.10,OG-1.0	str
32	date_modified	20250921T144511	str
33	disclaimer	Data has not been reviewed and is provided AS-IS.	str
34	file_version	2.71	str
35	keywords	Water Temperature, Conductivity, Salinity, Den...	str
36	keywords_vocabulary	NASA/GCMD Earth Science Keywords Version 6.0.0.0	str
37	license	These data may be redistributed and used witho...	str
38	acknowledgment	National Science Foundation OCE Division Grant...	str
39	contributer_email	null@null.com	str

[10]:

# Print to screen a table of the variables and variable attributes
plotters.show_contents(ds_OG1,'variables')

information is based on xarray Dataset

[10]:

	dims	units	comment	standard_name	dtype
name
CNDC	N_MEASUREMENTS	S/m	Conductivity corrected for anomalies	sea_water_electrical_conductivity	float32
CNDC_QC	N_MEASUREMENTS		Whether to trust each corrected conductivity value	status_flag	float32
CNDC_RAW	N_MEASUREMENTS	S/m	Uncorrected conductivity		float32
CNDC_RAW_QC	N_MEASUREMENTS		Whether to trust each raw conductivity value	status_flag	float32
DAVG_CURR_EAST	N_MEASUREMENTS	m/s	Eastward component of the dive depth-average current based on hdm	eastward_sea_water_velocity	float32
DAVG_CURR_NORTH	N_MEASUREMENTS	m/s	Northward component of the dive depth-average current based on hdm	northward_sea_water_velocity	float32
DEPTH	N_MEASUREMENTS	m	from science pressure and interpolated	depth	float64
DEPTH_Z	N_MEASUREMENTS	meters	Depth calculated from pressure using gsw library, positive up.	depth	float64
DOXY	N_MEASUREMENTS	micromoles/kg	Dissolved oxygen concentration, calculated from optode tcphase corrected for salininty and depth	mole_concentration_of_dissolved_molecular_oxygen_in_sea_water	float32
DOXY_QC	N_MEASUREMENTS		Whether to trust each optode dissolved oxygen value	status_flag	float32
EAST_DISPLACEMENT	N_MEASUREMENTS	meters	Eastward displacement from hdm		float32
GLIDER_HORZ_VELO_MODEL	N_MEASUREMENTS	cm/s	Vehicle horizontal speed based on hdm		float32
GLIDER_VERT_VELO_MODEL	N_MEASUREMENTS	cm/s	Vehicle vertical speed based on hdm		float32
GLIDE_ANGLE	N_MEASUREMENTS	cm/s	Glide angle based on hdm		float32
GLIDE_SPEED	N_MEASUREMENTS	cm/s	Vehicle speed based on hdm		float32
GLIDE_SPEED_QC	N_MEASUREMENTS		Whether to trust each hdm speed value	status_flag	float32
HEADING	N_MEASUREMENTS	degrees	Vehicle heading (magnetic)		float32
LATITUDE	N_MEASUREMENTS	degrees_north	Latitude of the sample based on hdm DAC	latitude	float64
LATITUDE_GPS	N_MEASUREMENTS	degrees_north		latitude	float64
LONGITUDE	N_MEASUREMENTS	degrees_east	Longitude of the sample based on hdm DAC	longitude	float64
LONGITUDE_GPS	N_MEASUREMENTS	degrees_east		longitude	float64
NORTH_DISPLACEMENT	N_MEASUREMENTS	meters	Northward displacement from hdm		float32
OXYSAT	N_MEASUREMENTS	micromoles/kg	Calculated saturation value for oxygen given measured presure and corrected temperature, and salinity		float32
PHASE	N_MEASUREMENTS				float64
PHASE_QC	N_MEASUREMENTS				int64
PITCH	N_MEASUREMENTS	degrees	Vehicle pitch		float32
PITCH_CTL	N_MEASUREMENTS				float32
PRES	N_MEASUREMENTS	dbar	Uncorrected sea-water pressure at pressure sensor		float32
PROFILE_NUMBER	N_MEASUREMENTS				float64
PSAL	N_MEASUREMENTS	1e-3	Salinity corrected for thermal-inertia effects (PSU)	sea_water_salinity	float32
PSAL_QC	N_MEASUREMENTS		Whether to trust each corrected salinity value	status_flag	float32
PSAL_RAW	N_MEASUREMENTS	1e-3	Uncorrected salinity derived from temperature_raw and conductivity_raw (PSU)		float32
PSAL_RAW_QC	N_MEASUREMENTS		Whether to trust each raw salinity value	status_flag	float32
ROLL	N_MEASUREMENTS	degrees	Vehicle roll		float32
ROLL_CTL	N_MEASUREMENTS				float32
SIGMA_T	N_MEASUREMENTS	g/m^3	Sigma based on density	sea_water_sigma_t	float32
SIGTHETA	N_MEASUREMENTS	g/m^3		sea_water_sigma_theta	float32
TEMP	N_MEASUREMENTS	degrees_Celsius	Termperature (in situ) corrected for thermistor first-order lag	sea_water_temperature	float32
TEMP_QC	N_MEASUREMENTS		Whether to trust each corrected temperature value	status_flag	float32
TEMP_RAW	N_MEASUREMENTS	degrees_Celsius	Uncorrected temperature (in situ)		float32
TEMP_RAW_QC	N_MEASUREMENTS		Whether to trust each raw temperature value	status_flag	float32
THETA	N_MEASUREMENTS	degrees_Celsius	Potential temperature based on corrected salinity	sea_water_potential_temperature	float32
TIME	N_MEASUREMENTS	seconds since 1970-01-01T00:00:00Z	Time of CTD sample in GMT epoch format	time	datetime64[ns]
TIME_DOXY	N_MEASUREMENTS		time for Aanderaa 4330 in GMT epoch format	time	datetime64[ns]
VBD_CC	N_MEASUREMENTS				float32
aanderaa4330_instrument_dissolved_oxygen	N_MEASUREMENTS	micromoles/kg	Dissolved oxygen concentration reported from optode corrected for salinity		float32
absolute_salinity	N_MEASUREMENTS	g/kg	Absolute salinity per TEOS-10		float32
buoyancy	N_MEASUREMENTS	g	Buoyancy of vehicle, corrected for compression effects		float32
conservative_temperature	N_MEASUREMENTS	degrees_Celsius	Conservative termperature per TEOS-10		float32
ctd_pressure	N_MEASUREMENTS	dbar	Pressure at CTD thermistor	sea_water_pressure	float32
density_insitu	N_MEASUREMENTS	g/m^3	Sea water in-situ density based on pressure		float32
divenum	N_MEASUREMENTS				int32
eng_GC_phase	N_MEASUREMENTS				float32
eng_aa4330_AirSat	N_MEASUREMENTS		As reported by the instrument		float32
eng_aa4330_CalPhase	N_MEASUREMENTS		As reported by the instrument		float32
eng_aa4330_O2	N_MEASUREMENTS	micromoles/L	Dissolved oxygen as reported by the instument, based on on-board calibration data, assuming optode temperature but without depth or salinity correction		float32
eng_aa4330_TCPhase	N_MEASUREMENTS		As reported by the instrument		float32
eng_aa4330_Temp	N_MEASUREMENTS	degrees_Celsius	As reported by the instrument		float32
eng_rec	N_MEASUREMENTS				float32
PLATFORM_MODEL	string
PLATFORM_SERIAL_NUMBER	string
TRAJECTORY	string
WMO_IDENTIFIER	string

Convert mission from a local directory of basestation files

For local data in the directory input_dir
Creates a plot of ctd_depth against ctd_time.

[11]:

# Specify the input directory on your local machine
input_dir = data_path + '/demo_sg005' ### chose the input directory with your data

# Load and concatenate all datasets in the input directory
# Optionally, specify the range of profiles to load (start_profile, end_profile)
list_datasets = readers.load_basestation_files(input_dir, start_profile=1, end_profile=5)

# Convert the list of datasets to OG1
ds_OG1, var_list = convertOG1.convert_to_OG1(list_datasets)

# Generate a simple plot
plotters.plot_profile_depth(ds_OG1)
plotters.show_contents(ds_OG1,'attrs')

Loading datasets: 100%|██████████| 5/5 [00:00<00:00, 39.42file/s]
Processing datasets:   0%|          | 0/5 [00:00<?, ?dataset/s]Variable 'vert_speed_gsm' not in OG1 vocabulary.
Variable 'speed_gsm' not in OG1 vocabulary.
Variable 'sound_velocity' not in OG1 vocabulary.
No conversion information found for micromoles/kg to micromoles/kg
Variable 'north_displacement_gsm' not in OG1 vocabulary.
Variable 'longitude_gsm' not in OG1 vocabulary.
Variable 'latitude_gsm' not in OG1 vocabulary.
Variable 'horz_speed_gsm' not in OG1 vocabulary.
Variable 'glide_angle_gsm' not in OG1 vocabulary.
No conversion information found for cm s-1 to cm s-1
Variable 'eng_wlbb2f_VFtemp' not in OG1 vocabulary.
Variable 'eng_tempFreq' not in OG1 vocabulary.
Variable 'eng_sbe43_O2Freq' not in OG1 vocabulary.
Variable 'eng_elaps_t_0000' not in OG1 vocabulary.
Variable 'eng_elaps_t' not in OG1 vocabulary.
Variable 'eng_depth' not in OG1 vocabulary.
Variable 'eng_condFreq' not in OG1 vocabulary.
Variable 'east_displacement_gsm' not in OG1 vocabulary.
No conversion information found for micromoles/kg to micromoles/kg
Variable 'depth' not in OG1 vocabulary.
Variable 'density' not in OG1 vocabulary.
Variable 'buoyancy' not in OG1 vocabulary.
Processing datasets: 100%|██████████| 5/5 [00:02<00:00,  2.01dataset/s]

information is based on xarray Dataset

[11]:

	Attribute	Value	DType
0	title	OceanGliders trajectory file	str
1	id	sg005_20080606T180738_delayed	str
2	platform	sub-surface gliders	str
3	platform_vocabulary	https://vocab.nerc.ac.uk/collection/L06/curren...	str
4	naming_authority	edu.washington.apl	str
5	institution	School of Oceanography\nUniversity of Washingt...	str
6	geospatial_lat_min	61.41231666666666	ndarray
7	geospatial_lat_max	61.57591666666667	ndarray
8	geospatial_lon_min	-8.747133333333332	ndarray
9	geospatial_lon_max	-8.273983333333332	ndarray
10	geospatial_vertical_min	-0.3214989667970032	ndarray
11	geospatial_vertical_max	845.8311973927603	ndarray
12	time_coverage_start	20080606T180256	str
13	time_coverage_end	20080607T080838	str
14	site	Multiple transects of Faroe-Iceland Ridge uppe...	str
15	project	Iceland Scotland Ridge June 2008	str
16	contributor_name	Charlie Eriksen, Peter Rhines	str
17	contributor_role	PI, Principal investigator	str
18	contributor_role_vocabulary	http://vocab.nerc.ac.uk/search_nvs/W08,	str
19	contributor_email	eriksen@uw.edu,	str
20	contributing_institutions	University of Washington - School of Oceanogra...	str
21	contributing_institutions_vocabulary	https://edmo.seadatanet.org/report/1434,	str
22	contributing_institutions_role	PI,	str
23	contributing_institutions_role_vocabulary	http://vocab.nerc.ac.uk/collection/W08/current/,	str
24	uri	9e33a22e-a959-11e3-b35f-0026bb609360	str
25	rtqc_method	No QC applied	str
26	rtqc_method_doi	n/a	str
27	comment	Processing start:\n20:13:32 11 Mar 2014 UTC: I...	str
28	start_date	20080606T180738	str
29	date_created	20140311T200332	str
30	featureType	trajectoryProfile	str
31	Conventions	CF-1.10,OG-1.0	str
32	date_modified	20250921T144514	str
33	disclaimer	Data provided AS-IS.	str
34	file_version	2.71	float32
35	keywords	Water Temperature, Conductivity, Salinity, Den...	str
36	keywords_vocabulary	NASA/GCMD Earth Science Keywords Version 6.0.0.0	str
37	license	These data may be redistributed and used witho...	str
38	acknowledgment	National Science Foundation, OCE Division, Gra...	str
39	contributer_email	null@null.com	str

Convert mission from the NCEI server (with p*nc files)

Data from the sg015 mission in the Labrador Sea (https://www.ncei.noaa.gov/access/metadata/landing-page/bin/iso?id=gov.noaa.nodc:0111844), dataset identifier gov.noaa.nodc:0111844.

[12]:

# Specify the server where data are located
server = "https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/033/20100903/"

# Load and concatenate all datasets from the server, optionally specifying the range of profiles to load
list_datasets = readers.load_basestation_files(server, start_profile=1, end_profile=19)

# Convert the list of datasets to OG1
ds_OG1, var_list = convertOG1.convert_to_OG1(list_datasets)

Loading datasets:  42%|████▏     | 8/19 [00:00<00:00, 35.19file/s]Downloading file 'p0330011_20100905.nc' from 'https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/033/20100903/p0330011_20100905.nc' to '/home/runner/.cache/seagliderOG1'.
Downloading file 'p0330012_20100905.nc' from 'https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/033/20100903/p0330012_20100905.nc' to '/home/runner/.cache/seagliderOG1'.
Loading datasets:  63%|██████▎   | 12/19 [00:00<00:00, 14.74file/s]Downloading file 'p0330013_20100906.nc' from 'https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/033/20100903/p0330013_20100906.nc' to '/home/runner/.cache/seagliderOG1'.
Downloading file 'p0330014_20100906.nc' from 'https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/033/20100903/p0330014_20100906.nc' to '/home/runner/.cache/seagliderOG1'.
Loading datasets:  79%|███████▉  | 15/19 [00:01<00:00, 10.39file/s]Downloading file 'p0330016_20100906.nc' from 'https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/033/20100903/p0330016_20100906.nc' to '/home/runner/.cache/seagliderOG1'.
Downloading file 'p0330017_20100906.nc' from 'https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/033/20100903/p0330017_20100906.nc' to '/home/runner/.cache/seagliderOG1'.
Loading datasets:  89%|████████▉ | 17/19 [00:01<00:00,  8.21file/s]Downloading file 'p0330018_20100906.nc' from 'https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/033/20100903/p0330018_20100906.nc' to '/home/runner/.cache/seagliderOG1'.
Downloading file 'p0330019_20100906.nc' from 'https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/033/20100903/p0330019_20100906.nc' to '/home/runner/.cache/seagliderOG1'.
Loading datasets: 100%|██████████| 19/19 [00:01<00:00,  9.71file/s]
Processing datasets:   0%|          | 0/19 [00:00<?, ?dataset/s]Variable 'eng_depth' not in OG1 vocabulary.
Variable 'eng_aa4330_Temp' not in OG1 vocabulary.
Variable 'longitude_gsm' not in OG1 vocabulary.
Variable 'speed_gsm' not in OG1 vocabulary.
Variable 'vert_speed_gsm' not in OG1 vocabulary.
Variable 'eng_aa4330_TCPhase' not in OG1 vocabulary.
Variable 'eng_sbect_tempFreq' not in OG1 vocabulary.
Variable 'eng_sbect_condFreq' not in OG1 vocabulary.
Variable 'eng_rec' not in OG1 vocabulary.
Variable 'north_displacement_gsm' not in OG1 vocabulary.
Variable 'horz_speed_gsm' not in OG1 vocabulary.
No conversion information found for micromoles/kg to micromoles/kg
Variable 'eng_aa4330_O2' not in OG1 vocabulary.
Variable 'eng_aa4330_AirSat' not in OG1 vocabulary.
Variable 'latitude_gsm' not in OG1 vocabulary.
Variable 'aanderaa4330_instrument_dissolved_oxygen' not in OG1 vocabulary.
Variable 'eng_elaps_t_0000' not in OG1 vocabulary.
Variable 'buoyancy' not in OG1 vocabulary.
Variable 'east_displacement_gsm' not in OG1 vocabulary.
Variable 'sound_velocity' not in OG1 vocabulary.
Variable 'density_insitu' not in OG1 vocabulary.
Variable 'density' not in OG1 vocabulary.
Variable 'eng_aa4330_CalPhase' not in OG1 vocabulary.
Variable 'eng_GC_phase' not in OG1 vocabulary.
No conversion information found for cm s-1 to cm s-1
Variable 'conservative_temperature' not in OG1 vocabulary.
Variable 'glide_angle_gsm' not in OG1 vocabulary.
Variable 'eng_elaps_t' not in OG1 vocabulary.
Variable 'absolute_salinity' not in OG1 vocabulary.
No conversion information found for micromoles/kg to micromoles/kg
Variable 'ctd_pressure' not in OG1 vocabulary.
Variable 'depth' not in OG1 vocabulary.
Processing datasets: 100%|██████████| 19/19 [00:12<00:00,  1.51dataset/s]

Saving data

Due to problems with writing xarray datasets as netCDF when attributes are not of a specified type (str, Number, np.ndarray, np.number, list, tuple), a function was written save_dataset.

[13]:

# Write the file
# This writer catches errors in data types (DType errors) when using xr.to_netcdf()
# The solution is to convert them to strings, which may be undesired behaviour
output_file = os.path.join(data_path, 'demo_test.nc')
if os.path.exists(output_file):
    os.remove(output_file)

writers.save_dataset(ds_OG1, output_file);

/home/runner/work/seagliderOG1/seagliderOG1/seagliderOG1/writers.py:47: UserWarning: Times can't be serialized faithfully to int64 with requested units 'seconds since 1970-01-01T00:00:00+00:00'. Resolution of 'nanoseconds' needed. Serializing times to floating point instead. Set encoding['dtype'] to integer dtype to serialize to int64. Set encoding['dtype'] to floating point dtype to silence this warning.
  ds.to_netcdf(output_file, format="NETCDF4")

[14]:

# Load the data saved
ds1 = xr.open_dataset(output_file)

# Generate a simple plot
#plotters.show_contents(ds_all,'attrs')
plotters.plot_depth_colored(ds1, color_by='PROFILE_NUMBER')

Run multiple missions

[15]:

# Add these to existing attributes - update to your details
contrib_to_append = vocabularies.contrib_to_append
print(contrib_to_append)

{'contributor_name': 'Eleanor Frajka-Williams', 'contributor_email': 'eleanorfrajka@gmail.com', 'contributor_role': 'Data scientist', 'contributor_role_vocabulary': 'http://vocab.nerc.ac.uk/search_nvs/W08', 'contributing_institutions': 'University of Hamburg - Institute of Oceanography', 'contributing_institutions_vocabulary': 'https://edmo.seadatanet.org/report/1156', 'contributing_institutions_role': 'Data scientist', 'contributing_institutions_role_vocabulary': 'http://vocab.nerc.ac.uk/search_nvs/W08'}

[16]:

# Specify a list of servers or local directories
input_locations = [
    # Either Iceland, Faroes or RAPID/MOCHA
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/005/20090829/", # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/005/20080606/", # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/005/20081106/", # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/012/20070831/", # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/014/20080214/",  # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/014/20080222/", # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/016/20061112/",  # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/016/20090605/", # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/016/20071113/", # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/016/20080607/",  # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/033/20100518/", # done
    "https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/033/20100903/", # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/101/20081108/",     # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/101/20061112/",    # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/101/20070609/",   # done
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/102/20061112/",  # done
    # Labrador Sea
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/015/20040924/",
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/014/20040924/",
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/008/20031002/",
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/004/20031002/",
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/016/20050406/",
    # RAPID/MOCHA
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/033/20100729/",
    #"https://www.ncei.noaa.gov/data/oceans/glider/seaglider/uw/034/20110128/",
]

for input_loc in input_locations:
    # Example usage
    ds_all = convertOG1.process_and_save_data(input_loc, output_dir=data_path, save=True,  run_quietly=True)

Loading datasets: 100%|██████████| 1/1 [00:00<00:00, 34.30file/s]
Processing datasets:   0%|          | 0/1 [00:00<?, ?dataset/s]Variable 'eng_depth' not in OG1 vocabulary.
Variable 'eng_aa4330_Temp' not in OG1 vocabulary.
Variable 'longitude_gsm' not in OG1 vocabulary.
Variable 'speed_gsm' not in OG1 vocabulary.
Variable 'vert_speed_gsm' not in OG1 vocabulary.
Variable 'eng_aa4330_TCPhase' not in OG1 vocabulary.
Variable 'eng_sbect_tempFreq' not in OG1 vocabulary.
Variable 'eng_sbect_condFreq' not in OG1 vocabulary.
Variable 'eng_rec' not in OG1 vocabulary.
Variable 'north_displacement_gsm' not in OG1 vocabulary.
Variable 'horz_speed_gsm' not in OG1 vocabulary.
No conversion information found for micromoles/kg to micromoles/kg
Variable 'eng_aa4330_O2' not in OG1 vocabulary.
Variable 'eng_aa4330_AirSat' not in OG1 vocabulary.
Variable 'latitude_gsm' not in OG1 vocabulary.
Variable 'aanderaa4330_instrument_dissolved_oxygen' not in OG1 vocabulary.
Variable 'eng_elaps_t_0000' not in OG1 vocabulary.
Variable 'buoyancy' not in OG1 vocabulary.
Variable 'east_displacement_gsm' not in OG1 vocabulary.
Variable 'sound_velocity' not in OG1 vocabulary.
Variable 'density_insitu' not in OG1 vocabulary.
Variable 'density' not in OG1 vocabulary.
Variable 'eng_aa4330_CalPhase' not in OG1 vocabulary.
Variable 'eng_GC_phase' not in OG1 vocabulary.
No conversion information found for cm s-1 to cm s-1
Variable 'conservative_temperature' not in OG1 vocabulary.
Variable 'glide_angle_gsm' not in OG1 vocabulary.
Variable 'eng_elaps_t' not in OG1 vocabulary.
Variable 'absolute_salinity' not in OG1 vocabulary.
No conversion information found for micromoles/kg to micromoles/kg
Variable 'ctd_pressure' not in OG1 vocabulary.
Variable 'depth' not in OG1 vocabulary.
Processing datasets: 100%|██████████| 1/1 [00:00<00:00,  1.48dataset/s]
File /home/runner/work/seagliderOG1/seagliderOG1/data/sg033_20100903T182416_delayed.nc already exists. Exiting the process.

File /home/runner/work/seagliderOG1/seagliderOG1/data/sg033_20100903T182416_delayed.nc already exists. Exiting the process.

seagliderOG1 demo

Reading basestation files

Load single sample dataset

Load datasets from a local directory

Load datasets from a remote directory (URL)