SBE Data Processing • The importance of accurate data • Potential errors in CTD data • How SBE instruments are designed to minimize errors • Why processing CTD data improves data quality • Detailed lessons on SBE Data Processing

Why worry about Absolute Accuracy in Sensors? • In oceanography, we measure physical quantities to calculate many parameters necessary to analyze the ocean – Thermodynamic properties are needed in ocean/climate models

• Calculated parameters (salinity) rely on accurate measurement of temperature, conductivity, and pressure • Small errors in original measurements can lead to large errors in calculated parameters – These small errors can lead to big errors in data analysis and interpretations

Types of Errors in CTD Data • Dynamic Errors – Errors incurred while sampling on moving platforms or on moorings where conditions are changing rapidly – Response time of sensors

• Static Errors (discussed already) – Initial and post-deployment calibration accuracy – Reported on calibration and specification sheets

• Sensor Drift Characteristics (discussed already) – Sometimes reported on specification sheet – Does not address fouling drift

Dynamic Errors in Temperature • Response time of sensor to changing condition • Temperature response time largely determined by physical size and construction. – Response time for profiling CTDs • 0.070 sec (SBE 9-11plus and 25plus) • 0.5 sec (SBE 19plus)

– Response time for moored CTDs 0.5 sec

• Corrected in data processing

Dynamic Errors in Conductivity • Response time of sensor to changing temperature AND salinity – Depends on flow speed through cell – Depends on thermal mass of materials that make up sensor

• For both high and low accuracy (0.1 to 0.002 PSU), response time and thermal lag errors can be significant • On free-flushed sensors where flow rates are always changing, these errors cannot be avoided or corrected • Only on flow-controlled CTDs can these errors be corrected for

Dynamic Errors in Salinity: 1. Not Sampling Same Water Parcel • T and C sensors measuring different parcels of water Example of Non-plumbed SBE sensors

Other Company’s T and C sensors COND CTD Tilted

t1

C O N D t1 t0

t0 T E Physical M Misalignment P

TEMP

Dynamic Errors in Salinity: 2. Mismatched Response Times T

C

S

Pressure

• Non-Equal response times between T and C • T and C used to compute salinity • This causes “spiking”

0 db

C leads T 0.084 db

Dynamic Errors in Salinity: 3. Temperature Error caused by Thermal Mass of Conductivity Cell Conductivity measurement is very sensitive to thermal mass of the cell – 90% of the C signal is dependent on temperature Image of a cold cup that changes color after putting hot coffee into it

Time it takes to change the temperature to new value is the thermal mass lag

Rough Seas Can Affect Data Quality

Sea-Bird Solution to Reducing Dynamic Errors 1. Make measurements on the same water parcel at stations along a pump-controlled flow path 2. This allows us to postprocess data to significantly reduce dynamic errors

Time 4 Time 3

Time 2

Time 1

Time 5

Flow Control • Forces sensors to measure same water parcel, but at different times • Can correct for differences in time each sensor takes its reading because we have constant flow (speed) and sample rate • Provides constant response time for sensors that are flow dependent (conductivity and oxygen) • Constant flow and sample rate allows for response time adjustments between sensors (similar to time of sample adjustment) • Reduces thermal mass amplitude and lag PLUS allows us to correct for it • Can separate alignment issues from ship heave • By adjusting flow speed, we can better match response times of sensors (T and C) on SBE 9plus - Lag is fixed and can be removed with high precision (SBE 19plus, 19plus V2, 25, 25plus)

Dynamic Processing Modules • All of the processing modules are explained in the manual • Default parameters for each instrument are provided in the manual

Key SBE Data Processing Modules for Profiling CTDs • Data Conversion converts data from hexadecimal to engineering units •

•

Wild Edit or (Median) Filter to remove outliers Align CTD coordinates measurements of T, C and P on same parcel of water –

Other variables as well as needed

• Filter refines response time of sensors and smoothes digital noise in Pressure data • Loop Edit (Optional) reduces ship heave effects by marking scans “badflag” if the • • • • • •

scan fails the minimum velocity criteria set by the user Cell Thermal Mass corrects conductivity sensor thermal lag error for a given flow rate determined by pump speed or estimated based on descent rate Derive takes the newly aligned and corrected independent variables (T, C, P, Oxvolts) and computes the dependent variables (Salinity, Density, Oxygen Concentration) Bin Average statistically averages data blocks into bins that are evenly spaced or interpolated pressure, depth, scan count or time blocks Split separates up and down casts ASCII In transforms an ASCII Text file with columns of data into a SBE formatted .cnv (converted) file for processing and plotting in SBE software ASCII Out transforms a SBE formatted file and outputs a simple column file of data in text format…this can be used in Excel and other non-SBE programs

Converting Dependent Quantities vs. Raw Independent Quantities • Salinity and Oxygen are computed quantities – They are what we call Dependent Variables as they rely on Independent Variables (T,C, P, OXVOLTS) • For successful computation of Dependent Variables, inputs need to be accurately measured AND accurately coordinated on a point in space, and secondarily coordinated in time response • If these Independent Variables are measured or coordinated incorrectly, this will have ripple effect in other computed quantities – density, buoyancy frequency, etc.

For Example: How to get salinity with only 10% of signal • Electrical measurement of conductivity – 90% of signal from temperature – 10% from salinity based on conducting ion content of seawater • 1% error in Temperature causes 10% error in Salinity • Always compute Salinity AFTER we process Temperature, Conductivity, and Pressure

Activity: Data Conversion Choose one data file • Use SBE Data Processing to convert data from SBE 9plus, in preparation for further processing; see notes for instructions • Or, if you use an SBE 19plus CTD, process data from SBE 19plus; see notes for instructions

Filtering Data for Matching TC Sensor Time Responses • Conductivity cell has a time constant (Tau) that depends on pumping rate – Temperature response is not flow dependent

• SBE 9plus pump and TC duct controls Tau of C to match Tau of T – No Filtering required

• SBE 25 and 25plus pump is slower than pump used with 9plus – Filtering T and C optional for SBE 25 – Filtering T and C recommended for SBE 25plus

• Tau for SBE 19plus and 19plus V2 T and C are not as well matched; require filtering of T and C

Filtering Pressure to Remove Digital Noise • Filtering pressure data removes digitization noise • Filter pressure data if: – You are going to use Loop Edit to remove data artifacts, and/or – You are interested in fine scale in SBE 9plus

Example: Filtering Pressure Pressure filtered with 0.15-sec time constant

Raw data 120

21.0

TEMPERATURE

26.0 120

S

21.0

T

TEMPERATURE S

P

26.0

T

P

dbars

dbars

170

170 34.95

SALINITY

35.15

34.95

SALINITY

35.15

Filtering Converted Data Filter Time Constants • SBE 9plus – Filter A 0.15 sec for P

• SBE 25plus – Filter A 0.1 sec for C and T – Filter B 0.25 sec for P

• SBE 25 (optional) – Filter A 0.1 sec for C and T – Filter B 0.5 sec for P

• SBE 19plus or 19plus V2 – Filter A 0.5 sec for C and T – Filter B 1.0 sec for P

Sensor Alignments • Distance between sensors P-T P-T to C-Star Transmissometer

• Time of travel of water parcel through plumbing P-T to C P-T to DO

• Align big discrepancies in response time (i.e., DO)

Symptoms of T and C Misalignment in Data 0 db

0 db

T

C

S

Pressure

• Evidence of mismatch seen in salinity spikes and density inversions • Correction via pressure shifting of conductivity

T Step = 0.05 C; C Step = 0.015 S/m Pressure Alignment Perfect 0 db

T

S

S Pressure

C

Pressure

T

C

C lags T 0.084 db

C leads T 0.084 db

Symptoms of Misalignment in Data • Mismatch of up and down cast values with depth due to: – Slow response times – Distance between sensors (as shown here)

Advancing Data in Time to Remove Misalignment • Alignment on T and C is done automatically in 11plus Deck Box • Alignment can change from factory default due to changes in plumbing that increase or decrease pumping speed • Use Align CTD module to match temperature and conductivity data streams in post processing – On 19plus and 25plus CTDs

How Do I Know How Much to Advance or Slow a Data Channel? 1. Use factory defaults –

Sea-Bird has done tests for standard configurations that provide default alignments

2. By knowing the flow rate and path distance between sensors, compute a time delay 3. By looking at your data – Find a spot in your data with a sharp salinity shift and/or unrealistic spiking – Experiment with alignment values to minimize salinity spiking

Example Data From the Faroe Islands

Sigma-t

T

S Descent rate

Subset of Example

ship heave T C mismatch Sigma-t

S

Descent rate

Activity: Align and Derive Choose one data file • Use SBE Data Processing to align data from SBE 9plus, derive calculated parameters, and plot results; see notes for instructions • If you use SBE 19plus, repeat process for data from SBE 19plus; see notes for instructions Note: You have already run Data Conversion on the file you will Align. The 19plus adds a Filtering Step… After you Align the data, you will Derive Salinity to check on spiking in Sea Plot

Alignment of T and C Conductivity Advanced 1 Scan (0.042s)

Original Data

Density, 2 [sigma-t, Kg/m^3 ]

Density, 2 [sigma-t, Kg/m^3 ]

27.900 27.925 27.950 27.975 28.000 28.025 28.050 28.075 28.100

27.900 27.925 27.950 27.975 28.000 28.025 28.050 28.075 28.100

500

500

525

525

550

550

575

625

density

575

salinity

600

Pressure, Digiquartz [db]

Pressure, Digiquartz [db]

600

salinity

625

density

650

650

675

675

700

700

34.750 34.775 34.800 34.825 34.850 34.875 34.900 34.925 34.950

34.750 34.775 34.800 34.825 34.850 34.875 34.900 34.925 34.950

Salinity, 2 [PSU]

Salinity, 2 [PSU]

Conductivity Advanced 2 Scans (0.084s) Density, 2 [sigma-t, Kg/m^3 ]

Conductivity Advanced 3 Scans (0.125s) Density, 2 [sigma-t, Kg/m^3 ]

27.900 27.925 27.950 27.975 28.000 28.025 28.050 28.075 28.100

27.900 27.925 27.950 27.975 28.000 28.025 28.050 28.075 28.100

500

500

525

525

550

550

575

575

Pressure, Digiquartz [db]

625

density

600

Pressure, Digiquartz [db]

salinity

600

625

salinity density

650

650

675

675

700

700 34.750 34.775 34.800 34.825 34.850 34.875 34.900 34.925 34.950

Salinity, 2 [PSU]

34.750 34.775 34.800 34.825 34.850 34.875 34.900 34.925 34.950

Salinity, 2 [PSU]

Example of T C Alignment for SBE 19plus

Dissolved O2 Alignment • Sensor time constants ~ 2 - 5 seconds, depending on temperature • Plumbing delay < 2 seconds, depending on location of sensor in flow path • Delays add for ~ 4 seconds total • Hysteresis in DO profiles is caused by plumbing delays, temperature mismatch, and sensor response time – Recommend corrections for deep ocean pressure > 1000 dbar

Hysteresis in Dissolved Oxygen Profiles 1.5 0

2.0

Oxygen, SBE 43 [ml/l] 2.5 4.0 3.0 3.5

Oxygen, SBE 43 [ml/l]

4.5

2.00 2.25 2.50 2.75 3.00 3.25 3.50 3.75 4.00 4.25 4.50

5.0

0

10 50

20

100

40

T

50

DO

60

Pressure, Strain Gauge [db]

Pressure, Digiquartz [db]

30

150

T

DO

200

70 80

250

90

300

100 17.5

20.0 22.5 25.0 27.5 Temperature [ITS-90, deg C]

30.0

15.0

17.5

27.5 20.0 22.5 25.0 Temperature [ITS-90, deg C]

30.0

T vs DO plot 27 27.5

26 25

25.0

Temperature [ITS-90, deg C]

Temperature [ITS-90, deg C]

24 23

22.5

22 21

20.0

20 19

17.5

18 2.0

2.5

3.0 3.5 4.0 Oxygen, SBE 43 [ml/l]

4.5

5.0

15.0 2.0

2.5

3.0 3.5 4.0 Oxygen, SBE 43 [ml/l]

4.5

5.0

Activity: Align and Derive DO Choose one data file

• Use SBE Data Processing to convert data from SBE 9plus, align oxygen, derive oxygen, and plot results; see notes for instructions • If you use a 19plus, process data from SBE 19plus; see notes for instructions Run Data Conversion Run Align CTD 3 times on converted file Run Derive on the 3 aligned files Look at data in Sea Plot

Dissolved Oxygen Advanced 0, 2, 4, 6 Seconds Relative to Pressure No advance

2 sec advance

4 sec advance

6 sec advance

Effect of Conductivity Cell Thermal Mass • Glass conductivity cell stores heat • A warm cell warms water moving through it – Will read warm of correct (High)

• A cold cell cools water moving through it – Will read cold of correct (Low)

• Water in cell is a different temperature than the thermometer measured a moment earlier • When salinity is computed, it will be in error

Cell Thermal Mass Example

Upcast Downcast T

S

SBE 9plus Salinity, with and without CTM Correction Green Salinity processed with Cell Thermal Mass (CTM) correction; Black Salinity unprocessed

Corresponding Temperature (green) and descent rate (blue)

~+0.02 psu ~0.8 ˚C step

Note: Downcast only, Data not LoopEdited

Dz/dt ~ 0.40 m/s

Removing Effect of Conductivity Cell Thermal Mass

Activity: Remove Conductivity Cell Thermal Mass Effect • Use SBE Data Processing to convert data from SBE 9plus, apply cell thermal mass correction, derive salinity, and plot results; see notes for instructions Run Data Conversion Skip the Align CTD step for now Run Cell Thermal Mass on converted file, append C to filename Run Derive on CTM file and original Converted file Plot data in Sea Plot to see what CTM processing did

Cast Corrected for Cell Thermal Mass

Corrected

How to Remove Ship Heave Effects on CTD Data • Data errors caused by CTD profiling reversals are flagged by scan line • We can choose to omit these “flagged” data from averaging and plots

Data Artifacts Caused by the Underwater Package Rapid Descent

• Ship heave causes underwater package to loop through water • Accelerations and decelerations caused by ship heave cause water entrained within package to blow by sensors

Ship Heave Slows Descent Rapid Descent Resumes

Turbulent Wake Wake is Shed Downward Sensor Path Goes Through Shed Wake

Ship Heave Effects Enlargement of plot at left

T

S

Descent rate

Descent rate

T

S

Salinity Spiking due to Ship Heave Profiling through Temperature Gradient

Removing Package-Induced Data Artifacts • Data errors introduced this way must be flagged in the data file or deleted – There is no fix

• Loop Edit flags scans of data that experienced reversals or loops caused by ship heave • Wild Edit removes data that fall outside of user-specified limits

Removing Package-Induced Data Artifacts: Loop Edit

Removing Package-Induced Data Artifacts: Wild Edit

Activity: Remove Loops Use SBE Data Processing • Run Data Conversion – Convert data from SBE 9plus

• Run Filter on converted file – Filter pressure

• Run Loop Edit on convertedfiltered file – Remove loops

• Run Sea Plot on convertedfiltered-loopedited results See notes for instructions

Removing PackageInduced Data Artifacts, Loop Edit Original

Edit by fixed speed (.25m/s)

Edit by % mean speed (20%)

Ancillary Data Processing • Data editing – Section • Retrieves a portion of a cast

– Split • Separates upcast from downcast

• Filtering – Window Filter • Offers a variety of window shapes

Bin Averaging • Reduces size of a data set by statistically estimating data values at even intervals (e.g., every meter or 10 meters) • Can work in depth (meters), pressure (decibars), time, or by scan • Can bin average upcast, downcast, or both – If bin averaging upcast and downcast, keeps upcast bins and downcast bins separate

• The surface bin is treated separately

Bin Averaging Protocol: Pressure Interpolated • A linear estimate of variable Xi at bin pressure Pi ( Xc − Xp) * (Pi − Pp ) Xi = + Xp ( Pc − Pp ) Pp =average pressure of previous bin Xp =average value of variable in previous bin Xc =average value of variable in current bin Pc =average pressure of current bin Pi = center value for pressure in current bin surface = 0 db Minimum first bin = bin size - (bin size/2) = 5 db First bin Bin size=10 db Sum and average all data within bin, then interpolate to calculate value of variable at center of bin

Center (target) first bin = bin size = 10 db Maximum first bin = bin size + (bin size/2) = 15 db

Bin Average Protocol: Pressure, Not Interpolated • Data within a bin is averaged by summing and dividing by number of points within bin surface = 0 db Minimum first bin = bin size - (bin size/2) = 5 db First bin Bin size=10 db Sum and average all data within bin

Center (target) first bin = bin size = 10 db Maximum first bin = bin size + (bin size/2) = 15 db

The Surface Bin • Surface bin constrained by user data entries: minimum, maximum, and assigned pressure or depth Surface bin Bin size=3 db

minimum surface bin = 0 db target surface bin = 0 db maximum surface bin = 3 db Minimum first bin = bin size - (bin size/2) = 5 db

First bin Bin size=10 db

Center (target) first bin = bin size=10 db Maximum first bin = bin size + (bin size/2) = 15 db

File Selection and Data Setup

Bin Average: Output Data # binavg_bintype = meters # binavg_binsize = 1 # binavg_excl_bad_scans = yes # binavg_skipover = 0 # binavg_surface_bin = no, min = 0.000, max = 5.000, value = 2.500 # file_type = ascii *END* 1.000 24.9124 35.2455 100 0.0000e+00 2.000 24.9582 35.2463 90 0.0000e+00 3.000 25.0029 35.2477 36 0.0000e+00

Activity: Bin Average Data Choose one data file

• Use SBE Data Processing to bin average converted data from SBE 9plus; see notes for instructions • Or, if you use SBE 19plus, process data from SBE 19plus; see notes for instructions

Activity: Bin Average Data Raw data converted, but no additional processing

Same data bin averaged (downcast only)

Data Processing Notes • Best data is collected at highest rate instrument is capable of • Data should not be reprocessed • Calculation of derived parameters and bin averaging should be done last

Processing Steps: SBE 9-11plus Data •

•

•

•

•

• •

Data Conversion – Output up /downcasts of all parameters – Only process independent parameters (T,C,P, OXVOLTS, Modulo Errors, etc.) – Output converted variables (salinity, DO concentration) if comparing to water samples Align CTD – SBE 11plus Deck Box usually advances primary C +0.073 sec, sometimes secondary C – Align Dissolved Oxygen (DO) (2-3 sec) and other sensor data accordingly in post processing Filter – Only if continuous time series and no Pressure outliers – Filter Pressure at +0.15 sec Loop Edit – Only if ship heave a problem (if you see loops or high standard deviations in descent rate) – Use minimum fall speed from CTD descent rate plots Cell Thermal Mass – ALWAYS apply this correction in Saltwater applications on moving platforms (not moored) – Do NOT apply this correction in Freshwater applications – Parameters: Alpha = 0.03 and Tau = 7 sec Derive – Compute Salinity, DO concentration, and other dependent variables (Density, Specific Conductance, etc.) Bin Average – Average data into depth, time, or pressure bins AFTER DERIVING computed variables

Processing Steps: SBE 19plus Data •

• •

•

•

• •

Data Conversion – Output up /downcasts of all parameters. – Only process on independent parameters (T,C,P, OXVOLTS, etc.) – Output converted variables (salinity, DO concentration) if comparing to water samples Align CTD – Advance Temperature +0.5 sec, Conductivity 0-0.1 sec, and Dissolved Oxygen Voltage 3-5 sec Filter – Only if continuous time series and no outliers – Filter Pressure at +1.0 sec, and Temperature and Conductivity at +0.5 sec Loop Edit – Only if ship heave a problem (if you see loops or high standard deviations in descent rate) – Use minimum fall speed from CTD descent rate plots Cell Thermal Mass – ALWAYS apply this correction in Saltwater applications on moving platforms (not moored) – Do NOT apply this correction in Freshwater applications – Parameters: Alpha = 0.04 and Tau = 8 sec Derive – Compute Salinity, DO concentration, and other dependent variables (Density, Specific Conductance, etc.) Bin Average – Average data into depth, time, or pressure bins AFTER DERIVING computed variables

Processing Steps: SBE 25plus Data •

• •

•

•

• •

Data Conversion – Output up /downcasts of all parameters. – Only process on independent parameters (T,C,P, OXVOLTS, etc.) – Output converted variables (salinity, DO concentration) if comparing to water samples Align CTD – Advance Conductivity +0.1 sec and Dissolved Oxygen (DO) Voltage 3-5 sec Filter – Only if continuous time series and no P outliers – Filter Pressure at +0.5 sec Loop Edit – Only if ship heave a problem (if you see loops or high standard deviations in descent rate) – Use minimum fall speed from CTD descent rate plots Cell Thermal Mass – ALWAYS apply this correction in Saltwater applications on moving platforms (not moored) – Do NOT apply this correction in Freshwater applications – Parameters: Alpha = 0.04 and Tau = 8 sec Derive – Compute Salinity, DO concentration, and other dependent variables (Density, Specific Conductance, etc.) Bin Average – Average data into depth, time, or pressure bins AFTER DERIVING computed variables

Batch Processing • Batch processing frees you from processing each cast individually • Batch processing is done from a command line prompt – Win2000/XP run “command” from Start -> Run dialog gives you an MSDOS window – Win95/98 use an MSDOS window – Run SBEBatch directly from Start -> Run dialog

• Format for sbebatch is: – sbebatch filename parameters

Batch Processing • Batch processing uses an application that runs other applications (i.e., data processing applications) • You may use the Windows Scripting Host or an application Sea-Bird provides, SBEBatch • The applications that the batch processor runs are listed in a text file that you make with a text editor like Notepad – A list of applications are shown in your notes

• SBEBatch reads each line of the text file and runs each application in turn

Batch Processing • Each line of your batch file contains – Name of the application – Name of the files to operate on – Any additional parameters needed to do the job

• Parameters are denoted by the ‘/’ character and an identifier; a table of parameters is shown in your notes • For example, a batch processing file that runs Data Conversion on 1 data file looks like: DatCnv /iC:\MyData.dat /cC:\MyCTD.con

- Input file is C:\MyData.dat, designated by /i - Configuration file is C:\MyCTD.con, designated by /c - This will cause Data Conversion to use last .psa file, substituting .dat and .con file from batch file for files specified in .psa file, and create MyData.cnv

Batch Processing Script • To process all the files in a folder use a wildcard: the ‘*’ character • For example, a batch processing file that runs Data Conversion on all data files in a folder looks like: datcnv /iC:\Data\*.dat /cC:\Data\MyCTD.con

- Input files are all .dat files in C:\Data\ - Configuration file is C:\Data\MyCTD.con

Running SBEBatch • SBEBatch is run from the command line • Following sbebatch is the name of the batch file that SBEBatch will open and execute • For example: sbebatch c:\MyBatch.txt - Causes SBEBatch to open MyBatch.txt and run the applications a line at a time

Batch Processing Script • Remember that the format for running SBEbatch is: sbebatch filename parameters • You can operate on files in different folders with the same batch file by using command line parameters • These are entered after the batch file name and are denoted by the ‘%’ character and a number – The first command line parameter is %1, the second is %2, etc.

• Your batch file must have entries that use the ‘%’ parameters

Batch Processing Script • For example, a batch file that has this line in C:\MyBatch.txt DatCnv /i%1\*.dat /c%1\MyCTD.con Executed with this command line SBEBatch C:\MyBatch.txt C:\Data (C:\Data is the %1 parameter) Will cause Data Conversion to be run like this: DatCnv /iC:\Data\*.dat /cC:\Data\MyCTD.con All the .dat files in C:\Data will be converted • For the same batch file, if the command line is SBEBatch C:\MyBatch.txt C:\NewData All the .dat files in C:\NewData will be converted

Activity: Batch Process Data Do on your own • Use SBE Data Processing to batch process data a large number of data files from one CTD; see notes for instructions