WARFARE CENTERS CORONA

WARFARE CENTERS CORONA

Metrology and Calibration (METCAL) Overview

NAVAL SURFACE WARFARE CENTER, CORONA DIVISION MEASUREMENT SCIENCE DEPARTMENT Bob Fritzsche (MS 20) 951.273.5244 [email protected] 1 Keeping America’s Navy #1 in the World

1

Measurement WARFARE CENTERS CORONA

“When you can measure what you are speaking about and express it in numbers, you know something about it; and when you cannot measure it, you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind. It may be the beginning of knowledge, but you are scarcely in your thought advanced to the stage of a science.” Lord Kelvin

You don’t know anything about anything unless you can measure it. Keeping America’s Navy #1 in the World

2

Overview WARFARE CENTERS CORONA

• NSWC Corona Background • Why Calibrate? • Metrology and Calibration Program • Measurement Reliability • Measurement-Related Standards • Conclusion Keeping America’s Navy #1 in the World

3

NSWC Corona Mission WARFARE CENTERS CORONA

To serve as the Navy’s independent assessment agent throughout the lifecycle:

"To gauge the warfighting capability of ships and aircraft, from unit to battlegroup level, by assessing the suitability of design, the performance of equipment and weapons, and the adequacy of training."

Ref: OPNAVNOTE 5450

Keeping America’s Navy #1 in the World

4

Navy Organization WARFARE WARFARECENTERS CENTERS CORONA CORONA

MISSION

Secretary Secretary of of the the Navy Navy

Operate the Navy's full spectrum research, development, test and evaluation, engineering and fleet support Center for ship hull, mechanical and electrical systems, surface ship combat systems, coastal warfare systems, and other offensive and defensive systems associated with surface warfare.

Chief Chief of of Naval Operations Naval Operations

Naval Naval Sea Sea Systems Command Systems Command

Naval Naval Surface Surface Warfare Warfare Center Center

Reference: OPNAVNOTE 5450 Ser 09B22/1U510577 of 23 Dec 1991

SHIPS AND SHIP SYSTEMS

SURFACE SHIP COMBAT SYSTEMS

Carderock Division

Dahlgren Division

NAVSSES, NAVSSES NAVSSES,, Philadelphia Philadelphia

CSS, CSS, Panama Panama City City CDSA, CDSA, Dam Dam Neck Neck

Commandant Commandant of of the Marine Corps the Marine Corps

Crane Division

Pt. Pt. Hueneme Division

Det., Det., Louisville Louisville

Det., Det., Virginia Virginia Beach Keeping America’s Navy #1 inBeach the World

ORDNANCE

ASSESSMENT

Indian Head Division

Corona Division

NAVSSES, NAVSSES, Seal Seal Beach Beach Philadelphia Philadelphia Det. Det.

5

NSWC Corona Division WARFARE CENTERS CORONA

Commanding Commanding Officer Officer CAPT CAPT RR.. Shafer Shafer

Technical Technical Director Director Vacant Vacant

Deputy Deputy Technical Technical Director Director JJ.. Stone ( Act) ) Act Stone (Act)

Measurement Measurement Science Science

A. A. Hovakemian Hovakemian

Product Product Engineering Engineering Assessment Assessment

DD.. Sugg Sugg

Performance Performance Assessment Assessment

DD.. Bergstrom Bergstrom

Quality Quality Assessment Assessment

Force Force Training Training Assessment Assessment

DD.. Costlow Costlow

M Act) Act) M.. Gammon Gammon ((Act)

Command Command

JJ.. Cardany Cardany

Keeping America’s Navy #1 in the World

Readiness Readiness Assessment Assessment

JJ.. Watson Watson

6

Measurement Science WARFARE CENTERS CORONA

Measurement Science Department Arman Hovakemian

MS00

Measurement Requirements Assessment Ken Felix

MS10

Air Programs MS11

A. Fritzsche

FMS Programs MS12

I. Garcia

Ground Programs MS13

L. Poling

Requirements Analysis MS14

M. Doerr

METCAL In-Service Assessment Bob Fritzsche

MS20

METCAL Systems MS21

R. Greenhalge

Metrology Engineering Assessment Vacant

MS30

L. Rice

Fleet METCAL In-Service MS23

R. Zaja

MS31

S. Buckman

Microwave/ Electro-Optics MS32

Ed Trovato

Physical/Mechanical MS33

SSP METCAL In-Service MS24

John Griffith

MS40

DC/Low Frequency

NAVSEA METCAL In-Service MS22

Advanced Measurement Systems

R. Figuerres

B. Heglar

R&D Coordinator MS40A

J. Walden

CBNR Warfare Systems Metrology MS41

M. Bishop

Advanced Automated Calibration MS42

R. Schumacher

Sensors and Special Projects MS43

A. Luh

Analytical Metrology MS25

J. Davis

Keeping America’s Navy #1 in the World

7

Definitions WARFARE CENTERS CORONA

Metrology: The Science of Measurement

Calibration: VIM: Set of operations that establish, under specified conditions, the relationship between values of quantities indicated by a measuring instrument or measuring system, or values represented by a material measure or a reference material, and the corresponding values realized by standards. Navy: Calibration is the comparison of a measurement system or device of unverified accuracy, to a measurement system or device of known or greater accuracy, to detect and correct any variation from required performance specifications. Keeping America’s Navy #1 in the World

8

Measurement Possibilities WARFARE CENTERS

GO AL

CORONA

Not Precise/

Precise /

Precise /

Not Accurate

Not Accurate

Accurate

Keeping America’s Navy #1 in the World

9

Why Calibrate? WARFARE CENTERS CORONA

• Ensure the readiness of test equipment to perform accurate measurements and provide valid data • Limit the number of erroneous test decisions • Maintain overall measurement integrity and traceability Keeping America’s Navy #1 in the World

10

Why Calibrate? – Data Integrity WARFARE CENTERS CORONA

• How do we know the data is good? – – – –

Accurate? Precise? Stable? Can you trust the person collecting the data? Can you trust the measurement instrument? What proof do you have that the numbers reflect reality?

• Can you trust the decision you are about to make using the collected data? • What is the cost if you are wrong? In Tolerance

Out of Tolerance Difference?

Keeping America’s Navy #1 in the World

11

Uniform Measurement Accuracies WARFARE CENTERS CORONA

Keeping America’s Navy #1 in the World

12

NSWC Corona – METCAL Role WARFARE CENTERS CORONA

NSWC Corona serves as the Scientific and Technical Agent for the Navy Metrology and Calibration (METCAL) Program per OPNAVINST 3960.16A, Navy Test and Monitoring Systems (TAMS)

and NAVAIRINST 13640.1B NAVSEAINST 4734.1B SSPINST 4355.8B MARCOR ORDER 4733.1B

Keeping America’s Navy #1 in the World

13

Navy Calibration Program WARFARE CENTERS CORONA

METCAL Executive Director TAMS Executive Board NAVSEA TMDE

SISCAL

NAVAIR

SSP

USMC

METCAL

NPSL Shore Calibration Laboratories Afloat Calibration Laboratories Keeping America’s Navy #1 in the World

14

METCAL Processes and Products WARFARE CENTERS CORONA

Navy METCAL Program NIST

METCAL Technical Agent Acquisition Support

Acquisition

• Cal Rqmts Analysis • Cal Logistics

NPSL

In-Service

Test Equipment

Navy Cal Lab

Engineering • Procedures • Intervals • Standards • Training • Database Mgmt • Traceability • Technical Support

R&D

Maintenance

• Standards (State-of-the-art)

Mission Ensure that TMDE used to make measurements on Navy Systems is accurate and traceable to National Standards to reduce or eliminate the safety and cost impacts of wrong test decisions. Keeping America’s Navy #1 in the World

15

METCAL Products and Services WARFARE CENTERS CORONA

Metrology Engineering Products and Services

Keeping America’s Navy #1 in the World

16

METCAL Program Scope WARFARE CENTERS CORONA

Test Equipment Different Models: ~ 140K Models: 140K Different S/ ~ 1,850K S/N: 1,850K

Approved Calibration Standards (Different Model #s)

~ 1.85M

~5,000

Approved Calibration Procedures

Annual Engineering Inquiries/Actions

Annual Calibrations

~800K Calibration Laboratories

Shore 165 Ship 244

~5,000

~2,000 Keeping America’s Navy #1 in the World

409

17

Measurement Traceability WARFARE CENTERS CORONA

Weapon System (F/A-18)

National Institute of Standards & Technology

Support Equipment Custodians

Navy Primary Standards Lab

Keeping America’s Navy #1 in the World

Fleet “I” Level Calibration Labs

Depot “D” Level Calibration Labs

18

METCAL Acquisition Life Cycle WARFARE CENTERS CORONA

A

B

Concept & Technology Development • Metrology Guidance/ Consideration

C System Development & Demonstration

• Requirements Analyses

• Verify Contractor Traceability

• ILSMT • International/ National Measurement Technology Assessment

• LSA

• Technology Validation

• Logistics Document Reviews

• CMRS Development

• CMRS Guidance

• CSP Development • TEMP Inputs

• Metrology R&D • Evaluation Criteria

• SERD • CMRS Review, Approval • Calibration Support Plan Approval • Cal Standard Design, Develop, Procurement • Evaluation Criteria • Cal Procedure Development, Approval (Contractor, DoD)

IOC Production & Deployment

• Contractor’s Calibration System Evaluation (Follow-on) • Calibration Std ILS • Deploy Calibration Support

FOC Operational Support

• TMDE Calibration • Feedback Responses • Technical/Engrg/ Training Assists

• Interval Reliability Analysis • Measure Process Control

• Joint Service Support • Modification/Rqmt Planning

• Establish Cal Support Level

• Tech, Mgt, Pol. Docs • Facility (Incl Environ) Requirements • Training Requirements • Manpower, Workload Requirements • Interval Analyses

Keeping America’s Navy #1 in the World

19

Calibration Logistics Lifecycle WARFARE CENTERS CORONA

Maintenance Maintenance Requirement Requirement Identified Identified • Testing • Preventive Maintenance • Inspection • Repair

TMDE TMDE Requirement Requirement Identified Identified • Assist in identifying TMDE to support: – Testing – Preventive Maintenance – Inspection – Repair • Begin developing calibration support requirements

Calibration Calibration Requirements Requirements Analysis Analysis • Determine if current calibration support is in place • If not, determine proper calibration support for application • Identify R&D rqmts • Maintain calibration support • Identify replacement / replenishment for obsolete TMDE and Standards • Technical Support • Retire support

Keeping America’s Navy #1 in the World

Initial Initial Calibration Calibration Support Support

Calibration Calibration Support Support Sustainment Sustainment 20

Navy Metrology R&D Program WARFARE CENTERS CORONA

Provide Advanced Measurement Technology for Emerging Weapons Systems / Improve Effectiveness and Efficiency of Existing Measurement Systems

NAVY / MARINE CORPS

• Electrical

SEA AIR LAND SPACE

TECHNOLOGY AREAS • Physical/Mechanical

Calibration Standards & Traceability

• Electro-Optic • mm/Microwave • Analytic Metrology • Radiation • Chemical/Biological Defense

Keeping America’s Navy #1 in the World

21

Joint Service Metrology WARFARE CENTERS CORONA

Joint Chiefs – Joint Logistics Commanders OFFICES OFFICESOF OF PRIMARY PRIMARYRESPONSIBILITY RESPONSIBILITY (OPRs) (OPRs)

JOINT JOINTTECHNICAL TECHNICAL COORDINATING COORDINATINGGROUP GROUPon onCALIBRATION CALIBRATIONAND ANDMEASUREMENT MEASUREMENT TECHNOLOGY TECHNOLOGY (JTCG-CMT) (JTCG-CMT)

TMDE TMDEPROGRAM PROGRAM STANDARDS STANDARDS GROUP GROUP

CALIBRATION CALIBRATION COORDINATION COORDINATION GROUP GROUP(CCG) (CCG)

STANDARDS STANDARDSLAB LAB OPERATIONS OPERATIONS WORKING WORKINGGROUP GROUP

ANALYTICAL ANALYTICAL METROLOGY METROLOGYSUBGROUP SUBGROUP

ENGINEERING ENGINEERING WORKING WORKING GROUP GROUP

ELECTRICAL ELECTRICAL//ELECTRONIC ELECTRONIC SUBGROUP SUBGROUP

CHEMICAL CHEMICAL//BIOLOGICAL BIOLOGICAL SUBGROUP SUBGROUP

INTERSERVICE INTERSERVICE SUPPORT SUPPORT GROUP GROUP(IS) (IS)

CALIBRATION CALIBRATION PROCEDURES PROCEDURES GROUP GROUP

CHEM/BIO CHEM/BIO SUPPORT SUPPORT WORKING WORKINGGROUP GROUP

AUTOMATED AUTOMATED PROCESSES PROCESSES WORKING WORKINGGROUP GROUP

RF RF//MICRO MICRO//MM-WAVE MM-WAVE SUBGROUP SUBGROUP

PHYSICAL PHYSICAL//MECHANICAL MECHANICAL SUBGROUP SUBGROUP

Keeping America’s Navy #1 in the World

ELECTRO-OPTIC ELECTRO-OPTIC SUBGROUP SUBGROUP

RADIOLOGICAL RADIOLOGICAL SUBGROUP SUBGROUP 17 Jan22 2008

Industry/Professional Involvement WARFARE CENTERS CORONA

Represents the Navy in the Measurement Science Field National Conference of NATIONAL DEFENSE INDUSTRIAL ASSOCIATION

International National

Standards Laboratories International

DoD Navy METCAL JTCG-CMT

Joint Technical Coordinating Group Calibration and Measurement Technology

ASME

Keeping America’s Navy #1 in the World

23

False Accept/False Reject WARFARE CENTERS CORONA

Keeping America’s Navy #1 in the World

24

Calibration Testing WARFARE CENTERS CORONA

Calibration Tests

Test Instrument

Test Steps

ICP

• A Test Instrument is considered In Tolerance if it passes all the steps of an Instrument Calibration Procedure (ICP)

Keeping America’s Navy #1 in the World

25

True In-Tolerance WARFARE CENTERS CORONA

A Unit Under Test (UUT) is truly in tolerance if: Lower Tolerance Limit < UUT Bias < Upper Tolerance Limit

UUT Bias (True)

Lower Tolerance Limit (-L)

0

Upper Tolerance Limit (L)

The UUT Bias is unknown Keeping America’s Navy #1 in the World

26

True Out Of Tolerance WARFARE CENTERS CORONA

A Unit Under Test (UUT) is truly out of tolerance if: UUT Bias < Lower Tolerance Limit UUT Bias > Upper Tolerance Limit

or

UUT Bias (True)

Lower Tolerance Limit (-L)

0

Upper Tolerance Limit (L)

The UUT Bias is unknown Keeping America’s Navy #1 in the World

27

Observed In Tolerance WARFARE CENTERS CORONA

A Unit Under Test (UUT) is observed in tolerance if: Lower Tolerance Limit < Deviation < Upper Tolerance Limit

Deviation (Observed)

Lower Tolerance Limit (-L)

0

Upper Tolerance Limit (L)

The Deviation is the observed difference between the UUT and the CAL Keeping America’s Navy #1 in the World

28

False Accept WARFARE CENTERS CORONA

Deviation (Observed)

Calibration System Error

UUT Bias (True) Acceptance

-L

0

L

False Accept (FA): • The Deviation is observed in tolerance [ -L < Deviation < L ] • The UUT Bias is out of tolerance [ Bias > L or Bias < -L ] • The decision to accept the UUT is incorrect Keeping America’s Navy #1 in the World

29

Measurement Reliability WARFARE CENTERS CORONA

Measurement Reliability

•• Measurement MeasurementReliability Reliabilityisisthe theprobability probabilityof ofbeing beinginintolerance tolerance(the (the probability probabilityall allthe thecalibration calibrationprocedure proceduresteps stepspass). pass). •• Measurement MeasurementReliability Reliabilitydecreases decreaseswith withtime timeafter aftercalibration calibration(which (whichisis why whywe wecalibrate calibrateperiodically). periodically). 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 0

6

12

18

24

30

36

Time Since Calibration (Months) Keeping America’s Navy #1 in the World

30

WARFARE CENTERS

Calibration Intervals And Test Decisions

CORONA

• Good test decisions: –

Detect bad items and repair or replace them

–

Verify good items and return them to the Fleet

• Good test decisions require accurate test equipment • Accuracy in test equipment is maintained by periodic calibration –

TMDE accuracy, or measurement uncertainty degrades with time

–

TMDE “Out of Tolerance” (OOT) not obvious to user - insidious error

–

TMDE OOT can only be determined by calibration

–

Periodic calibration keeps measurement uncertainty at acceptable levels

• Accurate calibration intervals keep false accept risk (acceptance of bad test equipment) low

Keeping America’s Navy #1 in the World

31

Calibration Intervals WARFARE CENTERS CORONA

•• To Toobtain obtainaaCalibration CalibrationInterval, Interval,you youneed: need: –– AAMeasurement MeasurementReliability ReliabilityCurve Curve(Estimated (Estimatedusing usingCalibration CalibrationResults) Results) –– AAMeasurement MeasurementReliability ReliabilityTarget Target 100 %

Measurement Reliability vs. Calibration Interval

Measurement Reliability

90 % 80 % 70 % 60 % 50 % 40 % 30 % 20 % 10 % 0% 0

6

12

18

24

30

36

42

48

Calibration Interval (Months) Keeping America’s Navy #1 in the World

32

ISO 9001 – Calibration Requirements WARFARE CENTERS CORONA

7.6 Control of monitoring and measuring devices • Requires organization to assess/understand measurement needs • Requires that measuring equipment – be accurate to ensure product conformity to specification – be calibrated or verified at specified intervals, or prior to use, against measurement standards traceable to international or national measurement standards

Keeping America’s Navy #1 in the World

33

Z540-1 Withdrawal WARFARE CENTERS CORONA

Test & Measurement Equipment/Cal Lab Accreditation

1994

2005

2006

2007

ANSI/NCSL Z540.1

ANS/ISO/IEC 17025:2005

ANSI/NCSL Z540.1

Approved

Approved

ANSI/NCSL Z540.32006 Approved

Withdrawn

Z540.1 is replaced by Z540.3 and 17025

Keeping America’s Navy #1 in the World

34

9001, 17025, & Z540.3 Relationship WARFARE CENTERS CORONA

ENTERPRISE Quality System ISO 9001 Independent Calibration Lab Products / Services

ISO/IEC 17025

Measurement and Test Equipment ANSI/NCSL Z540.3

Internal Calibration Laboratory ISO/IEC 17025 (or Z540.3)

Keeping America’s Navy #1 in the World

35

Distribution of Z540.1 Requirements WARFARE CENTERS CORONA

•

17025

Z540.1 Part I – Cal Lab – Cal lab requirements – Cal lab competency demonstration (ISO G25)

•

–

–

General requirements for the competence of testing and calibration laboratories Calibration service component competency demonstration

Z540.3

Part II – Cal System – M&TE cal requirements – Cal system requirements

Keeping America’s Navy #1 in the World

– –

Cal service component technical requirements Optional use of ANS/ISO 17025 with SC 5.3 for Cal service component competency demonstration

– –

M&TE cal requirements Cal system requirements

36

Roles of New Standards WARFARE CENTERS CORONA

• ANSI/ISO/IEC 17025: – Competence of test and calibration laboratories – Demonstration requirements: • Able to operate a management system • Are technically competent • Able to generate technically valid results

Laboratory Competence Requirements

• ANSI/NCSL Z540.3 – Technical requirements for calibration of M&TE – System for control of the accuracy of M&TE: • Calibration system performance requirements • Calibration laboratory performance requirements • Link to use of 17025 accredited cal labs

METCAL Program Requirements

– ISO 17025 accredited cal lab “smart customer” Keeping America’s Navy #1 in the World

37

WARFARE CENTERS

Calibration Program Functions

CORONA

•

Calibration Requirements –

•

Calibration Procedure –

•

•

•

Development, Review, Approval

TMDE/Calibration Standards –

Inventory Control

–

Reliability, Interval Assessment

–

Specification, Evaluation, and Management

Calibration Laboratory –

Facility/Technical/Management Requirements

–

Assessment (Accreditation)

Calibration Data –

•

Determination, Assessment

Collection, Management, Assessment

Training –

Development / Assessment

•

Technical Support

•

Professional Society Involvement Keeping America’s Navy #1 in the World

38

So, Why Calibrate? WARFARE CENTERS CORONA

Test, Measurement, and Diagnostic Equipment Must Be … Supportable

Compatible

• Repair parts

• Requirements – Form-fit-function – Accuracy – Traceability – Repeatability

• Repair procedures

• Allowances

• Training

• Inventories

• ICPs • Standards

• Documentation

Affordable • Control proliferation • Prevent unauthorized substitution • Ensure adequate support

To operate and maintain safe and reliable systems Keeping America’s Navy #1 in the World

39