NOAA Local 3-Month Temperature Outlook (L3MTO)

Performance Evaluation Marina Timofeyeva1, Annette Hollingshead2, Scott Handel1, Melissa Ou1, Jonathan Gottschalck1, Matthew Menne3, and Claude Williams3 1 NOAA/NWS; 2 Wyle; 3 NOAA/NCDC Other Contributors: Dave Unger, Andrea Bair, and Jenna Meyers

CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO Facts

www.weather.gov/climate

• Produced in partnership: NCDC, CPC, CSD, Regional HQs and Local Offices • Official 3 Month T Outlooks downscaled to local level (~1200 U.S. sites) • Design based on research that identified barriers on forecast use • Includes multiple formats, text interpretations, and consistent help service CLIMATE SERVICE DIVISION / NWS / NOAA

Allan Murphy’s Lessons (1993) • Climate forecast long-term performance is absolutely necessary information for both producers and users • Reference* to Murphy’s “forecast goodness”: o CONSISTENCY: forecasts agree with forecaster’s true belief about the future weather [strictly proper] o QUALITY: correspondence between observations and forecasts [verification] o VALUE: increase or decrease in economic or other kind of value to someone as a result of using the forecast [decision theory] * Murphy, A.H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. and Forecasting 8, 281-293. CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Consistency 4-tier primary QC ensures forecast consistency: •

Each year an automated process kicks off annual calculations to update the L3MTO background data

•

Rigorous QC ensures: - All input data is current, continuous and serially complete - Input data, annual statistics and skill from one year to the next do not vary beyond acceptable thresholds - Scripts are working properly

•

In the event ANY portion of QC fails: Æautomated emails are sent to the appropriate parties Æcomputations halt Æno new data is pushed until errors are resolved CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Consistency QC1

Fail

Contact NCDC

Pass

Annual Quality Control: (4 Tiers)

QC2

Fail

Check regression equations

Fail

Check forecast equations, consult with CPC

Fail

Review scores, consult with CPC, CSD & NCDC

Pass

QC1. NCDC Homogenized data sanity check Normal distribution of temperature differences QC2. L3MTO Regression statistics check Significance & slope comparison

QC3 Pass

QC4 Pass

QC3. Archive category comparison Ensure station-CD long-term forecast consistency QC4. Skill score check Heidke Skill Score 10% deviation allowance

Diagnostic Report

Diagnostic Report

Push Data

End calculations

CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Consistency Jan, Feb Mar, Nov, Dec Oct, Apr

MEAN VARIANCE January ‐1.07 1.93 February ‐0.93 1.79 March ‐0.89 1.81 April ‐0.45 1.14 May ‐0.19 0.87 June ‐0.10 0.74 July ‐0.07 0.64 August ‐0.17 0.66 September ‐0.25 0.86 October ‐0.50 1.02 November ‐0.88 1.31 December ‐0.96 1.41

MEAN VARIANCE January 0.03 0.33 February 0.01 0.31 March 0.01 0.27 April 0.00 0.24 May ‐0.01 0.23 June 0.00 0.23 July ‐0.01 0.21 August 0.00 0.22 September ‐0.01 0.23 October 0.00 0.25 November 0.01 0.28 December 0.02 0.29

CLIMATE SERVICE DIVISION / NWS / NOAA

FLAG

PASS

L3MTO - Consistency 3MTO

L3MTO Consistency

Step 1: Enhanced category comparison - use matched sample archive years - sum occurrences where station and corresponding CD enhanced categories match - calculate %age that categories matched (i.e.: during FMA ,81% of the time categories matched for data set #1; 84% f or data set #2) Step 2: If ‘match’ percentages between the two data sets changed more than 10% Æ FLAG PASS Æ If FLAGs account for less than 10% of the series

2006 vs. 2009 CONSOLIDATED Season HardFlags Hard% SoftFlags FMA 0 0.00% 7 MAM 0 0.00% 11 AMJ 0 0.00% 22 MJJ 4 0.09% 49 JJA 6 0.13% 72 JAS 1 0.02% 48 ASO 3 0.06% 25 SON 1 0.02% 23 OND 1 0.02% 10 NDJ 1 0.02% 16 DJF 2 0.04% 6 JFM 0 0.00% 4

Soft% 0.15% 0.24% 0.48% 1.06% 1.56% 1.04% 0.54% 0.50% 0.22% 0.35% 0.13% 0.09%

Sample Station 2006 vs. 2009 CONSOLIDATED Season Old New %age FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ DJF JFM

80.99 58.16 73.05 53.19 64.54 67.38 76.6 88.65 80.14 75 92.14 95

CLIMATE SERVICE DIVISION / NWS / NOAA

83.8 60.28 73.05 56.74 64.54 68.09 76.6 88.65 80.85 75 92.14 95

2.81 2.12 0 3.55 0 0.71 0 0 0.71 0 0 0

L3MTO - Consistency

CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Quality L3MTO quality is measured by correspondence between forecasts and observations (verification) Three skill scores are utilized for a robust evaluation: 1. Heidke Skill Score (HSS): to assess L3MTO improvement of forecast over 1971-2000 climatology using 3-categorical forecast format (Pie Chart). 2. Continuous Rank Probability Skill Score (CRPSS): measures L3MTO accuracy, skill and resolution. CRPSS estimates both the average correspondence between the individual pairs of observations and forecasts and the accuracy of forecasts relative to the accuracy of forecasts produced by a standard method (POE Chart). 3. Reliability diagrams: measures the correspondence of the conditional mean observation and conditioning forecast, averaged over all forecasts; identifying the most long-term performance details for each site and forecast 3-month period than a single score.

CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Quality Verification Background 2 types of input archive data available: 1.Consolidated Archive Forecasts: output from Consolidated forecast model (CFS, CCA, SMLR, OCN) Æ Overconfident - forecasters tend to back off on consolidated guidance 2.Published Archive Forecasts: Actual outlooks that were issued in the past Æ Undercuts archive data by not including a portion of the consolidated guidance CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Quality Consolidated

Archived

HSS

CRPSS

CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Quality

CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Quality Continuous Rank Probability Skill Score Comparisons

CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Quality Reliability Skill Comparisons

Observed Relative Frequency

1

0.8

0.6

0.4

0.2

0 0

0.2

0.4

0.6

0.8

1

Forecast Probability

CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Value • “Pre-approved” feedback form • Periodic (first in June 2009, 18 months period) assessment of American Customer Satisfaction Index (ACSI), which is an economic indicator based on modeling of customer evaluations of the quality of products and services. o ACSI computations use a variation of Partial Least Squares (PLS) Regression to determine impacts when many different causes (i.e., quality components) simultaneously effect an outcome (e.g., Customer Satisfaction) o ACSI has a “proven relationship with: Customer spending, shareholder value, cash flow, business performance, and GDP growth” Reference to publications summarized by Russ Merz (2006), CFI Group CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Value Impact: 1.9 L3MTO

75

Navigation

77

Functionality

74

Comprehension

74

40

50

60

70

Primary Reason for Accessing: n=1433 Casual Browsing: 22% Decision-making: 8% Research: 6% Disseminating to others: 4% I am not familiar with this product/Do Not Use: 58% Other: 2%

80

NWS Climate Services Blended Score CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Value Usefulness Text Interpretation Naviagation Terms and Definitions Clarity Plot Type Tabs Organization Completeness Colors Titles Blend 70

72

74

76

78

ACSI

Pie Chart

T Range

CLIMATE SERVICE DIVISION / NWS / NOAA

PoE

80

L3MTO - Value Tercile Classes for Temperature

%

Feeling Associations: Cool, Neutral, Warm

25%

Computations: Lower Tercile, Middle Tercile, Upper Tercile

8%

Statistic in the middle Tercile: Below Median, Near Median, Above Median

31%

Climate Variable Measurement: Less than 50 degrees F, Between 50 degrees F and 65 degrees F, Greater than 65 degrees F

29%

Other

8%

Tercile Classes for Precipitation

%

Feeling Associations: Dry, Neutral, Wet

27%

Computations: Lower Tercile, Middle Tercile, Upper Tercile

7%

Statistic in the middle Tercile: Below Median, Near Median, Above Median

33%

Climate Variable Measurement: Less than 2”, Between 2” and 7”, Greater than 7”

26%

Other

6%

CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Value Reported L3MTO-based decision • • • • • • • • • • • • • • •

“Academic decisions and farm decisions. “ “Agricultural decisions. Outdoor activity planning. “ “Casual decision making. “ “Daily and extended forecasts and recommendations for area events. “ “Emergency severe winter weather shelter planning. “ “Forecast information, likelihood of a record event, historical context. “ “Forecast to animal diseases. “ “Hurricane evacuation probability. Severe storm impact on attending events. Vacation and travel. Securing property prior to severe weather. “ “Provide information for other decision makers. “ “Rainfall seasonality estimation in context with climate change. “ “Snowplowing when to start, how much snow we will get, how many pieces of equipment to use, etc. “ “Tropical cyclone activity seasonal forecast. “ “Use data for research, to answer media and customer (mainly public) questions, and to see what is going on with the weather. “ “Watershed management and water conservation. “ “Whether to plant or not, any type of outside work or play. “

CLIMATE SERVICE DIVISION / NWS / NOAA

L3MTO - Value Additional sites for Local 3-Month Temperature Outlook: Columbus, NE. De Witt, Iowa Kalamazoo, MI Lompoc, CA Please cover NE Oregon. South America locations

Useful if NWS offered a Local 3-Month Precipitation Outlook

%

Useful if NWS offered a local climate change product displaying rate of change in different climate variables

%

Yes

86%

Yes

84%

No

14%

No

16%

Useful if NWS offered a 3-Month Outlook of El Nino/La Nina impacts on local climate variables

%

Yes

88%

No

12%

CLIMATE SERVICE DIVISION / NWS / NOAA

Summary L3MTO employs all three Murphy’s measures • Consistency – monthly and annually: o Use of homogenized data o 4-tier primary Quality Controls and o 2-fold secondary Quality Controls

• Quality assessment (Verification) - annually: o Three score evaluating L3MTO different formats

• Value – two-year period: o Periodic surveys resulting in ACSI CLIMATE SERVICE DIVISION / NWS / NOAA