Acceptance Testing for Digital Land Mobile Radio Systems

IWCE 2010 - W07 Acceptance Testing Acceptance Testing for Digital Land Mobile Radio Systems Presented by: Jay Jacobsmeyer, P.E. Pericle Communicatio...
3 downloads 3 Views 993KB Size
IWCE 2010 - W07 Acceptance Testing

Acceptance Testing for Digital Land Mobile Radio Systems Presented by:

Jay Jacobsmeyer, P.E. Pericle Communications Company [email protected]

Karl Reardon, P.Eng. Planetworks Consulting [email protected]

So What’s New with Digital? Many more systems, software, and technology between the user and their expectations Users who expect the “same performance …. but better”

Analog Audio Circuits Expectations

Audio Vocoder IP Backhaul Routing / Transport

Audio Vocoder Data Decoding and Assembly Multi-Level Demodulator Analog Audio Circuits



Antennas / Radio Path



Antennas / Radio Path

Multi-Level Modulator


Data Packetization/Encoding


Audio Vocoder

Analog Audio Circuits

Analog Audio Circuits




“Radio Was Fun Until They Put Software In It” - Anonymous

Acceptance Test Elements „

RF Coverage


System Capacity




Resiliency /Fault Recovery


Features and Functionality


Management and Administration

RF Coverage Testing

Vocoder „

IMBE™ by DVSI is the P25 Vocoder – Model-based like virtually all vocoders


Optimized for Low Bit Rate Apps (< 4.8 kbps)


Breakdown – 20 ms frame (50 frames per second) – 88 vocoder bits – 56 error control bits (unequal protection) – 48 signaling bits – 192 bits x 50 frames/s = 9600 bps


BER Estimator Used for Frame-by-Frame Decisions – Smoothed estimator weighs past, present, future frames – Has implications for performance metrics

ATP Implications „

Four Choices for Each Vocoder Frame – – – –


Accept frame (even if some errors detected) Repeat frame Adaptively smooth (interpolate) frame Mute frame

BER is Helpful, But – Frame Error Rate is More to the Point – I.e., the same BER can occur for a variety of frame error distributions

Multi-Level Modulation „

C4FM is a Form of 4-ary FSK


2 Bits per Symbol, Four Possible Carriers 11



1.2 kHz



ATP Implications „

Modulation Fidelity is an Option for Testing


Can Be Alternative to BER – Is a measure of eye closure – Does not require taking a channel out of service – Tracks very well to BER


RF Coverage Performance „

RF Fundamentals Still Apply to Digital – Mean C/(I+N) in fading must still achieve minimum level


But Digital Introduces New Elements – – – – – – –


Ability of forward error correction to deal with error events Especially error events caused by delay spread Vocoder performance with bit errors / burst errors Vocoder performance in presence of audio noise Channel entry and exit criteria (roaming algorithms) Infrastructure data routing and assembly algorithms Backhaul network performance

These New Elements Can Create Poor Performance Despite Seemingly Adequate C/(I+N)

Characteristics of the Signal „

Multipath Fading Dominates the Mobile Radio Channel – – – –


λ Fade rate is a function of the doppler frequency, V/λ E.g., for 860 MHz at 60 mph, fade rate is roughly 75 Hz Amplitude is assumed to be Rayleigh-distributed Channel can be time-dispersive due to delay spread

Radio Specifications are Typically for Mean Signal – Ideally, corresponds to a particular delivered audio quality (DAQ) – Additional margin is needed to operate in fading – Delay spread performance often poorly addressed or ignored


E.g., Typical P25 Phase I Radio (C4FM): – 12.5 kHz channel – Static sensitivity = -118 dBm (5% BER) – Fading sensitivity = -105 dBm (DAQ = 3.4)

Rayleigh Fading (V = 30 mph, f = 850 MHz) Amplitude normalized to mean, dB






-40 0.00




Time in seconds




Multipath Delay Spread „

Simulcast Networks Aside, the Signal at the Antenna is Really Multiple Reflected Signals with Random Delays – Extent of problem varies with topography


Delay Spread Causes Bit Errors Even When Signal is Strong – Weak correlation with signal strength remains – Line-of-sight paths tend to have lower delay spread


Unlike Simulcast, the Delay Profile is Time-Varying – Adaptive equalizers are effective tool – No guarantee manufacturer is using one, however


I.e., Performance is Implementation-Dependent – And it is unlikely that the mfr. has tested for all possible environments

Coverage Test Prerequisites „

Complete All Radio Site Acceptance Tests – Including site noise, desense, and intermod tests


Verify Backhaul Network Routing and Stability – Backhaul BER, packet drop rates, jitter etc.


Verify Switch Configuration and Fucntionality


Verify Radio Configuration Parameters – Including roaming parameters, channel lists, site adjacency lists


Understand and Account for Localized Noise and Interference Sources

What to Measure?



Farthest from User Experience

RSS = Received signal strength BER = Bit error rate FER = Frame error rate (= block error rate) DAQ = Digital audio quality


DAQ Closest to User Experience

•Method •Signal





Test can be automated •+ Specialized test equipment not req’d •+ Aligns well with coverage prediction

Does not consider localized noise and interference environments •- Does not verify digital system components and overall system operation •- Far removed from user experience (i.e. may not be reflective of real voice quality) •- Typically only used to measure one path direction


Error Rate or •Frame Error Rate


Test can be automated •+ Good measure of C/(I+N) •+ Verifies some of the digital elements


Specialized test equipment required •- Does not verify all system elements •- Somewhat abstract from actual user experience •- Typically only used to measure one path direction




(DAQ) Testing

Reflects end user experience •+ Incorporates roaming impacts •+ Automated data collection •+ Specialized test equipment not req’d •+ Exercises all system elements


Data evaluation is manual , subjective and time consuming, or -- Automated using expensive software •- Difficult to ‘mine’ data to identify problem origins (should they exist)

Signal Strength „

Signal Strength to be Measured in All Cases – Adequate C/N is necessary, but not sufficient condition – Measured simultaneously with other parameters


Figure of Merit = Mean Signal Power in Fading – – – – –


Must be linear average, not log average At least 50 subsamples over at least 40 wavelengths Fast receiver needed (25 -100 readings per second) Receiver sensitivity should match or exceed user radio Also must have good selectivity & dynamic range

For More on This Subject – See Jacobsmeyer & Weimer, “How to Conduct a Drive Test Survey,” IWCE 2007

Bit Error Rate Testing (Or Frame Error Rate Testing) Remove RF Channel from Service and Place in Test Mode


– –

May be problematic for busy, operational systems Need to verify consistent interpretation of the test bit sequence and value of status bits to avoid baseline errors

Digital analyzer or Instrumented Radio Used for Drive Test


Consider any differences between analyzer or instrumented radios and the radios which will be used

Collect Data


– – –

Configure for sufficient settling time prior to each measurement to ensure valid BER tests Traditional sampling rates (> 40 wavelengths) for measurements in fading environments still apply At normal speeds, the above sampling period will collect sufficient bit samples to be statistically valid

Analyze Data


Use processes recommended in TSB-88 to segment and analyze test data

Voice Quality Testing 1. 2.

Select and Use an Exclusive Talkgroup Automated Voice Play and Recording Preferred – Duration consistent with a typical voice call – Harvard sentences are appropriate – Use both male and female voice recordings


Collect Data Uniformly – Collect at least one test recording per tile – Eliminate duplicates per tile by arbitrary rule


Score Data – Train listeners for consistent scoring – Score each recording per DAQ definition – Note: Audio scoring criteria should be agreed to prior to test execution as the TSB -88 definitions subject to interpretation

DAQ & MOS Definitions DAQ




DAQ 1.0 Unusable. Speech present but not

MOS 1 Bad. Very

DAQ 2.0 Understandable with considerable effort.

MOS 2 Poor.

DAQ 3.0 Speech understandable with slight effort.

MOS 3 Fair. Slightly

DAQ 3.4 Speech understandable with repetition

MOS 4 Good.


Frequent repetition due to noise/distortion. Occasional repetition required due to noise/distortion. only rarely required. Some noise/distortion.

DAQ 4.0 Speech easily understood. Occasional noise/distortion.


Annoying. annoying.

Impairments perceptible but not annoying.

MOS 5 Excellent.

Impairments imperceptible.

MOS = Mean Opinion Score. Note that repetitions are not part of definition. No correlation between DAQ and MOS is implied by this table.

Scoring Methods „

Even With Training, Some Variance Occurs


One Method that Works: – Decide on passing threshold (E.g., DAQ = 3.4) – Pick three scorers: » Customer operations » Customer technical » Vendor technical

– Score PASS if majority (2 of 3) score above threshold „

Decide on Definition of Repeats – Purely subjective? – Or actually allow for a second call at each tile, but how many repeats are allowed, 10%, 20%? – Or, avoid repeat problem by defining pass as DAQ 4.0


Problem: Manual Scoring is Labor-Intensive – Training is required – Variability of scores is tough to eliminate


Possible Solution = PESQ – Perceptual Evaluation of Speech Quality (PESQ) – Standardized as ITU-T Recommendation P.862 – Commercial software available (appx. $12k/license)


Issue: PESQ Scores Do Not Match DAQ Scores – Option 1: Agree on mapping from PESQ to DAQ – Option 2: Abandon DAQ and use PESQ score threshold

Audio File Samples

Sample 1

Sample 2

Sample 3

Pass/Fail Criteria And Use of Statistics

Estimate for SAR (Service Area Reliability)

SAR(%) =

Tp Tt


where Tp is the number of tiles passed Tt is the total number of tiles

Minimum Number of Samples


2 zα / 2ofp(1 − 2



Where p is the value of the SAR, zα/2 is the argument of the unit normal distribution for a confidence of 1-α and d is one-half of the confidence interval [5]. Substitute zα for “greater than” test. This expression results from the normal approximation to the binomial distribution for large n.

Arguments of Unit Normal

Confidence Level 90%

Zα 1.28

Zα/2 1.64







Example (d = .02, p = 95%)

Confidence Level

Greater Than Test

Confidence Window Test


Zα 1.28

n 195

Zα/2 1.64

n 321











Greater Than test for these parameters means that if the measured SAR is at least 97% and we collect at least 643 samples, then we are at least 99% confident that the actual SAR is above 95%.

Some Purchasing Pitfalls „

Pitfall #1 - Failure to Specify Measurable Performance Standards


Pitfall #2 - Failure to Specify Standards that Measure Digital Voice Quality Directly – Signal strength is not enough – Digital radio requires subjective measure of voice quality


Pitfall #3 - Failure to Specify the Statistical Test – “Greater Than” or “Confidence Window”? – Number of samples required?


Pitfall #4 - Failure to Include Test Plan in the Contract – Otherwise, endless finger-pointing when things go wrong – Test for the actual user operating states

Summary „

An Acceptance Test Plan Must – – – –


Use measurable standards, Specify precisely how each standard is to be measured, Include the statistical test to be used, and Be part of the system contract

Clarify the Standards Where Necessary

Other Acceptance Test Elements

System Capacity „

System Capacity Goes Beyond the Radio Link


Capacity Often Difficult to Measure Directly – – –

Sheer volume of messages Unavailability of load generators Distribution of load sources


Scope Dependent on Whether System Is Delivered As Components, Subsystems, or Turnkey


Paper Analysis and Discrete System Tests Are Often the Only Practical Method

System Capacity Specs. 1.

Define the Voice and Data Usage Models – – –

Use busy hour traffic estimates Consider fault / system recovery message loads Watch for coordinated traffic events (shift starts, etc) which may be outside the normal busy hour


Define the Call Models (Group, Individual, Interconnect, Etc.) and Members


Define the Number and Approximate Locations


Determine Loads Per Channel, Site, Backhaul Link, Router, and System Switch

System Capacity Testing „

Distribute Between Factory and System Acceptance Tests


Use Commercial Network Test Tools to Stress Backhaul Components


Use Development Simulators (if available) to Test System Load Capacity


Appropriate Suite of Tests are Unique to Each System

System Reliability/ Resiliency „

Consider Interaction Between All Systems – Momentary packet loss in one subsystem can cause lengthy reset in another – Identification of error situations which cause ‘thrashing’ (two subsystems with independent retransmission algorithms) – Identification of race conditions


Acceptance Tests Should Be Developed Through Identification and Subsequent Test of Each of the Failure Modes

Features and Functionality „

Centralized Conformance Testing for Open Standards Assists With This Area


However, Conformance Tests Still Not at Stage Where These Tests Can Be Relegated to Others – Subtle differences between operation in different frequency bands – Different manufacturer interpretations of specifications – Manufacturers opt to implement different options within the specifications

Management & Administration „

Consider Information You Require to Support and Manage the Network – Channel, site, link and system traffic, loading and blocking statistics and reports – Outage and error reports


Acceptance Procedures Should Include Generation of Applicable Reports Using Data Generated During Execution of Other ATP’s


Consider Need to Consolidate Reports and Data From Disparate Subsystems

Final Thought The view is nice but you really should only have to go out once.


Early Land Mobile Radio - 1924 Source: Bell Labs,

References [1] W.C.Y. Lee, Mobile Communications Design Fundamentals, 2nd Ed., Wiley, 1993. [2] W.C.Y. Lee, “Estimate of Local Average Power of a Mobile Radio Signal,” IEEE Transactions on Vehicular Technology, February, 1985, pp. 22-27. [3] M. M. Peritsky, “Statistical Estimation of Mean Signal Strength in a Rayleigh Fading Environment,” IEEE Transactions on Communications, November, 1973, pp. 1207-1213. [4] G. C. Hess, Land Mobile Radio System Engineering, Artech House, 1993. [5] R. J. Larsen, M. L. Marx, An Introduction to Mathematical Statistics and its Applications, Prentice-Hall, 1986, pp. 281. [6] TIA-TSB-88-C, “Wireless Communications Systems – Performance in Noise and Interference-Limited Situations, etc.” 4/29/2008.

Points of Contact Jay M. Jacobsmeyer, P.E. Pericle Communications Company 1910 Vindicator Drive, Suite 100 Colorado Springs, CO 80919 (719) 548-1040 Fax: (719) 548-1211 [email protected] Karl Reardon, P.Eng. Planetworks Consulting Suite 650 - 220 Cambie Street Vancouver, BC, Canada V6B 2M9 Main: +1-604-638-3000 Fax: +1-604-638-3001 [email protected]

Suggest Documents