Voice Gateway

T H E TO L LY G R O U P No. 201102 March 2001 Lucent Technologies Lucent MAX TNT MultiVoice versus Cisco Systems, Inc. AS5300/Voice Gatewa...
Author: Avice Randall
1 downloads 0 Views 77KB Size
T

H

E

TO L LY

G

R

O

U

P

No. 201102

March 2001

Lucent Technologies

Lucent MAX TNT MultiVoice versus Cisco Systems, Inc. AS5300/Voice Gateway

Test Summary

VoIP Gateway Competitive Evaluation Premise: Carriers looking to implement Voice over IP (VoIP) solutions in their network need to ensure that the desired system will maintain high call completion rates and that these rates are consistent and are not lowered after any length of time. The VoIP system should also offer acceptable International Telecommunications Union (ITU) Perceptual Speech Quality Measurement (PSQM) scores and the flexibility to service every port on a network.

L

ucent Technologies, Inc. commissioned The Tolly Group to evaluate its MultiVoice VoIP Solution that includes MAX TNT WAN Access Switches, Release 9.0.0, and a Lucent MultiVoice Access Manager Release 3.1.0. Lucent also commissioned The Tolly Group to test Cisco Systems, Inc. AS5300/Voice Gateway devices, IOS version 12.1(5)T with a Cisco 3640 Gatekeeper version 12.1 (5a). Competitive tests were conducted to determine the average percent of call completion every six hours for a total of up to 72 hours across both the Lucent and the Cisco gateways. Engineers also conducted tests measuring the average PSQM as defined

Test Highlights m Maintains a 99.59% overall call completion rate in tests lasting more than 48 hours compared to the Cisco AS5300s with average call completion rates that begin to fall at 18 hours, and continue to fall for up to 48 hours m Averages International Telecommunications Union PSQM scores well under 1.0 for G.711 and 2.0 for G.729a m Offers Universal Port support for voice, ISDN, transparent fax, real-time fax, and analog modem m Supports Multiple Logical Gateways allowing one device to handle a variety of different call types by the ITU for each device when using G.711, G.729 or G.729a vocoders. The Tolly Group verified via pass/fail testing, that the Lucent MultiVoice Solution was capable of providing a gateway for any service to any port (universal port) and multiple logical gateways by demonstrating four different call types via one device. Testing was performed from December 2000 to January 2001.

Test results show that the Lucent MultiVoice Solution demonstrates 99% call completion rate across a 72-hour testing period. Cisco's AS5300 gateway also had a 99% call completion rate for approximately 18 hours but then its call completion percentage began to drop, and continued to drop for the next 54 hours. In PSQM tests, Lucent demonstrated lower (better voice quality) average PSQM scores than Cisco when using a G.711 voice

Percent of average call completion rate

Voice Over IP Gateway: Call Completion Rate, Average of Total Test Duration (6 Hour Intervals) As Reported by Empirix Hammer

120 100 80 60 40 20 0 6

12

18

24

30

36

42

48

54

60

66

72

Test duration (in hours) Lucent Source: The Tolly Group, March 2001

© 2001 The Tolly Group

Cisco Figure 1

Page 1

The Tolly Group

Lucent Technologies

MAX TNT MultiVoice

Voice Over IP Gateway Call Completion Rate,* Average of Total Test Duration (6 Hour Intervals) As Reported by Empirix Hammer Hours Hours Hours Hours Hours Hours Hours Hours Hours Hours Hours Hours 0-6 6-12 12-18 18-24 24-30 30-36 36-42 42-48 48-54 54-60 60-66 66-72 Lucent Test 1

100

100

100

100

100

99.5

100

100

97.9

97.9

97.7

98.2

Lucent Test 2

100

99.4

100

100

100

100

100

100

100

100

99.7

99.8

Cisco Test 1

99.9

99.8

99.9

99.7

99.7

99.9

99.9

99.9

94.5

54.7

51.2

50.3

Cisco Test 2

99.9

99.9

99.9

48.3

41.3

41.3

38.8

38.9

37.8

36.5

36.3

36.7

Cisco Test 3

99.9

99.9

99.9

99.9

99.9

99.9

96.1

52.6

N/A

N/A

N/A

N/A

Test 3 was executed for a duration of 48 hours due to testing schedule constraints * Percentage of calls launched that are successfully completed. Source: The Tolly Group, March 2001

Finally, upon Lucent's request, Tolly engineers verified that the Lucent MultiVoice Solution had the flexibility to offer any service dynamically on any port–meaning that it can implement successful call completion from a single port for voice, ISDN, transparent fax, real-time fax and analog modem. Tolly engineers also verified that the MultiVoice Solution supported Multiple Logical Gateways allowing it to receive the following four call types on a single unit: two-stage dialing with Automatic Number Identification (ANI); two-stage dialing with Personal Identification Number (PIN); single-stage dialing with ANI; and single-stage dialing.

Results Call Completion Rate The Tolly Group verified that when the Lucent MultiVoice Solution was tested for call completion rate for a period of up to 72 hours, it consistently showed 99% of average call completion across the entire testing period.

drop precipitously from 99% to as low as 36% by the end of the test. See figures 1 and 2; and table 1.

Voice Quality Measurement Tolly engineers tested the voice quality measurement of both the Lucent and the Cisco VoIP gateways. Each device was tested using voice code algorithms as defined by the ITU. The PSQM is from the ITU-T P.861 recommendation for an objective method of estimating the subjective quality of voice-band speech

codecs. The ITU PSQM scale runs from 0 to 6.5 with lower being better. Each gateway tested G.711 vocoders and the Lucent gateway tested G.729a vocoder, while the Cisco gateway tested using G.729 vocoder. Tests were conducted to measure the average PSQM across a VoIP gateway using Empirix Hammer VoIP test system. Results showed that when testing voice quality using a G.711 voice coder, the Lucent MultiVoice Solution showed an average PSQM of 0.52, 0.42, 0.49, and 0.63 for a man, woman, boy, and girl,

Voice Over IP Gateway Call Completion Rate, Average of Total Test Duration (minimum of 48 hours) As Reported by Empirix Hammer 99.59%

100% Percent of average call completion rate

coder, but showed slightly higher (worse voice quality) average PSQM scores using G.729a voice coder compared to its competitor when using G.729 voice coder. Slightly higher PSQM are expected for G.729a versus G.729 because G.729a is more efficient on MIPS at the expense of a slightly higher (i.e., worse) PSQM score.

Table 1

79.63%

80% 60% 40% 20% 0%

In the same test, the Cisco AS5300 gateway demonstrates the same average call completion rate for the first 18 hours, but then call completion rates

© 2001 The Tolly Group

Lucent

Cisco

Systems under test Source: The Tolly Group, March 2001

Figure 2

Page 2

The Tolly Group

Lucent Technologies

respectively. In the same set of tests, the Cisco AS5300 gateway demonstrated an average PSQM of 0.61, 0.45, 0.68, and 0.82 for a man, woman, boy, and girl, respectively. When testing the voice quality measurements using a G.729a voice coder, the Lucent MultiVoice Solution demonstrated an average of 2.07, 1.70, 2.14, and 2.18 for a man, woman, boy, and girl, respectively. The Cisco results using the G.729 vocoder showed average PSQM scores of 1.64, 1.44, 1.73, and 1.87 for a man, woman, boy, and girl, respectively. See figure 3. All scores were well below the generally acceptable PSQM voice quality score of 3.0 as defined by the ITU.

Verification of Universal Port Upon Lucent's request, The Tolly Group engineers verified that the Lucent MultiVoice Solution was capable of offering any service to any port. Using an ISDN modem, an analog modem, fax machines (to test transparent and real-time fax transmission) and analog telephones, Tolly engineers used a single port on the MAX TNT and validated that it can handle successful call completion without reconfiguration. See figure 4.

Multiple Logical Gateways Lucent also requested that The Tolly Group validate the Lucent MultiVoice Solution’s capability to implement Multiple Logical Gateways. Using Empirix Hammer, Tolly engineers generated the following four different call scripts: two-stage dialing with Automatic Number Identification (ANI); two-stage dialing with Personal Identification Number (PIN); single-stage dialing with ANI; and single-stage dialing. Results demonstrate that a single Lucent MAX TNT Access Switch with MultiVoice Access Manager simultaneously supported each of the four different call types tested. See figure 5.

Analysis How a product performs over extended periods of time is an area of concern for

© 2001 The Tolly Group

those selecting a carrier-class solution. Tests demonstrate that although both products were comparable in performance characteristics during three-hour test durations, a large variation in performance occurs during longer duration tests. The Tolly Group found that the two products exhibit extremely different results under shortand long-term testing. When tests ran in excess of 48 hours, the Cisco device reached a point at which failed calls exceeded the successful calls, thus driving the call completion rate (current 6 hour average) down as low as 36.3% during one of the tests. In order to validate that the degraded call completion rates were not attributable to the test tool, at completion of the 72-hour duration tests, the Cisco device was rebooted and a five-minute test was executed. During this five-minute test, 100% call completion rates were restored. The PSQM is from the ITU-T P.861 recommendation for an objective method of estimating the subjective quality of voice-band speech codecs. The ITU PSQM scale runs from 0 to 6.5 with lower being better. Tests demonstrate that the average PSQM scores for the Lucent MultiVoice Solution were lower than its competitor when using the G.711 uncompressed voice codec. This demonstrates that the Lucent gateway may represent better voice quality. However, when testing with the G.729a voice coder on the Lucent MultiVoice Solution–and testing the G.729 voice coder on the Cisco AS5300 gateway–the Lucent solution showed a slightly higher PSQM, demonstrating that the voice quality may be slightly worse. G.729a voice codec uses a more efficient algorithm for coding/decoding voice at the expense of slightly degraded PSQM scores–which are not discernible to the human ear. Regardless, all PSQM scores were lower than 3.0 thus generally accepted as being business-quality. By testing man, woman, boy and girl voices, results showed how each system, with vocoders of different standards, could handle a range of pitch frequencies. Testing demonstrates that the Lucent MultiVoice Solution was capable of successfully performing different types of calls over the same physical T1 card and possibly over the same DSP. With the singular configuration of the MAX TNT, each call was completed

MAX TNT MultiVoice Lucent Technologies, Inc. Lucent MultiVoice VoIP Solution Competitive Evaluation

Lucent Technologies, Inc. Lucent MultiVoice VoIP Solution Product Specifications* MultiVoiceTM VoIP services: m Retail · Calling card · 1+ long distance · 1010 Dial around · PC-to-phone m Wholesale toll-tandem replacement m Clearinghouse m Outsourced business services · Web-to-ACD · PBX trunk replacement

Data dial-up services: m m m m

Retail ISP Internet call diversion Port wholesale Outsourced VPN

For more information contact: Lucent Technologies Phone: (888) 426-2252 E-mail: [email protected] URL: http://www.lucent.com/ins/ products/multivoice/multivoiceformaxtnt.html URL: http://www.lucent.com/ins/ products/maxtnt/index.html *Vendor-supplied information not verified by The Tolly Group

Page 3

The Tolly Group

Lucent Technologies

MAX TNT MultiVoice

Voice Over IP Gateway: Voice Quality Measurement using G.711, G.729 and G.729a Vocoders As Reported by Empirix Hammer 3.0

Average PSQM

2.5 2.14 2.18

2.07

2.0

1.70

1.5

1.73

1.64

1.87

1.44

1.0 0.5

0.52

0.42

0.49

0.63

0.61

0.68

0.82

0.45

0.0 Lucent G.711

Cisco G.711 Man

Lucent G.729a Woman

Boy

Source: The Tolly Group, March 2001

successfully by the gatekeeper based on the DNIS (Dialed Number Identification Service). It is with this information that the gatekeeper selects the appropriate profile to apply to the call being made. During tests, a digital ISDN call was active throughout voice, analog modem, transparent and real-time fax calls. With this design, an end-user is permitted the flexibility to handle a multitude of different call types without the addition of expensive hardware. Service providers that are looking for a solution that offers flexibility to their product offerings are concerned that no additional hardware will be required. There is the up front capital investment that needs to be considered, but additional hardware means additional real estate. Renting space in a central switching office comes at a premium price. Traditionally, service providers had to purchase four separate platforms to offer four different types of service, but the Multiple Logical Gateway (MLG) feature allows a single device to partition off different types of service. With MLG, the Lucent MAX TNT was capable of applying four different call options to all inbound calls simultaneously.

Test Configuration and Methodology Two of each of the devices under test were configured in a test bed environment

© 2001 The Tolly Group

Cisco G.729

Girl Figure 3

that included a gatekeeper and a trafficgenerating device. Lucent Technologies, Inc. tested two Lucent MAX TNT WAN Access Switches, both 16-slot chassis', model TNT-AC-H, Release 9.0.0. Each device was connected to a 666-MHz Pentium III Dell OptiPlex GX110 with 128 Mbytes of RAM running Lucent MultiVoice Access Manager Release 3.1.0. The operating platform was also running Microsoft Corp. Windows NT Workstation 4.0 Service Pack 5. The PC was equipped with a 3Com Corp. EtherLink 10/100 PCI NIC (3C905C-TX) and was running Netscape Communicator 4.73 Web browser. Also under test were two Cisco Systems, Inc. AS5300/Voice Gateway devices, both IOS version 12.1(5)T. Both devices connected to a Cisco 3640 Gatekeeper version 12.1(5a). For testing, the systems under test connected to an Empirix Hammer VoIP test system via four T1 connections using up to 96 ports. See figure 6. For universal port testing, a Lucent/Ascend Pipeline ISDN model P75 ISBRI was used along with an Intel/Xircom CardBus Ethernet 100+ Modem 56, an analog modem, running on a 500-MHz Pentium III IBM ThinkPad with 128 Mbytes of RAM running Windows NT Workstation 4.0 Service Pack 6.

For testing, Tolly engineers used the following two fax machines: a HewlettPackard Co. OfficeJet and a Brother MFC 1970MC. A General Electric 2-9316C analog phone line was also used for these tests. To test the performance of the gateways for percent of call completion on a full-duplex Fast Ethernet IP backbone, Tolly engineers configured the Hammer test tool to load test up to 96 ports on the device under test. The Hammer device established a connection, engineers waited for the configured pause and repeated the test. Engineers ran tests to benchmark the average calls per second; and the call completion rate of both the Lucent and Cisco gateways under test. The duration of the test was a total of 72 hours and engineers ran three iterations and averaged the results. To verify call completion, the Hammer test tool line is seized, a call is placed, and a far gateway answers. The far gateway plays a prompt while the near gateway listens for prompt. There is a pause of three seconds. The near gateway plays a prompt and the far gateway listens for a prompt. The far gateway releases the call and the near gateway waits for a disconnection. The script then prepares to run again. If anything fails (as the script knows what to expect next and within given time limits) the call is counted as failed. If

Page 4

The Tolly Group

Lucent Technologies

everything occurs without failure, the call is counted as complete.

MAX TNT MultiVoice

Voice Over IP Gateway: Universal Port Verification of the Lucent MultiVoice Solution

In order to measure voice quality scores, Tolly engineers configured the Hammer test tool to perform voice quality measurements on the systems under test. Engineers configured the Hammer Test Editor for the following steps: establish a connection; play a voice file of a man, woman, boy or girl; end the connection; and repeat. The Hammer tool then made comparisons and issued PSQM value. In order to validate the Lucent MultiVoice Solution universal port capabilities, Tolly engineers configured the device for ISDN, analog modem, VoIP, transparent fax, and real-time fax on any port. Using a Lucent/Ascend Pipeline modem, a connection was established on one of the channels. Without disrupting or disconnecting the ISDN connection, a connection was established from an analog modem. With two channels committed, a VoIP call was placed on the MAX TNT and finally, a transparent fax was sent. Regardless of the voice codec selected, a transparent fax is handled the same as a standard voice packet. Engineers repeated the test sending a real-time fax while the three other call types were in progress. A real-time fax is derived from the ITU T.38 standard where the fax does not go through the voice coding process. Rather, the inbound fax tones go through a demodulation process where they are stripped down to bit form and are forwardedacross the IP network to the gateway on the far side. This gateway then encodes the bits to turn them back to fax tone and the fax process is completed. In order to verify that the MultiVoice Solution was capable of supporting Multiple Logical Gateways, engineers configured it for specific call types. Using an Empirix Hammer, Tolly engineers configured scripts for the following call types: two-stage dialing with Automatic Number Identification (ANI); two-stage dialing with Personal Identification Number (PIN); single-stage dialing with ANI; and finally, single-stage dialing. All four scripts were launched concurrently and call completion was verified.

Voice

Ö

ISDN

FaxÖ Transparent

A single port can handle different inbound call types

Ö

FaxReal time

Ö

Analog Modem

Ö =PASS

Source: The Tolly Group, March 2001

Figure 4

Voice Over IP Gateway: Universal Port Verification of the Lucent MultiVoice Solution

Ö

Two-Stage Dialing with ANI ANI-Automatic Number Identification

Ö

Two-Stage Dialing with PIN PIN-Personal Identification Number

Ö Single-Stage Dialing with ANI A single port can handle different inbound call types

Ö

Single-Stage Dialing

Ö =PASS Source: The Tolly Group, March 2001

© 2001 The Tolly Group

Ö

Figure 5

Page 5

The Tolly Group

Lucent Technologies

MAX TNT MultiVoice

Equipment Acquisition and Support

Test Bed Diagram Lucent MultiVoice Access Manager/Cisco 3640 Gatekeeper

The Tolly Group contacted executives at Cisco Systems, Inc. and invited them to provide a higher level of support than available through normal channels. Cisco representatives did not respond to several contact attempts.

Gatekeeper Up to 96 ports

Up to 96 ports

Lucent MAX TNT WAN Access Switches or Cisco AS5300/Voice Gateways

Up to 4 T1's

Up to 4 T1's Empirix Hammer VoIP test system Source: The Tolly Group, March 2001

Figure 6

The Tolly Group verified product release levels and shared test configurations with Cisco executives in order to give them an opportunity to optimize their devices for the tests. Cisco did not respond. However, Cisco standard support was utilized through standard support contracts that Lucent had in place. Results were shared with Cisco executives who neither acknowledged, nor disputed, their accuracy. For a more complete understanding of the interaction between The Tolly Group and Cisco Systems, Inc. check out the Technical Support Diary for Competitive Products Tested posted on The Tolly Group's World Wide Web site at http://www.tolly.com (see document 201102).

The Tolly Group gratefully acknowledges the providers of test equipment used in this project. Vendor Empirix

Product Hammer VoIP Since its inception, The Tolly Group has produced highquality tests that meet three overarching criteria: All tests are objective, fully documented and repeatable. We endeavor to provide complete disclosure of information concerning individual product tests, and multiparty competitive product evaluations.

As an independent organization, The Tolly Group does not accept retainer contracts from vendors, nor does it endorse products or suppliers. This open and honest environment assures vendors they are treated fairly, and with the necessary care to guarantee all parties that the results of these tests are accurate and valid. The Tolly Group has codified this into the Fair Testing Charter, which may be viewed at http://www.tolly.com.

Web address http://www.empirix.com

Project Profile Sponsor: Lucent Technologies, Inc. Document number: 201102 Product Class: VoIP gateway Products under test: · Lucent MAX TNT Access Switch Release 9.0.0 · Lucent MultiVoice Access Manager Release 3.1.0. · Cisco AS5300/Voice Gateway Version 12.1(5)T · Cisco 3640 Gatekeeper Version 12.1(5a) Testing window: November 2000 through January 2001 Additional information available: · Technical Support Diary · Configuration Files For more information on this document, or other services offered by The Tolly Group, visit our World Wide Web site at http://www.tolly.com, send E-mail to [email protected], call (800) 933-1699 or (732) 528-3300.

Internetworking technology is an area of rapid growth and constant change. The Tolly Group conducts engineering-caliber testing in an effort to provide the internetworking industry with valuable information on current products and technology. While great care is taken to assure utmost accuracy, mistakes can occur. In no event shall The Tolly Group be liable for damages of any kind including direct, indirect, special, incidental, and consequential damages which may result from the use of information contained in this document. All trademarks are the property of their respective owners. The Tolly Group doc. 201102 rev. kco 15 March 01

© 2001 The Tolly Group

Page 6