Indoor Baseline Test and Analysis Conducted on August 30, 2013 Washington State Convention & Trade Center ZK Services, LLC 4980 S. Alma School Rd. Suite 2-421 Chandler, AZ 85248 www.zkservices.net 1-650-279-2403 October 20, 2013 October 20, 2013
1
Table of Contents 1.0
Executive Summary .................................................................................................1 1.1 1.2 1.3 1.4 1.5
Operators Measured .............................................................................................................1 Areas Measured ...................................................................................................................1 Coverage ..............................................................................................................................2 Capacity ...............................................................................................................................2 Data Throughput ..................................................................................................................2
2.0
Introduction..............................................................................................................3
3.0
Scope........................................................................................................................3
4.0
Testing Plan..............................................................................................................4
5.0
Washington State Convention Center and Conference Center ................................4
6.0
Test Setup.................................................................................................................5 6.1 6.2 6.3 6.4
7.0
Data Analysis ...........................................................................................................9 7.1 7.2
8.0
Scanner Setup ......................................................................................................................6 Device Setup ........................................................................................................................7 Floor Plan Setup...................................................................................................................8 Post-Processing ....................................................................................................................9
Throughput...........................................................................................................................9 7.1.1 Throughput Plots .................................................................................................10 Coverage ............................................................................................................................14 7.2.1 Coverage Plots.....................................................................................................14 7.2.2 Conference Center ...............................................................................................21
Conclusion .............................................................................................................23
Table of Contents
October 20, 2013
1
ZK Services, LLC
Washington State Convention Center Test and Analysis
1.0 Executive Summary Testing of the cellular network data performance and coverage was done during the live event Penny Arcade eXpo (PAX) in Seattle on August 30, 2013, the first day of the show. Data throughput tests along with scanning data was collected in the Washington State Convention and Conference Centers for 4 major wireless operators Verizon, AT&T, TMobile and Sprint. The ZK-MPS/B product manufactured by ZK Celltest was used to collect the data.
1.1 Operators Measured The top four Operators in the market were measured for coverage and 3G/4G data performance.
1.2 Areas Measured The following areas were measured: • 1st Floor - Shops & Services • 2nd Floor - Meeting Rooms • 3rd Floor - Meeting Rooms • 4th Floor - Convention • 6th Floor - Ballrooms • Conference Center - Levels LL through L3
October 20, 2013
1
ZK Services, LLC
Washington State Convention Center Test and Analysis
1.3 Coverage Signal levels were measured for all the Operators for 2G, 3G and 4G technologies using both the connected handset and the internal scanners. Coverage was poor for all technologies and frequencies measured. 2G/3G coverage was slightly better than 4G coverage which was extremely poor. Based on our understanding there is no internal DAS and thus the buildings are served by external macro sites. Verizon did install a mobile cell site near the sky bridge and it helped somewhat but marginally.
1.4 Capacity We did not have a complete list of frequency channels of all the Operators so we are unable to make accurate assumptions in terms of the available total spectrum per Operator. However, based on the data we can make a fair assumption that the cellsites are getting overwhelmed. The data throughput rates are very poor. Obtaining a data channel is difficult. There are not enough sectors to handle the traffic. There were over 70,000 people at the event with about 250,000 SF of exhibit space which is equivalent to an average size NFL football stadium on gameday. The macro cells outside the buildings are not designed for, nor could they be enhanced to handle this amount of demand in such a small condensed space.
1.5 Data Throughput Data throughput tests were performed on all four Operators. In most areas it was difficult to obtain a data channel. When a data channel was obtained the throughput was extremely low which would make it unusable from the user perspective.
October 20, 2013
2
ZK Services, LLC
Washington State Convention Center Test and Analysis
2.0 Introduction Wireless coverage and performance indoors is becoming increasingly demanding by consumers. It becoming one of the top considerations in choosing where to stay and where to have an event. Convention owners are putting greater emphasis on enhancing and maintaining the wireless experience by installing Distributed Antenna Systems indoors that extend the coverage and capacity of the wireless network inside their facility. A convention that is jammed with thousands of people in a condensed space simply cannot be served effectively by the outdoor network. Highly condensed areas of high demand require a specific indoor network design.
FIGURE 1. Washington State Convention Center
3.0 Scope This paper focuses on measuring and analyzing the coverage and performance of the top four Seattle wireless Operators inside the Washington State Convention and Conference Centers. The buildings do not have an existing DAS. All network traffic is handled by the outdoor macro network and temporary mobile cellsites during events on an ad hoc basis. In determining whether to purchase and install a DAS the building management asked ZK Services to conduct a survey during a major event in order to determine a baseline of userperceived performance of the four major wireless Operators. The test was conducted on August 30, 2013 during the first day of the Penny Arcade eXpo (PAX). This event was attended by over 70,000 of which approximately 90% of them
October 20, 2013
3
ZK Services, LLC
Washington State Convention Center Test and Analysis
were on the 4th level halls (~200,000 SF) and 6th level halls (~40,000 SF). That’s about 63,000 people in 240,000 SF of space. This is similar to an NFL football stadium.
4.0 Testing Plan ZK Services engineers used the ZK Celltest ZK-MPS/B product to walk the convention and conference buildings and measure the coverage, quality and performance of operators AT&T, Verizon, T-Mobile and Sprint. The test plan consisted of phones running data transfers and scanners scanning the Operators’ frequencies. Areas walked include: • 1st Floor - Shops & Services • 2nd Floor - Meeting Rooms • 3rd Floor - Meeting Rooms • 4th Floor - Convention • 6th Floor - Ballrooms • Conference Center - Levels LL through L3
5.0 Washington State Convention Center and Conference Center The Washington State Convention Center is a 344,000 SF facility with over 205,700 SF of dedicated heavy-load exhibit space divisible into 6 halls, 45,000 SF of carpeted ballroom space divisible into 4 halls, 57,000 SF of meeting space divisible into 61 versatile meeting rooms along with retail shops, outdoor gathering areas and restaurants. The adjoining Conference Center has 71,000 SF of meeting, exhibit, banquet and pre-function space. Place. It is located in the heart of downtown Seattle.
October 20, 2013
4
ZK Services, LLC
Washington State Convention Center Test and Analysis
6.0 Test Setup The ZK-MPS/B was used as the walk-around testing device. The product contains dual high-speed scanners that can scan multiple technologies on 700/850/1900/AWS frequency bands. It also allows the user to connect up to eleven (11) cellular handsets and/or modems (Devices) in order to capture voice and data calling performance information.
FIGURE 2. ZK-MPS/B
The ZK-MPS/B unit integrates scanning hardware and computer processing and display into a single integrated unit. An internal dual battery system provides 4 - 8 hours of power for the entire system including powering the scanners, display interface and devices. The unit weighs 10 lbs including batteries and comes with a side/backpack carrying case that can swing to the side to make connections then swing to the back for walking. It has many compartments for devices and cables. The user does not have to worry about charging multiple batteries since a single dual battery system powers everything. Additional batteries are included along with an external charger so they can be easily swapped when required. It took three hours to walk all areas of the Convention and Conference Center. Log files were stored to the ZK internal memory card. After completion of the walk test the log files were copied to a computer and brought back to the office for further analysis.
October 20, 2013
5
ZK Services, LLC
Washington State Convention Center Test and Analysis
6.1 Scanner Setup The ZK-MPS/B was configured to scan the following channels and technologies:
Table 1: Channels scanned with MPS/B
A channel list was not provided by the client so ZK Service engineers determined the available channels by connecting phones to the MPS/B and using the channels that the phones locked to. We then entered these channels into the scan list and scanned them with the scanner. In addition, the connected handsets also logged measurement information on these channels. There are more channels being transmitted that we did not know however they were most likely being transmitted on the same base station antennas so we can assume that the measurements coming from one of the channels off the same antenna are very similar to the others. With that said there are channels that we did not scan, however based on our knowledge of the system and the measurements made we can safely assume our measurements provide more than enough information to accurately come to the conclusions in this report.
October 20, 2013
6
ZK Services, LLC
Washington State Convention Center Test and Analysis
6.2 Device Setup The following phones were connected to the MPS/B via USB cables. • Verizon HTC Incredible 4 EVDO/LTE Device • T-Mobile Samsung S3 UMTS/LTE Device • Sprint Samsung S3 EVDO/LTE Device • AT&T Samsung S3 UMTS/LTE Device The devices were running the ZK Metrics Android app that performs FTP download transfers. The FTP server is located in California and has an unlimited bandwidth pipe into it. The phones were set to download a 24MB file recursively (over and over). Data throughput values along with channel and signal level information were logged via the phones to the memory card on the ZK-MPS/B on each of the Operator’s networks.
October 20, 2013
7
ZK Services, LLC
Washington State Convention Center Test and Analysis
6.3 Floor Plan Setup Floor plan images for each of the areas of the convention center were imported into the ZK-MPS/B. The user displays the floorplan image on the screen while walking the area and drops breadcrumbs by moving the cursor and pressing a button. A sample of the lower level image is shown below:
FIGURE 3. Floorplan image of exhibit area used for data collection
October 20, 2013
8
ZK Services, LLC
Washington State Convention Center Test and Analysis
6.4 Post-Processing Although ZK log files have an open, ASCII format and can be imported with many other third party software packages including Actix, Windcatcher and IBWave, this data analysis was performed using the ZK KPI Mapper software and ZK Benchmarker software. The KPI Mapper software converts the ZK log data and images into MapInfo tables and workspaces. Maps of KPI's are auto-generated simplifying the analysis tasks. KPI Mapper also converts the condensed ZK log file into separate CSV files for each device and scanner band/technology. This makes it easier for other software to import the ZK data. The ZK Benchmarker software was used to mine the data and create tabular comparison information for VP level type reports.
7.0 Data Analysis The goal of the analysis was to baseline the coverage and the data performance of the Operators AT&T, Verizon, Sprint and T-Mobile. Voice calls were not specifically performed but the voice quality can be induced from the data and coverage measurements.
7.1 Throughput Throughput tests were conducted for all Operators. When available 4G LTE technology was tested. Sprint does not offer LTE so it was only 3G EVDO. FTP downloads were performed recursively downloading 24MB files from ZK’s FTP Server located in a 24 hour secure facility in California with unlimited bandwidth.
October 20, 2013
9
ZK Services, LLC
Washington State Convention Center Test and Analysis
7.1.1 Throughput Plots Below are LTE data throughput plots for AT&T T-Mobile and Verizon and an EVDO throughput plot for Sprint. Sprint does not provide LTE service. FTP download tests were conducted into the high speed ZK Server located in a collocation facility. There are no limitations on bandwidth going to and from the server. The following throughput plots are for the main exhibition hall. At the time of the test the hall was filled with show attendees. It was so crowded in the hall that it was very difficult for the ZK engineer to make his way through the aisles. Based on the following data throughput plots data service was basically not available. All four Operators had very poor data performance and large areas of no data service at all.
FIGURE 4. AT&T LTE Throughput (kbps)
October 20, 2013
10
ZK Services, LLC
Washington State Convention Center Test and Analysis
FIGURE 5. T-Mobile LTE Throughput (kbps)
October 20, 2013
11
ZK Services, LLC
Washington State Convention Center Test and Analysis
FIGURE 6. Verizon LTE Throughput (kbps)
October 20, 2013
12
ZK Services, LLC
Washington State Convention Center Test and Analysis
FIGURE 7. Sprint EVDO Throughput (kbps)
The net result is that users will be frustrated at the non-performance of data. These plots show that data rates and performance are extremely poor. It is unusable as a user. LTE data rates should be between 5Mbps and 30Mbps. EVDO rates should be in the 800Kbps to 1Mbps range. When a data channel was available these data rates don’t come close to what the average user expects.
October 20, 2013
13
ZK Services, LLC
Washington State Convention Center Test and Analysis
7.2 Coverage Of the provided list of channels the strongest ones for each frequency band per Operator are presented. This represents the best case for Operator coverage. 7.2.1 Coverage Plots LTE (4G) Coverage . The following plots show the 4G LTE coverage for AT&T, T-Mobile and Verizon. Sprint has no 4G network in the area. The 4th floor was the main exhibit area. The 4G coverage for all three Operators was poor and not strong enough for proper performance. Verizon installed a mobile cellsite near the skybridge which helped coverage levels in a section of the south hall.
FIGURE 8. AT&T LTE RSRP 4th floor exhibit area 700MHz channel 5780
October 20, 2013
14
ZK Services, LLC
Washington State Convention Center Test and Analysis
FIGURE 9. T-Mobile LTE RSRP 4th floor exhibit area AWS Channel 2250
October 20, 2013
15
ZK Services, LLC
Washington State Convention Center Test and Analysis
FIGURE 10. Verizon LTE RSRP 4th floor exhibit area 700MHz Channel 5230
October 20, 2013
16
ZK Services, LLC
Washington State Convention Center Test and Analysis
EVDO/UMTS (3G) Coverage. The following plots show the 3G LTE coverage for AT&T, Sprint, T-Mobile and Verizon. AT&T had good coverage in 3/4 of the exhibition hall and the mobile cell helped the Verizon coverage. Sprint and T-Mobile had poor coverage and all of the Operators did not cover the exhibition hall adequately.
FIGURE 11. AT&T UMTS Coverage - 4th Floor Exhibit Hall
October 20, 2013
17
ZK Services, LLC
Washington State Convention Center Test and Analysis
FIGURE 12. Sprint EVDO Coverage - 4th Floor Exhibit Hall
October 20, 2013
18
ZK Services, LLC
Washington State Convention Center Test and Analysis
FIGURE 13. T-Mobile UMTS Coverage - 4th Floor Exhibit Hall
October 20, 2013
19
ZK Services, LLC
Washington State Convention Center Test and Analysis
FIGURE 14. Verizon EVDO Coverage - 4th Floor Exhibit Hall
October 20, 2013
20
ZK Services, LLC
Washington State Convention Center Test and Analysis
7.2.2 Conference Center The following plots show the data throughput (or lack thereof) of the Conference Center building levels LL, L1 and L2. There was no service at all in the lower level and extremely spotty service in the upper levels. During our test we were stopped by a user that said that he could not send or receive texts and could not access email or the web from their phone.
FIGURE 15. AT&T Throughput - Conference Center
October 20, 2013
21
ZK Services, LLC
Washington State Convention Center Test and Analysis
FIGURE 16. Conference Center AT&T Throughput
FIGURE 17. Sprint Throughput - Conference Center
FIGURE 18. T-Mobile Throughput - Conference Center
October 20, 2013
22
ZK Services, LLC
Washington State Convention Center Test and Analysis
FIGURE 19. Verizon Throughput - Conference Center
8.0 Conclusion The goal was to perform a baseline study of data performance and coverage of the four major Carriers during a live event at the Washington State Convention Center and Conference Center. The ZK-MPS/B unit performed all phone and scanning measurements across 4 frequency bands for LTE, CDMA/EVDO and UMTS technologies. ZK’s KPI Mapper and Benchmarker software was used for analyzing the log data. Although Verizon installed a temporary cellsite which perhaps added marginal improvement in a section of the exhibit area all of the Operators did not provide adequate data channel access and throughput and coverage. This result was not unexpected as there is no DAS. The users are being served by the outdoor macro network. It is somewhat surprising on actually how poor the networks performed. Data was unusable and although voice access and quality was not specifically measured it most likely was equally bad.
October 20, 2013
23
ZK Services, LLC
Washington State Convention Center Test and Analysis
There is only one real solution if the desire is to provide quality wireless service which is to install a DAS. A DAS will provide high capacity and high quality wireless service through its dedicated antenna technology and implementation. ZK Services is a unbiased third party company that has been in the wireless testing business for over 30 years. Our experienced engineers are extremely knowledgeable on indoor and outdoor wireless system testing and analysis. Please note that data and plots are available for the other areas of the Convention Center upon request.
October 20, 2013
24
