EMC Proven™ Professional Knowledge Sharing Final Paper

Serial ATA Drives in CLARiiON® Arrays

EMC Proven Professional™ Knowledge Sharing October, 2007

Victor Franco EMC Corporation [email protected]

1

EMC Proven™ Professional Knowledge Sharing Final Paper

Table of Contents Introduction ............................................................................................................................................ 3 Serial ATA Introduction .......................................................................................................................... 3 To Performance through Power Consumption ...................................................................................... 4 Serial ATA vs. Fiber Channel Drives ..................................................................................................... 5 What do we call random access? .......................................................................................................... 6 Concerning Data Safety......................................................................................................................... 6 Concerning Performance ....................................................................................................................... 7 Final words.............................................................................................Error! Bookmark not defined. Bibliography ........................................................................................................................................... 8 Author’s Biography................................................................................................................................. 8

Disclaimer: The views, processes or methodologies published in this compilation are those of the author. They do not necessarily reflect EMC Corporation’s views, processes or methodologies.

2

EMC Proven™ Professional Knowledge Sharing Final Paper Introduction

Serial ATA drives have been used for some time in CLARiiON arrays. Though very recent, if we compare them to Fiber Channel drives, they have already completed one important technology revision (the so called “SATA II” drives). EMC recommendations on how to use these drives is clearly stated in the “CLARiiON Best Practices” series of documents available in PowerLink. These recommendations differ from how we would normally use Fiber Channel drives, so let’s introduce the technology, and after giving a look at what is happening beneath the surface. Where is the Hard Disk industry aiming its research? •

Increase density of data. This allows the vendors to produce either drives with higher capacity, or drives with the same capacity but smaller size (required to be used in a mobile phone for example). This has been achieved with technologies such as perpendicular recording or pixie dust layer.

•

Decrease power consumption. This is required to increase the battery life of mobile devices (or decrease size, weight or noise) or to sustain the Uninterrupted Power Supplies needed to feed, in the case of a power failure, the increasing amount of storage devices. Lower voltages are enabled, as in the Low Voltage Differential Signaling, and updated algorithms are in place to move the same amount of data in fewer operations, as with Command Queuing technology.

•

Increase the data transfer speed. We need to specify two areas: the first would be disk buffer to/from platter, which is the main bottleneck. Here we will again mention the Command Queuing technology, and add the rotational speed. The second area would be the disk interface to/from buffer, which won’t usually result in increasing performance, but rather increases the number of drives serviced by a controller. So, FC drives interfaces have moved from 1Gbit/s to 4 Gbit/s, and older Parallel ATA drives without DMA capabilities to newer Serial ATA drives with roadmap to reach 6Gbit/s.

Serial ATA Introduction Serial ATA is a computer bus technology designed to transfer data to and from a hard disk. This technology maintains the ATA protocol, but replaces the older parallel bus architecture with a serial one. It also includes improvements in electrical design, clocking strategy, cabling, connectors, termination, PCB routing and signaling. Thus all the older AT Attachment standard (ATA) based drives are now named by Parallel ATA (PATA). The organization behind this technology is now called Serial ATA International Organization or SATA-IO (www.sata-io.org). There are three main specifications (with revisions) regarding this technology, the last remains a Work In Progress: • • •

Generation 1: 150 MByte/sec (Fall 2002) Generation 2: 300 MByte/sec (Mid-2004) Generation 3: 600 MByte/sec (Mid-2007)

3

EMC Proven™ Professional Knowledge Sharing Final Paper Though the second generation specifications were documented in the middle of 2004, implementations did not occur in the market until a year or two later. So, recently EMC has started supporting SATA II drives, though this naming is improperly used as it does not mean 3Gbit/s (it was the name of the committee formed to author the SATA specifications –now renamed).

We tend to identify the maximum supported bandwidth of the specification with the technology. This is not accurate as: •

The theoretical limit is not real, as SATA drives use 8B/10B encoding, meaning the efficiency of the data transmission is 80% of the actual bandwidth. So, a SATA drive communicating at 1.5 Gbit/s will be delivering an approximate maximum bandwidth of 150 MByte/s. Even though, it is a worthy tradeoff as we eliminate the need for a separate clock signal (its advantages are also used by Ethernet, Fiber Channel or PCI Express).

•

Some of the more powerful features of the SATA specification are not described or implicit to the naming, but rather related to the vendor implementation. To name some of the more important ones: Native Command Queuing (NCQ), Hot Plug and Staggered Spin-up.

•

This is why we should not rely on the SATA 2 or SATA/300 naming, but rather reach deeper into the drive specifications (for example, SATA drive with 3Gbit/s interface and supporting NCQ and Hot Plug).

So, the main change in the new SATA II CLARiiON drives is not only the interface speed. Native Command Queuing allows the drives to be natively dual ported and no longer requires paddleboards to implement the connection to both backend loops and the hot-swap functionality.

To Performance through Power Consumption There is a very easy experiment that can be deployed to measure the impact of command queuing technology, and under what circumstances it is useful to have it enabled. We connect a NCQ SATA drive to a computer running software capable of sending NCQ commands. We also connect a current meter to the power supply feeding the disk, and measure the power consumption in different scenarios. If we produce sequential requests, the power consumption is constant and sustained while data is being serviced. No difference is noticed whether NCQ is enabled or not. But if we change to random patterns, power consumption increases, and is sustained over a longer period of time. This effect though, is reduced up to 66% with NCQ enabled.

4

EMC Proven™ Professional Knowledge Sharing Final Paperfor this difference is that in addition to the energy used in spinning the platter, we The reason use additional energy to move the actuator arm to service random requests. This effect is reduced by the queuing mechanism, as data is transferred to/from the disk internal buffer, not in the order it was requested, but reordered to minimize the distance the actuator arm needs to be moved. That is why this technology is also called “elevator seeking”, as an elevator stops on the different floors not in the order it was called, but starting in the closer one and minimizing overall time to service and power needed to do it.

We may conclude that having drives with command queuing technology is a requirement for performance in a random environment.

Serial ATA vs. Fiber Channel Drives If we compare the performance of Serial ATA Disks and FC (SCSI) Disks, we notice that despite the obvious differences in rotational speed, we achieve very similar transfer rates. Why then, are we told that Serial ATA drives performance ranges from 25% to 90% of FC drives? The answer depends on the application. It is very similar for sequential access (at the same rotational speed), and very different for random access. But again, did not both technologies implement command queuing? The underlying reason is that FC drives implement a Command Queuing technology called Tagged Command Queuing (TCQ) with an effective reordering of the first 8 IOs queued. On the other hand, Serial ATA drives do not implement this or if they do (SATA II drives may use Native Command Queuing NCQ), do it with an effective queue depth of 2. The given numbers apply to hard drives in CLARiiON boxes, as the theoretical limits for the specifications are 216 queued commands for TCQ, and 32 for NCQ. In real life scenarios, these numbers are reduced for two reasons: •

If we measure the number of tracks crossed by the actuator arm (travel distance) as the reordering queue size increases, we notice that it is reduced up to 70% as it’s increased to 8 requests. By doubling that number we don’t even get an extra 10%. This effect is seen in a lot of other applications (L1 and L2 cache size inside a CPU, read cache size of a CLARiiON, number of backup threads writing on a CDL…).

•

We need to make sure that we don’t lose any disk buffers’ queued data in case of a power failure, and the amount of commands to manage an NCQ queue is more restricted than a TCQ (thus the development of Serial Attached SCSI –SAS-).

This means that the Serial ATA drives used in the CLARiiON cannot be effectively used in random environments, as the performance highly degrades in that environment compared to the Fiber Channel drives.

5

EMC Proven™ Professional Knowledge Sharing Final Paper

What do we call random access? We use ‘random access’ from the point of view of the physical sectors on the hard drive. That is, producing requests that make the actuator arm move. So even when our application is delivering serial data, we might find that the drives are receiving data as random in the following cases: • • •

Filesystem fragmentation. A RAID Group has several LUNs accessed at the same time. We have a striped MetaLUN with LUNs of the same RAID group.

What happens if we randomly access SATA drives? There are two implications of misusing this technology: •

Data safety. More head movement means more heat. More heat increases the likelihood of failure. Also, longer rebuild times means more data exposure.

•

Performance. More head movement means more time to get data serviced.

To compensate for CLARiiON arrays using Serial ATA drives, a number of countermeasures have been taken:

Concerning Data Safety •

Modified Sniff Verify from Release 19. The Sniff rate has been increased from 64KB requests every 0.5 seconds in older Release 16 to 512KB requests every second. This change was needed since higher capacity drives take longer to verify, and the chance of a SATA drive failing is higher. We must detect media errors on drives as soon as possible. This parameter can no longer be changed by the user.

•

New DAE2P and DAE3P enclosures. The “performance” enclosures do not deliver more bandwidth (comparing DAE2 with DA2P), but with the use of a Cut Through Switch (CTS) they isolate the data traffic for every disk on the virtual loop. This feature reduces loop latency and provides better isolation in error conditions.

•

Increased number of hot spares. Best practices recommend having one hot spare per enclosure (15 drives) if we work with SATA drives, while the requirement for FC is one hot spare per pair of enclosures.

•

SATA Drives not allowed in a vault area. If a vault drive fails in a default configuration, the CLARiiON’s write cache CLARiiON is disabled (and we wouldn’t want that happening). So we’ll be restricting that area for the use of FC drives.

6

EMC Proven™ Professional Knowledge Sharing Final Paper

Concerning Performance •

Optimized RAID 3 algorithm. The amount of data moved per operation in the loop has been increased. The algorithm has been optimized for SATA drives.

•

Optimized RAID 5 algorithm. From FLARE 22, parity data is kept on the same disk for several stripes before moving onto another RAID Group disk. This way the behavior of RAID 5 gets closer to the RAID3.

•

We have already mentioned that new SATA II drives featuring Native Command Queuing are natively dual ported.

Conclusion Sequential access, then, is the proper use of Serial ATA Drives. To implement sequential access to the drive and not to the LUN, follow these recommendations (whenever possible): •

Bind one LUN per RAID Group to avoid producing disk random access.

•

Use RAID 3 in 4+1 Groups. Smaller RAID Groups will hold less LUNs, and hopefully will not be accessed concurrently. RAID3 is the optimal algorithm for SATA drives.

•

Ensure that the same SP owns all LUNs on RG. This is especially critical in SATA drives as they are not natively dual ported, and use a paddleboard that would block traffic from one Storage Processor (SP) if the other SP was already talking to a LUN in the same RAID Group (RG).

•

Use multiple streams per RG; four to five work best.

•

Use a page size of 16 KB for CLARiiON cache as we are expecting sequential requests and big block sizes. In a mixed environment, we may program and script changes to this size (for example, increasing the size for backup operations, and setting it back to a lower level once backup is finished).

•

Change the write-aside value to cache large writes. If sequential big block writes are in place, we don’t want to miss the write cache.

•

Bypass cache if many drives are writing in parallel, but make sure data is aligned.

I hope that you have found this information useful, and wish you continued good luck!

7

EMC Proven™ Professional Knowledge Sharing Final Paper

Bibliography Internet resources • Serial ATA International Organization (www.sata-io.org) • Seagate (www.seagate.com) • Storage Review (www.storagereview.com) • TOMs Hardware (www.tomshardware.com). To search for reviews on specific hard drives. • Wikipedia (www.wikipedia.org). Useful concepts and links when searching for “hard disk, Serial ATA, Native Command Queueing, perpendicular recording, pixie dust, antiferromagnetism…” White Papers • Serial ATA – A Comparison with Ultra ATA Technology – By SATA-IO • NCQ for Power Efficiency - February 10, 2006 - By Yun Wang • Serial ATA Native Command Queuing – July 2003 – By Intel & Seagate

Author’s Biography Mr. Franco has been working at EMC since 2001. He is an EMEA CLARiiON Lead Instructor, delivering mainly CLARiiON customer courses. He has performed a training delivery role for companies including IBM, CORITEL, Scientific Engineering Studies and NOVELL, and has been a local Storage Area Network Administrator for companies including CHUBB Insurance and Madrid Polytechnic University.

8