FASTER SAP TESTING MEANS BUSINESS AGILITY AN EASY-TO-USE, CLOUD-BASED, SECURE SOLUTION FOR TESTING SAP SYSTEMS

02.04.2012

FASTER SAP TESTING MEANS BUSINESS AGILITY If you have an SAP ERP installation at your company, or run one for a client, one of the biggest challenges is avoiding negative side effects to the production system when changing or adding business processes, customizations, patches, or upgrades. This is the main reason SAP requires businesses to run a separate instance of its ERP system in order to conduct testing. Regression testing helps ensure that changes do not impact production systems. Thorough testing slows the pace of needed changes and reduces agility. If you could speed up the testing process significantly, you could move faster without increasing risk. Most organizations test their SAP applications manually before they move packages of changes, called transports, to production systems. Manual testing is labor-intensive, requiring hours of tedious work. There are a few automated testing solutions on the market, but for the most part they are expensive, difficult to implement, and time-consuming to use and maintain. Many administrators would like a more cost effective and simplified solution to test SAP applications. There is now a third, emerging generation of testing solutions that focuses specifically on packaged applications and reduces the effort of capturing and running tests scenarios by a factor of 10. This new breed of testing solutions also offers increased peace of mind, knowing that test coverage is complete and that everything will run smoothly. In this paper, we track the development of SAP testing techniques through these three generations and explain how the latest generation of testing tools represents a giant step forward in addressing a challenge that all SAP customers face.

THE THREE GENERATIONS OF SAP TESTING Software testing methodologies and tools have evolved to support the demands of the business world. In this section we discuss the pros and cons of three generations of testing solutions: manual, automated, and intelligent.

-1-

THREE GENERATIONS OF SAP TESTING TOOLS GENERATION OF TESTING TOOLS

CREATING SCRIPTS

RUNNING SCRIPTS

MAINTAINING SCRIPTS

FIRST GENERATION: MANUAL TESTING

Manual

Manual

Manual

SECOND GENERATION: AUTOMATED TESTING

Semiautomated

Automated

Manual

THIRD-GENERATION: INTELLIGENT TESTING

Automated

Automated

Automated

FIRST GENERATION: MANUAL TESTING The first generation of testing, manual testing, is still the method most companies use for the bulk of their testing. Because business processes trigger functions in SAP, manual tests are often created and run by power users or SAP functional experts, who view testing as an extra step in the release process. They see it as time-consuming and tedious, with very little payoff. Manual testing causes friction between end users, power users, functional experts, and IT managers. While IT managers worry about system stability, users in all categories see testing as a burden that is added on top of their regular responsibilities. More than half of the organizations surveyed don‟t use a testing solution at all. Of those who do test, 74% don‟t use an automated solution, and 53% use general-purpose business software such as Microsoft Word and Excel. (Source: 2011 SAP Testing Survey) Despite the investment level of most companies in SAP ERP and its importance to their business, testing still gets short shrift. Manual testing remains the most common method of SAP testing. On the one hand, it is relatively economical, at least from the perspective of purchasing software and staffing. Because no special skills—besides understanding the SAP application and having a great deal of patience—are required, and no special software needs to be purchased, many organizations continue to test this way, creating test plans and executing them.

-2-

On the other hand, manual testing is a costly and ineffective way to limit risk to an expensive investment. Manual testing is time-consuming, inefficient, and highly prone to errors. It‟s impossible to achieve 100% coverage for all possible usage scenarios. The result is that frequently little quality testing gets done when manual testing is the only option. In many ways, this slows down a business‟s ability to innovate and to keep up to date on the latest software releases. Concerned about the cost and delays involved with upgrades and testing, many organizations are slow to make changes to their SAP installations. This limits opportunities provided by SAP‟s „best practices‟ enhancements to innovate and become more efficient on a business level; on the technical system front it actually heightens the risk of a catastrophic system failure or non-supported technical problems. PROS AND CONS OF MANUAL REGRESSION TESTING PROS

 Can handle any script and playback permutation scenario using “human intelligence”

 Economical in terms of software purchases— most manual testers use standard business software that is already in-house

 Simple—the only requirement is to accurately record the results of changes made to the SAP software

 Economical on a per-resource basis—little skill

CONS

 Not comprehensive—limits the ability of an organization to roll out changes to SAP

 Labor-intensive—high labor cost relative to level of risk control provided

 Risky and error prone because of human limitations related to extended focus, accuracy, and repetition

 Slow—Since all testing is done by humans, thorough testing takes time. The tradeoff is whether to sacrifice quality or delay the release

required

SECOND GENERATION: AUTOMATED TESTING The second generation of SAP testing delivered a significant leap forward. Quality assurance (QA) platforms were built to test software, offering several advantages. Business users or functional experts no longer needed to devote significant time to running the tests, and the tests could be run continuously with minimal disruption. The ability to automate a far greater number of tests offered new potential to diagnose issues with SAP installations and accelerate rollout schedules. There were still significant disadvantages, however.

-3-

COMPLEXITY AND EFFORT

Test automation is a resource-intensive undertaking, requiring programmers for home-baked or vendor solutions. The typical test automation project includes setting up dedicated, on-premise hardware and software and hiring a full-time test automation programmer and administrator to manage the project. The deployment of these systems offers substantial configurability but it is quite complex. EXPENSE

Organizations could spend close to $300,000 before a single test is run. Often, the tests run aground before any useful information is collected. PROS AND CONS OF AUTOMATED REGRESSION TESTING PROS

 Business users not as heavily consumed in tedious testing execution

 Increased test execution efficiency as greater volume of test repetitions can be run in a given time period

 Testing depth and accuracy improves—for the constructed scripts

 Once tests are created, they can be run again and again (subject to maintenance work)

 Offshore testing resources can help develop tests

CONS

 Complexity—Takes up to a year to implement. Coverage is limited because of time and effort to create scripts for each business process

 Effort—full-time test automation developers are needed  Expense—as much as $300,000 before the first test is run  Tools are general-purpose; not explicitly designed for testing SAP systems—requires tedious specification and programming work by both functional experts and test engineers

 Disconnect between functional experts and testing pros (external and offshore) can lead to coordination issues and even logical errors in the created test scripts

 Deployment costs—an expensive solution with regard to effort to deploy and system landscape requirements

 Brittle scripts—Once created, scripts are hard to maintain because even a small change breaks them

 Developing scripts using test automation tools requires scarce and expensive engineering talent

-4-

THE STATE OF TESTING TODAY According to the 2011 SAP Testing Survey, only about 26% of organizations have implemented automated testing of their SAP systems. Even at organizations that have adopted such tools, automated tests still do a fraction of the work. Those organizations using automated testing tools automate only about 20% of their testing processes, meaning that the bulk of testing is still done manually—organizations write test scripts and keep track of results using Word documents and Excel spreadsheets. Even more telling with regard to testing is the apparent lack of satisfaction with the solutions. Nearly half of the respondents to the survey are unhappy with their current situation; of the 26% who have tried automated solutions, 15% are not satisfied with what they've achieved using them. Of the 74% who have yet to automate testing, about 34% plan to do so. With nearly half unhappy with their current situation, there is a need for a new take on the very old problem of testing SAP landscapes.

THE STATE OF TESTING AUTOMATION (SOURCE: 2011 SAP TESTING SURVEY)

-5-

THIRD GENERATION: INTELLIGENT TESTING FOR SAP SYSTEMS The third generation of SAP testing is a software-as-a-service (SaaS) testing solution designed specifically for SAP. First, this solution does an impact analysis to show you what areas of the system are impacted by the changes and where to focus testing. Then, business users or functional experts generate testing scripts on the fly by running the business process once on their actual system. From then on, anyone can automatically run the test scenarios whenever needed. The generated test scripts can be executed automatically and they self-adjust, based on test results, with no need to re-record. Alongside the automated script, a natural-language step-by-step document is created, which is useful for management, reviews, and manual execution if needed. As a result, customers can reduce the effort associated with running SAP ERP tests by up to 80%. PROS AND CONS OF INTELLIGENT TESTING PROS

 Reduced time for testing staff—up to 80% less time spent testing

 Reduced time-to-market for SAP changes of all

CONS

 Organizations with an investment in automated testing might find it challenging to justify a second investment

types (days instead of months)

 Software-as-a-Service—no hardware or software to install—set-up in 20 minutes

 Cloud-based library stores 70,000+ test scripts, and adds more every day

 Regenerative learning—learns from and logs errors; skips them on the next run

 Simple and intuitive—business users can record their daily activities, then automate test scripts

-6-

THE PANAYA PROPOSITION Panaya is the leading third-generation SAP testing solution. Panaya‟s cloud approach to SAP testing is different. Instead of weeks or months to begin testing, Panaya can be up and running in 20 minutes. Panaya is an intelligent testing system that combines the human intelligence from manual testing with the efficiency and adaptability of computers. Panaya‟s customers perform their manual tests only once. Panaya records their keystrokes as they work through business processes and creates scripts. The next time that they want to test the process, the scripts run automatically. Panaya creates both manual and automated test scenarios in minutes. You don‟t need test automation engineers on staff to replicate and test your business processes. The money you save will come from spending less on dedicated testing staff. The time you save will reduce the time-to-market for the next iteration of your SAP ERP application. And finally, the increased test coverage improves the quality of your SAP ERP application and provides additional peace of mind. Here are a few of the value propositions of Panaya.

BUSINESS CLASS The level of effort required to set up most testing scripts is beyond the reach of the business user and requires dedicated programmers. Panaya is different. With Panaya, business users—the people who use SAP every day—can start recording and testing within minutes. The key here is that the business user doesn‟t feel like they‟re preparing tests because they are simply using the ERP application, just like they always do.

SAAS = SIMPLICITY, SECURITY, AND STABILITY Panaya is delivered as a SaaS solution. That means no machines to host, no dedicated administrators, and no installation routines. Screens are simple and intuitive. Setup takes about 20 minutes. Creation and execution of tests is always initiated and controlled by the customer; Panaya doesn‟t have any access to the customer landscape or to customer data. Panaya runs on Amazon.com‟s EC2 cloud, which has the strongest security certification of any cloud offering available. Even FDA-regulated companies and financial institutions with strict security requirements use Panaya.

-7-

THE WISDOM OF THE CROWD There are more than 20,000 companies using SAP. Before rolling out changes, they need to generate regression-testing scripts and perform regression testing. Because there are many shared processes across SAP instances, many of the scripts generated at individual installations are redundant. So why spend time creating them? At Panaya, we have analyzed more than 2,500 SAP instances and have created, stored, and catalogued more than 70,000 test scripts. Our algorithm uses the wisdom of the crowd, more thoroughly automating your scripts than any automation test engineer who could possibly work for you. Think of Panaya as the Google Trends of ERP testing. The entire SAP community benefits from a knowledge base that is more powerful than any one company could create.

REGENERATIVE LEARNING AND SELF-HEALING No test run is likely to be perfect, but every glitch is a learning opportunity. Panaya can learn from errors, and avoid repeating them the next time. Here are three common scenarios where Panaya excels and other solutions fall short. SMART PARAMETERIZATION

Suppose you are creating a purchase order and then use this purchase order for goods receipt. SAP always generates unique purchase order numbers, so every time you run this test, you will have to use the order number you received while running the test. Manual testers simply take notes of the order number they received and enter it manually. Test automation engineers must have the business knowledge and do some fair amount of programming to get such processes working, parameterizing the purchase order number. Panaya automatically knows to pass document numbers exactly as you would expect it to work without additional effort on anyone's part. CROWD-BASED WORKAROUNDS

Test recordings naturally cover only one specific case. When running the test, the system may behave slightly differently. This is where the cloud can work for you. Panaya‟s software already knows how other customers reacted in the very same situations. So when unexpected behavior occurs, Panaya automatically looks up a way to handle it via the community knowledge base and keeps running. If there is no workaround available, it switches to manual mode to let the user deal with the unexpected situation.

-8-

For example, if the user name field content is different from the recording, Panaya knows that‟s not a problem. However, if you suddenly have a different item category, it will alert you of a potential error. TESTING AT THE PUSH OF A BUTTON

There may be no better example of Panaya‟s advantage than the following diagram. This diagram represents the difference between Panaya and a generic testing solution. We‟ve collapsed some of the sub-tasks for clarity, but think you‟ll get the picture.

To capture a business process and run a test script in a general-purpose test-automation tool, a test-automation expert must negotiate his way through 32 steps and the business user must be involved in 11 of those steps. With Panaya‟s SAP-specific testing solution, the same process can be automated, by business user alone, in just two steps: record and play.

PANAYA STREAMLINES TESTING

“AN INCH WIDE AND A MILE DEEP” Generic testing solutions are made to work on every Windows application in the enterprise; they are a “mile wide and an inch deep.” Panaya was built using domain-specific knowledge, explicitly for SAP ERP systems—we‟re “an inch wide and a mile deep.” Our solution brings a new depth to testing software—the way people actually work on SAP.

-9-

DOMAIN AWARE: INHERENTLY UNDERSTANDS SAP PROCESSES

Let‟s go back to that purchase order for a minute. Suppose you are attempting to test the integrity of a purchase order process in SAP. With a generic testing solution, you would write a long script that would run the same process, with allowances for every expected amount ($10,000 increments, for example). Now, suppose there are certain business rules that must occur in order to accept certain numerical increments: If an order is more than $10,000, it should be approved by the site manager; if an order is more than $20,000, it needs the regional manager‟s approval. In a generic testing solution, you would need to write a highly complex branch script to cover these business cases. If the approval protocol changes, you have to change the script. That‟s a lot of “if‟s.” With Panaya, you quickly record a short script for each approval condition—so changing the protocol for any one increment won‟t break the entire order processing routine.

CONCLUSION: FROM QUANDARY TO QUALITY Panaya is a high-quality, low-impact, easy-to-use ERP testing solution. Manual testing is time consuming and an ineffective use of resources. Homegrown and generic commercial automated testing is expensive and difficult to set up. With Panaya, the intelligent testing solution for SAP, you don‟t have to choose between time-to-market, cost reduction, and risk reduction. You can: Reduce the effort and duration of regression tests by a factor of 10 Increase the number of useful scripts by a factor of 10 Commit internal IT staff to higher-value projects Let SAP users test their own functions Introduce changes more frequently, while maintaining or reducing risk levels Test 10 times a year, rather than two By using Panaya to test your SAP ERP system, you can be more innovative, agile, and meet changing business conditions head-on. By helping you move faster, Panaya speeds up business and moves testing from a roadblock to a competitive advantage.

- 10 -

DISCLAIMER AND TRADEMARK NOTICES This report is provided by Panaya Ltd. It is completely independent of and not affiliated with Oracle or SAP AG. Oracle is a registered trademark of Oracle. Oracle and other Oracle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Oracle. SAP is a registered trademark of SAP AG. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries. All other product and service names mentioned are the trademarks of their respective companies. DISCLAIMER OF WARRANTY

Panaya Ltd. makes no representation or warranties, either express or implied by or with respect to anything in this document, and shall not be liable for any implied warranties of merchantability or fitness for a particular purpose or for any indirect special or consequential damages. COPYRIGHT NOTICE

No part of this publication may be reproduced, stored in a retrieval system or transmitted, in any form or by any means, photocopying, recording or otherwise, without prior written consent of Panaya Inc. No patent liability is assumed with respect to the use of the information contained herein. While every precaution has been taken in the preparation of this publication, Panaya Inc. assumes no responsibility for errors or omissions. This publication is subject to change without notice. © 2012 Panaya Ltd. All rights reserved.

ABOUT PANAYA Panaya‟s software-as-a-service helps companies that use SAP or Oracle reduce 80% of their upgrade and testing risk and effort. Utilizing a cloud-based supercomputer, Panaya simulates the upcoming upgrade, automatically pinpointing which custom programs will break as a result of the upgrade and automatically fixing most of these problems. Panaya‟s testing solutions dramatically expedite ERP testing and eliminate the need for manual test script maintenance. Seamlessly capturing business knowledge in the background, as users work with the ERP applications, Panaya automatically generates plain-English test scripts that are rapidly executed and continually self-adjust based on test results.

- 11 -