®

IBM Software Group

WebSphere Application Server Tuning Fundamentals Matt Goshgarian: [email protected] Nishant Thakkar: [email protected]

WebSphere® Support Technical Exchange

IBM Software Group

Purpose of meeting § Create performance awareness 4Discuss tuning fundamentals 4Show impacts of performance 4Teach customers how to test performance themselves

WebSphere ® Support Technical Exchange

2

IBM Software Group

Agenda § Background on performance 4 Definitions 4 Out-of-box vs. tuned application performance 4 Performance dependencies § Key tuning parameters 4 WebSphere Application Server 4 Application 4 Hardware § Performance impacts of monitoring 4 WebSphere Application Server 4 Operating System 4 Network § Performance tools 4 PMI 4 Tivoli Performance Viewer 4 Advisors § Conclusion 4 Questions 4 Links

WebSphere ® Support Technical Exchange

3

IBM Software Group

Definitions § Tuning: 4 To modify system parameters in order to optimize performance

§ Throughput (requests / second): 4 A quantitative unit to measure performance. The higher the better.

§ Response time (milliseconds): 4 A quantitative unit to measure performance. The lower the bette r.

§ Trade: 4 A J2EE benchmark application used to measure performance of an application server.

§ Client workload driver: 4 An application that sends requests to stress the server.

WebSphere ® Support Technical Exchange

4

IBM Software Group

Testing methodology § Ensure repeatability 4No unexpected processes running on system 4Run on a controlled network § Stress WebSphere Application Server to near 100% CPU utilization § Test runs: 4Perform warm-up 4Test steady state 4Execute multiple runs of each test •Variance is expected

WebSphere ® Support Technical Exchange

5

IBM Software Group

Background on meeting – Out-of-box vs. Tuned application performance Non-tuned vs. tuned throughput performance Througput (higher is better) 500 431

450

requests per second

400 350 300

260

250 200 150 100 50 0 Out-of-box Trade on WAS 6.0.2

Tuned Trade on WAS 6.0.2

66% faster performance after tuning!! Caution – Different environments will yield different results

WebSphere ® Support Technical Exchange

6

IBM Software Group

Performance dependencies § § § § § §

Hardware Environment Network Complementing software Application Application server

WebSphere ® Support Technical Exchange

7

IBM Software Group

Agenda § Background on performance 4 Definitions 4 Out-of-box vs. tuned application performance 4 Performance dependencies § Key tuning parameters 4 WebSphere Application Server 4 Application 4 Hardware § Performance impacts of monitoring 4 WebSphere Application Server 4 Operating System 4 Network § Performance tools 4 PMI 4 Tivoli Performance Viewer 4 Advisors § Conclusion 4 Questions 4 Links

WebSphere ® Support Technical Exchange

8

IBM Software Group

CAUTION

Be sure to apply tuning in a test environment first!!

WebSphere ® Support Technical Exchange

9

IBM Software Group

JVM § Set proper heap size

4Initial heap size / Max heap size 4Monitor with verbose GC (Garbage Collection) 4Ideal GC cycle interval should be:

•No more than 2 seconds in length •Longer than 10 seconds apart 4Increasing size increases GC length but reduces cycle frequency

WebSphere ® Support Technical Exchange

10

IBM Software Group

JVM – Screen shot § How to set from Administrative Console:

4Servers > Application Servers > (server name) > Java and Process Management > Process Definition > Java Virtual Machine

WebSphere ® Support Technical Exchange

11

IBM Software Group

ORB § Pass by reference

4Disabled by default (Pass by value) 4Enabled only if EJB client and server are on same WebSphere Application Server instance

•ex: web container & EJB container use same JVM process 4Use caution when using this parameter!!! •Can cause unexpected behavior •Verify that the application is designed to use this parameter •An object’s reference is modified by both the caller and callee

WebSphere ® Support Technical Exchange

12

IBM Software Group

ORB – Screen shot §

How to set from Administrative Console:

4Servers > Application Servers > (server name) > Container Services > ORB Service

WebSphere ® Support Technical Exchange

13

IBM Software Group

Web container § Thread pool 4Use 5-10 threads per server CPU as starting point (Min and Max) 4Use Tivoli Performance Viewer to get the optimal values

§ Keep-Alive 4Increase the value to fit your application’s needs 4Most customers set to unlimited

§ Sessions 4Number of in-memory sessions 4Disable session overflow in production 4Set session timeout 4Session replication

WebSphere ® Support Technical Exchange

14

IBM Software Group

Web container thread pool – Screen shot §

How to set from Administrative Console:

4Servers > Application Servers > (server name) > Thread Pools > WebContainer

WebSphere ® Support Technical Exchange

15

IBM Software Group

Web container keep-alive – Screen shot §

How to set from Administrative Console:

4Servers > Application Servers > (server name) > Web Container Settings > Web container transport chains > (select name associated with respective port) > HTTP Inbound Channel

WebSphere ® Support Technical Exchange

16

IBM Software Group

Web container sessions – Screen shot §

How to set from Administrative Console:

4Servers > Application Servers > (server name) > Web Container Settings > Session management

WebSphere ® Support Technical Exchange

17

IBM Software Group

JDBC § JDBC driver

4Type 2 vs. type 4

•Type 4 typically performs faster 4Use type 4 if your vendor supports it •http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/topic/com.ibm.websphere.base.doc/info/aes/ae/rdat_mi nreq.html

§ Connection pools

•Start off using the default •Set max and min connections using Tivoli Performance Viewer to find the optimal amount

WebSphere ® Support Technical Exchange

18

IBM Software Group

JDBC driver – Screen shot §

How to set from Administrative Console:

4Resources > JDBC Providers > (provider name) > Data sources > (Data source name)

WebSphere ® Support Technical Exchange

19

IBM Software Group

JDBC connection pool – Screen shot §

How to set from Administrative Console:

4Resources > JDBC Providers > (provider name) > Data sources > (Data source name) > Connection pool properties

WebSphere ® Support Technical Exchange

20

IBM Software Group

Application development § Dynamic cache WebSphere specific service 4 Can intercept calls to cacheable servlets • Developing application with dynacache can improve performance over 100% 4 Can be used to create a map that is distributed throughout a cluster 4 http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.nd.doc/info/wel come_nd.html § Avoid large objects 4 Causes fragmentation 4 Keep objects less than 200K in size 4 Smaller objects allow the JVM to fully utilize memory § Synchronization issues 4 When system won’t scale 4 Detect using thread dumps 4 Keep synchronization blocks small 4 Use different objects as locks when possible § Session 4 Keep session sizes small 4 Good rule of thumb is less than 100K

WebSphere ® Support Technical Exchange

21

IBM Software Group

Hardware § If software performance boost still isn’t enough, upgrade hardware 4Number of tiers

•1-tier: Application server and database on one machine •2-tier: Application server on one machine, database on another •3-tier: Same as 2-tier, but with HTTP server on a third machine 4Scaling

•Vertical – Multiple processors across single machine •Horizontal – Cluster across multiple machines 4Hyperthreading

•If supported, will logically increase the number of physical processors

WebSphere ® Support Technical Exchange

22

IBM Software Group

CAUTION

Remember to apply tuning in a test environment first!!

WebSphere ® Support Technical Exchange

23

IBM Software Group

Agenda § Background on performance 4 Definitions 4 Out-of-box vs. tuned application performance 4 Performance dependencies § Key tuning parameters 4 WebSphere Application Server 4 Application 4 Hardware § Performance impacts of monitoring 4 WebSphere Application Server 4 Operating System 4 Network § Performance tools 4 PMI 4 Tivoli Performance Viewer 4 Advisors § Conclusion 4 Questions 4 Links

WebSphere ® Support Technical Exchange

24

IBM Software Group

Performance impacts of Monitoring § Why would you want to always run monitoring? 4 Catch problems before they cause user problems. 4 Gather important documentation during the first failure. § Different Levels 4 WebSphere Application Server 4 Operating System 4 Network 4 Application 4 HTTP Server 4 Database 4 Etc… WebSphere ® Support Technical Exchange

25

IBM Software Group

WebSphere Application Server Monitoring Items § Tracing enabled 4 Trace string set to none • This is the default 4 Allows tracing to a file to be enabled without restarting the application server. § verboseGC enabled 4 Allows for detection of memory leaks. 4 Allows for detection of Garbage Collection related performance problems. 4 Do not run this in production unless you are debugging a problem

WebSphere ® Support Technical Exchange

26

IBM Software Group

Graph of Tracing enabled Difference between tuned and tracing enabled:

0.93%

500.0 450.0

431.0

427.0

Tuned

Trace enabled

requests per second

400.0 350.0 300.0 259.7 250.0 200.0 150.0 100.0 50.0 0.0 Not Tuned

WebSphere ® Support Technical Exchange

27

IBM Software Group

Graph of verboseGC enabled Difference between tuned and verbose GC enabled: 0.47% 500.0 450.0

431.0

429.0

Tuned

verboseGC enabled

requests per second

400.0 350.0 300.0 259.7 250.0 200.0 150.0 100.0 50.0 0.0 Not Tuned

WebSphere ® Support Technical Exchange

28

IBM Software Group

Operating System Monitoring Items § Process CPU/Memory statistics 4 Tool: ps, perfmon 4 Used to determine if processes are using the expected cpu and memory. § Overall CPU/Memory statistics 4 Tool: vmstat, perfmon 4 Used to detect swapping and verify overall cpu/memory usage. § Local Network stats 4 Tool: netstat 4 Used to verify the number of connections to and from the machine. WebSphere ® Support Technical Exchange

29

IBM Software Group

OS Monitoring Script: Sample only. Not for production use. #!/usr/bin/sh while true; do date >> ps.out echo ------------------------------------------------------------ >> ps.out ps avwwg >> ps.out echo ------------------------------------------------------------ >> ps.out date >> netstat.out echo ------------------------------------------------------------ >> netstat.out netstat -an >> netstat.out echo ------------------------------------------------------------ >> netstat.out date >> vmstat.out echo ------------------------------------------------------------ >> vmstat.out vmstat 5 12 >> vmstat.out echo ------------------------------------------------------------ >> vmstat.out date >> vmstat.out done WebSphere ® Support Technical Exchange

30

IBM Software Group

Graph of OS Monitoring Difference tuned and operating system script:

1.93%

500.0 450.0

431.0

422.7

requests per second

400.0 350.0 300.0 259.7 250.0 200.0 150.0 100.0 50.0 0.0 Not Tuned

WebSphere ® Support Technical Exchange

Tuned

OS scipt

31

IBM Software Group

Other Monitoring Items § Network 4 Tools: Ping, telnet, or a simple HTTP client 4 Used to verify that the network isn’t dropping packets, isn’t overloaded, and is able to properly route HTTP requests. § Database 4 Tools: Vendor specific, and OS level tools 4 Used to verify that the database isn’t overloaded and that it’s working properly. § HTTP Server 4 Tools: Vendor specific, and OS level tools 4 Used to verify that the HTTP server isn’t overloaded and that it’s properly routing requests. WebSphere ® Support Technical Exchange

32

IBM Software Group

Network Monitoring Script: Sample only. Not for production use. #!/usr/bin/sh while true; do date >> ping.out echo ------------------------------------------------------------ >> ping.out ping $1 >> ping.out echo ------------------------------------------------------------ >> ping.out date >> ping.out sleep 45 done

§ § §

Usage: script ip-address This should be run between all sections of the network Example: 4 Client pings HTTP Server and HTTP Server pings Client. 4 HTTP Server pings App Server and App Server pings HTTP Server 4 App Server ping Database and Database pings App Server

WebSphere ® Support Technical Exchange

33

IBM Software Group

Graph of Network Monitoring Difference tuned and network script:

0.62%

500.0 450.0

431.0

428.3

Tuned

Network Script

requests per second

400.0 350.0 300.0 259.7 250.0 200.0 150.0 100.0 50.0 0.0 Not Tuned

WebSphere ® Support Technical Exchange

34

IBM Software Group

Recommended Items § Why we recommend some and not others? 4 Performance penalty verse gains in problem determination. § Should be enabled during test, and can be enabled in production 4 verboseGC

< 2% performance difference

• Do not enable in production, unless debugging a problem. 4 tracing (with no trace string)

< 2% performance difference

4 Operating System monitoring

< 2% performance difference

4 Network monitoring

< 2% performance difference

4 Database monitoring

no tests run with this enabled

4 HTTP Server monitoring no tests run with this enabled § Be careful when enabling any monitoring in production § Note: Run tests before enabling these in production. Not all systems will have the same performance results.

WebSphere ® Support Technical Exchange

35

IBM Software Group

Graph of combined items Difference tuned and monitoring items enabled:

1.31%

500.0 450.0

431.0

425.3

Tuned

verboseGC, trace, OS script and network script

requests per second

400.0 350.0 300.0 259.7 250.0 200.0 150.0 100.0 50.0 0.0 Not Tuned

WebSphere ® Support Technical Exchange

36

IBM Software Group

Agenda § Background on performance 4 Definitions 4 Out-of-box vs. tuned application performance 4 Performance dependencies § Key tuning parameters 4 WebSphere Application Server 4 Application 4 Hardware § Performance impacts of monitoring 4 WebSphere Application Server 4 Operating System 4 Network § Performance tools 4 PMI 4 Tivoli Performance Viewer 4 Advisors § Conclusion 4 Questions 4 Links

WebSphere ® Support Technical Exchange

37

IBM Software Group

WebSphere Application Server Performance Tools § Performance Monitoring Infrastructure (PMI) 4 Infrastructure for gathering performance metrics from the application server. § Performance Viewers 4 Applications that allow for viewing of PMI data 4 Examples: • Tivoli Performance Viewer - Included with WebSphere Application Server • Other Tivoli products • 3rd party products § Performance Advisors 4 Applications that give tuning advice based on PMI data 4 Examples: • Included with WebSphere Application Server - Tivoli Performance Viewer advisor - Runtime Performance Advisor • 3rd party products

WebSphere ® Support Technical Exchange

38

IBM Software Group

Enabling PMI § Go to the specified location: 4 App server: • Servers > Application servers > server_name > Performance > Performance Monitoring Infrastructure (PMI) 4 Node agent: • System Administraton > Node Agents > node_agent_name > Additional Properties > Performance Monitoring Infrastructure (PMI) § Under the configuration tab 4 Check “Enable Performance Monitoring Infrastructure (PMI)” 4 Click “OK” or “Apply” § Save the changes § Restart the server

WebSphere ® Support Technical Exchange

39

IBM Software Group

WebSphere ® Support Technical Exchange

40

IBM Software Group

How to setup monitoring §Go to: Servers > Application servers > server_name > Performance > Performance Monitoring Infrastructure (PMI) > Runtime §Select the desired level §Click Apply

WebSphere ® Support Technical Exchange

41

IBM Software Group

Performance Viewers § To properly tune an environment one needs to find performance bottlenecks § Tuning settings at random takes longer and can lead to sub-optimal performance.

WebSphere Application Server

Performance Metrics

Performance Viewer/Advisor

Tuning Changes

WebSphere ® Support Technical Exchange

42

IBM Software Group

TPV – How to access § Go to: Monitoring and Tuning > Performance Viewer > Current Activity > server_name

WebSphere ® Support Technical Exchange

43

IBM Software Group

TPV – How to setup logging § Within TPV go to: server_name > Settings > Log

WebSphere ® Support Technical Exchange

44

IBM Software Group

WebSphere ® Support Technical Exchange

45

IBM Software Group

Tivoli Performance Viewer – Table View

WebSphere ® Support Technical Exchange

46

IBM Software Group

Tivoli Performance Viewer Graph View

§ SVG must be installed to view the graph. § Firefox 1.5 has SVG support. § IE Users can download an SVG plugin from Adobe.

WebSphere ® Support Technical Exchange

47

IBM Software Group

Tivoli Performance Viewer advisor § Gives advice on resources 4 Threads 4 Database Connections 4 Data sources 4 Session utilization § Can advise on configuration and runtime performance

WebSphere ® Support Technical Exchange

48

IBM Software Group

WebSphere ® Support Technical Exchange

49

IBM Software Group

Tivoli Performance Viewer advisor – Session advice

WebSphere ® Support Technical Exchange

50

IBM Software Group

Conclusion § Background on performance 4 Performance can increase significantly after WebSphere is tuned for the application and environment. § Key tuning parameters 4 Tuning WebSphere is only the beginning. Further tuning can be done to the application environment. § Performance impacts of monitoring 4 Doing a little monitoring does not have a heavy performance hit, and can save time fixing problems. § Performance tools 4 Monitoring performance metrics enables you to reach optimal tuning with minimal effort.

WebSphere ® Support Technical Exchange

51

IBM Software Group

Additional WebSphere Product Resources

§

Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: www.ibm.com/developerworks/websphere/community/

§

Learn about other upcoming webcasts, conferences and events: www.ibm.com/software/websphere/events_1.html Join the Global WebSphere User Group Community: www.websphere.org Access key product show-me demos and tutorials by visiting IBM Education Assistant: ibm.com/software/info/education/assistant

§ § § §

Learn about the Electronic Service Request (ESR) tool for submitting problems electronically: www.ibm.com/software/support/viewlet/probsub/ESR_Overview_viewlet_swf.html Sign up to receive weekly technical My support emails: www.ibm.com/software/support/einfo.html

WebSphere ® Support Technical Exchange

52

IBM Software Group

Backup

WebSphere ® Support Technical Exchange

53

IBM Software Group

Parameters we used to tune Trade 4Pass by reference = true 4Web container thread pool = 50 / 50 4Max persistent keep-alive requests = Unlimited (-1) 4Heap size 1024 / 1024 4Java security = disabled 4PMI service = disabled 4Uninstall default applications

WebSphere ® Support Technical Exchange

54

IBM Software Group

Our 2-tier environment § Server: 4 IBM p630 4 4-way @ 1.0 GHz CPU 4 8 GB memory 4 AIX 5.2 4 WebSphere Application Server 6.0.2 Base

§ Database: 4 IBM p570 4 4-way @ 1.6 GHz CPU 4 8 GB memory 4 AIX 5.3 4 DB2 V8.2 JCC Type 4 (XA) driver

§ Client: 4 IBM x330 Intel P3 Xeon 4 2-way @ 1.2 GHz CPU 4 2 GB memory 4 Linux SLES 9.0

WebSphere ® Support Technical Exchange

55

IBM Software Group

Other Useful Links § Performance Tuning Guide: 4 http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/topic/com.i bm.websphere.base.doc/info/aes/ae/welc6toptuning.html § Tivoli Performance Viewer: 4 http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?t opic=/com.ibm.websphere.nd.doc/info/welcome_nd.html § Advisors: 4 http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?t opic=/com.ibm.websphere.nd.doc/info/ae/ae/tprf_tuningentrypoin t.html § Trade: 4 https://www14.software.ibm.com/webapp/iwm/web/preLogin.do? source=trade6 § WebSphere Application Server – Information Center: 4 http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp

WebSphere ® Support Technical Exchange

56

IBM Software Group

Any Questions?

WebSphere ® Support Technical Exchange

57