How To Add a New Node To An Existing MDM WebSphere Cluster Supporting MDM v11.0+ and WebSphere v8.5.0.2+

Perform the following steps in the order given: 1. 2. 3. 4.

Create a new node, attach it to the existing cluster, add an member server instance. Create a new JDBC provider and data source. Configure InfoSphere MDM on the new node. Delete and re-create EBA to propagate all assets to all nodes.

You will need access to your application server's file system, database, and WebSphere console. Be prepared to modify files, to update database records, and to run 'madconfig.sh' utility. At the end of this document, you will find a troubleshooting section that lists common errors and resolutions during the update process.

Create a new node, attach it to the existing cluster, add an member server instance A node in WebSphere Application Server is defined as a profile. It can exist on a separate computer or on a computer that already has one or more existing profiles configured. To create a new node and attached it to an existing cluster, you can issue the following command from a command line on the application server. cd /opt/IBM/WebSphere/AppServer/bin ./manageprofiles.sh -create -profileName AppSrv03 -profilePath /opt/IBM/WebSphere/AppServer/profiles/AppSrv03 -templatePath /opt/IBM/WebSphere/AppServer/profileTemplates/managed -federateLater false -dmgrAdminUserName perf01 -dmgrAdminPassword perf01 -dmgrHost warhol-10ge -dmgrPort 8879 Replace profileName, path, dmgrHost, UserName, Password and dmgrPort with your own values. Do the same throughout this document. Once that command completes, you should see an additional node added in the WebSphere Application Server console, as shown below (System administration → Nodes):

Now, to add a new server instance to the cluster on the new node, navigate to WebSphere Application Server clusters > SECluster > Cluster members, and click on New. Give a name to the new member

and select the new node from the drop down, and leave all other as default. Save and synchronize.

Once added, you should see the new member in the list, WebSphere application server clusters > SECluster > Cluster members:

Now we need to update properties in the new application server. First, update the Classpath and Generic JVM arguments. Navigate to Middleware servers > MDMServer2SECluster > Process definition > Java Virtual Machine. Classpath = $ {USER_INSTALL_ROOT}/installedAssets/com.ibm.mdm.server.resources.propertiesE001.jar/BASE/com.ibm.mdm.server.resources.properties-E001.jar Generic JVM arguments = -Dorg.eclipse.emf.common.util.URI.archiveSchemes=wsjar -Xmso512k

Next, create the custom properties for the JVM. Navigate to Middleware servers > MDMServer2SECluster > Process definition > Java Virtual Machine > Custom properties, create the following: • • • • • • • •

MDM_DEPLOYMENT_NAME = ${MDM_DEPLOYMENT_NAME} MDM_INSTANCE_NAME = ${MDM_INSTANCE_NAME} com.ibm.security.jgss.debug = off com.ibm.security.krb5.Krb5Debug = off com.ibm.ws.eba.fast.bundles = true com.ibm.ws.eba.pmi.enabled = false felix.fileinstall.dir = ${mad.root.dir}/conf felix.fileinstall.filter = (.*)\.cfg

• •

mad.root.dir = /opt/IBM/WebSphere/AppServer/profiles/AppSrv03/installedApps/warholCell01/MDMnative-E001.ear/native.war org.apache.aries.blueprint.header.mandatory = true org.osgi.framework.library.extensions = .a,.so

•

webservices.unify.faults = false

•

Next, create two environment entries on the JVM. Navigate to Middleware servers > MDMServer2SECluster > Process definition > Environment Entries. Create two entries as follows: • •

LD_LIBRARY_PATH = ${mad.root.dir}/lib MAD_ROOTDIR = ${mad.root.dir}

Restart nodgeagent for the new node.

Create a new JDBC provider and data source In InfoSphere MDM v11, the “MDM” datasource isn't automatically created for the new server. You must first create a JDBC provider, then create a data source having the same properties as the ones used by the existing nodes. Before we can add a JDBC provider and data source, we need to update a few of WebSphere variables. They were created as part of adding the new node. These variables are used by data sources and other JVM properties later. Navigate to Environment → WebSphere variables, and set scope to the new server. There needs to be 5 variables. You may see one or two already created for you, so please make sure all are created, as shown, note the references to the newly created server 3:

Navigate to Resources → JDBC → JDBC providers, and change the scope to just the newly added server,

Click on New and use the following values • • • •

Database type = User-defined Implementation class name = com.ibm.mdm.jdbc.DB2DataSource Name = JDBC provider for Virtual MDM Description = JDBC provider for Virtual MDM

Then click on Next. In the Class path box, enter the following: ${User-defined_JDBC_DRIVER_PATH}/databaseBinary.jar ${mad.root.dir}/lib/madjdbc.jar These values should be the same as ones used in existing nodes. You should always double-check the ones from your environment. Finish the steps and Save and synchronize. Once done, you should see the new provider as shown:

Next, create a data source using this JDBC provider. Navigate to the Data sources pane off the new JDBC provider, JDBC providers > JDBC provider for Virtual MDM > Data sources, click on New. Use the following values in the fields: • •

Data source name = MDM JNDI name = jdbc/MDM

Click on Next and keep the default setting. Click on Next and choose a working authentication alias for this data source to sign into database. If you haven't created it, you can do so by clicking on the link on this page, “Global J2C authentication alias”.

Finish, Save and synchronize. Next, create custom properties required for the data source. Navigate to Data sources > MDM > Custom properties for the new node. Click on New to create the following properties and values: • • • • • •

databaseName = your database name serverName = data server host name portNumber = database port, e.g., 60000 user = database login user name password = database user password dynamicSections =1000

Once created, they should look like this:

At this point, you should have 3 “MDM” data sources, one for each server, as shown.

You can now test connection on all of them. Make sure they all connect successfully.

If you have trouble getting data sources to work successfully, refer to the Troubleshooting section for more information.

Configure InfoSphere MDM native components on the new node WebSphere Application Server will automatically deploy the InfoSphere MDM installed applications onto the new node, but you must configure the InfoSphere MDM native components. This is done through creating a set of configuration files and create ODBC connection properties on the application server's file system. The easiest way to do this is to copy over a working set of files from an existing node. Run the following command: cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv03/installedApps/warholCell01/MDMnative-E001.ear/native.war/conf cp /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/warholCell01/MDMnative-E001.ear/native.war/conf/*.cfg . cp /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/warholCell01/MDMnative-E001.ear/native.war/conf/*.ini . There should be now 8 *.cfg files and one .ini file: -rw-r--r--rw-r--r--rw-r--r--rw-r--r--rw-r--r--rw-r--r--rw-r--r--rw-r--r--rw-r--r--

1 1 1 1 1 1 1 1 1

root root root root root root root root root

root root root root root root root root root

105 148 101 45 49 150 184 232 602

Jan Jan Jan Jan Jan Jan Jan Jan Jan

17 17 17 17 17 17 17 17 16

01:14 01:14 01:14 01:14 01:14 01:14 01:14 01:14 19:16

com.ibm.mdm.mds.entity.manager.cfg com.ibm.mdm.mds.job.manager.cfg com.ibm.mdm.mds.event.manager.cfg com.ibm.mdm.mds.messages.cfg com.ibm.mdm.mds.net.logic.cfg com.ibm.mdm.mds.search.index.cfg com.ibm.mdm.mds.jdbc.cfg com.ibm.mdm.mds.jni.cfg odbc.ini

Go through every cfg file, and update any reference to node to the current new node, e.g., changing AppSrv01 to AppSrv03. Keep everything else the same. Then we need to create an ODBC data source. Open a command-line prompt, and change directory to /installedApps//MDM-native-XXX.ear/native.war/scripts, cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv03/installedApps/warholCell01/MDMnative-E001.ear/native.war/scripts ./madconfig.sh register_odbc ./madconfig.sh create_datasource -Dmad.db.type=DB2 -Dmad.db.name=bluedog -Dmad.db.port=60000 -Dmad.db.host=rembrandt-10ge -Dmad.db.dsn=bluedog_E001

Where: DBTYPE: is your database type; specify DB2, DB2Z, ORACLE, or MSSQLU DBHOST: is your database host name or IP address DBPORT: is your database port DBNAME: is your database name DSN: is the data source name; default naming convention is _, where MDM_INSTANCE_IDENTIFIER usually is "E001".

The last step in this section is to update the database with the addition of the new server. The table APPINSTANCE stores all InfoSphere MDM application server names and their sequence numbers. Your existing APPINSTANCE table should have two nodes: db2 "select * from appinstance" INSTANCE_ID DEPLOYMENT_ID -------------------- -------------------1 1000 2 1000

NAME -------------------------MDMServer1SECluster MDMServer2SECluster

2 record(s) selected.

You must run an insert command to add the third instance. db2 "INSERT INTO appinstance (INSTANCE_ID, DEPLOYMENT_ID, NAME, LAST_UPDATE_DT, LAST_UPDATE_USER) VALUES (3, 1000, 'MDMServer3SECluster', CURRENT DATE, 'installer')"

At the end, you should have all 3 nodes in the table: INSTANCE_ID DEPLOYMENT_ID -------------------- -------------------1 1000 2 1000 3 1000

NAME ------------------------MDMServer1SECluster MDMServer2SECluster MDMServer3SECluster

3 record(s) selected.

Delete and re-create EBA to propagate all assets to all nodes In order for InfoSphere MDM EBA to propagate properly onto all nodes, it is the best to delete it, redeploy and re-map roles to users. This step requires both the WebSphere Application Server console and running the utility called 'madconfig.sh' on the file system. For these updates to work, we recommend increase the deployment manager's max heapsize from the default size of 256MB to 1GB. You can change it in the WebSphere Application Server console as shown:

Restart the Dmgr. Stop the cluster.

Next, delete the business-level application EBA-E001:

Save and synchronize. Next, delete all the internal bundles Environment->OSGi Bundle->Internal bundle repository:

Save and synchronize. Next, delete assets for the InfoSphere MDM operational server EBA. Select all the items in the screen shown and click Delete:

Save and synchronize. Now, we are ready to re-deploy the operational server again. Run the following two commands from a command line on the application server. copy /opt/IBM/MDM/properties/install_mdm_eba.properties to /opt/IBM/MDM/mds/scripts, and rename it to "local.properties" /opt/IBM/MDM/mds/scripts/madconfig.sh install_mdm_eba

copy /opt/IBM/MDM/properties/install_prop_file_jar.properties to /opt/IBM/MDM/mds/scripts, and rename it to "local.properties" /opt/IBM/MDM/mds/scripts/madconfig.sh install_prop_file_jar /opt/IBM/MDM/mds/scripts/madconfig.sh map_roles_to_users /opt/IBM/MDM/mds/scripts> ./madconfig.sh map_roles_to_users Buildfile: build.xml map_roles_to_users: Creating new property file: /opt/IBM/MDM/properties/map_roles_to_users.properties Updating property file: /opt/IBM/MDM/properties/map_roles_to_users.properties Updating property file: /opt/IBM/MDM/properties/map_roles_to_users.properties Updating property file: /opt/IBM/MDM/properties/map_roles_to_users.properties Updating property file: /opt/IBM/MDM/properties/map_roles_to_users.properties Updating property file: /opt/IBM/MDM/properties/map_roles_to_users.properties # # Enter EBA prefix: # com.ibm.mdm.hub.server.app Updating property file: /opt/IBM/MDM/properties/map_roles_to_users.properties # # Enter WAS application name: # MDM-web-services-E001 Updating property file: /opt/IBM/MDM/properties/map_roles_to_users.properties # # Enter BLA user: # mdmadmin Updating property file: /opt/IBM/MDM/properties/map_roles_to_users.properties

# # Enter BLA password: # mdmadmin Updating property file: /opt/IBM/MDM/properties/map_roles_to_users.properties WASX7209I: Connected to process "dmgr" on node warholCellManager01 using SOAP connector; The type of process is: DeploymentManager WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: "[--blaName, MDM-operational-server-EBA-E001, --ebaPrefix, com.ibm.mdm.hub.server.app, --blaUser, mdmadmin, --wsName, MDM-web-services-E001, --blaPwd, mdmadmin]" Configuring application security --------------------------------------------------------------AdminBLA: List composition units in a BLA BLA ID (or BLA name): MDM-operational-server-EBA-E001 Optional parameter: Display description: Usage: AdminBLA.listCompUnits("MDM-operational-server-EBA-E001", "") Return: List the composition units within the specified business-level application. --------------------------------------------------------------cuID is WebSphere:cuname=com.ibm.mdm.hub.server.app-E001_0001.eba Mapping roles to users Mapped roles to users for EBA ADMA5075I: Editing of application MDM-web-services-E001 started. ADMA5058I: Application and module versions are validated with versions of deployment targets. ADMA5005I: The application MDM-web-services-E001 is configured in the WebSphere Application Server repository. ADMA5005I: The application MDM-web-services-E001 is configured in the WebSphere Application Server repository. ADMA5005I: The application MDM-web-services-E001 is configured in the WebSphere Application Server repository. ADMA5005I: The application MDM-web-services-E001 is configured in the WebSphere Application Server repository. ADMA5113I: Activation plan created successfully. ADMA5011I: The cleanup of the temp directory for application MDM-web-services-E001 is complete. ADMA5076I: Application MDM-web-services-E001 edited successfully. The application or its web modules may require a restart when a save is performed. Mapped roles to users for WS-EAR Mapping Run As roles to users Mapped Run As roles to users for EBA Mapped Run As roles to users for WS-EAR Saving the configuration. BUILD SUCCESSFUL Total time: 6 minutes 26 seconds

Once successfully deployed, perform a Full Resynchronize on all nodes, as shown below:

Your newly expanded cluster is ready to GO!

Troubleshooting Error: Newly created data source fails on test connection. Symptoms: You may get the following exception on “test connection”. [1/14/14 17:52:13:058 EST] 000000e9 DSConfigHelpe W DSRA0174W: Warning: GenericDataStoreHelper is being used. [1/14/14 17:52:13:062 EST] 000000e9 DataSourceCon E DSRA8040I: Failed to connect to the DataSource. Encountered "": java.lang.IncompatibleClassChangeError at com.initiatesystems.db.jdbcx.db2base.BaseDataSource.getConnectionProperties(Unknown Source) at com.initiatesystems.db.jdbcx.db2base.ddz.a(Unknown Source) at com.initiatesystems.db.jdbcx.db2base.ddz.getXAConnection(Unknown Source) at com.ibm.mdm.jdbc.DB2DataSource.getXAConnection(Unknown Source) at com.ibm.ws.rsadapter.DSConfigHelper$1.run(DSConfigHelper.java:1251) at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5459) at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:558 5) at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255) at com.ibm.ws.rsadapter.spi.ServerFunction$6.run(ServerFunction.java:565) at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118) at com.ibm.ws.rsadapter.DSConfigHelper.getPooledConnection(DSConfigHelper.java:1266) at com.ibm.ws.rsadapter.DSConfigHelper.getPooledConnection(DSConfigHelper.java:1174) at com.ibm.ws.rsadapter.DSConfigurationHelper.getConnectionFromDSOrPooledDS(DSConfigur ationHelper.java:1975)

Resolution: Restart node agent for all nodes. Error: Madconfig.sh fails with some environment variable not found. Resolution: Simply view the madconfig.sh file and change that reference to the environment variable to a hard-coded path. The is only used rarely. Error: SIB errors appear in the SystemOut.log and ffdc. Symptoms: You may get the following error. [1/16/14 13:38:57:737 EST] FFDC Exception:com.ibm.websphere.sib.exception.SIErrorException SourceId:com.ibm.ws.sib.processor.impl.DestinationManag er.createDestinationLocalization ProbeId:1:3747:1.508.1.8 Reporter:com.ibm.ws.sib.processor.impl.DestinationManager@1fa0af15 com.ibm.websphere.sib.exception.SIErrorException: CWSIP0006E: An internal destination configuration error occurred to destination MDM.MESSAGING.BACKOU T in DestinationManager, 1:4859:1.508.1.8 at com.ibm.ws.sib.processor.impl.DestinationManager.checkQueuePointContainsLocalME(DestinationManager.java :4856) at com.ibm.ws.sib.processor.impl.DestinationManager.checkValidLocalizationConfig(DestinationManager.java:4 167) at com.ibm.ws.sib.processor.impl.DestinationManager.createDestinationLocalization(DestinationManager.java:

3810)

at com.ibm.ws.sib.processor.impl.DestinationManager.createDestinationLocalization(DestinationManager.java: 3709) at com.ibm.ws.sib.processor.impl.AdministratorImpl.createDestinationLocalization(AdministratorImpl.java:44 4) at com.ibm.ws.sib.admin.impl.JsLocalizer.loadLocalizations(JsLocalizer.java:518) at com.ibm.ws.sib.admin.impl.BaseMessagingEngineImpl.loadLocalizations(BaseMessagingEngineImpl.java:2126) at com.ibm.ws.sib.processor.impl.MessageProcessor.startInternal(MessageProcessor.java:1584) at com.ibm.ws.sib.processor.impl.MessageProcessor.start(MessageProcessor.java:1346) at com.ibm.ws.sib.admin.impl.JsMessagingEngineImpl.start(JsMessagingEngineImpl.java:627) at com.ibm.ws.sib.admin.impl.HAManagerMessagingEngineImpl.activate(HAManagerMessagingEngineImpl.java:1242) at com.ibm.ws.sib.admin.impl.JsActivationThread.run(JsActivationThread.java:94) ==> Performing default dump from com.ibm.ws.sib.utils.ffdc.SibDiagnosticModule :Thu Jan 16 13:38:57 EST 2014 SIB FFDC dump for::com.ibm.websphere.sib.exception.SIErrorException: CWSIP0006E: An internal destination configuration error occurred to destination M DM.MESSAGING.BACKOUT in DestinationManager, 1:4859:1.508.1.8 Platform Messaging :: Messaging engine::[MDM.SIB.SECluster:SECluster.000-MDM.SIB.SECluster] Platform Messaging :: Release name: :WAS85.SIB Platform Messaging :: Level name: :cf021305.01 com.ibm.ws.sib.processor.impl.DestinationManager:@(#) SIB/ws/code/sib.processor.impl/src/com/ibm/ws/sib/processor/impl/DestinationManager.java, SIB.pro cessor, WAS85.SIB, cf021304.02 1.508.1.8 com.ibm.ws.sib.processor.impl.AdministratorImpl:@(#) SIB/ws/code/sib.processor.impl/src/com/ibm/ws/sib/processor/impl/AdministratorImpl.java, SIB.proce ssor, WAS85.SIB, cf021304.02 1.117 com.ibm.ws.sib.admin.impl.JsLocalizer:@(#) 1.84 SIB/ws/code/sib.admin.impl/src/com/ibm/ws/sib/admin/impl/JsLocalizer.java, SIB.admin, WAS85.SIB, cf021 304.02 09/04/07 11:14:09 [2/6/13 15:23:09]

Resolution: on the database, drop all SIB tables using the following command. db2 "select 'db2 drop table '||tabname from syscat.tables where tabschema = 'PERF01' and tabname like 'SIB%'" 1 ---------------------------------------------------------------------------------------------------------------------------------------------db2 drop table SIB000 db2 drop table SIB001 db2 drop table SIB002 db2 drop table SIBCLASSMAP db2 drop table SIBKEYS db2 drop table SIBLISTING db2 drop table SIBOWNER db2 drop table SIBOWNERO db2 drop table SIBXACTS

Restart the cluster. SIB tables will be re-created and updated with the new cluster configuration. Error: Error deploying EBA – database connection error. Resoultion: Changes are the encrypted password in the local.properties file is used as literals in this utility. So you will need to change that manually to a clear text password. /opt/IBM/MDM/properties> cat ../mds/scripts/local.properties #Madconfig Response Property File #Fri Nov 01 18:22:56 EDT 2013 wasUser=perf01 blaName=MDM-operational-server-EBA-E001

dbType=DB2 wasHome=/opt/IBM/WebSphere/AppServer propFileJarName=com.ibm.mdm.server.resources.properties-E001.jar dbHost=rembrandt-10ge dbName=bluedog wasHost=warhol-10ge wasPwd={xor}LzotOW9u dbPort=60000 cluster.flag=true cluster=SECluster cell=warholCell01 wasPort=8879 trustFile=/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/etc/DummyClientTrustFile.jks trustFilePwd={xor}CDo9Hgw=

Change the “wasPwd” to clear text password. Try the utility again. Error: When re-deploying BLA, Dmgr hangs. Cannot access the WebSphere Application Server console any more. Symptoms: You may get the following Errors. com.ibm.websphere.management.exception.ConnectorException: ADMC0009E: The system failed to make the SOAP RPC call: invoke at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplateOnce(SOAPConnectorClient.java:88 6) at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:682) at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:672) at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:658 ) at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:480 ) at com.sun.proxy.$Proxy0.invoke(Unknown Source) at com.ibm.ws.management.AdminClientImpl.invoke(AdminClientImpl.java:224) at com.ibm.ws.management.cmdframework.impl.ClientTaskCommand.processTaskParameters(ClientTaskCommand.java: 184) at com.ibm.websphere.management.async.client.AsyncCommandClient.processCommandParameters(AsyncCommandClien t.java:140) at com.ibm.mdm.de.config.appserver.core.MDMAppDeploy.createAsset(MDMAppDeploy.java:273) at com.ibm.mdm.de.config.appserver.core.configurator.MDMPhAppConfigurator.configure(MDMPhAppConfigurator.j ava:37) at com.ibm.mdm.de.config.appserver.remoteclient.command.DeployMDMPhAppRemoteCommand.performInstall(DeployM DMPhAppRemoteCommand.java:30) at com.ibm.mdm.de.config.appserver.remoteclient.command.DeployMDMPhAppRemoteCommand.execute(DeployMDMPhApp Re

Resolution: Increase Dmgr maximum heap to 1GB, and restart Dmgr.