OGF25/EGEE User Forum

Programming with DRMAA DRMAA + WS + JSDL

Krzyszof Kurowski Paweł Lichocki Mariusz Mamoński {krzysztof.kurowski,lichocki,mamonski}@man.poznan.pl

Agenda •introduction and motivations •DRMAA •idea •overview •routines •Academic point of view •running numerical algorithms •boosting performance •exemplary application •Industrial point of view •developing middleware •enhanced security, reliability and functionality •real-life applications

Introduction … back in Gridlab times •What we liked: • people, parties and results ;-) • idea = GAT + middleware • new scenarios and user driven

use cases, e.g. Zakopane/migration • a feedback we provided for O(G)GF •What we did not like: • the way middleware was integrated with DRMS • script-based job managers, new versions/releases, ... • security model, no support for advanced AAA scenarios • poor quality, no docs, many bugs and problems with deployment and support (at that time ;-) • we had to wait for OGSA, WSRF, … and we liked C and WS 3

Motivations •GridLab feedback: •GAT -> a new OGF standard: SAGA •Middleware -> OGF DRMAA, JSDL, HPC-Profile, ... •What was really important for our middleware: • performance (e.g. one million job submissions per day)

stability and portability (e.g. ANSI C not Java) • simple well accepted standards not only recommendations or best practices • flexibility, new security models, … • support for local and external users performing relatively simple job submission and monitoring operations (more advanced scenarios we were discussing under a new working group in OGF), thus DRMAA 4

DRMAA idea Application (binary file)

specific script

specific script

Application (binary file)

specific script

DRMAA application

unified API specific commands

specific commands

specific commands

vendor DRMAA lib

vendor DRMAA lib

vendor DRMAA lib

DRMS 1

DRMS 2

DRMS 3

DRMS 1

DRMS 2

DRMS 3

Overview •technically DRMAA may be seen as a substitute for the

dedicated scripts for job handling •from a broader perspective DRMAA is a type of parallel programming model (like OpenMP or MPI) •benefits •provides compatibility with all major DRMSs •eases the implementation effort and deployment •good at use cases such as workflows or parameter sweep •pitfalls •DRMAA application must be run from the submit host of the chosen DRMS •DRMAA application uses only one DRMS at a time

Routines in a nutshell •session handling •drmaa_init •drmaa_exit •job submission •job_template routines •drmaa_submit_job •drmaa_submit_bulk_jobs •job monitoring and control •drmaa_job_ps •drmaa_control •job synchronization •drmaa_wait •drmaa_synchronize

init

set template

submit job (template)

wait (job)

exit

DRMAA academic use case •Imagine an MPSD algorithm •It does not saturate the provided computational power •There exist different versions, each of them suitable in

other specific case •It is a crucial part of a very time-demanding application •It is (practically) impossible to know a priori which version to use (in order to minimize the execution time) •Such algorithms exist, for example •Simplex - a popular algorithm for numerical solution of the linear programming problem •Hyper-heuristics - an approach of choosing appropriate heuristic basing on instance characteristic

Example •Observations •many methods - standard, revised, Bartels-Golub, sparse

Bartels-Golub, Reid’s Method, ... •each method suits best different type of data (e.g. sparse or dense matrices) •the differences in execution time may be huge •Problem •one should analyze the input and choose the method which seems to be most suitable, but this takes time and might be incorrect since there are no clear criterias •Solution •run all methods, wait for the first to finish, gather results, terminate other methods

Application flow and assumpptions •the scheme presents the idea of

the DRMAA application •in the source code we assume •the binary file name is simplexN •the binary takes as an argument the path to the input file •the input file is named data.in •the input file is accessible on all execution hosts in the working directory (otherwise a file-staging must be used) •we ignore potential errors and failures of DRMAA calls

init for i = [0..N) set i-th template submit job (i-th template) j = wait (ANY) for i = [0..N) and i != j terminate job (i-th) exit

Application source code 1/2 int i; char e[DRMAA_ERROR_STRING_BUFFER]; size_t s = DRMAA_ERROR_STRING_BUFFER; drmaa_init( NULL, e, s ); drmaa_job_template_t *jt = NULL; drmaa_allocate_job_template( &jt, e, s ); const char *args[2] = { "data.in", NULL }; drmaa_set_vector_attribute( jt, DRMAA_V_ARGV, args, e, s );

init

allocate template set generic template

char jobid[ N ][DRMAA_JOBNAME_BUFFER]; for (i = 0; i < N; ++i) { for i = [0..N) char cmd[] = "simplex_"; cmd[6] = i + 48; set i-th template drmaa_set_attribute( jt, DRMAA_REMOTE_COMMAND, cmd, e, s ); drmaa_run_job( jobid[ i ], DRMAA_JOBNAME_BUFFER, jt, e, s ); submit job (i-th template) } drmaa_delete_job_template( jt, e, s );

delete template

Application source code 2/2 for (i = 0; i < N; ++i) { for i = [0..N) char jobid_out[DRMAA_JOBNAME_BUFFER]; int status = 0, aborted = 0, exited = 0; drmaa_attr_values_t *rusage = NULL; drmaa_wait( DRMAA_JOB_IDS_SESSION_ANY, jobid_out, j = wait (ANY) DRMAA_JOBNAME_BUFFER, &status, DRMAA_TIMEOUT_WAIT_FOREVER, &rusage, e, s ); drmaa_wifaborted( &aborted, status, NULL, 0 ); if (aborted != 1) { check if job exited normally drmaa_wifexited( &exited, status, NULL, 0 ); if (exited == 1) if yes do not wait again break; } }

for i = [0..N) /*and i != j*/ for (i = 0; i < N; ++i) drmaa_control( jobid[ i ], DRMAA_CONTROL_TERMINATE, e, s ); drmaa_exit(e, s);

terminate job (i-th) exit

Advanced programming in DRMAA •The solution was to use drmaa_wait with

DRMAA_JOB_IDS_SESSION_ANY. However waiting for any “normally terminated” job is not that straightforward •Other limitations •The previous approach allows us to run many algorithms simultaneously, but uses only one DRMS •The DRMAA application must be run from the submission host of the chosen DRMS •DRMAA does not cover issues regarding resource requests •Questions •What if we have access to many separated clusters? •What about security? •Solution •map DRMAA functionalities to web-services along with JSDL

Industrial approach - SMOA Computing • Successor of the OpenDSP (Open DRMAA Service Provider) • Web Service interface to DRMAA compliant systems • Adds authentication, authorization and accounting layers which are out of scope DRMAA specification • Robust implementation (C, gSOAP toolkit) • Modular architecture - new scenarios can be realized by addition of new C/Python modules) • Use of standard interfaces (JSDL, DRMAA, ODBC, BES HPC Basic Profile) - easier integration and maintenance • https://sourceforge.net/projects/smoa-project

JSDL to DRMAA mapping • ➞ DRMAA_JOB_NAME • ➞ DRMAA_REMOTE_COMMAND •* ➞ DRMAA_V_ARGV •* ➞ DRMAA_V_ENV • ➞ DRMAA_WD • ➞ DRMAA_INPUT_PATH • ➞ DRMAA_OUTPUT_PATH • ➞ DRMAA_ERROR_PATH • other JSDL elements if needed can be mapped to DRMAA_NATIVE_SPECIFICATION by dedicated SMOA Computing JSDL Filter module

BES to DRMAA mapping (interfaces) •CreateActivity ➞ drmaa_run_job •GetActivitiesStatuses ➞ drmaa_job_ps, drmaa_wait •TerminateActivities ➞ drmaa_control

Example SMOA realization My organization 2. 6. 3. 1.

Cluster A SMOA Computing

DRMAA Compliant System

Cluster B SMOA Computing

1 - Subscribe 2, 3 - Create Activity 4, 5 - Notify 6 - Terminate Activity

4.

5.

SMOA Notification

DRMAA Compliant System

OpenDSP use case G-Render - Grid-based Image Processing System •used in TeleHVEM •virtual laboratory for High Voltage Electron Microscope •e-science •Server system •Sun Grid Engine (SGE) for computational grid •OpenDSP for DRMAA web service •Client •Various image processing features

sge_schedd sge_execd sge_qmaster

OpenDSP

GRender GUI

GRender

GRender Agent

gSOAP stubs

Summary & Future • The best way to learn how to program in DRMAA is to start

writing your own DRMAA application instead of using native programming interfaces or scripts for a specific DRMS • If you have access to many computing resources managed by different DRMSs you may want to use SDKs to DRMAA Service Provider (now SMOA Computing) we developed for C/C++, Java, .NET as well as example tools like Vine/GS toolkit, Jabber clients, … • We have already extended DRMAA with a set of generic advanced reservation APIs (testing now with LSF, SGE, PBSPro) • We integrated our middleware with well known programming and execution environments ProActive (Java) and OpenMPI (C/ C++/Python) to manage cluster-to-cluster parallel apps • It should not be difficult to implement a new SAGA adaptor for SMOA Computing

Links and literature •[1] OGF DRMAA Working Group, http://www.drmaa.org •[2] DRMAA Specification, http://www.ogf.org/documents/GFD.133.pdf •[3] DRMAA C bindings, https://forge.gridforum.org/sf/docman/do/downloadDocument/

projects.drmaa-wg/docman.root.ggf_13/doc5545 •[4] Grid Engine HOWTOs, http://gridengine.sunsource.net/howto/howto.html#DRMAA •[5] FedStage DRMAA Wiki, http://wiki.fedstage.com/FedStage%20DRMAA •[6] FedStage DRMAA for PBS PRO, http://sourceforge.net/projects/pbspro-drmaa •[7] FedStage DRMAA for LSF, http://sourceforge.net/projects/lsf-drmaa •[8] JSDL Working Group, http://forge.gridforum.org/sf/sfmain/do/viewProject/ projects.jsdl-wg •[9] JSDL Specification, http://www.gridforum.org/documents/GFD.56.pdf •[10] gSOAP project, http://www.cs.fsu.edu/~engelen/soap.html •[11] FedStage Open DRMAA Service Provider, http://sourceforge.net/projects/opendsp •[12] TeleHVEM, http://goc.pragma-grid.net/wiki/images/2/2e/Hvem-yeom.pdf

•[13] SMOA Project, http://sf.net/projects/smoa-project

Thank you