OGF25/EGEE User Forum
Programming with DRMAA DRMAA + WS + JSDL
Krzyszof Kurowski Paweł Lichocki Mariusz Mamoński {krzysztof.kurowski,lichocki,mamonski}@man.poznan.pl
Agenda •introduction and motivations •DRMAA •idea •overview •routines •Academic point of view •running numerical algorithms •boosting performance •exemplary application •Industrial point of view •developing middleware •enhanced security, reliability and functionality •real-life applications
Introduction … back in Gridlab times •What we liked: • people, parties and results ;-) • idea = GAT + middleware • new scenarios and user driven
use cases, e.g. Zakopane/migration • a feedback we provided for O(G)GF •What we did not like: • the way middleware was integrated with DRMS • script-based job managers, new versions/releases, ... • security model, no support for advanced AAA scenarios • poor quality, no docs, many bugs and problems with deployment and support (at that time ;-) • we had to wait for OGSA, WSRF, … and we liked C and WS 3
Motivations •GridLab feedback: •GAT -> a new OGF standard: SAGA •Middleware -> OGF DRMAA, JSDL, HPC-Profile, ... •What was really important for our middleware: • performance (e.g. one million job submissions per day)
stability and portability (e.g. ANSI C not Java) • simple well accepted standards not only recommendations or best practices • flexibility, new security models, … • support for local and external users performing relatively simple job submission and monitoring operations (more advanced scenarios we were discussing under a new working group in OGF), thus DRMAA 4
DRMAA idea Application (binary file)
specific script
specific script
Application (binary file)
specific script
DRMAA application
unified API specific commands
specific commands
specific commands
vendor DRMAA lib
vendor DRMAA lib
vendor DRMAA lib
DRMS 1
DRMS 2
DRMS 3
DRMS 1
DRMS 2
DRMS 3
Overview •technically DRMAA may be seen as a substitute for the
dedicated scripts for job handling •from a broader perspective DRMAA is a type of parallel programming model (like OpenMP or MPI) •benefits •provides compatibility with all major DRMSs •eases the implementation effort and deployment •good at use cases such as workflows or parameter sweep •pitfalls •DRMAA application must be run from the submit host of the chosen DRMS •DRMAA application uses only one DRMS at a time
Routines in a nutshell •session handling •drmaa_init •drmaa_exit •job submission •job_template routines •drmaa_submit_job •drmaa_submit_bulk_jobs •job monitoring and control •drmaa_job_ps •drmaa_control •job synchronization •drmaa_wait •drmaa_synchronize
init
set template
submit job (template)
wait (job)
exit
DRMAA academic use case •Imagine an MPSD algorithm •It does not saturate the provided computational power •There exist different versions, each of them suitable in
other specific case •It is a crucial part of a very time-demanding application •It is (practically) impossible to know a priori which version to use (in order to minimize the execution time) •Such algorithms exist, for example •Simplex - a popular algorithm for numerical solution of the linear programming problem •Hyper-heuristics - an approach of choosing appropriate heuristic basing on instance characteristic
Example •Observations •many methods - standard, revised, Bartels-Golub, sparse
Bartels-Golub, Reid’s Method, ... •each method suits best different type of data (e.g. sparse or dense matrices) •the differences in execution time may be huge •Problem •one should analyze the input and choose the method which seems to be most suitable, but this takes time and might be incorrect since there are no clear criterias •Solution •run all methods, wait for the first to finish, gather results, terminate other methods
Application flow and assumpptions •the scheme presents the idea of
the DRMAA application •in the source code we assume •the binary file name is simplexN •the binary takes as an argument the path to the input file •the input file is named data.in •the input file is accessible on all execution hosts in the working directory (otherwise a file-staging must be used) •we ignore potential errors and failures of DRMAA calls
init for i = [0..N) set i-th template submit job (i-th template) j = wait (ANY) for i = [0..N) and i != j terminate job (i-th) exit
Application source code 1/2 int i; char e[DRMAA_ERROR_STRING_BUFFER]; size_t s = DRMAA_ERROR_STRING_BUFFER; drmaa_init( NULL, e, s ); drmaa_job_template_t *jt = NULL; drmaa_allocate_job_template( &jt, e, s ); const char *args[2] = { "data.in", NULL }; drmaa_set_vector_attribute( jt, DRMAA_V_ARGV, args, e, s );
init
allocate template set generic template
char jobid[ N ][DRMAA_JOBNAME_BUFFER]; for (i = 0; i < N; ++i) { for i = [0..N) char cmd[] = "simplex_"; cmd[6] = i + 48; set i-th template drmaa_set_attribute( jt, DRMAA_REMOTE_COMMAND, cmd, e, s ); drmaa_run_job( jobid[ i ], DRMAA_JOBNAME_BUFFER, jt, e, s ); submit job (i-th template) } drmaa_delete_job_template( jt, e, s );
delete template
Application source code 2/2 for (i = 0; i < N; ++i) { for i = [0..N) char jobid_out[DRMAA_JOBNAME_BUFFER]; int status = 0, aborted = 0, exited = 0; drmaa_attr_values_t *rusage = NULL; drmaa_wait( DRMAA_JOB_IDS_SESSION_ANY, jobid_out, j = wait (ANY) DRMAA_JOBNAME_BUFFER, &status, DRMAA_TIMEOUT_WAIT_FOREVER, &rusage, e, s ); drmaa_wifaborted( &aborted, status, NULL, 0 ); if (aborted != 1) { check if job exited normally drmaa_wifexited( &exited, status, NULL, 0 ); if (exited == 1) if yes do not wait again break; } }
for i = [0..N) /*and i != j*/ for (i = 0; i < N; ++i) drmaa_control( jobid[ i ], DRMAA_CONTROL_TERMINATE, e, s ); drmaa_exit(e, s);
terminate job (i-th) exit
Advanced programming in DRMAA •The solution was to use drmaa_wait with
DRMAA_JOB_IDS_SESSION_ANY. However waiting for any “normally terminated” job is not that straightforward •Other limitations •The previous approach allows us to run many algorithms simultaneously, but uses only one DRMS •The DRMAA application must be run from the submission host of the chosen DRMS •DRMAA does not cover issues regarding resource requests •Questions •What if we have access to many separated clusters? •What about security? •Solution •map DRMAA functionalities to web-services along with JSDL
Industrial approach - SMOA Computing • Successor of the OpenDSP (Open DRMAA Service Provider) • Web Service interface to DRMAA compliant systems • Adds authentication, authorization and accounting layers which are out of scope DRMAA specification • Robust implementation (C, gSOAP toolkit) • Modular architecture - new scenarios can be realized by addition of new C/Python modules) • Use of standard interfaces (JSDL, DRMAA, ODBC, BES HPC Basic Profile) - easier integration and maintenance • https://sourceforge.net/projects/smoa-project
JSDL to DRMAA mapping • ➞ DRMAA_JOB_NAME • ➞ DRMAA_REMOTE_COMMAND •* ➞ DRMAA_V_ARGV •* ➞ DRMAA_V_ENV • ➞ DRMAA_WD • ➞ DRMAA_INPUT_PATH • ➞ DRMAA_OUTPUT_PATH • ➞ DRMAA_ERROR_PATH • other JSDL elements if needed can be mapped to DRMAA_NATIVE_SPECIFICATION by dedicated SMOA Computing JSDL Filter module
BES to DRMAA mapping (interfaces) •CreateActivity ➞ drmaa_run_job •GetActivitiesStatuses ➞ drmaa_job_ps, drmaa_wait •TerminateActivities ➞ drmaa_control
Example SMOA realization My organization 2. 6. 3. 1.
Cluster A SMOA Computing
DRMAA Compliant System
Cluster B SMOA Computing
1 - Subscribe 2, 3 - Create Activity 4, 5 - Notify 6 - Terminate Activity
4.
5.
SMOA Notification
DRMAA Compliant System
OpenDSP use case G-Render - Grid-based Image Processing System •used in TeleHVEM •virtual laboratory for High Voltage Electron Microscope •e-science •Server system •Sun Grid Engine (SGE) for computational grid •OpenDSP for DRMAA web service •Client •Various image processing features
sge_schedd sge_execd sge_qmaster
OpenDSP
GRender GUI
GRender
GRender Agent
gSOAP stubs
Summary & Future • The best way to learn how to program in DRMAA is to start
writing your own DRMAA application instead of using native programming interfaces or scripts for a specific DRMS • If you have access to many computing resources managed by different DRMSs you may want to use SDKs to DRMAA Service Provider (now SMOA Computing) we developed for C/C++, Java, .NET as well as example tools like Vine/GS toolkit, Jabber clients, … • We have already extended DRMAA with a set of generic advanced reservation APIs (testing now with LSF, SGE, PBSPro) • We integrated our middleware with well known programming and execution environments ProActive (Java) and OpenMPI (C/ C++/Python) to manage cluster-to-cluster parallel apps • It should not be difficult to implement a new SAGA adaptor for SMOA Computing
Links and literature •[1] OGF DRMAA Working Group, http://www.drmaa.org •[2] DRMAA Specification, http://www.ogf.org/documents/GFD.133.pdf •[3] DRMAA C bindings, https://forge.gridforum.org/sf/docman/do/downloadDocument/
projects.drmaa-wg/docman.root.ggf_13/doc5545 •[4] Grid Engine HOWTOs, http://gridengine.sunsource.net/howto/howto.html#DRMAA •[5] FedStage DRMAA Wiki, http://wiki.fedstage.com/FedStage%20DRMAA •[6] FedStage DRMAA for PBS PRO, http://sourceforge.net/projects/pbspro-drmaa •[7] FedStage DRMAA for LSF, http://sourceforge.net/projects/lsf-drmaa •[8] JSDL Working Group, http://forge.gridforum.org/sf/sfmain/do/viewProject/ projects.jsdl-wg •[9] JSDL Specification, http://www.gridforum.org/documents/GFD.56.pdf •[10] gSOAP project, http://www.cs.fsu.edu/~engelen/soap.html •[11] FedStage Open DRMAA Service Provider, http://sourceforge.net/projects/opendsp •[12] TeleHVEM, http://goc.pragma-grid.net/wiki/images/2/2e/Hvem-yeom.pdf
•[13] SMOA Project, http://sf.net/projects/smoa-project
Thank you