gridengine.info

In this mailing list post (from which I am about to quote wholesale from ...) Dan gives a very good explanation on why the DRMAA API is not available on windows.

"...SFU is an API for Windows that looks similar to UNIX. When we ported the SGE execution daemon to Windows, we didn't do a native port; we ported it to SFU, which was considerably less work. In order for the execution daemon to function, it requires SFU to be installed. Otherwise, it won't run at all. You will have to install SFU on every execution machine. Starting with Vista, SFU is renamed SUA and is part of the base OS.

DRMAA is not currently available on Windows, but could be if there was a demonstrable need. The reason why it's not available is that it would also require SFU, which means that the DRMAA library would be a .so instead of a .dll. Because it would be a .so, any application that wanted to use DRMAA would also have to use SFU, and no sane person would start out writing a Windows application using SFU. A quick work-around is to write a simple daemon that uses RESTful web services to expose the DRMAA operations."

Enrico Sirola reports that an updated python module for interacting with DRMAA-compliant distributed resource management ("DRM") systems has been released.

The DRMAA working group website is http://www.drmaa.org/ for those looking for additional information.

%  jrunscript -cp $SGE_ROOT/lib/drmaa.jar -f drmaa.js

Job 2 submitted
Job 2 has ended
Job terminated abnormally
%

Richard Hierlmeier seems to have joined the ranks of Sun Bloggers and has a facinating post up documenting how he used the javascript engine that ships with Java 6 to bind to drmaa.jar.

The post is here:

http://blogs.sun.com/rhierlmeier/entry/java_drmaa_binding_with_javascript

Cheng Soon Ong mentions on the developer mailing list:

I've been playing around with python and sge, and translated the drmaa tutorial at http://gridengine.sunsource.net/project/gridengine/howto/drmaa.html to a python version. Look at sge_tutorial.rst in the attached archive.

The newly translated tutorial will likely find a home on gridengine.sunsource.net soon but in the mean time you can:

• Read the original mailing list post
• Read the follow-up post containing a patch
• Download the tutorial files

Most casual SGE users and admins probably find little cause to monitor the Grid Engine developer mailing list. A nice little success story has played out on the list recently with a user assisting the SGE dev team in quickly discovering, isolating and fixing a memory leak that has been in the codebase since the DRMAA 1.0 API release.

A user posted this message to the developer list, showing what appears to be a memory leak in in drmaa_run_job(). Andreas then replied asking if it was possible for the user to recreate the issue while running under the valgrind instrumentation framework.

In this follow-up thread, the user-provided valgrind data allowed Andreas to pinpoint the problem, file Issue #2497 with the bug tracking database and then post a preliminary patch that fixes the problem.

The patch still needs to undergo code review before it makes it officially into the Grid Engine codebase. Overall this is a nice little success story where a user was able to go the extra mile (by instrumenting under valgrind) in order to provide the developers exactly what they needed to quickly identify and fix things.

Kudos to James & Andreas.

The 22nd Open Grid Forum -- OGF22
Hyatt Regency Cambridge
Cambridge, MA USA
February 25-28, 2008
Website: http://www.ogf.org/OGF22/

Coworker Chris Dwan and I will be attending this event and one of us will likely end up speaking. Both of us are known to be somewhat cynical of the "big G" Grid Computing world so we'll be bringing our industry-centric views and bias towards practical solutions into the forum.

(I swear, when you speak to some of these "big G grid" supercomputing or academic folks you get the sense that they think that everyone has 100 million in government funding and a petabyte-scale single namespace storage solution to apply to the problems at hand....)

Among the other attendees that I know about, Chris Smith from Platform Computing will also be there -- he handles Platform's involvement with standards bodies and is another person on my "smart people that I learn a lot from" list. Should be an interesting event.

And finally, some discount registration offers for readers of this blog:

• "Buy one 1 pass and get a 2nd for free"
• "$150 discount off the purchase off the full day pass"

DanT has an interesting blog post up where he talks about two different methods getting down into invoking product-specific DRM features when using DRMAA clients. The article is a result of a SGE mailing list thread where someone found themselves wanting to invoke SGE-specific abilities that are not explicitly part of the DRMAA spec. Dan talks about the two main ways to go about doing this.

The full post is here:
http://blogs.sun.com/templedf/entry/using_drm_specific_functionality_via

On a related note, the "Writing Grid Enabled Applications" lab that Dan presented at JavaOne is now available online. You will have to register at the Sun Developer Network in order to view the video or download the materials. A nice walkthrough of DRMAA usage and the API.

There have been lots of interesting Wiki work and CVS commits over the past week or so, some highlights:

DRMAA 1.0 Java Bindings

Grid Engine CVS for the 6.1 release has been updated with Java bindings for DRMAA 1.0 support. People interested in Java language bindings for DRMAA have a very short time to review the 1.0 specifications and provide feedback. (via DanT's blog)

{draft} ARCo Howto's for MySQL and Oracle 10g committed to CVS

No formal announcement has been made regarding these documents so beware that they are drafts subject to change. The documents likely refer to versions of the Analysis and Reporting Console (ARCo) that will only be available in the upcoming SGE 6.1 snapshot and beta releases. That said though ...

• ARCo and MySQL Howto
• ARCo and Oracle 10g Howto | (CVS Head)

Boolean Expression Support - Implementation Details

A very detailed wiki page has been created to document the specifics of the new more flexible host and string resource expressions that will be supported in both the next point release of Grid Engine (6.0u10) and the upcoming 6.1snapshot2 releases:

http://wiki.gridengine.info/wiki/index.php/EvaluationExpressionSupport

Update: 13 June 2006
Andreas has started a formal Ruby-DRMAA project:

…the DL based Ruby wrapper for shared DRMAA C bindings now has it’s own home page and CVS repository under

http://drmaa4ruby.sunsource.net/

Andreas writes:

  http://gridengine.sunsource.net/files/documents/7/89/drmaa_ruby.tar.gz

my first version of a DRMAA binding for Ruby. Thanks to Ruby/DL it was
not necessary to use SWIG. Open issues are

  * assumes DRMAA 0.95 instead of DRMAA 1.0
  * memory leak due to JobTemplate destructor not yet functioning
  * a number of functions are not yet supported (drmaa_ps_job(),
    drmaa_control(), ...)

In related news, it appears that people are using and beginning to extend the Ruby-based SGE log-file analysis script mentioned previously on this site.

Interesting announcement on the mailing lists today regarding http://sourceforge.net/projects/opendsp.

From the website:

"OpenDSP is an open architecture implementation of SOAP Web Service multi-user access and policy-based job control using DRMAA (Distributed Resource Management Application API) routines supported by various distributed resource management systems."

For version v0.1 RC1 the feature list is pretty interesting. Hopefully I'll have time to experiment with this a bit.