gridengine.info

Congratulations to the SGE development, release and testing teams who have announced today that Grid Engine 6.2u5 is officially out! This is a "new features" release as well which is always excellent.

As usual, read the full announcement text here:
http://gridengine.sunsource.net/news/SGE62u5-announce.html

For me, the highlight new features are going to be the array job throttling tools (previously available in undocumented form) and slotwise preemption. I suspect others are really going to like the new ability to do topology aware job placement via the new job2core features.

And continuing in a trend of putting some value-adding features only in the commercial version of SGE available directly from Sun, there are some features that won't be built into the freely available courtesy binaries:

• Enhancements to the sgeInspect GUI including more graphical wizards for PE and cloud adapter configuration
• Formal support for Amazon EC2 cloud instances via a cloud adaptor that integrates with Sun's SDM service domain management stack
• Native support for Hadoop/SGE integration (!!)

Most compelling of the "commercial only" features for me is probably the seamless Hadoop integration -- I need to take a long look at seeing how that functions and performs. The sgeInspect tool is fantastic in action but not something I'd truly consider essential to have. Same goes for the SDM/Cloud/EC2 stuff -- it's nice to see it happening but I have yet to get my own head around how I'd deploy it in a production setting.

Sun has done the right thing and left the special features in the public codebase. There is nothing stopping determined people from building their own versions but I'd argue if you are at the point where you are building these options from source and using them in a production setting your use of SGE is probably important enough to merit a support contract or some other formal arrangement with Sun.

My coworkers and I do build some of these for our own use and we have made an intentional decision not to share the methods online - it's important that Sun make money on this product, especially with the Oracle merger hanging overhead.

The link to the materials is here:
http://blog.bioteam.net/2009/09/28/sge-training-slides/

Comments & feedback welcome, my company is still (of course) offering customized SGE training aimed at both users or administrators in timeslots ranging from 2-hours to 2-days depending on need and interest. Look for more training announcements in early 2010.

Photos from the 2009 Grid Engine Workshop in Germany. Please help me name and tag the participants!

Also, my talk on "Grid Engine & Amazon Compute Cloud" is online here:
http://blog.bioteam.net/2009/09/09/grid-engine-amazon-ec2/. The other talks and training slides are slowly making their way onto the net. We should have video from all the talks as well.

Important Note: Sun has changed the license terms for this release. The full release from Sun.com can only be used for 90 days for free. The courtesy binaries are still free for all to use but the distribution will not include the Amazon EC2 cloud adaptor or the excellent "sgeinspect" tool. Source code for both of these components is available under the SISSL license so theoretically community members can build versions for themselves.

The full release announcement is here:
http://gridengine.sunsource.net/news/SGE62u3-announce.html

For me, the most important new features are the SGEInspect tool (screenshots of which you can view online at http://www.flickr.com/photos/chrisdag/sets/72157617805352910/ and the exclusive host scheduling feature which now removes the need for PE-based 'hacks' to achieve the same goal.

The license change is interesting, I need to see how hard it is to build sgeinspect from source code, it really is a powerful new tool. It's a shame that this won't be part of the free distribution but then again I want Sun to make product and support revenue off of SGE so I can see the point.

A few weeks back I recorded a large and clear screencast of my most recent laptop install using the SGE GUI and then processed the heck out of it so that it is a decent quicktime movie weighing in at only about 3.1MB for download.

Not major news but may be of interest to people who have not seen the new graphical installer in action yet. Of particular note the things that I like about the SGE GUI install methods:

• Wildcards and ranges for hostnames (not possible in other SGE install methods)
• Does remote installs when passwordless SSH is available
• When things fail it actually collects decent log and debug information in a nice central location
• Does some pre-install sanity checking and warns of potential issues
• The final summary of your install with options to print or save the key details is a fantastic new addition

Click on the image to start the download:

Sun has published a new flash video demo (registration required for viewing) here: http://www.sun.com/offers/details/sun_grid_engine.html

Description:

Sun envisions a world of many clouds, both public and private, that are open, compatible, and designed for all types of applications—including high performance computing. Sun is extending cloud computing to High Performance, building not just a cloud, but an entire Cloud eco-system. Through this narrated demonstration, learn how Sun Grid Engine fulfills the promise of cloud computing in HPC.

Grid Engine 6.2 update 1 has been released, the official announcement page does not seem to be up yet but you can find the older "available" notice here.

The list of fixed issues is significant and can be viewed here:
http://gridengine.sunsource.net/project/gridengine/62patches.txt

Another discussion happening recently on the SGE user list concerns how best to handle virtualization. That thread can be browsed here.

In a followup, Andreas is soliciting feedback from the wider community on how you want to see this area handled in future revisions of Grid Engine. Time to speak up if you have an opinion!

Read Andreas's request for feedback here.

Dan T has a new whitepaper entitled "Beginners Guide to Sun Grid Engine 6.2 Installation & Configuration" up online. Direct link is here (registration required).

Also available by going to:
http://www.sun.com/software/gridware/support.xml
... and looking under the Whitepaper section.

Unofficial word is that the official release of SGE 6.2 is coming on Tuesday, August 5th.

To read up on why this is news, check out Dan's excellent essays:

• Why upgrade to 6.2? (Part I)
• Why upgrade to 6.2? (Part II)