These are notes for running Gaussian serially on the TeraGrid. I'll look at IU's BigRed and NCSA's Abe, Mercury, and Cobalt. Surprisingly this wasn't documented anywhere that I could find with Google (I had to resort to the Help Desk, which was helpful but this is missing the point). And of course running things on each machine requires a different incantation. You may wonder why the TeraGrid doesn't make its environments more consistent for these common applications. I suggest posting hypotheses on this subject as comments.
IU's Big Red
Gaussian is a famous piece of Quantum Chemistry software. I'll assume you have an input file, your_input.inp.
1. Request to be added to the Gaussian group. I found this through non-standard routes (thanks to Ray Sheppard) but try contacting the TeraGrid help desk. You have to do this before you can proceed.
1.5 Add gaussian to your .soft file. For more on the SoftEnv, see http://www.teragrid.org/userinfo/jobs/environment.php
My .soft file looks like this:
more ~/.soft
#
# This is the .soft file.
# It is used to customize your environment by setting up environment
# variables such as PATH and MANPATH.
# To learn what can be in this file, use 'man softenv'.
#
#
@bigred
@teragrid-basic
@globus-4.0
@teragrid-dev
+gaussian
----------------------
Either type "resoft" or just exit and login again.
2. Make a Loadleveler script like the one below:
---------------
# @ output=stdout.txt
# @ error=stderr.txt
# @ wall_clock_limit=5:00:00
# @ account_no=YOUR_ACCOUNT_NUMBER
# @ queue=default
export g03root=/N/soft/linux-sles9-ppc64/gaussian/g03-d.02
. $g03root/g03/bsd/g03.profile
g03 $HOME/your_input.inp $HOME/your_output.out
---------------
A real script would move stuff in and out of scratch space and stuff like that (Gaussian creates very large files while you run it), but I intend to automate this through Globus+Condor-g. Also this only will submit serial jobs.
3. Submit with llsubmit, track with llq. The IU Knowledge Base has a nice comparison of Loadleveler and PBS commands and directives here: http://kb.iu.edu/data/axpz.html
NCSA's Abe
Unlike the other machines, you definitely do not want "+gaussian" in your .soft file on abe. This will produce the following error:
g03: error while loading shared libraries: /usr/local/intel/mkl/10.0.3.020/lib/em64t/libmkl.so: invalid ELF header
Make a PBS script like the one below. Again this is only a serial job.
[mpierce@honest2 ~]$ more gaussTest.pbs
#PBS -o stdout.txt
#PBS -e stderr.txt
#PBS -A YOUR_ACCOUNT_NUMBER
#PBS -q normal
#PBS -l walltime=05:00:00
soft add +intel-mkl
setenv g03root /usr/apps/chemistry/gaussian/g03
source $g03root/g03/bsd/g03.login
g03 $HOME/your_input.inp $HOME/yourOutput.out
----------------------
Alternatively, I could have added intel-mkl to my .soft file. I had to get the NCSA Help Desk to tell me this (thanks go to John Estabrook @ NCSA for a quick and accurate response to my ticket). They pointed me in the right direction, but why not just have this documented?
NCSA also has a command-line tool, /usr/local/bin/qg03, that will make and submit a better PBS script than the one above.
globusrun -o -r grid-abe.ncsa.teragrid.org/jobmanager-pbs '&(executable=/u/ncsa/mpierce/gaussian.abe.ncsa.sh)(arguments=/u/ncsa/mpierce/input.in /u/ncsa/mpierce/mepjunk12.out)(project=YOUR-TG-ACCOUNT)(queue=debug)(host_types=himem)(host_xcount=1)(xcount=8)'
or
globusrun -o -r grid-abe.ncsa.teragrid.org/jobmanager-pbs '&(executable=/u/ncsa/mpierce/gaussian.abe.ncsa.sh)(arguments=/u/ncsa/mpierce/input.in /u/ncsa/mpierce/mepjunk12.out)(project=YOUR-TG-ACCOUNT)(queue=debug)(count=1)(hostCount=8)(minMemory=16000)'
I think these are equivalent.
NCSA's Mercury
This still works but who knows how long it will be before the system/application admins break it. As with Abe, best to at least use the qg03 command (it should be in your path) to generate a sample PBS script.
1. Add "+gaussian" to your .soft file (in $HOME) as above.
2. Make a PBS script like the one below. Again this is only a serial job.
[mpierce@honest2 ~]$ more gaussTest.pbs
#PBS -o stdout.txt
#PBS -e stderr.txt
#PBS -A YOUR_ACCOUNT_NUMBER
#PBS -l walltime=05:00:00
g03 $HOME/your_input.inp $HOME/yourOutput.out
----------------------
3. To invoke this with Globus clients or equivalent, you should use a command like
globusrun -o -r grid-abe.ncsa.teragrid.org/jobmanager-pbs '&(executable=/u/ncsa/mpierce/gaussian.cobalt.ncsa.sh)(arguments=/u/ncsa/mpierce/input.in /u/ncsa/mpierce/mepjunk12.out)(project=YOUR_ACCOUNT)(queue=debug)(minMemory=1600)'
or
globusrun -o -r grid-abe.ncsa.teragrid.org/jobmanager-pbs '&(executable=/u/ncsa/mpierce/gaussian.mercury.ncsa.sh)(arguments=/u/ncsa/mpierce/input.in /u/ncsa/mpierce/mepjunk12.out)(project=YOUR_ACCOUNT)(queue=debug)(host_types=himem)(host_xcount=1)(xcount=1)'
NCSA's Cobalt
This has changed and so previous instructions (identical to running Gaussian on Mercury above) no longer work. *DELETED TIRADE*. Setting +gaussian in your .soft no longer seems to work. Use the following minimal scriptlet
#!/bin/csh
setenv g03root /usr/apps/chemistry/gaussian/G03/pp5_e01
source $g03root/g03/bsd/g03.login
$g03root/g03/g03 $1 $2
You can invoke this with a globus run command like the one above.
PSC's Pople
You have to fill out a PDF form and mail or fax it to PSC to get access to Gaussian. I did not pursue this.
Subscribe to:
Post Comments (Atom)
4 comments:
Marlon, my name is Shawn Brown and I am the person in charge of quantum chemistry software at the PSC. There is a perfectly good explanation on why people need to sign a PDF to run Gaussian at the PSC. PSC has a site license that allows anyone to use the program free of charge on our resources and we have continual updates of the program (i.e. the latest and greatest available on that resource). Gaussian was originally developed at CMU, and so this is the consequence of that history. The requirement of the PDF is so that the user, prior to running, has taken responsibility for the fairly restrictive End-User License that the Gaussian uses. This has to be signed and faxed back otherwise it is not legal, hence not being in 2009.
There are explicit instructions on how to run the code at the following site: http://www.psc.edu/general/software/packages/gaussian/
Gaussian is only available on certain architectures (the XT3 is meant for highly scalable work and does not run Gaussian).
Before you post inflammatory remarks on your blog about our center and the software we provide, I suggest you contact us first, as we would for the Indiana facilities, to makes sure you have the proper information. We are happy to take constructive criticism and are endeavoring always to make our documentation more accessible and readable. If you have suggestions that may be helpful to this, we would be more than happy to take them into consideration.
Shawn's comments refer to an older version of the blog, which I have edited. My comments and opinions expressed here are mine alone and not my employer's.
My original complaint is that there is not a more automated or streamlined mechanism for requesting access to Gaussian, not the fact that an end-user license was required.
Hello Marlon, I am about to install a Gaussian 03 on condor. I would like to know whether I need a special version of G03 (the "Linda") or is the regular G03 will work fine as well.
Sincerely,
Amir
Gaussian is pretty hard line about what you can and can't do with it. For instance, benchmarking it and comparing the results with any other comparable software will get you a shiny new entry on their black list. That being said, it's gotten much easier to run codes like Gaussian through the different gateways popping up. The TeraGrid (now XSEDE) Common User Environment group has also done a lot of work to get modules in place across all sites. While their advertising and documentation could use some work, their product is solid. That, combined with the environment endpoints bundled into the User Profile Service, go a long way towards getting your environment ready to run. When XSEDE is a bit further down the road you will see some more tools show up in the user portal to specifically make these kinds of questions even easier to answer. As for the difficulty of scripting these things up and throwing them into queue...that's why you get the big bucks ;-)
Post a Comment