Marlon Pierce's Community Grids Lab Blog

Security Flaw with IU CAS-Enabled Web Sites

2011-08-08T12:02:00.000-07:00

There is a security flaw in many of IU's online services (such as the time off system and the document workflow approval system--see http://kb.iu.edu/data/akui.html): they use CAS for single-sign on but don't provide a CAS "log off". If you log into one of these services, you are logged into every CAS-enabled service until your session expires, perhaps without realizing it. The only way I know to log off after CAS login is to go through IU's OneStart portal, which does have a CAS log-off.

For example, if you login to the EPTO service to record your time off, you are also logged into OneStart and everything else. If you leave your laptop or workstation unattended while the CAS session is still valid, I can sneak over and access all of your private employee information by pointing your browser to OneStart.

GCE11 Workshop Scheduled

2011-07-27T17:28:00.001-07:00

The Gateway Computing Environments 2011 workshop will be at Supercomputing 2011. Submissions are open and welcome. See http://www.collab-ogce.org/gce11/index.php/Main_Page for more information.

Changing Apache Rave's backing database

2011-07-27T10:08:00.000-07:00

Apache Rave provides software for running a social networking portal. The default download uses the H2 in-memory relational database on the backend. To change this to MySQL, PostgreSQL or another external database, you need to do the following:

Edit the file rave-portal/src/main/webapp/WEB-INF/dataContext.xml to use the desired JDBC driver; and
Download the appropriate database driver jar file and place it in the WEB-INF/lib directory of your portal's tomcat server.

If you are starting the server with "mvn cargo:start", this is rave-portal/target/tomcat6x/webapps/portal/WEB-INF/lib/|
You probably will want to install the JDBC driver jar in your local Maven repository and put the dependency in rave-portal/pom.xml.

It is also worth reviewing the dataContext.xml file. The H2 database default user name and password are here (you may want to update) and has a console running on port 11111. You can point your browser to http://localhost:1111/ and login. You can also change the default behavior of H2 to use files for persistence.

Implementing account creation in Apache Rave

2011-06-23T11:09:00.000-07:00

Apache Rave (http://incubator.apache.org/rave/) is an Apache Incubator project to build a Web portal and services on top of Google's OpenSocial and W3C's Widget specifications. We're writing Rave from scratch using the Spring (version 3) framework. These are notes on adding a basic user registration feature. The code described below is open source and will soon be in Rave's SVN. I'll assume general familiarity with Spring. If not, check out Rave's source code and walk through it. The notes below will tell you were to look, but I won't include all the code.

Rave's user management is built around the following classes:

User: this is a data model class that represents a user (name, password, etc). Spring (and JPA) manage the object-relational mappings between User objects and the relational database storage.
UserService: this is an interface that allows you to interact with the User data model.
UserRepository: this is an interface that allows you to interact with the backend database. We use JPA for object-relational mapping.
DefaultUserService: this implements the UserService interface and interacts with the UserRepository (and thus the JpaUserRepository).
JpaUserRepository: this implements the UserRepository interface and provides access to the javax.persistence.EntityManager, which handles the database persistence.

There are similar classes and interfaces for Rave's widget, page, and region data models. Spring of course handles all the dependency injection (makes sure DefaultUserService has an instance of the UserRepository implementation, for example). JPA takes care of the crud.

We'll now need to implement two pieces of code: an account registration form (newaccount.jsp) and a Spring controller class to intercept our form actions (NewAccountController.java). The registration form and associated action URL both need to be accessed by unauthenticated users, so we modify Rave's applicationContext-security.xml file accordingly (see Rave code for full example).

NewAccountController needs to do three things: get access to the UserService implementation (Spring injects this), create a new User object, and use the UserService to insert the new User object into the repository. Once a user is registered, NewAccountController redirects to the login page.

This is pretty basic functionality, and there's a lot left to do. As an Apache Software Foundation incubator project, Rave is not only open source but community driven. If you want to participate, checkout the code from SVN, create an account on Rave's Jira system (https://issues.apache.org/jira/browse/rave), create issues and submit patches. Frequent contributors can become full project members.

Chembiogrid and Chembiocloud

2011-05-11T06:41:00.000-07:00

We had some adventures in domain hosting. If you are looking for the correct version of the ChemBioGrid website, please see http://www.chembiogrid.info or http://www.chembiocloud.org. This contains links to Web services, databases, and other useful stuff from the NIH funded CICC project. The chembiogrid.org site is currently outside our control.

A Jersey REST Example with Maven

2010-12-13T10:59:00.000-08:00

I finally got around to trying out Jersey for REST services. Below is an end-to-end example that will build a war file that can be deployed into a Tomcat server. I'm using Java 1.6 and Tomcat 6.0.20.

Assumption: I will assume you can create and compile a project with Apache Maven. I used Maven 2.2.1 below.

Installing: Jersey doesn't have to be installed in a conventional sense. You just need to have the correct jars in your web application's $CLASSPATH. Maven will automate this for you. Here is your POM file:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>yourGroup</groupId>
  <artifactId>yourExample</artifactId>
  <packaging>war</packaging>
  <version>1.0</version>
  <name>${artifactId}</name>
<repositories>

<repository>
<id>maven2-repository.dev.java.net</id>
<name>Java.net Repository for Maven</name>
<url>http://download.java.net/maven/2/</url>
<layout>default</layout>
</repository>
<repository>
<id>maven-repository.dev.java.net</id>
<name>Java.net Maven 1 Repository (legacy)</name>
<url>http://download.java.net/maven/1</url>
<layout>legacy</layout>
</repository>
  </repositories>
  <dependencies>

<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-server</artifactId>
<version>1.4</version>
</dependency>
<dependency>
<groupId>com.sun.grizzly</groupId>
<artifactId>grizzly-servlet-webserver</artifactId>
<version>1.9.18-i</version>
</dependency>
<dependency>
<groupId>com.sun.jersey.jersey-test-framework</groupId>
<artifactId>jersey-test-framework-grizzly</artifactId>
<version>1.4</version>
<scope>test</scope>
</dependency>
  </dependencies>
</project>

Configuring: There seem to be a few options for your web.xml file. The following worked for me. You have to change the line in bold.
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE web-app
   PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.2//EN"
   "http://java.sun.com/j2ee/dtds/web-app_2.2.dtd">
<web-app>
  <servlet>
<servlet-name>Jersey Web Application</servlet-name>
<servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>
   <init-param>
   <param-name>com.sun.jersey.config.property.packages</param-name>
   <--You need to change this line to match your package name -->
   <param-value>org.yourproject.restservices</param-value>
   </init-param>
   <load-on-startup>1</load-on-startup>
  </servlet>
  <servlet-mapping>
<servlet-name>Jersey Web Application</servlet-name>
<url-pattern>/*</url-pattern>
  </servlet-mapping>
</web-app>

The Code: The following is a simple Java code that takes supports both GET and POST operations with arbitrary key-value pairs. It doesn't however support parameters with multiple values. I show a sample code snippet below. From other examples that I've seen, the UriInfo class seems to be the easiest way to get a Map of the request parameters with GET methods. With POST, you have more direct access to the map.

//This package declaration should match what you used in the web.xml above.

package org.yourproject.restservices;

//These are Jersey jars

import javax.ws.rs.GET;

import javax.ws.rs.POST;

import javax.ws.rs.Produces;

import javax.ws.rs.Consumes;

import javax.ws.rs.Path;

import javax.ws.rs.QueryParam;

import javax.ws.rs.core.Context;

import javax.ws.rs.core.MultivaluedMap;

import javax.ws.rs.core.UriInfo;

import java.net.URLEncoder;

import java.util.Iterator;

@Path("/echo") public class MyService{

   @GET 

   @Produces("text/html") 

   public String get(@Context UriInfo ui) {

      MultivaluedMap queryParams=ui.getQueryParameters();

      return showQueryParams(queryParams);      

}

   @POST

   @Consumes("application/x-www-form-urlencoded")

   @Produces("text/html") 

   public String post(MultivaluedMap queryParams) {

 return showQueryParams(queryParams);    

}

   //Extract the parameters.    

   protected String showQueryParams(MultivaluedMap queryParams) {

 Iterator it=queryParams.keySet().iterator();

 String theKey=null;

 String returnString="";   

 try {

      while(it.hasNext()) {

       theKey=(String)it.next();

       returnString += URLEncoder.encode(theKey,"UTF8")

                 +"  "

                 +URLEncoder.encode((String)queryParams.getFirst(theKey),"UTF-8");

}

}

 catch (Exception ex) { 

    //This will probably be an UnsupportedEncodingException.

           //Handle more approrpriately. 

    ex.printStackTrace();

           returnString=ex.getMessage();

}

 return returnString;

}

}

Compile and Invoke: Use "mvn clean install" to generate a war file, then copy this from your project's target directory to a waiting Tomcat server.  Better yet, add the clean up and installation code to your POM using Ant tasks.  

You can invoke this service any number of ways: directly call it in a web browser, write an HTML page that uses it as a FORM action, or invoke on the command line with curl or wget. Here is a curl example that uses POST:

curl http://your.server.org:8080/yourExample/echo -d "hollow=world&true=false"

Plotting with dygraphs JavaScript

2010-10-04T11:26:00.000-07:00

I stumbled across dygraphs at lunch. This is perfect for plotting line plots--anything you care to export from a spreadsheet. For input, you only need a file with comma separated values. For free, you get mouseover point values and drag-and-zoom.

HTML5 Messaging and PMRPC

2010-10-04T07:50:00.000-07:00

From a suggestion by David Braun, I took a quick look at PMRPC as a way to send messages between browser windows and gadgets using HTML5 standards. One of PMRPC's selling points is that it promises to hide the details of the much lower level HTML5 API. Oh, computer science....

Since I'm not a great JavaScript programmer, I found the provided examples to be a little opaque. Here are simple working examples, at least in Google Chrome on Mac. They did not work for Firefox 3.6.

Window-IFrame Example

In this example, you have an HTML page that contains an iFrame, and you want to send a message from the container to the child iFrame. Here's the parent's HTML. I dubbed this file frame2.html.

<html>
<head>
<script src="json2.js" type="text/javascript"></script>
<script type="text/javascript" src="pmrpc.js"></script>
<script>
function dothing() {
// call the exposed procedure
pmrpc.call( {
  destination : window.frames["iFrameA"],
  publicProcedureName : "HollowPMRPC",
  params : ["Hollow World!"] } );
}
</script>
</head>
<body>
<iframe id="iFrameA" title="iFrameA" src="http://your.server.base/frame1.html"></iframe>
This is Frame B.
<input type="button" name="send message" onClick="dothing()">
</body>
</html>

The embedded iFrame (called frame1.html above) looks like this:
<html>
<head>
<script src="json2.js" type="text/javascript"></script>
<script type="text/javascript" src="pmrpc.js"></script>
<script>
//function init(){
// expose a procedure
alert("Init");
pmrpc.register( {
  publicProcedureName : "HollowPMRPC",
  procedure : function(printParam) { alert(printParam); } } );
//}
</script>
</head>
<body>
This is Frame A.
</body>
Click the button and send the alert. Welcome to 2010.

IFrame to IFrame Communication
To get two peer windows (two embedded iFrames in this case) to send messages to each other, we need to use the "publish" destination in frame1.html. We'll also delete the iframe tag. The new HTML looks like this:
<html>

<head>
<script src="json2.js" type="text/javascript"></script>
<script type="text/javascript" src="pmrpc.js"></script>
<script>
function dothing() {
// call the exposed procedure
pmrpc.call( {
  destination : "publish",
  publicProcedureName : "HollowPMRPC",
  params : ["Hollow World!"] } );
}
</script>
</head>
<body>
This is Frame B.
<input type="button" name="send message" onClick="dothing()">
</body>
</html>

We don't need to make any modifications to frame2.html. Finally, we need a simple HTML page to contain the iFrames for frame1.html and frame2.html. I call this container.html:
<html>

<body>

</body>

</html>

Using SVN Externals as Organization Links

2010-07-27T07:50:00.000-07:00

Here's the problem: you have a large SVN code base with many sub-projects. Setting up your SVN as a straightforward hierarchical organization (each sub-projects is a sub-folders in your grand scheme) will eventually cause problems, since you will want to significantly reorganize components over time without breaking builds that may be dependent on the older directory layouts.

We've found SVN externals to be a good way to handle the problem of linking across far-flung directories within the same SVN repository. In this setup, each major sub-component still gets its own directory. The organization appropriate for your current system is a simple organizing/aggregating directory. Sub-components are then linked into this directory using svn:external properties that are defined in a properties file (this is the only file you need in your aggregating directory).

The easiest way to do this is to create a file (svn_ext_val.txt) in your new aggregating directory and define properties within it. Call your aggregating directory "myagg". Then for example (from our OGCE project)

[shell-prompt> svn co http://ogce.svn.sourceforge.net/svnroot/ogce/myagg

[shell-prompt> cd myagg

[shell-prompt>cat svn_ext_val.txt

sgfac https://ogce.svn.sourceforge.net/svnroot/ogce/ogce-xsul-services/sgfac/

xregistry https://ogce.svn.sourceforge.net/svnroot/ogce/ogce-xsul-services/xregistry/

ogce-xbaya https://ogce.svn.sourceforge.net/svnroot/ogce/ogce-xbaya-gui/

gadget-container https://ogce.svn.sourceforge.net/svnroot/ogce/ShindigOGCE/ishindig-trunk/

xregistryinterface https://ogce.svn.sourceforge.net/svnroot/ogce/ogce-gadget-interfaces/xregistryinterface/

experimentbuilder https://ogce.svn.sourceforge.net/svnroot/ogce/ogce-gadget-interfaces/experimentbuilder/

gadget-sharedlib https://ogce.svn.sourceforge.net/svnroot/ogce/ogce-gadget-interfaces/gadget-sharedlib/gadget-sharedlib/

Each property points to the appropriate component directory within SVN. Set these for your project with the command

svn propset svn:externals -F svn_ext_val.txt .

Verify everything is correct with

svn propget svn:externals

Then commit with "svn commit" as usual. Now run "svn update" and it should check out all the external components. If you edit code in any of the subcomponents and commit, the external directory will be updated correctly.

Calling javac from Ant Inside Maven Using AntRun

2010-04-26T09:00:00.000-07:00

We ran into a situation where we needed to call Ant's javac task to compile code generated by a legacy Ant task as part of a larger build. This exposed a very confusing feature of Maven: it apparently changes the java.home property to point to the JRE home. This is no problem on Macs but is on Linux.

There are several possible workarounds. The one that I currently prefer is to exploit the fact that Maven only overrides ${java.home} but not the original ${env.JAVA_HOME}. So your <javac> task inside the antrun plugin should look like this:

</javac>

In testing, the fork attribute needed to be set to true, but I don't know why.

Another workaround is to specify the location of tools.jar in the Antrun plugin's dependencies. This introduces a second problem: Mac and Linux put the tools.jar classes in different jars. For Linux, this is just in $JAVA_HOME/lib/tools.jar as you would expect, but Mac embeds these classes in classes.jar, located in $JAVA_HOME/../Classes/classes.jar. Thus you should use

<plugin>
<artifactId>maven-antrun-plugin</artifactId>
<dependencies>
   <dependency>
   <groupId>com.sun</groupId>
   <artifactId>tools</artifactId>
   <version>1.5.0</version>
   <scope>system</scope>
   <systemPath>${tools.jar}</systemPath>
   </dependency>
   </dependencies>

   </plugin>

Set ${tools.jar} depending on your operating system as described in the preceding paragraph. Obviously this introduces an undesirable need to either set or detect the operating system of the build.

Perhaps there is some deep reason for Maven to internally overwrite java.home, but I can't guess it. This really should be cleaned up.

Nagios and HTTPS Monitoring

2010-02-18T10:34:00.000-08:00

This doesn't seem to be configured in the basic Nagios installation, but it is easy to do. The following was done on a server running RedHat 5.2.

Go to your nagios top level directory, /usr/local/nagios.
Use the check_http command line tool and make sure it works: ./libexec/check_http -H your.server.hostname -p 8443 -S Here the port is 8443. -S is the option to use SSL.
Add a command, "check_https", to your etc/objects/commands.cfg file:

   # 'check_https' command definitiondefine command{
   command_name check_https
   command_line $USER1$/check_http -I $ARG1$ -p $ARG2$ -S
   }
Add the host to be checked to the appropriate config file (see my previous post for an example, or see below).

Make sure your don't have any typos: /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Reload nagios: /etc/rc.d/init.d/nagios reload

You are done. I put all of my Nagios checks into localhost.cfg, which is probably not a good idea, but it was simple to get going. Add the following to the bottom of your cfg file:

define host{

use linux-server ; Name of host template to use

host_name some-host ; The name we're giving to this host

alias Some Host to Check ; A longer name associated with the host

address 123.456.789 ; IP address of the host

hostgroups linux-servers ; Host groups this host is associated with

}

define service{

use local-service ; Name of service template to use

host_name some-host

service_description HTTP

check_command check_https!your.server.hostname!8443

notifications_enabled 1

}

Resetting Old Media Wiki Passwords in MySQL

2010-02-10T13:42:00.000-08:00

I needed to reset a forgotten user password in an older version of Mediawiki that predates the changePassword.php utility. Also, email notifications were disabled. These are some clarifications of the instructions from http://www.mediawiki.org/wiki/Manual:FAQ#How_do_I_reset_a_password.3F. Here are the steps:

Look at your LocalSettings.php file to remind yourself of the DB name, the DB prefix, user name and password associated with your Mediawiki installation. Let's say they are mywikidb, mywikiprefix, mywikiuser, and mywikipass, respectively.
Login from the command line to your mysql: mysql -u mywikiuser -pmywikipass
Change to the database at the mysql prompt: USE mywikidb;
Verify things: SHOW columns FROM mywikiprefixuser;
Verify more things: SELECT user_name, user_id, user_real_name, user_email FROM mywikiprefixuser;
Change the password: UPDATE mywikiprefixuser SET user_password = MD5(CONCAT(user_id, '-', MD5('newpass'))) WHERE user_name="someuser";

In step 6, change mywikiprefix to your prefix, newpass to the desired password, and someuser to the appropriate user of your system. In step 5, you should see the user_name value ("JimboWales"), so this is a useful check.

Some Nagios Notes for HTTP Monitoring

2010-01-07T14:10:00.000-08:00

I needed to monitor some non-standard HTTP ports with Nagios. Installation went well, but it took a little fooling around to get the HTTP monitoring the way that I wanted. [N. B. After writing this, I realized later that the default settings could probably do what I wanted. But here it is anyway.]

First, make sure your check_http command is working correctly. It is in $NAGIOS_HOME/plugins/check_http. Use --help to see your options. You'll note that there are several, but the actual command that Nagios runs is defined in the file commands.cfg. The default is

# 'check_http' command definition

define command{

        command_name    check_http

        command_line    $USER1$/check_http -I $HOSTADDRESS$ $ARG1$

}

I changed the command_line to

command_line $USER1$/check_http -I $ARG1$ -p $ARG2$

You could of course add additional options supported by check_http.

Next, you need to add hosts to be monitored. I just appended these to localhost.cfg. This is an example for monitoring http://my.machine.indiana.edu:9999 (with IP addres 123.456.789).

define host{

        use             linux-server            ; Name of host template to use

        host_name       my-machine       ; The name we're giving to this host

        alias           CICC Tomcat     ; A longer name associated with the host

        address         123.456.789           ; IP address of the host

        hostgroups      linux-servers           ; Host groups this host is associated with

        }

define service{
        use             local-service         ; Name of service template to use

        host_name       my-machine

        service_description     HTTP

        check_command   check_http!my.machine.indiana.edu!9999
        notifications_enabled           0

        }

That's it. Maybe there is a better way to do this but I could not find it easily.

Quick Guide to Using Open Social's Activity API with Google Friend Connect

2010-01-04T15:46:00.000-08:00

As I've discussed in some previous posts, Google Friend Connect gives you a simple way to access Open Social applications. The JavaScript API is pretty simple and is mainly useful for logging in, setting up small social networks, and rendering social gadgets. You can use the JavaScript API to decorate HTML with various social mini-gadgets for tagging, rating, commenting, and so on.

In this post, we'll look at how to directly call the Open Social API without going through a pre-existing gadget. Our motivation is the Open Social Activity API. Familiar social networking sites like Facebook and Twitter have "activity streams" in which each user posts status updates. These can also be automatically generated side effects (such as bookmarking a site with Del.icio.us's Twitter interface, for example).

An Open Social Activity API example using Google Gadgets is here: http://code.google.com/apis/opensocial/docs/0.8/devguide.html#Activities. In addition to being an HTML gadget, this relies on the Gadget API (see the call to gadgets.util.registerOnLoadHandler(startActivity) near the bottom). To do the same thing with Google Friend Connect, we need to take the following steps:

Load the FriendConnect JavaScript libraries with google.load.
Use the FriendConnect libraries to initialize an OpenSocial API call.
Pass in a callback function (startActivity() in our example) that the server side (at Google HQ, presumably) will invoke.
Implement the callback function using the OpenSocial Activity API.

The code for doing this is below, with comments.

<html>

<body>
<div id="content_div"></div>


<script type="text/javascript" src="http://www.google.com/jsapi"></script>


<script type="text/javascript">
    google.load('friendconnect', '0.8');
</script>

<script type="text/javascript">
          /* This will dynamically change the content to the new activity */
    var div=document.getElementById("content_div");

     /*No caching for development. */

    google.friendconnect.container.setNoCache(1);

     /* Use this to invoke Open Social methods */
    google.friendconnect.container.initOpenSocialApi({
    site: '##########################',   /* Replace this with your GFC ID */
    onload: function(securityToken) { startActivity(); }
    });
/* This is a callback invoked when we initial the Open Social libraries. It has a callback method of its own, called callback() */

function postActivity(text) {
    var params = {};
    params[opensocial.Activity.Field.TITLE] = text;
    var activity = opensocial.newActivity(params);

    opensocial.requestCreateActivity(activity, opensocial.CreateActivityPriority.HIGH, callback);

    div.innerHTML = "Activity title is: " + activity.getField(opensocial.Activity.Field.TITLE);
};

/**
* Server calls this function to indicate whether the activity post succeeded or failed.
*/
function callback(status) {
    if (status.hadError())
    {
      alert("Error creating activity.");
    }
    else
    {
      alert("Activity successfully created.");
    }
};

/**
   * Start the process of posting an activity.
   */
function startActivity() {
    postActivity("This is a sample activity, created at " + new Date().toString());
};

</script>

</body>
</html>

To use this, just save it as an HTML file on a Web server registered to Google Friend Connect and load the page. You will need to change "################" to use your registered site ID.

Quick Guide to Using Google's OpenID

2009-12-30T11:54:00.000-08:00

These notes are based on some work done by Jun Ji in our group. He's been converting the QuakeSim project's portlets into Google gadgets as a prelude to Open Social integration, as discussed in previous posts. One problem is that the gadgets need to identify and authenticate the user. We don't have excessive security requirements. We just need to save the users' interactions with the web interface as persistent sessions ("projects") so that they can be accessed again in later sessions.

This identification step is actually pretty simple to do with OpenID and openid4java, but you may not guess this from Google's documentation (see for example http://code.google.com/apis/apps/sso/openid_reference_implementation.html). To start, you will need to download the openid4java tar file (see http://code.google.com/p/openid4java/) and unpack it. The openid4java jars need to go in your webapps's WEB-INF/lib directory, as usual.

Next, you need to copy the provided samples consumer_redirect.jsp and consumer_returnurl.jsp to your webapp's directory. They are in ./samples/simple-openid/src/main/webapp/ of the openid4java download. You will need to modify them a little: consumer_redirect.jsp for example has a variable returnToUrl that needs to be changed to use your webapp's name. For our other modifications to these codes, see the source code at source forge: http://crisisgrid.svn.sourceforge.net/viewvc/crisisgrid/QuakeSim2/portlets_dev/Disloc3/src/main/webapp/

This completes the setup. You can now make simple login page like the one below (call it index.jsp or whatever). You need to do the following:

See if a logout action has been requested. If so, clean up relevant OpenID session variables.
See if the openid attribute has been set. If so, give the user an option of logging out or navigating on to other pages.
Otherwise, redirect the user to the Google OpenID login page.

Here's what the JSP page looks like:

<%@ page session="true" %>
<html>
<body>
<%
    if (request.getParameter("logout")!=null)
    {
        session.removeAttribute("openid");
        session.removeAttribute("openid-claimed");
%>
    Logged out!<p>
<%
    }
    if (session.getAttribute("openid")==null) {
%>
<form method="POST" action="consumer_redirect.jsp">
<strong><br>QuakeSim2 OpenID login test page:<br><br>
</strong>
<input type="hidden" name="openid" value="https://www.google.com/accounts/o8/id" size="60"/><br>
<input type="submit" value="log in into google"/>
</form>
<%
} else {
%>

Logged in as <%= session.getAttribute("openid") %><p>
email address <%= session.getAttribute("email")%><p>
<a href="?logout=true">Log out</a>

<% } %>

</body>
</html>

Note the form action sends the user to the consumer_redirect.jsp page we took from openid4java. The hidden parameter openid is sent to consumer_redirect.jsp and tells it to redirect to the Google login page. The consumer_returnurl.jsp page acts like a callback page and tells the Google login form where to redirect the browser request if login was successful. Your future requests will be filtered through this servlet, which can enforce verification.

Finally, note that several session variables will be set. Here are some sample name-value pairs (just use session.getAttributeNames() in index.jsp and list):

openid-disco OpenID2 OP-endpoint:https://www.google.com/accounts/o8/ud ClaimedID:null Delegate:null
email marpierc AT gmail.com
openid-claimed https://www.google.com/accounts/o8/id?id=ABCdefg-hIJKLMNop
openid https://www.google.com/accounts/o8/id?id=ABCdefg-hIJKLMNop

You may want to use these in other applications. We use the email attribute to identify the user for database purposes. You may also want to use these variables to associate a successful OpenID login with a legacy user entry in your database.

Getting Started with Friend Connect

2009-12-14T14:02:00.000-08:00

This blog post will cover two simple capabilities of Google Friend Connect: creating a social network and rating items in a table. First, however, let's provide some motivation for science portals.

We've developed the QuakeSim portal for several years to support earthquake modeling research and education. A typical scenario, common in many other gateways, is for a user to set up and launch an application through a Web interface. Web services are needed to both run the modeling code and preserve metadata about the run: the input parameters, the time the code was executed, the URLs of all the input, output, and error files, and so on. We organize these metadata fields as projects owned by a particular user. The metadata is modeled with JavaBeans and stored in object databases, although these are irrelevant technical details here. It is also possible, and appropriate, to model the metadata as Atom/RSS, for example.

Let's look first at creating a little GFC social network of people who may be interested in looking at the metadata feeds. You will need to create a Google account if you don't have one and register a URL with GFC. Just use your development machine's URL, as you would with Google Maps. Note the Site ID that you are assigned (it will be in the GFC URL of the projects you register). You'll need to use this in your GFC applications.

To get started, you can generate code for lots of GFC applications. Just save this generated HTML/JavaScript in files that can be loaded from your Web server. If you want to use these as standalone pages, be sure to add <html> and <body> tags to the beginning and end of the document. You can also embed the generated code into existing HTML pages. To wrap the code as a gadget, just paste it in the CDATA section of an "html" gadget.

First, now, we need a GFC code snippet to manage logins and mini-social networks. The GFC way for doing this is shown below:

<html>

<body>

  <!-- Include the Google Friend Connect javascript library. -->

  <script type="text/javascript" src="http://www.google.com/friendconnect/script/friendconnect.js"></script>

  <!-- Define the div tag where the gadget will be inserted. -->

  <div id="div-4719709228063302739" style="width:276px;border:1px solid #cccccc;"></div>

  <!-- Render the gadget into a div. -->

<script type="text/javascript">

var skin = {};

skin['BORDER_COLOR'] = '#cccccc';

skin['ENDCAP_BG_COLOR'] = '#e0ecff';

skin['ENDCAP_TEXT_COLOR'] = '#333333';

skin['ENDCAP_LINK_COLOR'] = '#0000cc';

skin['ALTERNATE_BG_COLOR'] = '#ffffff';

skin['CONTENT_BG_COLOR'] = '#ffffff';

skin['CONTENT_LINK_COLOR'] = '#0000cc';

skin['CONTENT_TEXT_COLOR'] = '#333333';

skin['CONTENT_SECONDARY_LINK_COLOR'] = '#7777cc';

skin['CONTENT_SECONDARY_TEXT_COLOR'] = '#666666';

skin['CONTENT_HEADLINE_COLOR'] = '#333333';

skin['NUMBER_ROWS'] = '4';

google.friendconnect.container.setParentUrl('/gadgets-repo/' /* location of rpc_relay.html and canvas.html */);

google.friendconnect.container.renderMembersGadget(

 { id: 'div-4719709228063302739',

   site: '16707400350554686557' },

  skin);

</script>

</body>

</html>

The site ID in the above example is one of my site IDs. You would need to replace this with your site ID.

This is just generated code from the GFC site. I only added the (necessary) HTML and BODY tags so that it would work as a standalone page. Load this new HTML page in your browser and use it to sign in and add people to your network. If you are already signed in to a Google site (say to iGoogle) on the same browser, then the GFC widget above should detect this through cookies. Note that this network should apply only to a specific site ID, so you can create multiple social networks. One difficulty here of course is that all of this is external to your legacy portal's authentication system.

Our next example is a rating widget that can be embedded into an existing page. Recall our motivating problem was an archive list of project metadata associated with a particular user's previous code submissions. In QuakeSim, we render this information using JSF datatables populated with JavaBeans pulled from an object database, but this detail is irrelevant here. In the browser, the generated page would look something like the HTML below:

<html>

<body>

<!-- Include the Google Friend Connect javascript library. -->

<script type="text/javascript" src="http://www.google.com/friendconnect/script/friendconnect.js"></script>

<!-- Define the div tag where the gadget will be inserted. -->

<table border="1">

  <tr>

     <th>Project</th><th>ID</th><th>Date</th><th>Input File</th><th>Output File</th><th>Recommend</th>

  </tr>

  <tr>

     <td>

        Northridge

     </td>

     <td>

123456789

     </td>

     <td>

        08:43 GMT, Dec 14, 2009

     </td>

     <td>

        <a href="http://www.quakesim.org/mpierce/northridge/input">Input</a>

     </td>

     <td>

        <a href="http://www.quakeims.org/mpierce/northridge/output">Output</a>

     </td>

     <td>

        <div id="div-1095870898733665271" style="width:100%;"></div>

     </td>

  </tr>

  <tr>

     <td>

        Plieto

     </td>

     <td>

987654321

     </td>

     <td>

        10:12 GMT, Dec 12, 2009

     </td>

     <td>

        <a href="http://www.quakesim.org/mpierce/plieto/input">Input</a>

     </td>

     <td>

        <a href="http://www.quakeims.org/mpierce/plieto/output">Output</a>

     </td>

     <td>

        <div id="div-123456789" style="width:100%;"></div>

     </td>

  </tr>

</table>

<script type="text/javascript">

var skin = {};

skin['HEIGHT'] = '21';

skin['BUTTON_STYLE'] = 'compact';

skin['BUTTON_TEXT'] = 'Recommend it!';

skin['BUTTON_ICON'] = 'default';

google.friendconnect.container.setParentUrl('/gadgets-repo/' /* location of rpc_relay.html and canvas.html */);

google.friendconnect.container.renderOpenSocialGadget(

 { id: 'div-1095870898733665271',

   url:'http://www.google.com/friendconnect/gadgets/recommended_pages.xml',

   height: 21,

   site: '16707400350554686557',

   'view-params':{"pageUrl":"http://www.quakesim.org/users/mpierce/northridge","pageTitle":"Northridge","docId":"recommendedPages"}

},

  skin);

google.friendconnect.container.renderOpenSocialGadget(

 { id: 'div-123456789',

   url:'http://www.google.com/friendconnect/gadgets/recommended_pages.xml',

   height: 21,

   site: '16707400350554686557',

   'view-params':{"pageUrl":"http://www.quakesim.org/users/mpierce/plieto","pageTitle":"Plieto","docId":"recommendedPages"}

},

  skin);

</script>

</body>

</html>

The main thing to notice is that each project has been given a URL. I use http://www.quakesim.org/users/mpierce as the base--see the pageUrl parameter in the code above. This assumes a REST style programming model for the metadata services, which unfortunately isn't implemented yet (but now I have some motivation to do it...). The javascript at the end is a slightly modified version of generated GFC code. The values for the view-params (pageURL, pageTitle) are generated and should be taken from entries in your project database.

Each entry in the table will have a button with "Recommend It" in the final column. You can get a list of recommended projects in a page like the one below (again, this is generated by an online GFC tool):

<html>
<body>

<script type="text/javascript" src="http://www.google.com/friendconnect/script/friendconnect.js"></script>

<div id="div-6585877907190446397" style="width:300px;border:1px solid #cccccc;"></div>

<script type="text/javascript">
var skin = {};
skin['BORDER_COLOR'] = '#cccccc';
skin['ENDCAP_BG_COLOR'] = '#e0ecff';
skin['ENDCAP_TEXT_COLOR'] = '#333333';
skin['ENDCAP_LINK_COLOR'] = '#0000cc';
skin['ALTERNATE_BG_COLOR'] = '#ffffff';
skin['CONTENT_BG_COLOR'] = '#ffffff';
skin['CONTENT_LINK_COLOR'] = '#0000cc';
skin['CONTENT_TEXT_COLOR'] = '#333333';
skin['CONTENT_SECONDARY_LINK_COLOR'] = '#7777cc';
skin['CONTENT_SECONDARY_TEXT_COLOR'] = '#666666';
skin['CONTENT_HEADLINE_COLOR'] = '#333333';
skin['HEADER_TEXT'] = 'Recommended stories';
skin['RECOMMENDATIONS_PER_PAGE'] = '5';
google.friendconnect.container.setParentUrl('/gadgets-repo/' /* location of rpc_relay.html and canvas.html */);
google.friendconnect.container.renderOpenSocialGadget(
{ id: 'div-6585877907190446397',
   url:'http://www.google.com/friendconnect/gadgets/recommended_pages.xml',
   site: '16707400350554686557',
   'view-params':{"docId":"recommendedPages"}
},
skin);
</script>
</body>
</html>

Integration Thoughts: the main issue I see in the above example is integrating the GFC authentication and authorization system with your portal. We'll look at this next. Also, one of the strengths of GFC is the simplicity of the JavaScript code exposed to the end developer. Gateways have long had a problem integrating in programming-literate computational scientists who just didn't care to learn Enterprise Java and similar things. To retain this advantage of GFC, we would probably need to consider building our own higher level gadget that queries our project database and is rendered with google.friendconnect.container.renderOpenSocialGadget() GFC call.

Building Simple Social Network Applications with Google Friend Connect

2009-12-10T08:09:00.000-08:00

Within the academic community, there has been a push by various groups to provide a "Facebook for Scientists", a "Facebook for Scholars", or similar things. This is to sell you something, of course. I am skeptical of this approach, since we already have Facebook, and I don't want to join and contribute to yet another social network, especially if I don't know its long term sustainability.

Let's try to follow a systematic approach. As a starting point, I would like to point you to a nice submission by Roger Curry, Cameron Kiddle and Rob Simmonds to our recent GCE09 workshop. The authors looked at using Facebook, Ning, and Elgg as three social networking platforms. You know Facebook, and you probably know that Ning is a social networking hosting service. Elgg was new to me: it is a downloadable software package, like Drupal or Mambo. Unlike Drupal, however, Elgg is designed from the beginning to be a social networking platform rather than a content management system.

Curry, Kiddle, and Simmonds's paper presents three detailed case studies. I would add some additional remarks on requirements. First, a new social networking site should be compatible with existing social networks and related standards. I should be able to import in my existing profile and social networking information. I should also be able to reuse my identity from another site. I don't want to remember yet another password. Similarly, I should be able to export my social networking information. I don't want to get locked in. OpenID, OpenSocial, OAuth, and related standards may make it easier to share one's identity and information across social networking services.

Second, development flexibility is important. Of the three social networks Curry et al reviewed, Ning's hosted service was generally the least favored. Part of this was because of the advertisements, but it was also noted that access to Ning source code was limited. If you are considering a hosted service, you need also to consider the development framework that you will be required to use, its strengths, and its weaknesses.

Third, I want to use the best available implementation of a capability, which may not be the one that comes in the software. For example, there are many wikis and calendaring tools. I want the ability to easily import these tools into a social networking framework. I don't want to rely upon the wiki or calendar plugin that comes with some particular framework.

For these reasons, I have become very interested in Google Friend Connect (GFC) as a way to build social networking science gateways (and credit goes to Raminder Singh and Gerald Guo in our lab for introducing this to me). Gadgets are particularly interesting in that they are a client-side technology: any Web development framework can be used to develop gadgets. We have developed an open source gadget container (see http://www.collab-ogce.org/ogce/index.php/OGCE_Gadget_Container) that can be used to host your own gadgets. We discuss these things in more detail and provide an implementation case study in our own workshop paper.

I have advocated using Open Social and Google Gadgets for some time (see for example http://grids.ucs.indiana.edu/ptliupages/publications/INGRIDFinal.pdf), but the JavaScript API is not particularly simple, and of course Facebook (the biggest US social networking site) doesn't participate in Open Social. Thus, while Open Social provides a nice and well documented interface, it is likely that the social network information that it can access is lower quality than Facebook's (at least for US users).

GFC solves those two problems. First, the developer API, although still in JavaScript, is much simpler than Open Social's. The GFC site provides a number of basic capabilities (chat, comment, rating, polling, etc) that have generated HTML and JavaScript code. You can build up more complicated applications by cutting and pasting these capabilities into a single mash-up, add them to an existing HTML page, or make them into a gadget. Second, GFC applications are linked to Google's single sign-on system. Google may not have the social networking depth of Facebook, but they do have untold millions of accounts through GMail, Groups, and other popular online tools. This allows you to create miniature social networks for a particular GFC application out of your gmail contact list. Each GFC application is associated with a particular URL (and generated site ID), so you can create multiple networks out of the pool of Google accounts.

In subsequent posts, I'll look at GFC in a little more detail to examine its potential for science gateway social networks.

Mass Imports of Maven Repositories into Archiva

2009-12-03T12:30:00.000-08:00

Credit goes to Fugang Wang for this crafty trick. Archiva is a Web-based system for managing your Maven repositories. Unfortunately the Web interface only allows you to upload one jar at a time, and you also need to provide metadata (artifactId, version, group, etc). This isn't very useful if you need to import a few dozen or more jars.

Here is how to import an existing local repository (that is, your $HOME/.m2 directory) into Archiva:

1. Create a new named repository, say cyberaide. Then you'll get a directory 'cyberaide' under the archiva installation directory.
2. Move the whole repo directory under the newly created directory.
3. Click the 'Repository scanning' button for your repository. This will 'repair' the artifacts db so the newly added artifacts would show up.

Karajan and GridFTP

2009-07-23T17:41:00.000-07:00

I owe big thanks to Mihael Hategan for some help here.

Karajan is a workflow system that is part of the Java COG Kit and also powers the Swift system. You can get it in the 4.1.5 version of the COG Kit, but this version is out of date, so it is better to get the code from the SourceForge SVN:

View it: http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/src/cog/

Get it: svn co https://cogkit.svn.sourceforge.net/svnroot/cogkit/trunk/current/src/cog

Compile with ant. Here's an example script that will run gridftp on a remote host and return the file listing and metadata to your screen. Run it with $COG_INSTALL_PATH/bin/cog-workflow gridftpExample.xml:

<project>
<include file="cogkit.xml"/>

<set name="l">
<file:list dir="." host="gridftp-hg.ncsa.teragrid.org" provider="gridftp"/>
</set>
<for name="file" in="{l}">
<set names="type,permissions,size,modified">
<file:info file="{file}"/>
</set>
<print message="{file} Type:{type} Permissions:{permissions} Size:{size} Modified:{modified}"/>
</for>
</project>

Mihael actually added some of these fields on my request (they are part of the underlying code).

Karajan also has its own scripting language that is much less awkward than XML for expressing the scripts:

import("sys.k")
import("task.k")

element(niceType, [type]
if (
type == FILETYPE:FILE, "File"
type == FILETYPE:DIRECTORY, "Directory"
type == FILETYPE:SOFTLINK, "Softlink"
type == FILETYPE:DEVICE, "Device"
"Unknown"
)
)

files := file:list(dir=".",host="gridftp-hg.ncsa.teragrid.org",provider="GT2")

for(f, files,
[type, perm, size, modified] := file:info(f)
print("name: {f}, type: ", niceType(type),
" ({type}), size: {size}, modified: {modified}, permissions: {perm}")
)

Blogger, Google, and Bing

2009-07-07T17:23:00.000-07:00

I noticed my inflammatory post on how to use Gaussian on the TeraGrid (http://communitygrids.blogspot.com/2009/07/running-gaussian-on-big-red.html) is the #3 match if you google "gaussian teragrid", but it doesn't make the Top 50 if you use Bing. I suppose Google is weighting the search results to direct you to Blogger/Blogspot content.

Running Gaussian on TeraGrid

2009-07-06T13:03:00.000-07:00

These are notes for running Gaussian serially on the TeraGrid. I'll look at IU's BigRed and NCSA's Abe, Mercury, and Cobalt. Surprisingly this wasn't documented anywhere that I could find with Google (I had to resort to the Help Desk, which was helpful but this is missing the point). And of course running things on each machine requires a different incantation. You may wonder why the TeraGrid doesn't make its environments more consistent for these common applications. I suggest posting hypotheses on this subject as comments.

IU's Big Red
Gaussian is a famous piece of Quantum Chemistry software. I'll assume you have an input file, your_input.inp.

1. Request to be added to the Gaussian group. I found this through non-standard routes (thanks to Ray Sheppard) but try contacting the TeraGrid help desk. You have to do this before you can proceed.

1.5 Add gaussian to your .soft file. For more on the SoftEnv, see http://www.teragrid.org/userinfo/jobs/environment.php

My .soft file looks like this:
more ~/.soft
#
# This is the .soft file.
# It is used to customize your environment by setting up environment
# variables such as PATH and MANPATH.
# To learn what can be in this file, use 'man softenv'.
#
#
@bigred
@teragrid-basic
@globus-4.0
@teragrid-dev
+gaussian
----------------------
Either type "resoft" or just exit and login again.

2. Make a Loadleveler script like the one below:
---------------
# @ output=stdout.txt
# @ error=stderr.txt
# @ wall_clock_limit=5:00:00
# @ account_no=YOUR_ACCOUNT_NUMBER
# @ queue=default

export g03root=/N/soft/linux-sles9-ppc64/gaussian/g03-d.02
. $g03root/g03/bsd/g03.profile

g03 $HOME/your_input.inp $HOME/your_output.out
---------------

A real script would move stuff in and out of scratch space and stuff like that (Gaussian creates very large files while you run it), but I intend to automate this through Globus+Condor-g. Also this only will submit serial jobs.

3. Submit with llsubmit, track with llq. The IU Knowledge Base has a nice comparison of Loadleveler and PBS commands and directives here: http://kb.iu.edu/data/axpz.html

NCSA's Abe

Unlike the other machines, you definitely do not want "+gaussian" in your .soft file on abe. This will produce the following error:

g03: error while loading shared libraries: /usr/local/intel/mkl/10.0.3.020/lib/em64t/libmkl.so: invalid ELF header

Make a PBS script like the one below. Again this is only a serial job.
[mpierce@honest2 ~]$ more gaussTest.pbs
#PBS -o stdout.txt
#PBS -e stderr.txt
#PBS -A YOUR_ACCOUNT_NUMBER
#PBS -q normal
#PBS -l walltime=05:00:00

soft add +intel-mkl
setenv g03root /usr/apps/chemistry/gaussian/g03
source $g03root/g03/bsd/g03.login
g03 $HOME/your_input.inp $HOME/yourOutput.out
----------------------

Alternatively, I could have added intel-mkl to my .soft file. I had to get the NCSA Help Desk to tell me this (thanks go to John Estabrook @ NCSA for a quick and accurate response to my ticket). They pointed me in the right direction, but why not just have this documented?

NCSA also has a command-line tool, /usr/local/bin/qg03, that will make and submit a better PBS script than the one above.

globusrun -o -r grid-abe.ncsa.teragrid.org/jobmanager-pbs '&(executable=/u/ncsa/mpierce/gaussian.abe.ncsa.sh)(arguments=/u/ncsa/mpierce/input.in /u/ncsa/mpierce/mepjunk12.out)(project=YOUR-TG-ACCOUNT)(queue=debug)(host_types=himem)(host_xcount=1)(xcount=8)'

or

globusrun -o -r grid-abe.ncsa.teragrid.org/jobmanager-pbs '&(executable=/u/ncsa/mpierce/gaussian.abe.ncsa.sh)(arguments=/u/ncsa/mpierce/input.in /u/ncsa/mpierce/mepjunk12.out)(project=YOUR-TG-ACCOUNT)(queue=debug)(count=1)(hostCount=8)(minMemory=16000)'

I think these are equivalent.

NCSA's Mercury

This still works but who knows how long it will be before the system/application admins break it. As with Abe, best to at least use the qg03 command (it should be in your path) to generate a sample PBS script.

1. Add "+gaussian" to your .soft file (in $HOME) as above.

2. Make a PBS script like the one below. Again this is only a serial job.
[mpierce@honest2 ~]$ more gaussTest.pbs
#PBS -o stdout.txt
#PBS -e stderr.txt
#PBS -A YOUR_ACCOUNT_NUMBER
#PBS -l walltime=05:00:00

g03 $HOME/your_input.inp $HOME/yourOutput.out
----------------------

3. To invoke this with Globus clients or equivalent, you should use a command like

globusrun -o -r grid-abe.ncsa.teragrid.org/jobmanager-pbs '&(executable=/u/ncsa/mpierce/gaussian.cobalt.ncsa.sh)(arguments=/u/ncsa/mpierce/input.in /u/ncsa/mpierce/mepjunk12.out)(project=YOUR_ACCOUNT)(queue=debug)(minMemory=1600)'

or

globusrun -o -r grid-abe.ncsa.teragrid.org/jobmanager-pbs '&(executable=/u/ncsa/mpierce/gaussian.mercury.ncsa.sh)(arguments=/u/ncsa/mpierce/input.in /u/ncsa/mpierce/mepjunk12.out)(project=YOUR_ACCOUNT)(queue=debug)(host_types=himem)(host_xcount=1)(xcount=1)'

NCSA's Cobalt
This has changed and so previous instructions (identical to running Gaussian on Mercury above) no longer work. *DELETED TIRADE*. Setting +gaussian in your .soft no longer seems to work. Use the following minimal scriptlet

#!/bin/csh
setenv g03root /usr/apps/chemistry/gaussian/G03/pp5_e01
source $g03root/g03/bsd/g03.login
$g03root/g03/g03 $1 $2

You can invoke this with a globus run command like the one above.

PSC's Pople
You have to fill out a PDF form and mail or fax it to PSC to get access to Gaussian. I did not pursue this.

Error in gsi-ssh and ssh on Mac 10.5: percent_expand: NULL replacement

2009-07-03T08:46:00.000-07:00

I found this error while trying to log onto the TeraGrid ("Tara Grid") with gsissh (google "teragrid single sign on" for context).

percent_expand: NULL replacement

Apparently this is a well-known bug with Mac 10.5's ssh. The workaround is to append "-i ~/.ssh/id_rsa" or "-i ~/.ssh/id_dsa" to the command line, like so:

gsissh login-hg.ncsa.teragrid.org -i ~/.ssh/id_rsa

I got this from http://www.nabble.com/ssh%3A-percent_expand%3A-NULL-replacement-to13439207.html

Some CML Fun

2009-06-30T19:23:00.000-07:00

These are some notes on some initial work to create a chemical properties pipeline. End result will be calculated properties in an RDF triple store that can be exported using ORE/Atom. If I don't write this stuff down, I'll forget it.

Step 0: get Jumbo CML converter tools from SourceForge. Peter Murray Rust was kind enough to write a little README, now in the SVN. I did an SVN checkout rather than download the tagged Jumbo versio 5.4 or 5.5.

svn co https://cml.svn.sourceforge.net/svnroot/cml/jumbo-converter/trunk jumbo-converter

Step 0.5: Compile as an executable jar. I used "mvn assembly:assembly". See http://www.springone2gx.com/blog/scott_leberknight/2008/06/creating_executable_jars_using_the_maven_assembly_plugin.html

Step 0.75: Run with sample data using the command (must set classpath):

java -jar target/jumbo-converters-jar-with-dependencies.jar -converter org.xmlcml.cml.converters.molecule.pubchem.PubchemXML2CMLConverter
-sd examples/input -odir ../output -is pubchem.xml -os pubchem.cml

Step 1: get XML descriptions of molecules from Pubchem. I'm not a chemist, so I did a structure similarity search on caffeine, applied the Rule of Five, and got 1298 matches. That's pretty good.

Step 2: Unfortunately the PubChem download gives everything in one big XML file, but Jumbo's converter expects each molecule to be in a separate file. I googled around and made a little Perl script to do this:

#!/usr/bin/perl

use File::Basename;

$file = @ARGV[0];

open(_FH, "< $file") or die "Unable to open file: $file\n";

$count=0;
$max_records=1;
$files_counter=0;

while(<_FH>)
{
if($count == 0)
{

my @suffix=qw(.pubchem.xml);
my ($basefilename, $path, $suffix)=fileparse($file,@suffix);
$filename=$basefilename . "_part_" . $files_counter . $suffix;
open(FH2, "> $filename") or die "Unable to open file: $filenam
e\n";
$count++;
print FH2 "\<\?xml version=\"1.0\"\?\>\n";
}

if (grep /<\/PC-Compound>/, $_ )
{
print FH2 $_;
$count++;
}
elsif (grep /<PC-Compound>/, $_ )
{
print FH2 "<PC-Compound xmlns=\"http://www.ncbi.nlm.nih.gov\"
xmlns:xs=\"http://www.w3.org/2001/XMLSchema-instance\" xs:schemaLocation=\"http:
//www.ncbi.nlm.nih.gov ftp://ftp.ncbi.nlm.nih.gov/pubchem/specifications/pubchem
.xsd\">\n";

}
else {
print FH2 $_;
}

if ($count == $max_records + 1)
{
$count = 0;
$files_counter++;
close(FH2);
}
}

I stole a lot of this from http://www.mysysad.com/2008/01/parse-xml-records-with-perl-script.html

Step 2.5: Rerun the converter command on the new files. Jumbo was able to handle most but barfed on a few, so I'll have to follow up on this.

Step 3: Go to lunch. Next need PMR and Co. to decide on the best way to change the CML into Gaussian input files.

Step 4: Realize you downloaded a bunch of 2D structures, so repeat steps 0-3 with 3D structures.

Step 5: Make Gaussian input files from CML with

java -jar target/jumbo-converters-jar-with-dependencies.jar -converter org.xmlcml.cml.converters.compchem.gaussian.input.CML2GaussianInputConverter -sd examples/output -odir ../gaussian -is pubchem.cml -os pubchem.gauss

Current SVN of jumbo-converters won't compile without first checking out and compiling also cmlxom and jumbo5 from SVN (they are peers in the directory structure).

Step 6. Run Gaussian on TeraGrid machines. See http://communitygrids.blogspot.com/2009/07/running-gaussian-on-big-red.html. Welcome to 2009!

Step 7. Get the data back and convert the Gaussian standard output to CML. I used the command

java -jar target/jumbo-converters-jar-with-dependencies.jar -converter org.xmlcml.cml.converters.compchem.gaussian.log.GaussianLog2CMLConverter -sd examples/gaussOut -odir ../gaussOut -is out -os cml

I put the Gaussian output files in the examples/gaussOut directory.

Step 8. Convert the output CML to RDF:

java -jar target/jumbo-converters-jar-with-dependencies.jar -converter org.xmlcml.cml.converters.compchem.gaussian.GaussianCML2OWLRDFConverter -sd examples/gaussOut -odir ../gaussOut -is cml -os rdf

Jobs at the Pervasive Technology Institute

2009-06-24T08:13:00.000-07:00

We are looking to fill two positions. Please see the full ad below.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The following positions based at IUB and reporting to Marlon Pierce, Assistant Director, Community Grid Labs, is being posted externally. If interested, you must apply online at http://www.jobs.indiana.edu/. Search for job listing 401 and refer to positions number #00034420 and #00034425. See also https://ola.indiana.edu/joblisting/index.cfm?jlnum=401&search=2

Principal Software Research Engineers - PAE4IT (Two positions available)

Description and Responsibilities:

Indiana University is looking for experienced individuals to fill multiple positions in Cloud computing , Grid computing, and message-oriented middleware development. The positions call for expertise in Web service software development, experience developing Java Messaging Service (JMS) applications and related Enterprise event bus technologies, experience with distributed systems, sensor webs, and/or audio-video systems research and advanced development.
Working independently, the incumbent provides expertise in the evaluation, development, implementation and deployment of specific Cloud Computing, Message-Oriented Middleware, and Grid-related technologies in conjunction with Pervasive Technology Institute and Research Technology staff and collaborators. Investigate research problems in distributed computing systems, develop software using best software engineering practices, publish peer-reviewed research articles in workshops and journals, evaluate existing technologies, supervise staff and graduate students, and oversee all development aspects of Cloud and Grid scientific data and application management. Work with scientific users to develop Cloud and Grid systems. Collaborate with key academic researchers and open source user community including faculty, scientific researchers, and grant stakeholders.
Indiana University's Pervasive Technology was recently awarded a 5 year extension to its core funding from the Lilly Endowment, following its highly successful initial 7 years (as the Pervasive Technologies Laboratories). PTI will be located in Indiana University's new state of the art Research Incubator facility. See http://www.pervasive.iu.edu/ for more information. To date, the PTI laboratories have published more than 880 peer-reviewed papers, have won over $180,000,000 in grants over and above their core funding, and have released and maintain numerous open source software packages. PTI researchers work closely with Indiana University's Research Technology group, who maintain IU's world class cyberinfrastructure, http://racinfo.indiana.edu/cyberinfrastructure/.
Qualifications:

Master's degree in Computer Science or relate technical discipline (such as physics, informatics, engineering) is required. PhD preferred. Ph. D. incumbents must have demonstrated (through peer-reviewed publications and software products) a primary research focus on one or more of the following: a) distributed, parallel, Grid and/or Cloud computing; b) high performance, distributed messaging systems; c) audio/video collaboration systems; and d) security in distributed systems. Incumbents with Master's degree must have five to seven years experience in software development and deployment in the appropriate field or fields.

Requirements include a proven ability to undertake and succeed in innovative research projects, and excellent communication skill (covering scientific article writing, presentation skills, and the ability to work with and lead teams). Experience developing software systems using standard software engineering practices (such as source code management software, build and test systems, etc). For Cloud and Grid incumbents, experience developing Web Services, REST services, scientific workflows, and distributed Web applications on Unix/Linux/Mac OSX and/or Windows. Experience developing applications on Grids using software such as Globus or Condor. Experience developing scientific and research Cloud applications using Amazon Web Services, Microsoft Azure, Google AppEngine, and related systems. Experience developing data-parallel scientific, machine learning, and/or information retrieval algorithms using systems such as Apache Hadoop and Microsoft Dryad. For audio/video stream incumbents, demonstrated knowledge and implementation of standards as well as innovative systems. For message-oriented middleware incumbents, demonstrated sophisticated experience with standard MOM software. MOM incumbents should have experience both with core MOM development (efficient routing, security, reliable delivery) as well as applications.
Ability to foster and maintain collaborations with internal and external academic researchers through a balanced, service-based approach required. Demonstrated ability to learn new technologies and tools. Demonstrated initiative in evaluating and adopting new technologies. Organizational and leadership skills required. Able to operate effectively in a complex and dynamic environment with the capability of functioning as a project leader. Excellent oral and written communication skills. Demonstrated conceptual, analytical, and logical abilities. Ability to interact successfully with staff and faculty in extended consultations; demonstrate enthusiasm for new technologies; and deliver excellent outcomes under high pressure and short deadlines. Excellent communication skills required to interact with people of varying levels of knowledge.

Indiana University is an Affirmative Action/Equal Employment institution.

OGCE Overview Presentation from E-Science 2008

2009-06-04T08:28:00.000-07:00

OGCE Project Overview

View more OpenOffice presentations from marpierc.

Slideshare tutorial on GTLAB tag libraries

2009-06-04T08:22:00.000-07:00

Gtlab Install

View more OpenOffice presentations from marpierc.

New Pervasive Technology Institutes Web Site

2009-04-24T07:02:00.000-07:00

I'm doing my part for page ranking: http://www.pervasive.iu.edu/. This is the Community Grids Lab's new grandparent organization. We remain the same, just have a slightly different container org chart.

MS Word's Formatting Palette on Mac

2009-04-16T10:19:00.000-07:00

To get this, just click "View->Formatting Palette". The pop-up has everything you need (fonts, bullets, pre-defined styles, etc) for formatting your text.

I post this because I'm surprised that more people (Mac users anyway) apparently don't use this.

Old Dog Learns Old Trick

2009-04-06T12:15:00.001-07:00

It's never too late. I needed to add Tomcat's FORM authentication to a webapp. The steps for doing this are surprisingly under-documented. Here are they are:

1. Write a login form page (login.jsp). The form action should point to "j_security_check". The input parameters for the form are "j_username" and "j_password". There are plenty of examples of this.

2. Add the following to your web.xml:

<login-config>
<auth-method>FORM</auth-method>
<realm-name>Example Form-Based Authentication Area</realm-name>
<form-login-config>
<form-login-page>/login.jsp</form-login-page>
</form-login-config>
</login-config>

I took this from the jsp-examples/security examples that come with Tomcat.

If you stop here, install your webapp, and try to login by going directly to login.jsp, it will not work. You will get an error like

The requested resource (/mywebapp/j_security_check) is not available

3. You need to also add the following to your webapp's web.xml file (taken from Tomcat's jsp-examples/ again):

<security-constraint>
<display-name>Example Security Constraint</display-name>
<web-resource-collection>
<web-resource-name>Protected Area</web-resource-name>

<url-pattern>/*</url-pattern>

<http-method>DELETE</http-method>
<http-method>GET</http-method>
<http-method>POST</http-method>
<http-method>PUT</http-method>
</web-resource-collection>
<auth-constraint>

<role-name>tomcat</role-name>
<role-name>role1</role-name>
</auth-constraint>
</security-constraint>

This should work. The dirty trick is that you have to let Tomcat's security constraints redirect you to your login page. You can't point your browser to this page directly.

Word Cloud for My Blog

2009-02-20T17:31:00.001-08:00

From Wordle, courtesy of Yu "Don't call me Marie on your stupid blog" Ma.

Some Xen command line stuff

2009-02-06T15:55:00.000-08:00

Here's a Xen command line cheat sheet. We'll assume Xen is installed.

1. Make sure you've booted from the Xen kernel.

2. To start the Xen daemon, use "xend restart".

3. To install a new image, use virt-install --paravirt --name vc8 -r 500 -f /var/lib/xen/images/vc5.img --file-size 8 -l http://fedora.fastsoft.net/pub/linux/fedora/linux/releases/8/Fedora/i386/os --nographics

The virt-install man page has more examples and an explanation of the above. The above command installs Fedora 8 on the image. You should be prompted to install the OS in the VM after the download. This can obviously take a while, so you'll want to clone these images.

4. To list vms, you can use "virsh list" or "xm list".

5. To start your VM, use "virsh start vc8". The image "vc8" was the image created above.

6. To connect to the above VM, use "xm console vc8". Replace vc8 with your VM name. This will boot the operating system, which should get you to a login prompt.

GRAM Job submission failed because data transfer to the server failed (error code 10)

2009-02-05T14:32:00.000-08:00

Problem: I had a working pre-Web service GRAM "fork" job manager but then needed to use LSF job manager for submissions to the scheduler on a cluster. The LSF job manager was not built when we deployed globus initially, which is unusual.

The LSF job manager was built with the commands

% gpt-build globus_gram_job_manager_setup_lsf-1.17.tar.gz
% ./setup-globus-gram-job-manager-lsf

However the command line tests didn't work. For example, the command

globusrun -o -r my.secret.machine/jobmanager-lsf '&(executable=/bin/date)'

threw the error

GRAM Job submission failed because data transfer to the server failed (error code 10)

This is unfortunately an all-purpose Globus error. You will sometimes see it associated with problems in the grid-mapfile, but again my fork jobmanager worked fine, so I had a different bug.

Unfortunately nothing useful turned up in the gsi-gatekeeper.log, even after I turned up the logging level.

Solution: the problem turned out to be that the LSF job manager files were not given the correct permissions during the deployment. These should be 755 (group and world readable and executable). Find them with a command like

find $GLOBUS_LOCATION -name "*lsf*"

I then made the changes manually, but you may also do some "find|xargs" trick.

Globus error 123 and Condor_g

2009-02-05T10:58:00.001-08:00

Thanks to Stu Martin and Todd Tannenbaum for independently providing the solution below.

Problem: Condor-G submission to a Globus pre-webservice GRAM failed and throws an error

 "Globus error 123 (could not write the job state file)"

This is described in more detail at http://www-unix.globus.org/mail_archive/discuss/2002/11/msg00131.html. You can reproduce it with this globusrun command:

globusrun -r my.secret.machine/jobmanager '&(executable=/bin/date)(save_state=yes)'

Solution: Edit $GLOBUS_LOCATION/etc/globus-job-manager.conf and change the value of the -state-file-dir to point to a local, non-NFS file system. For example:

-state-file-dir /usr/local/gram_job_state

Also, set permissions on this directory:

chmod ogu+rwxt /usr/local/gram_job_state

myproxy-logon error

2009-02-05T07:49:00.000-08:00

The problem: my favorite myproxy-logon command suddenly stopped working and I got the error barf below

Failed to receive credentials.
Error authenticating: GSS Major Status: Authentication Failed
GSS Minor Status Error Chain:
globus_gss_assist: Error during context initialization
globus_gsi_gssapi: Unable to verify remote side's credentials
globus_gsi_gssapi: Unable to verify remote side's credentials: Couldn't verify the remote certificate
OpenSSL Error: s3_pkt.c:1052: in library: SSL routines, function SSL3_READ_BYTES: sslv3 alert bad certificate SSL alert number 42

I tried the usual soluiton: made sure my certificates were correct in $HOME/.globus/certificates and /etc/grid-security/certificates but still got the error.

Solution: I had a proxy certificate from another (non-overlapping) Grid sitting around in the default X509 location (/tmp/x509up_nnn). I deleted this and myproxy-logon worked again. Apparently the myproxy-logon command tries to use this credential to authenticate to the MyProxy server. The errors are what you expect since MyProxy from Grid #1 doesn't trust the CA of Grid #2.

Delete the old credential and things should work again.

Simple OAuth Client and Server Examples

2009-01-14T18:35:00.000-08:00

The OAuth Google code (http://code.google.com/p/oauth/) is woefully undocumented, so I put together below a zeroeth order approximation to how things actually work. To use this, you need both the Open Social Java implementation (for the client) and the OAuth Java implementation. Check them both out from the Google Code SVNs as described in earlier posts.

In the examples below, we will use OAuth's two-legged authentication. This is the server-to-server authentication used by OpenSocial gadgets. It is appropriate when no human intervention is needed (or is possible) in the authentication process. Also, we are going to just look at shared secret key-style authentication, using the HMAC-SH1 algorithm and symmetric key encryption. The OAuth code will also support public/private asymmetric keys.

Finally, note that the steps for generating the OAuth consumer key and consumer secret are out of scope. We will just use the key and secret from the earlier OpenSocial gadget experiments. Generally, shared secrets will need to be communicated through a separate registration process. Presumably, the server would use the consumer key sent by the client to look up the secret key on the server side from a database, but check the OAuth specification for details.

Server Code
We'll implement a dummy server as a JSP. You can put this in an Apache Tomcat server under a webapp called OAuthTest. You will also need to put several jars in webapp/OAuthTest/WEB-INF/lib:

ls apache-tomcat-5.5.27/webapps/OAuthTest/WEB-INF/lib/
commons-codec-1.3.jar jetty-6.1.11.jar
commons-httpclient-3.1.jar jetty-util-6.1.11.jar
commons-logging-1.1.jar junit.jar
httpclient-4.0-beta1.jar oauth-core-20090108.jar
httpcore-4.0-beta2.jar

The above jars are all from OAuth/java/lib, plus the oauth-core jar that you generated when you compiled things (see previous post--use "mvn clean install").

Here at last is the actual JSP for the dummy service. I call this OAuthTest.jsp.

--------------------

<%
//Presumably this should actually be looked up for a given key. Use a real key.
String consumerSecret="123121212yourkeyhere18918";

//Presumably the key is sent by the client. This is part of the URL, after all. Use a real key.
String consumerKey="orkut.com:1231212122";

//Construct the message object. Use null for the URL and let the code construct it.
OAuthMessage message=OAuthServlet.getMessage(request,null);

//Construct an accessor and a consumer
OAuthConsumer consumer=new OAuthConsumer(null, consumerKey, consumerSecret, null);
OAuthAccessor accessor=new OAuthAccessor(consumer);

//Now validate. Weirdly, validator has a void return type. It throws exceptions
//if there are problems.
SimpleOAuthValidator validator=new SimpleOAuthValidator();
validator.validateMessage(message,accessor);

//Now what? Generate some JSON here for example.
System.out.println("It must have worked"); %>

-------------------

Client Code
To build an OAuth client, we can start from our earlier OpenSocial client. For convenience, we will leave in the OpenSocialUrl and OpenSocialHttpRequest classes, which help construct and execute the REST invocation, but one could easily eliminate this and use the standard java.net classes that underly these two.

Here's the client code. Save it in a file called MyOAuthClient.java in OpenSocial's java/samples directory.
---------------------
import org.opensocial.data.*;
import org.opensocial.client.*;
import net.oauth.*;
import java.util.*;

public class MyOAuthClient {

public static void main(String[] args) {
MyOAuthClient mosc=new MyOAuthClient();
}

public MyOAuthClient() {
String REST_BASE_URI=
"http://localhost:8080/OAuthTest/OAuthTest.jsp";
String CONSUMER_SECRET=
"uynAeXiWTisflWX99KU1D2q5";
String CONSUMER_KEY=
"orkut.com:623061448914";
// String VIEWER_ID=
// "03067092798963641994";
String VIEWER_ID="08354253340777199997";

try {
OpenSocialUrl requestUrl = new OpenSocialUrl(REST_BASE_URI);
OpenSocialHttpRequest request=new OpenSocialHttpRequest(requestUrl);
requestUrl.addQueryStringParameter("xoauth_requestor_id", VIEWER_ID);
requestUrl.addQueryStringParameter("st", "");

String requestMethod=request.getMethod();
String postBody = request.getPostBody();

OAuthMessage message =
new OAuthMessage(requestMethod, requestUrl.toString(), null);

OAuthConsumer consumer =
new OAuthConsumer(null, CONSUMER_KEY, CONSUMER_SECRET, null);
consumer.setProperty(OAuth.OAUTH_SIGNATURE_METHOD, OAuth.HMAC_SHA1);

OAuthAccessor accessor = new OAuthAccessor(consumer);
accessor.accessToken = "";
message.addRequiredParameters(accessor);

for (Map.Entry p : message.getParameters()) {
if (!p.getKey().equals(postBody)) {
requestUrl.addQueryStringParameter(
OAuth.percentEncode(p.getKey()),
OAuth.percentEncode(p.getValue()));
}
}

//Take a look at the signed URL
System.out.println("Signed REST URL: "+requestUrl.toString());

//Done with signing. Now back to OpenSocialBatch's submitRest()
//Finally, get the response. This is the meat of the getHttpResponse
//method, without the error checking.
request.execute();
}

catch(Exception ex) {
ex.printStackTrace();
}
}
}
------------------
Compile this from the OpenSocial SVN checkout's java directory with the command ant compile-samples.

Now set your classpath and execute. The following UNIX commands will do the trick:
export CP=`echo $HOME/opensocial-java-client-read-only/java/lib/*.jar | tr ' ' ':'`

export CP=`echo $HOME/opensocial-java-client-read-only/java/dist/*.jar | tr ' ' ':'`:$CP

java -classpath $CP:/Users/marlonpierce/opensocial-java-client-read-only/java/samples/bin/ MyOAuthClient

Check the output of your Tomcat server's catalina.out to see if it worked.

Obviously this has all been a simple exercise to see how to get things going and needs a lot of work to make it a real service.

Quick Note on Google Code OAuth SVN Checkout

2009-01-12T13:21:00.001-08:00

I'm looking at the Google OAuth code (http://code.google.com/p/oauth). It's pretty light on instructions. To compile the Java checkout from SVN, you also need to check out the Maven repositories:

mkdir oauth-fun; cd oauth-fun
svn checkout http://oauth.googlecode.com/svn/code/java
svn checkout http://oauth.googlecode.com/svn/code/maven

You compile with

cd java (assuming you are in oauth-fun)
mvn clean install

The problem is that my checkout today didn't compile. I had to add the maven repository to the top level pom.xml (the one in oauth-fun/java) like so:

<repositories>
<repository>
<id>oauth-local</id>
<url>file:../maven</url>
</repository>
</repositories>

Roll Your Own REST with Open Social Clients

2009-01-11T11:27:00.000-08:00

As in the last few posts, I've been looking at the OpenSocial REST/RPC client code. This can be used to access (for example) Orkut as a back-end service. The Java clients are here: http://code.google.com/p/opensocial-java-client/. The general (programming-language neutral) API is documented here: http://code.google.com/apis/orkut/docs/rest/developers_guide_protocol.html

The Java client examples show how to do simple stuff like access a person's profile data and his/her friends' information. The primary higher level API for this is the OpenSocialClient.java class. The problem is that this class only implements a subset of the possible REST/RPC commands at this time. Peeling back the covers a little, however, you can see that this class is really just constructing REST URLs or RPC commands using lower level classes, so you can in principle do this yourself and bypass OpenSocialClient entirely.

I decided to try this out. The code below uses the REST style invocations to duplicate the DisplayProfileData.java sample code. It turned out to be a little convoluted to do this because the OpenSocialClient class itself gets passed to lower level classes (particularly OpenSocialRequstSigner.java), which is not helpful. Hopefully the developers will reorganize this in the future.

Anyway, on to the code. I think it can be modified to construct any OpenSocial REST request, including PUT and DELETE (which are not currently in OpenSocialClient), but that is a test for another day. I don't claim this is beautiful. It is just my attempt to unravel everything.

I used Revision 33 of the SVN checkout. The OAuth snapshot jar in lib can cause problems if you use a different version. See comments below.

----------------

import org.opensocial.data.*;
import org.opensocial.client.*;
import net.oauth.*;
import java.util.*;

public class MyOpenSocialClient {

public static void main(String[] args) {
MyOpenSocialClient mosc=new MyOpenSocialClient();
}

public MyOpenSocialClient() {
String REST_BASE_URI=
"http://sandbox.orkut.com/social/rest";
String CONSUMER_SECRET=
"uynAeXiWTisflWX99KU1D2q5";
String CONSUMER_KEY=
"orkut.com:623061448914";
// String VIEWER_ID=
// "03067092798963641994";
String VIEWER_ID="08354253340777199997";

try {
//Construct the request object
OpenSocialRequest req=new OpenSocialRequest("people","people.get");
req.addParameter("groupId","@self");
req.addParameter("userId","08354253340777199997");

//Construct the request URL from the request object
//This is the OpenSocialBatch.submitRest() method.
OpenSocialUrl requestUrl = new OpenSocialUrl(REST_BASE_URI);
requestUrl.addPathComponent(req.getRestPathComponent());
if (req.getParameter("userId") != null) {
requestUrl.addPathComponent(req.getParameter("userId"));
}
if (req.getParameter("groupId") != null) {
requestUrl.addPathComponent(req.getParameter("groupId"));
}
System.out.println("This is the REST URL: "+requestUrl.toString());

//Now construct and sign the request
//This is the OpenSocialRequestSigner.signRequest() code.
OpenSocialHttpRequest request = new OpenSocialHttpRequest(requestUrl);
requestUrl.addQueryStringParameter("xoauth_requestor_id", VIEWER_ID);
requestUrl.addQueryStringParameter("st", "");

String postBody = request.getPostBody();
System.out.println("Post bod: "+postBody);
String requestMethod = request.getMethod();
// OpenSocialUrl requestUrl = request.getUrl();

OAuthMessage message =
new OAuthMessage(requestMethod, requestUrl.toString(), null);

if (postBody != null) {
message.addParameter(postBody, "");
}

OAuthConsumer consumer =
new OAuthConsumer(null, CONSUMER_KEY, CONSUMER_SECRET, null);
consumer.setProperty(OAuth.OAUTH_SIGNATURE_METHOD, OAuth.HMAC_SHA1);

OAuthAccessor accessor = new OAuthAccessor(consumer);
accessor.accessToken = "";

message.addRequiredParameters(accessor);

//VERSION WARNINIG: This works with SVN version 33 and oauth-20081115.jar but other
//versions may have problems.
for (Map.Entry p : message.getParameters()) {
if (!p.getKey().equals(postBody)) {
requestUrl.addQueryStringParameter(
OAuth.percentEncode(p.getKey()),
OAuth.percentEncode(p.getValue()));
}
}

//Take a look at the signed URL
System.out.println("Signed REST URL: "+requestUrl.toString());

//Done with signing. Now back to OpenSocialBatch's submitRest()
//Finally, get the response. This is the meat of the getHttpResponse
//method, without the error checking.
request.execute();
String responseString=request.getResponseString();
OpenSocialResponse resp=OpenSocialJsonParser.getResponse(responseString, req.getId());
//A little clunky--have to add this to the OpenSocialResponse
//object's "items" map with the "people" key.
resp.addItem("people",responseString);

System.out.println("Response: "+responseString);

//Now cast the response into an OpenSocialPerson object
OpenSocialPerson person=resp.getItemAsPerson("people");

System.out.println("----------");

// Output the name and ID of the requested person
System.out.println("Info. for " + person.getDisplayName());
System.out.println("ID: " + person.getId());

// Retrieve individual fields using the getField method; fields may be
// complex (objects or arrays) or simple (Strings), which you can
// determine by querying the object using the isComplex method.
// The thumbnail URL should be a simple field, so we'll output
// the value as a String
OpenSocialField thumbnailUrlField = person.getField("thumbnailUrl");
if (!thumbnailUrlField.isComplex()) {
System.out.println("Thumbnail URL: " +
thumbnailUrlField.getStringValue());
}

System.out.println("----------");

} catch (Exception e) {
System.out.println("Request failed:" );
e.printStackTrace();
}
}
}

OpenSocial REST and WGET: Not So Much

2009-01-09T13:54:00.000-08:00

The Open Social REST/RPC documentation (http://code.google.com/apis/orkut/docs/rest/developers_guide_protocol.html) says vaguely that you can play with the examples using the UNIX wget command, but I don't see it.

The problem is that the URLs have to be signed and carry proper security information. By fooling around with the example code, you can see that the OpenSocialClient class's fetchPerson() method is really just constructing a URL like the one below:

http://sandbox.orkut.com/social/rest/people/08354253340777199997/@self

The number 083542533407771999 is my orkut user ID. You can't just "wget" this URL or put it in your browser, however. Doing so will return the error

HTTP request sent, awaiting response... 401 The request did not have a proper security token nor oauth message and unauthenticated requests are not allowed

Fooling around with the client codes (see previous post) will reveal the actual, signed URL used in the REST operation:

http://sandbox.orkut.com/social/rest/people/08354253340777199
997/@self?oauth_consumer_key=orkut.com%3A623061448914&oauth_nonce=1231537930162003000&oauth_tim
estamp=1231537930&oauth_signature=0h%2FU49KtBplnmnc%2BhDKsDxFPR9k%3D&oauth_signature_method=HMA
C-SHA1&oauth_token=&xoauth_requestor_id=03067092798963641994&oauth_version=1.0

This URL is good for only one invocation (time-stamped).

To reproduce this, make the following change to DisplayProfileInfo.java:

// c.setProperty(OpenSocialClient.Properties.RPC_ENDPOINT,
// "http://sandbox.orkut.com/social/rpc");
c.setProperty(OpenSocialClient.Properties.REST_BASE_URI,
"http://sandbox.orkut.com/social/rest");

and then add the following line to OpenSocialBatch.java's submitRest() method.

OpenSocialRequestSigner.signRequest(request, client);
System.out.println("This is the post-signed REST url: " + request.getUrl().toString());

More soon. It turns out the the base OpenSocialClient.java class is actually just a wrapper around the other classes (OpenSocialRequest and so on). So although the client does not apparently support write operations, you can work around this by constructing the appropriate REST POST operation.

OpenSocial REST API Is Here

2009-01-08T12:25:00.001-08:00

I just noticed the Google-led OpenSocial REST/RPC APIs are finally available. These allow you to build Java, PHP, Python, and Ruby clients that can access and manipulate your social network data.

Blog announcement: http://opensocialapis.blogspot.com/

Java code: http://code.google.com/p/opensocial-java-client/

Orkut Sandbox information: http://code.google.com/apis/orkut/docs/rest/developers_guide_protocol.html

Client Examples
As you might expect, you can use Orkut's sandbox as your "backend" service. The Java code and examples are pretty simple but work. As a bonus, here is a recursive method that you can use in the DisplayProfileData.java example to display all information about a particular person. It will also work in DisplayFriends.java. You'll need to add imports for OpenSocialObject and OpenSocialField.

protected void processFields(OpenSocialObject osObj) throws Exception {
//See all fields
String[] fields=osObj.fieldNames();
for(String fieldName : fields) {
OpenSocialField afield=osObj.getField(fieldName);
if(!afield.isComplex()) {
System.out.println(fieldName+" "+afield.getStringValue());
}
else {
System.out.println(fieldName+" is complex:" );
Collection complexValues=afield.getValues();
for(OpenSocialObject compVal : complexValues) {
processFields(compVal);
}
}
}

Call it anywhere after creating the sample person, like so:

OpenSocialPerson person = c.fetchPerson("03067092798963641994");
...
processFields(person);

You may also want to change the Person ID in the above line. You get this ID when you add sample.xml social gadget to your sandbox. Mine, for example, is 08354253340777199997. This ID does not need to match the viewer ID discussed below.

Here is the sample output:
156-56-104-143:java mpierce$ java -classpath samples/bin:$CP DisplayProfileData
----------
Info. for API DWH
ID: 03067092798963641994
----------
Printing fields
----------
photos is complex:
value http://www.orkut.com/img/i_nophoto64.gif
type thumbnail
thumbnailUrl http://www.orkut.com/img/i_nophoto64.gif
name is complex:
givenName API
familyName DWH
id 03067092798963641994
isOwner false
isViewer true
------Done---------
Thumbnail URL: http://www.orkut.com/img/i_nophoto64.gif
----------

Modifying the Backend
Here's were the problems start. The sample clients above include a default view ID, consumer secret, and consumer key as part of the OAuth infrastructure:

c.setProperty(OpenSocialClient.Properties.CONSUMER_SECRET,
"uynAeXiWTisflWX99KU1D2q5");
c.setProperty(OpenSocialClient.Properties.CONSUMER_KEY,
"orkut.com:623061448914");
c.setProperty(OpenSocialClient.Properties.VIEWER_ID,
"03067092798963641994");

Of course you will want to replace these with your own values. To do this, you will need to deploy a gadget and verify that you own it. The steps are simple:

Write a gadget. You can use http://opensocial-resources.googlecode.com/svn/samples/rest_rpc/sample.xml for example. I'm not sure if the gadget in anyway affects the REST/RPC server side calls that you can use.
Get an Orkut sandbox account and log into sandbox.orkut.com.
Install your application into your Orkut account.
Verify that you own the gadget: https://www.google.com/gadgets/directory/verify

This last step is the problem. It involves two steps. First you provide the URL for your gadget and submit. You will receive a long signature string, which you need to add to your gadget. Replace the one in the sample.xml if you are using a copy of that. Save your gadget and click the verification web site button to complete the process. Presumably this will give you the consumer key and consumer secret that you need, but I have not gotten this to work for some reason.
See
http://groups.google.com/group/opensocial-orkut/browse_thread/thread/43640406754ab7f6

No airport hardware found

2008-12-04T17:46:00.000-08:00

From the stupid Mac tricks department: if you get the error in the title (i.e. wireless card not working),

power down,
take out your battery,
count "1 Steve Jobs, 2 Steve Jobs, 3 Steve Jobs",
reinsert battery,
power up,

and viola...voila, wireless will work.

The last guy on this thread deserves a Nobel prize: http://forums.macrumors.com/archive/index.php/t-173155.html

QuakeSim Stuff in Shakeout Youtube Movie

2008-11-13T11:06:00.001-08:00

About 1/2 though:
http://www.youtube.com/watch?v=7F3J_EtRFi0

WebDAV with Apache2 and Windows

2008-10-26T08:39:00.000-07:00

In case you haven't looked in a while, WebDAV modules are included with Apache 2 and 2.2. See for example http://www.serverwatch.com/tutorials/article.php/2176771 for an installation overview.

This will work great for Mac clients (from "Finder" hit apple-k to connect as a remote web folder) with HTTPD basic authentication, but they will fail for Windows XP clients (either mapping as a drive or adding network location). The reason, it turns out, is that Windows is actually working correctly: it won't do Basic Auth over an insecure connection.

To solve this, use SSL. There's an overview of the problem and variuos solutions at http://ulihansen.kicks-ass.net/aero/webdav/. See for example http://www.vanemery.com/Linux/Apache/apache-SSL.html for some Apache + SSL instructions.

One Final Trick: Windows clients seem to have a problem with the trailing "/", so in your Apache httpd.conf file, use something like

<location /dav>

and not

<location /dav/>

for your WebDAV folder name.

Abstract: Services and Components for the QuakeSim Project

2008-10-08T10:18:00.000-07:00

[Posting is part of our lab seminar series.]

We discuss two recent enhancements to the QuakeSim grid infrastructure. First, we describe our work to integrate QuakeSim Web services with large scale Globus deployments such as the US TeraGrid and Open Science Grid. Our implementation is based on the Condor BirdBath client libraries and provides a simplified and stateless API for accessing Globus resources. In addition to QuakeSim, we are developing a variation of this service to support the University of Minnesota’s VLAB project. We discuss integration of this service with TeraGrid information services for system load and queue wait time prediction, as well as security issues and prospects for deployment on NASA resources.

Our second focus is on work performed with the Scripps Orbit and Permanent Array Center’s archival data services for permanently deployed California GPS stations. We have developed a daily historical analysis Web service of the GPS data using RDAHMM. This service has been combined with an interactive Web interface that allows the user to view historical state changes in the GPS network. This archival data is available on a fourteen day delay, so we conclude with a discussion of the prospects for a nearer real-time network event detection system.

DAG Classes for Java

2008-09-24T15:13:00.000-07:00

It's a little surprising that even after more than a dozen years worth of bloat, Java still doesn't have a standard set of classes to handle standard graphs like DAGs and their associated operations.

I needed this recently and decided against writing from scratch. With a little Googling, I decided to use the Plexus Utilities. See the JavaDocs at http://plexus.codehaus.org/plexus-utils/apidocs/index.html

The Plexus-Utils overview is here: http://plexus.codehaus.org/plexus-utils/. The best way to get going is to grab the jars with Maven. Below is a dumb little example code:

import org.codehaus.plexus.util.dag.DAG;
import org.codehaus.plexus.util.dag.Vertex;
import org.codehaus.plexus.util.dag.CycleDetectedException;

import java.util.*;

public class DAGFun {
DAG dag;
Vertex from1, to1;
Vertex from2, to2;
public DAGFun() {
dag=new DAG();
from1=new Vertex("from");
to1=new Vertex("to");
}

public DAGFun(DAG dag) {
this.dag=dag;
}

public void addEdge(DAG dag, Vertex from, Vertex to) throws CycleDetectedException {
dag.addEdge(from,to);
}

public void processDag(){
List vertices=dag.getVerticies();
List roots=new ArrayList();
//Find all the roots. There may be more than one
for(int i=0;i<vertices.size();i++){
Vertex test=(Vertex)vertices.get(i);
if(test.isRoot()) roots.add(test);
}

//Now walk through all the DAGs
for(int i=0;i<roots.size();i++){
Vertex v=(Vertex)roots.get(i);
List children=v.getChildren();
for(int j=0;j<children.size();j++) {
processChild((Vertex)children.get(j));
}
}
}

public void processChild(Vertex vertex) {
if(vertex.isLeaf()) {
return;
}
List children=vertex.getChildren();
for(int j=0;j<children.size();j++) {
processChild((Vertex)children.get(j));
}
}

public static void main(String[] args) throws Exception{
Vertex from1, to1;
Vertex from2, to2;
from1=new Vertex("from1");
to1=new Vertex("to1");

from2=new Vertex("from2");
to2=new Vertex("to2");

DAG dag=new DAG();
DAGFun df=new DAGFun(dag);
df.addEdge(dag,from1,to1);
df.addEdge(dag,from2,to2);

df.processDag();
}
}

Yum install of SVN on CentOS

2008-09-08T12:53:00.001-07:00

yum install subversion mod_dav_svn

Sun RSS Portlet in GridSphere

2008-08-27T14:29:00.000-07:00

OK, thanks to Suresh and Marcus, I got the Sun RSS portlet working in GridSphere. Very good indeed, since the others that I found depended on Liferay or JBoss jars. Good grief, people, you are getting on my last nerve.

Here are the steps:

Get the portlet from here: http://wiki.java.net/bin/view/OpenPortal/RSSPortlet
Do the usual gridsphere stuff: you will need to modify web.xml, copy over the gridsphere-portlet.xml from a working portlet into WEB-INF, copy over a gridsphere-ui-tags.jar from a working portlet into WEB-INF/lib. Also a group.xml and layout.xml file in WEB-INF will help. Plenty of examples of this over at http://www.collab-ogce.org/.
Also touch webapps/gridsphere/WEB-INF/CustomPortal/portlets/rssportlet
For some reason, the Sun portlet doesn't include the necessary jars to support the portlet's tag libraries, so you will need to get these and put them into WEB-INF/lib: jstl-1.0.6.jar, standard-1.0.6.jar.

This should be everything. The missing jar problem in step 4 was the frustrating part. Here's a list:

content-0.4.1.jar jdom-1.0.jar rome-0.8.jar
gridsphere-ui-tags-2.1.jar jstl-1.0.6.jar standard-1.0.6.jar

Probably there is an easier way, but around here we like to do it the bassackwards way.

Some BLAHP Notes

2008-08-18T08:11:00.000-07:00

BLAHP is a plugin to Condor-G that enables Condor to submit jobs directly to schedulers like PBS/Torque. It is a variation of the GAHP scheduler that Condor-G uses to submit Globus jobs. Globus in turn provides the appropriate job manager for interacting with local scheduler. BLAHP (hopefully) will allow us to skip this step.

BLAHP is avaialble from the EGEE; see https://twiki.cnaf.infn.it/cgi-bin/twiki/view/EgeeJra1It/BLAH_guide for an overview. There is an obsolete version of this in the Condor release, but we must override.

We are interested in this because Condor-G and Birdbath provide a Web service programming interface for managing jobs. See other notes on this blog and at http://sangpall.blogspot.com/. We'd like to use this service for accessing machines that don't run Globus GRAM.

Basic steps for getting this going are the following. All steps are done on the same server.

Install Condor and optionally Globus.
Get the latest Condor C++ Classads, configure, make, make install
Check out BLAHP from CVS, configure, make, make install
Get Torque if you don't have a scheduler installed, configure, make, make install.

Notes are courtesy of Francesco Prelz, with some corrections from Yu Marie Ma. I'm building this on RHEL4.

Install Classads

Make an installation directory. I'll call this BLAHP.
wget ftp://ftp.cs.wisc.edu/condor/classad/c++/classads-0.9.9.tar.gz
Unpack and cd.
./configure --prefix $HOME/BLAHP/classads/ --enable-namespace --lib-name classad
make; make install

The trick is the configure step. You must enable the classad namespace (at least on RHEL4) for BLAHP to install (divined by Marie).

Install BLAHP
I also do these steps from the $HOME/BLAHP directory.

cvs -d :pserver:anonymous@glite.cvs.cern.ch:/cvs/jra1mw co org.glite
cvs -d :pserver:anonymous@glite.cvs.cern.ch:/cvs/jra1mw co -r glite-ce-blahp_R_1_11_1_0 org.glite.ce.blahp
cd $HOME/BLAHP/org.glite.ce.blahp/
./configure --with-classads-prefix=$HOME/BLAHP/classads/ --with-globus-prefix=$HOME/globus/ --prefix=$HOME/BLAHP/blahp

Install Torque

This assumes you don't have this already installed. The steps below will work for a non-root build.

cd $HOME/BLAHP
./configure --prefix=$HOME/BLAHP/torque --with-server-home=$HOME/BLAHP/torque
make; make install

Configure BLAHP
We now need to configure BLAHP to work with our local Condor and Torque installations.

TeraGrid RSS Feeds

2008-08-05T06:55:00.000-07:00

Here's a big list of TeraGrid RSS news feeds. A while back, I wrote a post complaining that these didn't work correctly in iGoogle, but I'm pleased to write now that this has been corrected. On the downside, you have to go through a few pages to find the RSS URLs.

You can guess the subject for each feed from the title (rss-indiana.rss is the IU feed, for example).

http://news.teragrid.org/feeds/rss-education.rss
http://news.teragrid.org/feeds/rss-event.rss
http://news.teragrid.org/feeds/rss-gpfs.rss
http://news.teragrid.org/feeds/rss-indiana.rss
http://news.teragrid.org/feeds/rss-lsu.rss
http://news.teragrid.org/feeds/rss-ncar.rss
http://news.teragrid.org/feeds/rss-ncsa.rss
http://news.teragrid.org/feeds/rss-nics.rss
http://news.teragrid.org/feeds/rss-networking.rss
http://news.teragrid.org/feeds/rss-ornl.rss
http://news.teragrid.org/feeds/rss-psc.rss
http://news.teragrid.org/feeds/rss-pr.rss
http://news.teragrid.org/feeds/rss-purdue.rss
http://news.teragrid.org/feeds/rss-sdsc.rss
http://news.teragrid.org/feeds/rss-gateways.rss
http://news.teragrid.org/feeds/rss-archives.rss
http://news.teragrid.org/feeds/rss-tacc.rss
http://news.teragrid.org/feeds/rss-general.rss
http://news.teragrid.org/feeds/rss-anl.rss
http://news.teragrid.org/feeds/rss-vis.rss

Old Windows XP on New Dell Latitude

2008-06-24T15:36:00.000-07:00

More adventures in system administration: I needed to install Windows XP on a new Dell Latitude laptop to support a demo. My old Windows XP installation CD crapped out, claiming it could not find any disk drives/devices.

The workaround (thanks to Rick McMullen) turned out to be to change from SATA AHCI to ATA in the BIOS. On the downside, it wiped out my old Vista installation, although switching back to AHCI seemed to recover it.

Useful TeraGrid REST Information Services

2008-04-30T13:12:00.001-07:00

Mildly Interesting KML+Google Maps Bug

2008-04-04T14:02:00.000-07:00

Google map keys are of course tied to a particular base URL. You can get a key for http://localhost for development, but you may have noticed this isn't necessary: maps will load correctly in http://localhost/ for any map key.

However, you can't load KML into your maps on localhost unless you are using a key for http://localhost. That is, your map will load but the KML overlay wont' be added. That is,

geoXml=new GGeoXml("http://mapgadgets.googlepages.com/cta.kml");
map.addOverlay(geoXml);

won't add the overlay to your map if you are using mismatched keys. To see this in inaction, try serving up the code sample http://code.google.com/apis/maps/documentation/examples/geoxml-kml.html
from your own web server (using your own key, of course). First try using localhost:8080 and then use http://your.real.server.name:8080/. This assumes the google key was generated for http://your.real.server.name:8080/.

Google's GData Java API

2008-03-28T11:58:00.000-07:00

This has been out for a while, but I recently realized that Google's GData API provides a way to interact with all Google services (Doc, Calendar, YouTube, Blogger, Picasso, etc) through your own server side code. One nice application for Science Gateways is to write a Web Service that submits grid jobs, writes service state changes to Google Calendar, spits out result synopsis to a blog, and posts visualizations to Picasso or YouTube.

Below is some code liberally borrowed from Blogger, YouTube, and Calendar API gdata examples to show how simple this is. Some notes:

Get the Java code from here: http://code.google.com/p/gdata-java-client/downloads/list
You will need a clientID and associated developer key. Get these from http://code.google.com/apis/base/signup.html (I think).
See any number of Google examples, including http://code.google.com/apis/base/javadevguide.html
You can get your blogger ID by examining the Atom feed. It will be in the header and look something like this: <id>tag:blogger.com,1999:blog-19457310</id> The blog ID to use in the code below would be 194573 in this example.

--------------------
Code example below
--------------------
import com.google.gdata.client.*;
import com.google.gdata.client.youtube.*;
import com.google.gdata.data.*;
import com.google.gdata.data.geo.impl.*;
import com.google.gdata.data.media.*;
import com.google.gdata.data.media.mediarss.*;
import com.google.gdata.data.youtube.*;
import com.google.gdata.data.extensions.*;
import com.google.gdata.util.*;

import com.google.gdata.client.*;
import com.google.gdata.data.*;
import com.google.gdata.util.*;
import java.io.IOException;
import java.net.URL;
import java.util.*;

//Google Calendar stuff
import com.google.gdata.client.calendar.*;
import com.google.gdata.data.acl.*;
import com.google.gdata.data.calendar.*;
import com.google.gdata.data.extensions.*;

public class TestBlogger {
String clientID="XXXXX";
String developer_key="XXXXXXX";

String feedUrl="http://gdata.youtube.com/feeds/api/videos/";
String scarFace5sec="http://gdata.youtube.com/feeds/api/videos/egwB7hVIIEc";

GoogleService googleService;
YouTubeService youTubeService;
CalendarService calendarService;

public GoogleService getGoogleService() {
return googleService;
}
public void setGoogleService(GoogleService googleService) {
this.googleService=googleService;
}

public YouTubeService getYouTubeService() {
return youTubeService;
}

public void setYouTubeService(YouTubeService youTubeService) {
this.youTubeService=youTubeService;
}

public void setCalendarService(CalendarService calendarService) {
this.calendarService=calendarService;
}

public CalendarService getCalendarService() {
return this.calendarService;
}

public TestBlogger() throws Exception {
//Replace these with real values.
String userName="me@gmail.com";
String password= "qwerty";

//Log into blogger.
GoogleService myService=new GoogleService("blogger","");
myService.setUserCredentials(userName,password);
setGoogleService(myService);

//Log into YouTube
YouTubeService service = new YouTubeService(clientID, developer_key);
service.setUserCredentials(userName,password);
setYouTubeService(service);

//Log into Calendar
CalendarService calService=new CalendarService(clientID);
calService.setUserCredentials(userName,password);
setCalendarService(calService);
}

public void writeToCalendar(String titleOfEvent,
String contentToPost) throws Exception {
URL postUrl =
new URL("http://www.google.com/calendar/feeds/"+userName+"/private/full");
CalendarEventEntry myEntry = new CalendarEventEntry();

myEntry.setTitle(new PlainTextConstruct(titleOfEvent));
myEntry.setContent(new PlainTextConstruct(contentToPost));

// DateTime startTime = DateTime.parseDateTime("2008-03-18T15:00:00-08:00");
System.out.println((new Date()).toString());
DateTime startTime = new DateTime(new Date());
// DateTime endTime = DateTime.parseDateTime("2008-03-18T17:00:00-08:00");
When eventTimes = new When();
eventTimes.setStartTime(startTime);
// eventTimes.setEndTime(endTime);
myEntry.addTime(eventTimes);

// Send the request and receive the response:
CalendarEventEntry insertedEntry = calendarService.insert(postUrl, myEntry);

CalendarEventEntry myEntry2 = new CalendarEventEntry();
String now=(new Date()).toString();
myEntry2.setContent(new PlainTextConstruct("Test post at "+now));
myEntry2.setQuickAdd(true);

// Send the request and receive the response:
CalendarEventEntry insertedEntry2 =calendarService.insert(postUrl, myEntry2);
}

public String getBloggerEntryString() throws Exception {
String returnString="";
URL feedUrl=new URL("http://communitygrids.blogspot.com/atom.xml");
Feed resultFeed = googleService.getFeed(feedUrl, Feed.class);

// Print the results
System.out.println(resultFeed.getTitle().getPlainText());
for (int i = 0; i < resultFeed.getEntries().size(); i++) { Entry entry = resultFeed.getEntries().get(i); returnString+=entry.getTitle().getPlainText()+"
";
}
return returnString;
}

public String getVideoFeedString() throws Exception {

String returnString;

VideoFeed videoFeed=youTubeService.getFeed(new URL(feedUrl),
VideoFeed.class);

VideoEntry videoEntry = youTubeService.getEntry(new URL(scarFace5sec),
VideoEntry.class);

returnString="Title: " + videoEntry.getTitle().getPlainText()+"
";
returnString+=videoEntry.getMediaGroup().getDescription().getPlainTextContent()+"
";
return returnString;
}

public Entry createPost(GoogleService myService, String blogID, String title,
String content, String authorName, String userName)
throws ServiceException, IOException {
// Create the entry to insert
Entry myEntry = new Entry();
myEntry.setTitle(new PlainTextConstruct(title));
myEntry.setContent(new PlainTextConstruct(content));
Person author = new Person(authorName, null, userName);
myEntry.getAuthors().add(author);

// Ask the service to insert the new entry
URL postUrl = new URL("http://www.blogger.com/feeds/" + blogID + "/posts/default");
return myService.insert(postUrl, myEntry);
}

public static void main(String[] args) throws Exception {
String blogId=1292929011;
String someName="test";

TestBlogger blog=new TestBlogger();
// System.out.println(blog.getBloggerEntryString());
// System.out.println(blog.getVideoFeedString());
String content="Some post test with the blogger Java api"+"
";
content=blog.getBloggerEntryString()+blog.getVideoFeedString();
blog.createPost(blog.getGoogleService(),
blogId,
"Testing Blogger API",
content,
"Marlon Pierce",
someName);
blog.writeToCalendar("Test Calendar Post",content);
}
}

Broken TeraGrid RSS News Feeds

2008-03-27T17:56:00.000-07:00

[Nagger's Update: the titles were fixed after I posted this. Hopefully reordering will be soon to follow.]

If you have a TeraGrid account, you probably have noticed the 6.02(10)^23 emails you get every day (Ranger is up! Ranger is down...). You can disable all of these by going to 'http://news.teragrid.org/subscribe-db.php. Even better, you can subscribe instead to RSS feeds and get these updates in a sensible fashion.

There are only two problems: the title fields in the feeds are broken and most of the feeds are in newest-entry-last order. The iGoogle RSS reader (for one) assumes that the first entry is the newest (not the oldest) entry.

Here is an example (to pick on the home team): http://news.teragrid.org/feeds/rss-indiana.rss I did notice that TACC had at least fixed their ordering, but most of the other centers have the same problem.

I reported these errors to the help desk like a good soldier, got my trouble ticket, and then...nothing. After a couple of weeks through some back channel emails, I found out that my ticket had been received and that they would fix it within the the year. I am not making this up. No one is willing to take five minutes to fix this simple problem. Probably they don't have any time left after all the meetings.

Testing Blogger API

2008-03-14T15:13:00.001-07:00

Two Nice Papers by Jim Grey
SSH Empty Passphrases
Password Protected RSS/Atom Feeds
Get Off My Cloud
Counting Unique IPs in Apache Log Files
XFN Test
BBQ Flickr Mashup
Emacs for C#/.NET/Mono
Running GOAP on PolarGrid Testbed
Polar Grid Matlab Note
More with Shindig and Open Social
Quick Start with Open Social, Shindig
A Little More JavaScript and FaceBook Hacking
Quick Facebook JavaScript and PHP API Notes
Portals and Web Applications
Polar Grid URL
Some Project Zero Thoughts
More Mac OS X Emacs
Birdbath and Globus RSL
Installing and Configuring Globus Services on Mac OS X
Condor-G, Birdbath and TransferOutputRemaps
File Retrieval with BirdBath and Condor-G
Condor-G Plus TeraGrid Example
Returning Standard Output in Condor-G
Globus Client+MyProxy+Pacman on FC7 Notes
Title: Scarface in 5 seconds
...

Two Nice Papers by Jim Grey

2008-03-12T13:55:00.000-07:00

Thanks to Shrideep for sending these:

G. Bell, J. Gray,  and A. Szalay, "Petascale Computational Systems:Balanced
CyberInfrastructure in a Data-Centric World," IEEE Computer, vol. 39, no 1,
pp. 110-112, Jan. 2006.
http://research.microsoft.com/~gray/papers/Petascale%20computational%20systems.pdf

J. Gray, "Distributed Computing Economics", Computer Systems Theory,
Technology, and Applications, A Tribute to Roger Needham, A. Herbert and K.
Sparck Jones eds., Springer, 2004,pp 93-101
ftp://ftp.research.microsoft.com/pub/tr/TR-2003-24.pdf

SSH Empty Passphrases

2008-02-29T13:11:00.001-08:00

Of course you should never set up empty passphrases on ssh keys, but if you do and it doesn't work, check the permissions on your .ssh directory and .ssh/authorized_keys file. These should not be readable or writable by anyone but the owner.

Password Protected RSS/Atom Feeds

2008-02-28T08:52:00.000-08:00

HTTP Authenticated feeds don't seem to work with iGoogle or Internet Explorer's feed reader, so I looked for a couple of alternatives. These work:

Sage: a FireFox plugin. Works fine on both Mac and Windows, as long as you have FireFox.
NewsGator: works great on Windows Vista

I also looked into UniveRSS, a 3D (!?) RSS reader for Vista that is based on Windows Presentation Foundation. Um, no thanks. Keep It Simple, Stupid.

Get Off My Cloud

2008-02-25T17:39:00.001-08:00

I'd like to put in a stake in the ground for anti-cloud computing. "Cloud computing" (i.e. Grid computing, only not gruesome) has a lot of buzz, and it is assumed that in the future we will all use computing and data storage as for-fee services virtually hosted on sites like Amazon. One brilliant guy compares it to the electrical power grid: http://www.nicholasgcarr.com/bigswitch/. Nicholas Carr, you owe Larry Smarr royalties.

In general, it is assumed that these online services will make the home computer become thinner and thinner until it no longer exists in its current form. In this vision, the next generation will be dominated by lightweight client devices (i.e. iPhones, Blackberries, Nokia N-series tablets, etc.), and most computing and data management will be performed by large online services. Existing service providing companies like Google will dominate, and Microsoft is desperately trying to get market share.

While this will happen, I don't think it is the whole story. The home PC isn't dead and (for many of us) will become a full fledged home appliance. Think of it as the long tail of cloud computing. Consider

There will be dozens of computing cores on relatively modest home computers within a decade (i.e. home computers will be mini parallel supercomputers).
Storage on these same machines will commonly be in the terabytes (already there on high end desktops).
High speed internet access will be available in most homes (probably already there for upper income households).

Actually, we are already there. The main thing that will change is the excessive parallel core horsepower that the next generation of PCs will have (and maybe we are there, too--check out this thing--72 cores on a desktop!). Major vendors are desperately trying to improve the software development environments to take advantage of these cores. Seems to me that Web servers are a natural fit for this.

So instead of everyone uploading photos and movies to Flickr and YouTube, why not provide "home editions" of these services that allow me to run my own web service? Or why not a "home edition" of Last.fm in which I can serve up and share my music collection with my friends and family? Ideally, I would also be able to federate my servers with friends and family into peer-networks and other such topologies so that I could share information transparently with friends while maintaining fine-grained control of the data at the same time. Obviously there is a lot of file and music sharing software already--I am thinking more of adding the social networking capabilities provided by various Web 2.o sites to this.

Another possible application for the home supercomputer is managing MyHouse@HOME (a hypothetical project). In Tom Igoe's book, Making Things Talk, he introduces the concept of object-oriented hardware (or, if you prefer, service-oriented hardware). Ubiquitous micro-computers and sensors combined with simple network protocols allow you to build interesting home applications, such as twitter-style blogs for your house. This could allow you, for example, to monitor your house while you are away or keep an eye on elderly family and friends. Event detection on this sort of sensor data is an interesting problem. This sort of sensor data obviously should go through a central Web server hub under your control.

More importantly and generally, with a home computing appliance, I can control my own personal data and make available as I choose instead of trusting large central repositories. Family medical records, genetic testing results, financial records, etc could be maintained locally and accessed by outside clients in ways that the owner controls.

One may object that there are many security problems. For example, the home computer becomes a bigger honey pot of valuable information than it is already, so it becomes a target for both internet break-ins and physical theft. And you would lose most of your records in a house fire that melted your computer. Hard drive failures would be a disaster. But all of these are already problems to some degree. Secure, encrypted hard drives are already available technology (in laptops for example), so just physically stealing the machine won't expose you to unnecessary risk.

Home denial of service attacks would be a new problem. I can also see the need for a secure "Swiss Bank Account" for data on your hard drive: every night you can rsync your data to a secure site, where it will be stored in numbered accounts in encrypted form via public/private keys and retrievable only by you (i.e. malicious employees don't know which account belongs to whom and can't read the data anyway). Probably this already exists in the financial world. Amazon's S3 and Micorsoft's SkyDrive are early examples, but security and reliability will need to be greatly enhanced.

The compelling thing to me is that none of this is particularly new. Context-aware sensors, parallel computing techniques and algorithms, network computing, social networks, and so on are old topics in the academic world. The key new concept is accessibility: do-it-yourself enthusiasts will broaden participation in these fields and will push researchers to deepen it. This will create the anti-cloud. Or if you prefer, everyone can have their own cloud.

What is missing seems to be the higher level building block software for building home cloud services (i.e. the home edition of Last.fm). Microsoft, in particular, seems to be missing the point. They are trying desperately to transform into a service provider (e.g. to compete with Google on search), but they are always chasing the leader (witness various Live projects, Virtual Earth, Skyserver. etc). Instead, why not concentrate on their core competency: home and business software, only now "cloud-enabling"? Note "cloud-enabling" is not "cloud-enabled". The former is server side (making use of the fat computing appliance in the home office). The latter is the thin client to some remote cloud service.

I'll conclude with a poke at the HPC and Grid world: yes, we were there first, just like the Vikings were the first Europeans in North America.

I note that the smart guys like Ray Ozzie at MS were way ahead of me with Live Mesh, which is debuting today (April 23).

Counting Unique IPs in Apache Log Files

2008-02-24T11:00:00.000-08:00

XFN Test

2008-02-22T12:04:00.000-08:00

Me: http://www.blogger.com/profile/11646814516542136925

Me: http://del.icio.us/marlon_pierce

Me: http://www.last.fm/user/marpierc/

Chuck: http://www.dr-chuck.com/csev-blog/

Marie: http://www.blogger.com/profile/01674561517458718071

Sangmi: http://www.blogger.com/profile/15257352896573946615

Romeo: http://www.blogger.com/profile/02475747864220945014

BBQ Flickr Mashup

2008-02-22T06:57:00.000-08:00

Thanks to Marie and Jun for constructing the best sort of Grid. I saw this and immediately asked if I could get one with BBQ. And it was made so.

I find it pleasing.

Emacs for C#/.NET/Mono

2008-02-21T07:08:00.000-08:00

Thought I'd give Mono a try with Mac OSX+Emacs. Of course Emacs won't have the nice formatting Lisp scripts for your code, but you can get them from here: http://www.cybercom.net/~zbrad/DotNet/Emacs/ sans instructions. The other link suggested by Mono (http://davh.dk/script/) was broken.

Here's how to get it to work:

Unzip the zip file from the link above.
Edit (or create) your $HOME/.emacs file and add the appropriate sections from zbrad.emac s. You could just add everything.
Add the line (add-to-list 'load-path "~/.emacs.d/site-lisp") at the beginning of your .emacs file.
Copy the "site-lisp" directory from the downloaded zip and there you go.

You should now be in business. I don't understand why Mr. Merrill didn't provide a README with his fine stuff. Gives emacs a bad name. Filling in these sorts of gaps made Tim O'Reilly famous.

Running GOAP on PolarGrid Testbed

2008-02-19T18:01:00.000-08:00

These notes assume you have the sample data and a snapshot of the code. These are unpacked in /home/polargrid/PolarGRID and /home/polargrid/code, respectively.

Configuring GOAP
These notes are Make sure that matlab is in your path:
export PATH=/usr/local/matlab/bin:$PATH
if necessary. Cd into the /home/polargrid/code directory and edit/check the ParamMCRDS_IU.m file. Make sure these parameters are OK:

param.in_path = '/home/polargrid/PolarGRID/raw/';
param.out_path = '/home/polargrid/PGOut/';
param.ref_path = '/home/polargrid/PolarGRID/150MHz_PD03_PD10/';
param.pos_path = '/home/polargrid/PolarGRID/';
param.pos_name = 'MCRDS_20070917_ALL_pos.mat';
param.prefix = 'data.20070917';

param.in_path should point to the .raw data files.
param.out_path can point anywhere you like, as long as the directory exists.
param.ref_path should point to the directory with the MCRDS_reference_* files. param.pos_path should point to the directory that contains the param.pos_name file.

You may also want to fool around with these parameters.

param.type = 'MCRDS';
param.file_idx_start = 25;
%param.file_idx_stop = 99;
param.file_idx_stop = 26;

These correspond to the data files *.0025.raw to *.0099.raw in /home/polargrid/PolarGRID/raw/. The above just runs the code with 2 input files (should take about 3 minutes) instead of 75.

I also had to use these parameters below.

param.sched_type = 'local';
%param.sched_type = 'jobmanager';
param.sched_name = 'localhost';
%param.sched_name = 'heimdall.cluster.cresis.ku.edu';

You can leave these as is for now. I also had to add the "exit" command to the end of the main script for force matlab to exit. Otherwise running the command script with the -r option (see below throws you back to the >> command prompt.

Running the Code
Do this from the /code/ directory. Run the command

matlab -nodisplay -nodesktop -r ParamMCRDS_IU

Don't use redirected standard inputs to read the matlab file (i.e. don't use "matlab < ParamMCRDS_IU.m") or you'll get

Warning: Type-ahead buffer overflow.
In distcomp.abstractjob.waitForState at 92
In StartScriptCluster at 28

See bottom of

https://engineering.purdue.edu/AAE/AboutUs/Help/wiki/RunningJobsOnTheServers

Also, the code won't create the final jpegs correctly this way. So...don't do that.

Where's the Data?
The output files go to the directory you specified above (PGOut). The final jpegs are placed in

PGOut/fk_data/

along with the header files (metadata, .txt) and .mat files.

Polar Grid Matlab Note

2008-02-19T10:03:00.002-08:00

If you get the error

[matlab@polargrid003 ~]$ /usr/local/matlab/bin/matlab
Warning: Unable to open display , MATLAB is starting without a display.
You will not be able to display graphics on the screen.
License checkout failed.
License Manager Error -95
MATLAB is unable to connect to the license server.
Make sure you can resolve the hostname of your machine.
If you are unable to resolve the hostname, contact your System Administrator.

Troubleshoot this issue by visiting:
http://www.mathworks.com/support/lme95a

Diagnostic Information:
Feature: MATLAB
License path: /usr/local/matlab/etc/license.dat:/usr/local/matlab/etc/*.lic:
FLEXnet Licensing error: -95,378. System Error: 115

This just means that the matlab license server isn't running. Start it as the matlab user with the command

/usr/local/matlab/etc/lmstart -l /tmp/eeee

More with Shindig and Open Social

2008-02-18T14:00:00.000-08:00

Get Going

After svn checkout, cd to $HOME/shindig/java/gadgets (or wherever you installed).
Run "mvn package" and then "mvn jetty:run-war" from this directory.
Point your browser at

http://localhost:8080/gadgets/files/container/sample1.html

or similar. Note this is an "aggregating client container" for collecting and displaying gadgets. The gadgets are loaded as XML from a specified remote URL and arranged using a layout manager. However, these containers don't know anything about your social networks--they are just for presentation and organization of content. If you actually want to host your own social gadgets you will need to deploy them into the "gadget hosting container"(e.g. Orkut or some other app). This is available on your Shindig server at

http://localhost:8080/gadgets/files/samplecontainer/samplecontainer.html
Note the aggregating client container just shows how to add gadgets from multiple providers. Any social networking information is NOT controlled or accessible by these containers. To run your own gadget and manipulate social network information (such as a user's profile and list of friends), you will need to run your gadget in a container.
If you want to add your own sample HTML file for aggregating gadgets, you need to put these in $HOME/shindig/javascript/container/. The maven package command will move these files into java/gadgets/target/gadgets/files/container/myjunk.html.

You'll then have to shutdown and restart jetty.

This is a little non-intuitive directory layout but see java/gadgets/pom.xml.
The $HOME/shindig/javascript/README is a good place to start.

Hosting your own gadgets.

Place your widget code in

$HOME/shindig/javascript/samplecontainer

or any subdirectory (like /examples/).
After compilation, these will be located in

./target/gadgets/files/samplecontainer/

or any subdirectory (/examples/ in the shindig SVN).
These will be served up from

http://localhost:8080/gadgets/files/samplecontainer/examples/myapp.xml
You can use the XML from the sample widget at

http://code.google.com/apis/opensocial/docs/0.6/devguide.html
Just download or cut and past this XML file onto your local file system.

For some reason, you will need to change the top:

<ModulePrefs title="Title of Your Application">
<Require feature="opensocial-0.6"/>
</ModulePrefs>

to
<ModulePrefs title="People API Howto">
<Require feature="opensocial-samplecontainer"></Require>
</ModulePrefs>

This seems to have been corrected in newer versions. Check the header of the sample gadget and use this as a guideline for your gadget.
Anyway, load the samplecontainer URL (http://localhost:8080/gadgets/files/samplecontainer/examples/myapp.xml) in your browser, and then load your gadget by pasting in the URL

http://localhost:8080/gadgets/files/samplecontainer/examples/myapp.xml
If you edit your gadget in any way, you must (apparently) restart your Jetty server.

This gadget is now ready to be displayed in any gadget container.

Putting Your Gadget Into Your Display Container
In the last previous two steps, you a) displayed existing gadgets using the gadget display container and b) set up your own gadget hosting container. The documentation doesn't very clearly distinguish these, but you should see this by now. The display container can load any gadgets (including production gadgets available from iGoogle). The gadget hosting container, on the other hand, is used to run your gadgets with your social network data.

We can now combine the two. Edit any of the sample HTML files in java/target/gadgets/files/container, changing the URL to point to the gadget you made in the previous example. Restart your Jetty server and you should see it correctly displayed.

Similarly, you should be able (if your Jetty server's URL is publicly accessible) to add your social gadget to other display containers.

Quick Start with Open Social, Shindig

2008-02-17T11:44:00.002-08:00

OpenSocial is of course the Google-led consortium of competitors to FaceBook, but until very recently most of its efforts appeared to be vaporware. This is starting to change.

One of my immediate concerns about the OS API was how easy it would be to build an OS compliant container. Otherwise, we would have to rely upon existing containers (Orkut, anyone?) to host our OS-based applications. Fortunately, the Apache Shindig incubator project (http://incubator.apache.org/shindig/) is something of an open source reference implementation for the OS API (written in Java with a Maven build no less).

Check it out with SVN and build from source with Maven (worked perfectly on version 628525). Run with Jetty server.

Point browser to http://localhost:8080/gadgets/files/container/sample1.html and look through other samples. You can also load these HTML files using file:// if you don't want to run the Jetty server.

Next time: can I use OS clients to query my LinkedIn profile?

A Little More JavaScript and FaceBook Hacking

2008-02-15T15:24:00.000-08:00

Facebook's JavaScript example is pretty minimal, so I thought I'd do a little more hacking around. I'm not a great JS programmer, so I like to unroll the function definitions.

You should grab the very minimal JS API documentation from here: http://wiki.developers.facebook.com/index.php/JavaScript_Client_Library. It is the "zipped archive" link near the bottom. This guide is very unfriendly at the current time, but in general you should see that the JS equivalents of the API list at http://wiki.developers.facebook.com/index.php/API replace the "." with a "_". So users.getInfo becomes users_getInfo(...).

Here's an example that uses the users_getInfo() method and some other stuff to get a user's profile information.


<textarea style="width:500px;height:300px;" id="_traceTextBox">
</textarea>
<script src="http://static.ak.facebook.com/js/api_lib/FacebookApi.debug.js" type="text/javascript">
</script>
<script type="text/javascript">

// Create an ApiClient object, passing app's api key and
// a site relative url to xd_receiver.htm
var api = new FB.ApiClient('<your_key_here>', '/xd_receiver.htm', null);

// Get friends list
function getResults(result,exception) {
Debug.dump(api.get_session().uid,'you');
Debug.dump(result,'The stuff');
}

function getInfo(result,exception) {
Debug.dump(result,'The info');
}

// require user to login
api.requireLogin(function(exception) {
api.friends_get(null,getResults);
var myinfo='last_name,first_name,hometown_location,work_history,pic_small';
api.users_getInfo(api.get_session().uid,myinfo,getInfo);

});
</script>

You need to update this stuff above to use your application's key, as indicated.

The output of this will be something like
---------------------
you: 627774031

The stuff: {Array}
[Deleted]

The info: {Array}
[0]: {Object}
first_name: Marlon
hometown_location: {Object}
city:
state:
country:
zip:
last_name: Pierce
pic_small: http://profile.ak.facebook.com/profile6/1797/98/t627774031_2463.jpg
uid: 627774031
work_history: {Object}

friendsResult from batch execution: {Array}
[Deleted]
notificationsResults from batch: undefined
----------------------

I guessed the values for the myinfo array by looking at the big XML file example shown at http://wiki.developers.facebook.com/index.php/Users.getInfo. That's the real URL for the tiny version of my FaceBook photo.

So in summary, you can use this approach to embed FaceBook profile information in your own web application. You'll of course have to do a bit of work to format all of this stuff. Note also that you can substitute another user's numeric ID in the users_getInfo() call (ie that of one of your friends). I used this to spy on Dave De Roure, who spends a lot more time on FaceBook than I do. And I won't even start on Dan Katz.

FQL and JS
Finally, note that all of the FaceBook API is basically a set of wrappers around their SQL like FQL query language. So you can, if you prefer, make custom methods out of FQL query strings. For example

var myQuery='SELECT name FROM user WHERE uid='+api.get_session().uid;
api.fql_query(myQuery,getFQLResponse);

will execute the indicated query string from JavaScript and pass it to your (developer defined) function getFQLResponse().

Groups
To list information about groups you are a member of, use the following. You can replace your uid with another UID.
api.groups_get(api.get_session().uid,null,getGroups);
api.groups_getMembers(18629081888,getGroupMembers);

In the second line, 18629081888 is the GID for the very inactive OGF Web 2.0 group.

Quick Facebook JavaScript and PHP API Notes

2008-02-14T13:41:00.000-08:00

JavaScript
Took a quick look at http://wiki.developers.facebook.com/index.php/JavaScript_Client_Library and found a few obfuscations. To get started, you need to do these things (described a little too briefly in the link above).

Register for an application key at http://indiana.facebook.com/developers/editapp.php?new (Don't hit enter yet)
Cllick the "Optional Fields" drop down and

Provide a callback URL (that is, the full URL of your HTML page containing the Facebook JavaScript application).
Choose "IFrame" as the canvas page URL.

When it is working, this little app just prints out your friend's Facebook IDs (numbers) onto the screen. When you load your page (http://some.host.name/mytest/index.html), you will be redirected to Facebook to login and then will go back to your URL.

PHP
Also took a quick tour of the PHP examples ("footprints" and some hello-world type stuff). Looks like the instructions for setting up the JS example assumed you were familiar with doing things the PHP way.

For footprints, you will need a web server (apache), PHP, and MySQL. Get, install, and start these in the usual ways. Hints: for Macs, get the nice disk image from MySQL. Also, read the PHP installation instructions before installing Apache, since PHP will require some non-default configuration options (particularly DSO support).

The web server provides the callback URL as well ("http://your.host.name/footprints"), which will need to be on a world accessible Apache (or other) server. Dump your application in htdocs/footprints and make sure paths are correct (index.php needs to point correctly to facebook.php). Note also that apps.facebook.com/footprints will be taken, so make a different name. The facebook app name does not have to match your callback app's name.

Create the MySQL database. The footprints PHP scripts don't do this for you.

Edit config.php. You will need the root password for your db.
From the command line, log into your db with /usr/local/mysql/bin/mysql -u root -p and provide the root password at the prompt. You will then need to create and switch to the footprints database.
mysql> CREATE DATABASE footprints;
mysql> USE footprints
mysql>CREATE TABLE `footprints` ( `from` int(11) NOT NULL default '0', `to` int(11) NOT NULL default '0', `time` int(11) NOT NULL default '0', KEY `from` (`from`), KEY `to` (`to`) );

Step 5 is just a cut and paste of the config.php file from footprints, but note that they omitted the final ";" on the statement.

When it is working, this little application allows you to step on your friends. It gives you a little text area for typing names that will be autocompleted as you type. Note this must run inside Facebook, unlike the previous example.

When setting up your PHP app, you'll notice you can choose between FBML and IFrames. FBML will (as you might have guessed) a nice FaceBook styled application and some extra goodies. The IFrame option will produce a pretty bare bones footprint application, but obviously you have more power to customize the styles.

Hosting Your Services
Here are some instructions on using Amazon Web Services to host your application.

http://developer.amazonwebservices.com/connect/entry.jspa?entryID=1044

Portals and Web Applications

2008-02-12T19:04:00.000-08:00

There are many tools for building web applications and portals, so it is useful try to classify these a bit. The field also suffers from overlapping terminology ("framework" providing the worst example) and overlapping functionality. I'll take an informal stab at trying to organize these things.

Programming Tools and Languages: this is the simplest group (hopefully) to define. PHP, PERL/CGI, Java Server Pages and Servlets are all examples. One could argue that JSP is more of a development framework (below), but these aren't sharp definitions. I'll err on the side of usage--most people don't write JSP using beautiful MVC architected code.

Goodies and Widgets: these are useful little or medium-sized extensions that can be easily plugged into several different development environments. PEAR/PHP and Java Tag Libraries seem to be best examples on the server side. Various JavaScript libraries (such as YUI) are good examples of client-side applications. These can be embedded into several different toolkits and frameworks. These are distinct from development framework modules (i.e. Ruby on Rails modules) since they are meant to work with both Programming Tools (above) and more specific, derived Development Frameworks (below). Thus one can put useful tags into JSP as well as JSF.

Development Frameworks: Frameworks are typically one level of abstraction above the common programming language tools. Java is full of these: Struts, JSF, Velocity, etc. Ruby on Rails is another famous example. DWR, Google Web Toolkit, and other tools also fall in here. ASP.NET and various compatible, open source implementations like Mono go here.

Frameworks typically encode a development methodology (MVC) and provide an extensibility framework so that all developers create code more or less in the same way. Frameworks thus allow you to develop reusable and useful little modules that can be shared with other developers using the same framework.

A framework can be used to build standalone applications, but it can also be used to develop a rich, self contained component that can be deployed into a container. In contrast, a container (after installation) comes up immediately with useful tools such as login modules, layout managers, and some user tools (say, a calendar or a blog authoring tool). In this example, one may use JSF (a framework) to build a web calendar or a web email application, but one would also typically want to embed these into a more comprehensive container (a portlet container in the Java world).

Application Containers: Joomla (php), Zope (python), Mambo, and Drupal (php) are examples. Java JSR 168 containers and various similar efforts also fit here: GridSphere, LifeRay, uPortal, Jetspeed2, etc. Sakai is another example--it supports the Java standard but also has its own component model. Interestingly, the non-Java containers mentioned above all began life as content management systems.

Arguably only Java's portlet standard really takes the container philosophy completely seriously. Joomla et al, really don't care about portability of codes across container vendors, although they do obviously care about shareable community developed modules. All Java portlet container providers (except Pluto, the reference implementation) don't just give you a container for managing your own portal components--they also give you lots of built in functionality and other goodies (calendars, chat tools, document managers, etc). LifeRay and Sakai are good examples of this.

Related Things: Web-based containers for content and document management, blogging, wikis, and learning management are related closely to portals. There are many different content management systems. Learning management varies from Moodle to Blackboard. Blogging is often provided as an online service (such as Google's Blogspot), but you can also install and run your own blogging service. WordPress is a prominent example of blogging software. Many frameworks (such as Drupal) come with blogging modules. Wikis are similar to content management and document management systems. The best known wikis (MediaWiki and Twiki) are standalone tools, but many containers come with Wiki modules (Sakai for example).

Services for Portals: One of the dangers of portal tools at any level is that they can become monolithic. The danger here is that you will be stuck with inflexible technology and can't respond to changing requirements. The current Enterprise vs. Web 2.0 conflict provides a compelling example. Many famous Web 2.0 sites (flickr, youtube, etc) are Web Services as much as they are Web sites, so modern portals must be able to easily integrate (rather than compete) with these large third party services.

External services: there is a great temptation of course to bundle everything into a particular portal for simplicity of installation and maintenance. However, this is not a good plan in the long term since you will (for example) tie your managed content to a particular tool rather than a portable standard.

When choosing a framework or container, a good starting point in the evaluation process is to see if it supports clients to the following standard services.

Authentication and Authorization: these are often tied to LDAP, CAS, or (in the Grid world) MyProxy. Shibboleth is an authorization mechanism. The basic idea is that the portal needs to go to an external database to get the login information. This DB is independent of the portal and may have other clients.
RSS/Atom feeds are particularly important in Web 2.0 applications, since most of them publish syndication in some form or another. Many containers provide RSS readers. This is any easy way to incorporate external content (say, YouTube videos) into a portal.This is particularly important topic in modern education portals, as for example one may embed publication feeds from Connotea and CiteULike, slides from SlideShare or ImageLooop, SciVee or YouTube for scientific visualizations and online lectures, etc. As can be seen from the previous list, there are dedicated, high quality online services for many education and science portal content that can be incorporated into a portal pretty generically as RSS/Atom. It is a bad idea to reinvent this stuff or tie it too closely to your portal framework.

We may also turn this problem upside down: if you run a resource management portal,
you must be able to export that portal's content to other user interfaces such as iGoogle
and NetVibes that users are more likely to use every day.

Calendars: Collaborative calendars are obviously best extracted from portals since there are so many calendar clients. This is also an example of a reversible service: you typically want to synch your online calendar with your local PC or mobile device. A relevant standard is iCal.
WebDAV servers for content management. WebDAV is an extension of HTTP and is supported by many Web servers (with the right module, of course).
Grid and workflow services for running scientific applications and accessing data.
Digital libraries: access to Fedora, etc. Don't reinvent this or jackbooted librarians will be goose stepping into your cubicle.

It is of course inadequate to simply support such services through clients, as the implementation must be useful and enjoyable. On the downside, however, it is also hard to know when one has crossed the line that separates a rich, useful client from a full fledged internal service that can't be easily extracted from the framework. I certainly can think of several examples that I've written.

Social Networking: the other challenge facing education portals is how much to integrate with social networking sites such as FaceBook or LinkedIn.

Polar Grid URL

2008-02-12T13:59:00.000-08:00

Using the power of my blog to shift google page ranks, I hereby declare that PolarGrid (aka Polar Grid) has the following official URL: http://www.polargrid.org.

Damn polar coordinate system.

Some Project Zero Thoughts

2008-01-20T11:29:00.001-08:00

IBM's little Project Zero is their effort to provide a Java-based Web 2.0 framework/toolkit/whatever for building your own applications. Several things look promising, such as the list of core Web 2.0 services (commenting, tagging, rating, etc) at http://www.projectzero.org/wiki/bin/view/Documentation/CoreGettingStartedZeroZone.

But looking at it raises more concerns and confusions. I could just be wrong, but here are some criticisms.

They seem to be providing their own Web server container, which is a really bad idea. How do I port this to some other Web Server like Tomcat or Jetty?
I managed to stumble on a bug in their simple "to-do" list application--only the last item in the list gets deleted properly.
They rely on Groovy as their Java Web framework (you can also develop in PHP). Groovy may be nice, and it does sound sort of like "Ruby", but there are also a lot of JSP developers out there. Why ignore them? Having seen a lot of Java ideas come and go, I am really reluctant to embrace anything but the lowest common denominator technology from them.
The packaging of the tutorials is nice and easy to get going (looks like maven or ant under the hood of some scripts), but I have to critique them a bit on their documentation (or maybe I didn't read carefully enough) here: it was not clear until I dug a little more that the methods implemented in
todos.groovy represent the universal Project Zero API for REST services: all services implement onList(), onCreate(), etc, and these are bound to various HTTP Methods (onList <--> GET). See http://www.projectzero.org/wiki/bin/view/Documentation/CoreDevelopersGuideREST.I think this is reasonable (after, REST is essentially a universal API, the equivalent of a one size fits all WSDL). I just wish the nice tables in the link above were shown in the tutorial.

More Mac OS X Emacs

2008-01-20T08:29:00.000-08:00

There is a nice list of emacs for Mac OS X here: http://www.emacswiki.org/cgi-bin/wiki/EmacsForMacOS.

I've been using the build from Porkrind (as I posted some time ago), but I have recently switched to EmacsApp. EmacsApp features much nicer integration with the Mac GUI widgets and supports both the classic Emacs keyborad shortcuts as well as standard Mac shortcuts (that is, paste can be either cntrl-y or apple-v). Moving text between Emacs and other Mac applications is much better implemented.

Birdbath and Globus RSL

2008-01-18T17:15:00.000-08:00

The condor_submit globusrsl parameter's "real" name is GlobusRSL (use condor_q -l to see this). Thus to specific in your Birdbath Java client an RSL parameter such as maxWallTime, use

new ClassAdStructAttr("GlobusRSL",
ClassAdAttrType.value2,
"\"(maxWallTime=120)(maxTime=120)\"")

Installing and Configuring Globus Services on Mac OS X

2007-12-31T14:09:00.000-08:00

* Download Mac binaries from Globus site. Probably the VDT will work fine also.

* Do the usual configure/make/make install business.

* Unfortunately, all Macs that I tried are missing Perl XML parsers and so will fail at the "make install" step. For some reason, Globus's ./configure isn't set up to catch this. Not sure why. I found the following nice instructions for installing Perl::XML and other stuff at

http://ripary.com/bundle-xml.html

This will take a while.

* Set up Simple CA. See

http://vdt.cs.wisc.edu/releases/1.8.1/installation_post_server.html#simpleca

and

http://www-unix.globus.org/toolkit/docs/4.0/admin/docbook/ch07.html#s-simpleca-admin-installing.

The VDT page above actually points to some older (v 3.2) instructions, but these are still OK also.

* Make your host certificate and sign it. I did all of this as root.

$GLOBUS_LOCATION/bin/grid-cert-request -host `hostname`

(note the backticks around hostname).

cd /etc/grid-security/
~/Globus-Services/bin/grid-ca-sign -in hostcert_request.pem -out hostsigned.pem

* Create your xinetd services.

cd /etc/inetd.d/
touch gsigatekeeper
touch gsiftp

* Here is my gsigatekeeper:
service gsigatekeeper
{
socket_type = stream
protocol = tcp
wait = no
user = root
env = LD_LIBRARY_PATH=/Users/mpierce/Globus-Services/lib
env = DYLD_LIBRARY_PATH=/Users/mpierce/Globus-Services/lib
server = /Users/mpierce/Globus-Services/sbin/globus-gatekeeper
server_args = -conf /Users/mpierce/Globus-Services/etc/globus-gatekeeper.conf
disable = no
}

And my gsiftp:
service gsiftp
{
instances = 100
socket_type = stream
wait = no
user = root
env += GLOBUS_LOCATION=/Users/mpierce/Globus-Services
env += LD_LIBRARY_PATH=/Users/mpierce/Globus-Services/lib
env += DYLD_LIBRARY_PATH=/Users/mpierce/Globus-Services/lib
server = /Users/mpierce/Globus-Services/sbin/globus-gridftp-server
server_args = -i
log_on_success += DURATION
nice = 10
disable = no
}

Note the LD_LIBRARY_PATH is not useful for Macs--you need DYLD_LIBRARY_PATH instead (see below). But I left it in anyway--you will need this for Linux installations.

* Start your services:

service gsiftp start
service gsigatekeeper start

* You may want to add these also to /etc/services
tail /etc/services
# Carstein Seeberg
# 48004-48555 Unassigned
com-bardac-dw 48556/udp # com-bardac-dw
com-bardac-dw 48556/tcp # com-bardac-dw
# Nicholas J Howes
# 48557-49150 Unassigned
# 49151 IANA Reserved
#gsiftp 2811/tcp
#gsigatekeeper 2119/tcp

* Check these with telnet:
telnet localhost 2811
telnet localhost 2119

* Note you must use DYLD_LIBRARY_PATH on the Mac or else the service will not start even though the "service" commands above will not complain. You will get errors like this if you telnet to the ports:
/Users/condor/execute/dir_8492/userdir/install/lib/libglobus_gss_assist_gcc32.0.dylib

* Requisite Globus Complaint: I had to do all the above configuration stuff by hand. Why not have a configuration post-installation "flavor" called "my first globus installation" that does all of this for you?

* Create a grid-mapfile and some usercerts, or just use your favorite grid-mapfile from some place else.

Condor-G, Birdbath and TransferOutputRemaps

2007-12-17T18:45:00.000-08:00

Here is the classad attribute structure you need for transferring your output back from the remote globus machine to specific files on your condor-g job broker/universal client. By default, the output goes to your cluster's spool directory. The example below shows how to move these to /tmp/ on the condor host.

Note that TransferOutput files are separated by commas, and TransferOutputRemaps files are separated by semicolons.

new ClassAdStructAttr("TransferOutput",
ClassAdAttrType.value2,
"\"testgeoupdate.index,testgeoupdate.node,testgeoupdate.tetra\""),

new ClassAdStructAttr("TransferOutputRemaps",
ClassAdAttrType.value2,"\"testgeoupdate.index=/tmp/testgeoupdate.index;
testgeoupdate.node=/tmp/testgeoupdate.node;testgeoupdate.tetra=/tmp/testgeoupdate.tetra\""),

...

The trick as usual is to get the condor true-names for the parameters you want to use. These are typically NOT the familiar names from condor_submit. Always create a command script for what you want to do before writing your birdbath api code, and then use condor_q -l to see the internal names that condor actually uses. These are the ones you will need to use in your classAdStructAttr array.

File Retrieval with BirdBath and Condor-G

2007-12-15T17:30:00.000-08:00

As I wrote earlier, output files (other than standard out) of Globus jobs submitted via Condor-G can be retrieved using the TransferOutput attribute. To reproduce this with the BirdBath Java API, you need an incantation something like below.

ClassAdStructAttr[] extraAttributes =
{
new ClassAdStructAttr("GridResource", ClassAdAttrType.value3,
gridResourceVal),
new ClassAdStructAttr("Out", ClassAdAttrType.value3,
outputFile),
new ClassAdStructAttr("UserLog", ClassAdAttrType.value3,
logFile),
new ClassAdStructAttr("Err", ClassAdAttrType.value3,
errFile),
new ClassAdStructAttr("TransferExecutable",
ClassAdAttrType.value4,
"FALSE"),
new ClassAdStructAttr("when_to_transfer_output",
ClassAdAttrType.value2,
"\"ON_EXIT\""),
new ClassAdStructAttr("should_transfer_files",
ClassAdAttrType.value2,
"\"YES\""),
new ClassAdStructAttr("StreamOut",
ClassAdAttrType.value4,
"FALSE"),
new ClassAdStructAttr("StreamErr",
ClassAdAttrType.value4,
"FALSE"),
new ClassAdStructAttr("TransferOutput",
ClassAdAttrType.value2,
"\"testgeoupdate.index,testgeoupdate.node,testgeoupdate.tetra\""),
new ClassAdStructAttr("Environment",ClassAdAttrType.value2,
"\"PATH=/home/teragrid/tg459247/geofest.binaryexec:/bin:/usr/bin\""),
new ClassAdStructAttr("x509userproxy",
ClassAdAttrType.value3,
proxyLocation)

The key problem with BirdBath is that it seems to require the parameter "true-names" rather than their values documented in the commandline condor_submit's man pages. For example, to specify the file to use for logging, you have to use the parameter "UserLog", not "log" or "Log". To see the actual "true-names" used by Condor internally, submit a job with condor_submit and then use condor_q -l. It will be in the job_queue.log file in the the spool directory as well.

With birdbath+condor-g there is a similar problem returning files from the remote globus server. You have to use "TransferOutput" as one of the attributes. Also I noticed birdbath did not set StreamOut and StreamErr, so I manually set these as well.

The output files (testgeoupdate.index, etc) will be uploaded to your job's spool directory. Note birdbath does not clean this up, unlike the condor_submit invocations (which delete your job's clusterxy.proc0.subproc0 directory on completion). To make birdbath mimic this behavior, you can set LeaveJobInQueue=TRUE in your classadstructattrs above.

Condor-G Plus TeraGrid Example

2007-12-13T14:06:00.000-08:00

Below is an example condor command script for running a job from my local grid client (a PC running FC7, globus, condor). I'm running a set of codes that I installed on the machine (lonestar) that I wanted to use. These codes are run by a perl script, so I had to set the PATH. I want to upload the input files from my desktop at submission time--the paths here are for my client machine. Note Condor-G will put these in $SCRATCH_DIRECTORY on lonestar, which doubles as your working directory (that is, autoref.pl will be executed here). To get the files back from lonestar to the PC, I used "transfer_output_files" and listed each file. Full paths for these aren't necessary. Condor will pull them back from $SCRATCH_DIRECTORY on the remote machine to your local directory.

# Here it is. Please gaze at it only through a welder's mask.
executable = /home/teragrid/myaccnt/geofest.binaryexec/autoref.pl
arguments = testgeoupdate rare
transfer_executable = false
should_transfer_files=yes
when_to_transfer_output=ON_EXIT
transfer_input_files =/home/mpierce/condor_test/Northridge2.flt,/home/mpierce/condor_test/Northr
idge2.params,/home/mpierce/condor_test/Northridge2.sld,/home/mpierce/condor_test/NorthridgeAreaMan
tle.materials,/home/mpierce/condor_test/NorthridgeAreaMantle.sld,/home/mpierce/condor_test/Northri
dgeAreaMidCrust.materials,/home/mpierce/condor_test/NorthridgeAreaMidCrust.sld,/home/mpierce/condo
r_test/NorthridgeAreaUpper.materials,/home/mpierce/condor_test/NorthridgeAreaUpper.sld,/home/mpier
ce/condor_test/testgeoupdate.grp
transfer_output_files=testgeoupdate.index,testgeoupdate.node,testgeoupdate.tetra
universe = grid
grid_resource = gt2 tg-login.tacc.teragrid.org/jobmanager-fork
output = test.out.$(Cluster)
log = test.log.$(Cluster)
environment = PATH=/home/teragrid/myaccnt/geofest.binaryexec/:/bin/:/usr/bin
queue

Returning Standard Output in Condor-G

2007-12-13T08:27:00.000-08:00

[The problems have been hard to reproduce, so none of this may be necessary. But it doesn't hurt.]

Here's a nugget of wisdom: start Condor-G as root and set CONDOR_IDS=(uid).(gid)(that is, see /etc/passwd and /etc/group for the values, should be something like 501.501) if you want to run as a user other than "condor".

I had the following problem: condor_submit worked correctly but standard output went to /dev/null. Checking the logs and spool, I saw that TransferOut was also FALSE and couldn't override this.

I solved this by restarting condor as root (it resets the process id to the CONDOR_IDS user). It also seems to work if you run condor as the correct user (that is, the one specified by CONDOR_IDS).

Globus Client+MyProxy+Pacman on FC7 Notes

2007-12-12T08:53:00.000-08:00

Here are my "hold your mouth right" instructions. Deviate at your own risk.

* Get pacman, untar, cd, and run setup.sh.

* Make an installation directory (like mkdir $HOME/vdt) and cd into it. Don't run Pacman in $HOME--it copies everything in the current working directory into post-install, which could be awful if you run from $HOME.

* Fedora Core 7 is not supported by the VDT, so do the pacman pretend:

cd $HOME/vdt (if not there already)
pacman -pretend-platform Fedora-4

* Install Globus clients (you are still in $HOME/vdt):
pacman -get http://vdt.cs.wisc.edu/vdt_181_cache:Globus-Client

Answer no to installing VDT certs. We will get TG certs in a minute, and the VDT certs seem to have some problems.

* Install MyProxy (still in $HOME/vdt):
pacman -get http://vdt.cs.wisc.edu/vdt_181_cache:MyProxy

* Install GSIOpenSSH (still in $HOME/vdt):
pacman -get http://vdt.cs.wisc.edu/vdt_181_cache:GSIOpenSSH

* Set your $GLOBUS_LOCATION and source $GLOBUS_LOCATION/etc/globus-user-env.sh.

* Get the TG certificates:
* wget http://security.teragrid.org/docs/teragrid-certs.tar.gz
Unpack these in $HOME/.globus/certificates/

* I didn't like the Condor installation I got from Pacman. This should be done for now with the old condor-configure command on a direct download.

* Also make sure pacman didn't set $X509_CERT_DIR. This had some self-signed certs that caused myproxy-logon to myproxy.teragrid.org to fail and overrode my .globus and /etc/grid-security certificate locations.

MyProxy, Globus Clients on Mac OSX: Use the VDT

2007-12-10T13:58:00.000-08:00

[NOTE: Read the comments. Mac binaries are available from www.globus.org; missing links were added.]

I spent a couple of hours trying to compile Globus source on my Mac OS X, only to have it fail with some mysterious error. Some things never change. Why don't they provide a pre-built Mac OS X binary? Here is the error that I get:

make: *** [globus_rls_server-thr] Error 2

Luckily, the VDT seems to do things the right way. I used Pacman to install globus clients (the command line tools only) and MyProxy. See the VDT documentation for instructions. Why doesn't Globus do this? Unbelievable. Anyway, here are the Pacman commands that I used:

pacman -get http://vdt.cs.wisc.edu/vdt_181_cache:Globus-Client

and

pacman -get http://vdt.cs.wisc.edu/vdt_181_cache:MyProxy

I answered "no" for installing certificates--I have the ones I needed, and answering "yes" caused a failure, probably because of write permission problems.

Set $GLOBUS_LOCATION and source $GLOBUS_LOCATION/etc/globus-user-env.sh as usual.

Condor Birdbath Compromises

2007-12-06T08:18:00.000-08:00

I have not been able to find the right incantation for using the "initialdir" attribute with the Birdbath web service. I'm trying to do the equivalent of the following submission script:

universe = vanilla
executable = /path/to/my/bin/a.out
output = theOutput
error = theError
log = theLog
arguments = 10 21
environment = "PATH=/path/to/my/bin/:/bin/:/usr/bin"
initialdir = /path/to/my/data
should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT
queue

Looking at spool/job.queue.log, you will see that this attribute's proper name is "Iwd". However, the classad structure (using Birdbath java helper classes) like below didn't work:

//Create a classad for the job.
ClassAdStructAttr[] extraAttributes =
{
new ClassAdStructAttr("Out", ClassAdAttrType.value3,
outputFile),
new ClassAdStructAttr("Err", ClassAdAttrType.value3,
errFile),
new ClassAdStructAttr("Log", ClassAdAttrType.value3,
logFile),
new ClassAdStructAttr("Environment", ClassAdAttrType.value2,
"\"PATH=/path/to/my/bin/:/bin:/usr/bin\""),
new ClassAdStructAttr("ShouldTransferFiles", ClassAdAttrType.value2,
"\"IF_NEEDED\""),
new ClassAdStructAttr("WhenToTransferFiles", ClassAdAttrType.value2,
"ON_EXIT"),
new ClassAdStructAttr("Iwd", ClassAdAttrType.value3,
"/path/to/my/data/")
};

The Iwd parameter is set correctly in the logs, but the executables can't find the input files in the initial directory correctly. My workaround solution was to uploading all the data files to the

File[] files={ new File("/local/data/path/file1.dat"),
new File("/local/data/path/file1.dat") };

And then run the usual way, except specify the files

xact.submit(clusterId, jobId, userName, universeType,
executable,arguments,"(TRUE)", extraAttributes, files);
xact.commit();
schedd.requestReschedule();

This will put the data (input and output) in condor's spool directory.

Running GeoFEST with Condor

2007-12-04T18:37:00.001-08:00

This is a prelude to using Condor-G. I used the following incantation:

-bash-3.00$ more meshgen.cmd
universe = vanilla
executable = /globalhome/gateway/geofest.binaryexec/autoref.pl
output = autoref.out
error = autoref.err
log = autoref.log
arguments = /globalhome/gateway/condor_test/testgeoupdate rare
environment = "PATH=/globalhome/gateway/geofest.binaryexec/"
#getenv = true
initialdir = /globalhome/gateway/condor_test
should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT
queue

The key thing here was the "environment" attribute, since the autoref.pl script is of course a perl script that calls other executables. These executables need to be in the PATH.

You can also do this by setting the PATH environment variable in your shell and then using the getenv attribute. This will work OK for the command line but will not be much use in a web service version. OK, you could get it to work, but this is the kind of thing that will break 6 months later when you move to a new machine.

None of the condor examples I could find discuss this, but it is well documented in the condor_submit manual page.

Compiling RDAHMM Correctly

2007-12-03T15:10:00.000-08:00

The correct value for the LDLIBS variable is

LDLIBS = -lda -lnr -lut -lcp /usr/lib/liblapack.a -lblas -lm

CIMA Portal Link

2007-11-30T10:17:00.000-08:00

Here's a link to the CIMA Crytallography Portal: http://cimaportal.indiana.edu:8080/gridsphere/gridsphere. This was hard for me to find with Google for some reason. The keyword combination "cima portal" did not work well, but it was the top hit if I googled "cimaportal".

Condor, Dagman for Running VLAB Jobs

2007-11-17T09:13:00.000-08:00

Here's how to run PWSCF codes with Condor. Assume you have the executables and that the input data comes to you in a .zip file. We need to do these steps:

Clean up any old data
Unpack the .zip
Run PWSCF's executables.

As always, see my del.icio.us "condor" tags (right side of blog) for useful links.

For each step, we need to write a simple Condor .cmd file. Condor can do parameter sweeps but you must use DAGMan to run jobs with dependencies. We will start with a data file called __CC5f_7.zip.

Here is clean_pwscf.cmd:

universe=vanilla
executable = /bin/rm
arguments= -rf __CC5f_7
output = clean.out
error = clean.err
log = clean.log
queue

Now unpack_pwscf.cmd:

universe = vanilla
executable = /usr/bin/unzip
arguments = __CC5f_7.zip
output = unpack.out
error = unpack.err
log = unpack.log
queue

Finally, run_pwscf.cmd:

universe=vanilla
executable = pw.x
input = Pwscf_Input
output = pw_condor.out
error = pw_condor.err
log = pw_condor.log
initialdir= __CC5f_7
queue

All of these are run in the same directory that contains the pw.x executable. The only interesting part to any of these scripts is the initialdir directive in the last one. This specifies that the script is executed in the newly unpacked __CC5f_7 directory, which contains the Pwscf_Input file and will be the location of the .out, .err, and .log files.

As with most scientific codes, PWSCF creates more than one output file. In this case they are located in the __CC5f_7/tmp directory. Presumably these would be preserved and copied back if I was running this on a cluster and not just one machine, although I may need to include the directives

should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT

Finally, we need to create our DAG. Here is the content of pwscf.dag:

Job Clean clean_pwscf.cmd
Job Unpack unpack_pwscf.cmd
Job Run run_pwscf.cmd

PARENT Clean CHILD Unpack
PARENT Unpack CHILD Run

The "job" portion associates each script with a nickname. The PARENT portion than defines the DAG dependencies. Submit this with
condor_submit_dag -f pwscf.dag

The -f option forces condor to overwrite any preexisting log and other files associated with pwscf.dag.

Running Condor on the Airplane

2007-11-10T14:16:00.001-08:00

Or anywhere without internet connection.

Install with condor_install as best you can.
Use 127.0.0.1 for your condor host.
Edit the annoying HOSTALLOW_WRITE to change the broken default. Why can't I set this during the installation phase?
Uncomment DEFAULT_DOMAIN_NAME. The value doesn't matter.
Uncomment NO_DNS=TRUE.
And don't forget to replace $(UWCS...) with $(TESTINGMODE...) in Part 3.

Using Maven 1 Repositories with Maven 2

2007-11-05T11:43:00.000-08:00

The wonderful Maven2 Dependency Plugin lets you grab your project's dependent jars and puts them into a local directory. Thanks to Srinath for showing me this.

This grabs all the jars in your dependency list and puts them in target/.../lib.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${basedir}/target/${pom.artifactId}-${pom.version}/lib</outputDirectory>
<overWriteReleases>false</overWriteReleases>
<overWriteSnapshots>true</overWriteSnapshots>
</configuration>
</execution>
</executions>
</plugin>

Emacs goto-column function

2007-11-05T09:58:00.000-08:00

Found this at http://www.icce.rug.nl/edu/1/cygwin/extra/dot.emacs.


Put it in your .emacs file.


(defun goto-column-number (number)
"Untabify, and go to a column number within the current line (1 is beginning
of the line)."
(interactive "nColumn number ( - 1 == C) ? ")
(beginning-of-line)
(untabify (point-min) (point-max))
(while (> number 1)
 (if (eolp)
     (insert ? )
   (forward-char))
 (setq number (1- number))))


To use it in emacs, use "Esc-X goto-column-number", enter, and type column number of the current line.

Maven 1 Repositories with Maven 2

2007-11-05T09:28:00.000-08:00

Add the following to your Maven 2 POM's repository section to download jars from an older Maven 1 repository. This is useful since Maven 1 repositories are much simpler to create than Maven2.

<repository>
<id>ogce-maven1-legacy-repo</id>
<url>http://www.collab-ogce.org/maven</url>
<layout>legacy</layout>
</repository>

Maven: Making a War and Jar at the Same Time

2007-11-01T15:31:00.001-07:00

Maven 2 nice automates building WAR files, but it places your compiled classes in WEB-INF/classes instead of making a new jar in /WEB-INF/lib.

If you want your stuff to be compiled as a .jar as well as a .war, you can do this by specifying the jar goal in the command line:

[shell>mvn clean jar install

Note this will make myproject.jar in target/, not in target/myproject/WEB-INF/lib, so you will need to use the Ant plugin to move this stuff around.

But this is not always an option: for deep, modular builds using the reactor, you may want to build your whole thing using one "mvn install". To do this, do the following:

Specify "war" packaging at the top of your pom.xml.
Then add the following to your build section.

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<executions>
<execution>
<id>make-a-jar</id>
<phase>compile</phase>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

And that's it. Do a " maven install" and you get a jar and a war.

Condor Collector Didn't Start

2007-10-10T14:06:00.000-07:00

OK, installing Condor on a local set of linux machines with NFS, no firewall problems, and real IP addresses. What could possibly go wrong? Used condor_install, edited the HOSTWRITE_ALLOW param, tried to start with condor_master, but condor_status gave the error

-bash-3.00$ ./bin/condor_status
CEDAR:6001:Failed to connect to
Error: Couldn't contact the condor_collector on myhost.myplace.myuniversity.edu.
....

The answer is found in the link below: you have to enable the additional daemons (particularly COLLECTOR) in the condor_config file.

https://www-auth.cs.wisc.edu/lists/condor-users-rc/2005-February/msg00119.shtml

Note this does not apply to firewalled machines with no access to DNS, no real IPs, etc. See my earlier post for this solution.

Why in the world was this not set in the config file for the master node by default?

NVO Summary

2007-10-07T11:05:00.000-07:00

I bookmarked several NVO related links on del.icio.us. See my tag role on this blog, or check http://del.icio.us/marlon_pierce. This blog post is intended to tie these together.

It looks like the Grid Summer Schools in 2004 and 2006 are the best sources for software. Ray Plante's nice but somewhat outdated summary, http://us-vo.org/pubs/files/PublishHowTo.html, is the best primer.

In general, it looks like one needs to do the following:

Get some data.
Set up a registry. NCSA's VORegistry-in-a-box looks like the best available software. They also interestingly seem to build on the Digital Library's OAI software.
Set up some data services. The simplest is a Cone Search service. This is available from the 2004 Summer School. This uses HTTP GET/URLS/REST, and apparently returns a list of links to existing whole images in the archive.
The SAIP service (aka SAI service) seems to be also available in the 2004 download. This implements the Simple Image Access Protocol. The services are typically HTTP GET/REST based but support more arguments that the Cone Service. For example, you can request image cutouts and other things that are created and returned on the fly, not just whole images.
The OpenSkyNode service is yet another service interface. This looks like it supports more complicated, SQL-like queries and can query multiple servers. But the JSkyNode software that Ray Plante mentions is no longer available.

For actually building the database, Roy Williams suggested http://amwdb.u-strasbg.fr/saada/.

Better TeraGrid Single Sign On

2007-10-02T07:46:00.000-07:00

Through the invisible hand of market forces, the world inevitably becomes optimized over time. Witness http://www.teragrid.org/userinfo/access/tgsso_native.php. This supersedes my earlier post, http://communitygrids.blogspot.com/2007/06/teragrid-single-sign-on.html.

Quote

2007-09-21T11:03:00.000-07:00

"ALL men of whatsoever quality they be, who have done anything of excellence, or which may properly resemble excellence, ought, if they are persons of truth and honesty, to describe their life with their own hand; but they ought not to attempt so fine an enterprise till they have passed the age of forty."

Benvenuto Cellini, one of Tom Sawyer's favorite authors, from Cellini's autobiography.

Windows Vista, Sockets, Java NIO, and TIME_WAIT

2007-09-18T14:14:00.001-07:00

Blogging on behalf of Shrideep Pallickara:

When you are working with NIO sockets on Microsoft Vista you may run into a
problem where you will sometimes be able to connect to a specific host/port
combination, and sometimes the following error related to a bind exception
would be thrown:
java.net.SocketException: Invalid argument: sun.nio.ch.Net.setIntOption
...


The reason this happens is that when a connection is closed it goes into a
TIME_WAIT state. This can be checked with a tool such as netstat. Sometimes,
it can take up to a couple of minutes to get out of this state. When you try
to establish connections to the same host/port combination at a later time
you may not be able to establish this connection, because the previous
connection is still in a timeout state.

To get around this problem you need to configure your socket so that it can
reuse addresses. With this fix you can bind a socket to the specified
Address even if there is a connection in the timeout state that utilizes the
socket's address or port.

Please note that on the client side you will need to configure the setup
BEFORE you bind the socket. The code below shows how you do this for NIO.
You will need to wrap this code-fragment in the appropriate try-catch block.


  SocketChannel sc = SocketChannel.open();
  sc.socket().setReuseAddress(true);
  sc.socket().setKeepAlive(true);
  sc.configureBlocking(false);
 
  InetSocketAddress ia = new InetSocketAddress(_hostName, _portNum);
  sc.connect(ia);

You will also need to do a similar configuration on the Server side of the
socket as well.

Restarting the ISCSI SAN

2007-09-12T09:16:00.000-07:00

The ISCSI SAN periodically dies and the file systems become inaccessible. Logwatch error symptoms can look like this:

 --------------------- Kernel Begin ------------------------

WARNING:  Kernel Errors Present
   connection0:0: iscsi: detected conn error (1011) ...:  3 Time(s)
  Buffer I/O error on device sdc1,  ...:  29 Time(s)
  EXT2-fs error (device sdc1): e ...:  171 Time(s)
  end_request: I/O error, dev sdc, sector ...:  2421 Time(s)
  lost page write due to I/O error on sdc1 ...:  29 Time(s)
  sd 3:0:0:0: SCSI error: return code = 0 ...:  2421 Time(s)
  sd 3:0:0:0: SCSI error: return code ueu ...:  1 Time(s)

To restart, follow these steps:

1. Log into the local NavSphere Express web utility running on the SAN.
2. Restart the SAN via NavSphere
3. Reboot the Linux server that mounts the SAN. If you don't want to reboot, try restarting the iscsi daemon and remounting the file system. Use fdisk -l to make sure that the iscsi devices are visible.

services iscsi restart
fdisk -l
mount /my/iscsi/partition

db4o and NFS problems

2007-08-09T18:53:00.000-07:00

Strange unexpected errors running db4o on NFS file system. Everything worked fine until a mysterious exception

com.db4o.ext.DatabaseFileLockedException: Database locked: '/globalhome/gateway/queueservice3.db'
at com.db4o.internal.JDK_1_4.lockFile(Unknown Source)
at com.db4o.internal.Platform4.lockFile(Unknown Source)
at com.db4o.io.RandomAccessFileAdapter.(Unknown Source)
at com.db4o.io.RandomAccessFileAdapter.open(Unknown Source)
at com.db4o.internal.IoAdaptedObjectContainer.open(Unknown Source)
at com.db4o.internal.IoAdaptedObjectContainer.(Unknown Source)
at com.db4o.internal.ObjectContainerFactory.openObjectContainer(Unknown Source)
at com.db4o.Db4o.openFile(Unknown Source)
at com.db4o.Db4o.openFile(Unknown Source)
at org.apache.jsp.junk_jsp._jspService(junk_jsp.java:49)

This only happens on NFS directories. Changing to a local directory, the same code works fine.

Google Maps: EGeoXML

2007-07-27T15:49:00.000-07:00

This is a wonderful little JS that allows you to work with KML files in your JavaScript code. Also overcomes the rendering limits of KML, so you can show many more placemarks.

http://www.econym.demon.co.uk/googlemaps/egeoxml.htm

Drupal Mac Install

2007-07-19T17:29:00.002-07:00

I'm doing this in a unixie way rather than looking for Mac specific packages.

0. sudo -H -u root /bin/bash

1. Get Mysql and install. Put this in /usr/local/msyql. The mysql manual is pretty good, so I won't duplicate. Make a db user ("drupaloompa") and db as drupal tells you.

2. Get Apache 2 and install. You'll need DSO support, so ./configure --enable-so. I wonder why they don't make this a default. Anyway, Apache 2.2 gave an error:

configure: error: Cannot use an external APR-util with the bundled APR

I decided to punt and go back to Apache 2.0. This worked fine. Probably I have another apache server pre-installed by Steve Jobs. Looks so, so be be careful that your paths point to /usr/local/apache2.

3. Get PHP 5. Configure/make/make install had one variation: ./configure failed unless I gave the full path to mysql, so I ran (in /usr/local/php-X.Y)

./configure --with-apxs2=/usr/local/apache2/bin/apxs --with-mysql=/usr/local/mysql

Noticed an unusual problem with PHP's "make install": it looked for libs in

/usr/local/mysql/lib/mysql/

but my libs were in

/usr/local/mysql/lib/

I rectified this straighforwardly (it's late) but probably this is some ./configure issue.

Checked php by creating a little file (phptest.php) with the line <? phpinfo() ?> in htdocs. This worked/

4. Copied drupal directory (after gnutarring) into htdocs. Note you want to do this:

cp -r /path/to/drupal-5.1/* /usr/local/apache/htdocs

You don't want the directory /usr/local/apache/htdocs/drupal-5.1. For some reason, my PHP files did not get parsed in this directory, but everything worked fine when I moved things to the top level htdocs directory.

5. Start apache and point to http://localhost/index.php. You will see a complaint about write access to settings.php. Do this:

chmod a+w sites/default/settings.php

Run this, use your Mysql user account and password, and you are good. Remove write permissions on settings.php.

chmod a-w sites/default/settings.php

Point your browser to http://localhost/index.php again and get going. Note that you will need to clear your browser cache if you loaded any Apache pages. Use shift-reload or just quit the browser and start a new one. In any case, http://localhost/ should show the index.php pages, not "It works" or any Apache feathers.

Some condor problems and solutions

2007-07-11T14:57:00.000-07:00

As in my previous post, here were some problems I encountered along the way to getting condor installed on a set of lab PCs running linux with no shared file systems and no DNS.

* Condor complains about missing libstdc++.so.5: install the compatibility patch for your OS. See earlier post.

* You get something like

Failed to start non-blocking update to <10.237.226.81:9618>.
attempt to connect to <10.237.226.81:9618 > failed: Connection refused (connect errno = 111).
ERROR: SECMAN:2003:TCP connection to <10.237.226.81:9618> failed

This probably means you have not set HOSTALLOW_READ and HOSTALLOW_WRITE correctly. Use the IP addresses you want to connect as the values. Also be sure you have enabled the NO_DNS and DEFAULT_DOMAIN_NAME parameters in your condor_config file.

* Jobs seem stuck in "idle". This could just mean that condor is really being unobtrusive. Try changing the START, SUSPEND, etc attributes in condor_config (see previous post and link).

* Jobs only run on the machine they are submitted from. Try adding

should_transfer_files=YES
when_to_transfer_output=ON_EXIT

to your job submission script.

* Need some general debugging help: here is a nice little link
http://www.cs.wisc.edu/condor/CondorWeek2004/presentations/adesmet_admin_tutorial/#DebuggingJobs

In short, look at the log files in /home/condor/log. Here are some useful commands:

1. condor_q -better-analyze
2. condor_q -long
3. condor_status -long

#1 will tell you why your job is failing to get matched.