Thursday, December 04, 2008

No airport hardware found

From the stupid Mac tricks department: if you get the error in the title (i.e. wireless card not working),
  1. power down,
  2. take out your battery,
  3. count "1 Steve Jobs, 2 Steve Jobs, 3 Steve Jobs",
  4. reinsert battery,
  5. power up,
and viola...voila, wireless will work.

The last guy on this thread deserves a Nobel prize:

Sunday, October 26, 2008

WebDAV with Apache2 and Windows

In case you haven't looked in a while, WebDAV modules are included with Apache 2 and 2.2. See for example for an installation overview.

This will work great for Mac clients (from "Finder" hit apple-k to connect as a remote web folder) with HTTPD basic authentication, but they will fail for Windows XP clients (either mapping as a drive or adding network location). The reason, it turns out, is that Windows is actually working correctly: it won't do Basic Auth over an insecure connection.

To solve this, use SSL. There's an overview of the problem and variuos solutions at See for example for some Apache + SSL instructions.

One Final Trick: Windows clients seem to have a problem with the trailing "/", so in your Apache httpd.conf file, use something like

<location /dav>

and not

<location /dav/>

for your WebDAV folder name.

Wednesday, October 08, 2008

Abstract: Services and Components for the QuakeSim Project

[Posting is part of our lab seminar series.]

We discuss two recent enhancements to the QuakeSim grid infrastructure. First, we describe our work to integrate QuakeSim Web services with large scale Globus deployments such as the US TeraGrid and Open Science Grid. Our implementation is based on the Condor BirdBath client libraries and provides a simplified and stateless API for accessing Globus resources. In addition to QuakeSim, we are developing a variation of this service to support the University of Minnesota’s VLAB project. We discuss integration of this service with TeraGrid information services for system load and queue wait time prediction, as well as security issues and prospects for deployment on NASA resources.

Our second focus is on work performed with the Scripps Orbit and Permanent Array Center’s archival data services for permanently deployed California GPS stations. We have developed a daily historical analysis Web service of the GPS data using RDAHMM. This service has been combined with an interactive Web interface that allows the user to view historical state changes in the GPS network. This archival data is available on a fourteen day delay, so we conclude with a discussion of the prospects for a nearer real-time network event detection system.

Wednesday, September 24, 2008

DAG Classes for Java

It's a little surprising that even after more than a dozen years worth of bloat, Java still doesn't have a standard set of classes to handle standard graphs like DAGs and their associated operations.

I needed this recently and decided against writing from scratch. With a little Googling, I decided to use the Plexus Utilities. See the JavaDocs at

The Plexus-Utils overview is here: The best way to get going is to grab the jars with Maven. Below is a dumb little example code:

import org.codehaus.plexus.util.dag.DAG;
import org.codehaus.plexus.util.dag.Vertex;
import org.codehaus.plexus.util.dag.CycleDetectedException;

import java.util.*;

public class DAGFun {
DAG dag;
Vertex from1, to1;
Vertex from2, to2;
public DAGFun() {
dag=new DAG();
from1=new Vertex("from");
to1=new Vertex("to");

public DAGFun(DAG dag) {

public void addEdge(DAG dag, Vertex from, Vertex to) throws CycleDetectedException {

public void processDag(){
List vertices=dag.getVerticies();
List roots=new ArrayList();
//Find all the roots. There may be more than one
for(int i=0;i<vertices.size();i++){
Vertex test=(Vertex)vertices.get(i);
if(test.isRoot()) roots.add(test);

//Now walk through all the DAGs
for(int i=0;i<roots.size();i++){
Vertex v=(Vertex)roots.get(i);
List children=v.getChildren();
for(int j=0;j<children.size();j++) {

public void processChild(Vertex vertex) {
if(vertex.isLeaf()) {
List children=vertex.getChildren();
for(int j=0;j<children.size();j++) {

public static void main(String[] args) throws Exception{
Vertex from1, to1;
Vertex from2, to2;
from1=new Vertex("from1");
to1=new Vertex("to1");

from2=new Vertex("from2");
to2=new Vertex("to2");

DAG dag=new DAG();
DAGFun df=new DAGFun(dag);


Monday, September 08, 2008

Yum install of SVN on CentOS

yum install subversion mod_dav_svn

Wednesday, August 27, 2008

Sun RSS Portlet in GridSphere

OK, thanks to Suresh and Marcus, I got the Sun RSS portlet working in GridSphere. Very good indeed, since the others that I found depended on Liferay or JBoss jars. Good grief, people, you are getting on my last nerve.

Here are the steps:
  1. Get the portlet from here:
  2. Do the usual gridsphere stuff: you will need to modify web.xml, copy over the gridsphere-portlet.xml from a working portlet into WEB-INF, copy over a gridsphere-ui-tags.jar from a working portlet into WEB-INF/lib. Also a group.xml and layout.xml file in WEB-INF will help. Plenty of examples of this over at
  3. Also touch webapps/gridsphere/WEB-INF/CustomPortal/portlets/rssportlet
  4. For some reason, the Sun portlet doesn't include the necessary jars to support the portlet's tag libraries, so you will need to get these and put them into WEB-INF/lib: jstl-1.0.6.jar, standard-1.0.6.jar.
This should be everything. The missing jar problem in step 4 was the frustrating part. Here's a list:

content-0.4.1.jar jdom-1.0.jar rome-0.8.jar
gridsphere-ui-tags-2.1.jar jstl-1.0.6.jar standard-1.0.6.jar

Probably there is an easier way, but around here we like to do it the bassackwards way.

Monday, August 18, 2008

Some BLAHP Notes

BLAHP is a plugin to Condor-G that enables Condor to submit jobs directly to schedulers like PBS/Torque. It is a variation of the GAHP scheduler that Condor-G uses to submit Globus jobs. Globus in turn provides the appropriate job manager for interacting with local scheduler. BLAHP (hopefully) will allow us to skip this step.

BLAHP is avaialble from the EGEE; see for an overview. There is an obsolete version of this in the Condor release, but we must override.

We are interested in this because Condor-G and Birdbath provide a Web service programming interface for managing jobs. See other notes on this blog and at We'd like to use this service for accessing machines that don't run Globus GRAM.

Basic steps for getting this going are the following. All steps are done on the same server.
  1. Install Condor and optionally Globus.
  2. Get the latest Condor C++ Classads, configure, make, make install
  3. Check out BLAHP from CVS, configure, make, make install
  4. Get Torque if you don't have a scheduler installed, configure, make, make install.
Notes are courtesy of Francesco Prelz, with some corrections from Yu Marie Ma. I'm building this on RHEL4.

Install Classads
  • Make an installation directory. I'll call this BLAHP.
  • wget
  • Unpack and cd.
  • ./configure --prefix $HOME/BLAHP/classads/ --enable-namespace --lib-name classad
  • make; make install
The trick is the configure step. You must enable the classad namespace (at least on RHEL4) for BLAHP to install (divined by Marie).

Install BLAHP
I also do these steps from the $HOME/BLAHP directory.
  • cvs -d co org.glite
  • cvs -d co -r glite-ce-blahp_R_1_11_1_0 org.glite.ce.blahp
  • cd $HOME/BLAHP/org.glite.ce.blahp/
  • ./configure --with-classads-prefix=$HOME/BLAHP/classads/ --with-globus-prefix=$HOME/globus/ --prefix=$HOME/BLAHP/blahp
Install Torque

This assumes you don't have this already installed. The steps below will work for a non-root build.
  • cd $HOME/BLAHP
  • ./configure --prefix=$HOME/BLAHP/torque --with-server-home=$HOME/BLAHP/torque
  • make; make install
Configure BLAHP
We now need to configure BLAHP to work with our local Condor and Torque installations.

Tuesday, August 05, 2008

TeraGrid RSS Feeds

Here's a big list of TeraGrid RSS news feeds. A while back, I wrote a post complaining that these didn't work correctly in iGoogle, but I'm pleased to write now that this has been corrected. On the downside, you have to go through a few pages to find the RSS URLs.

You can guess the subject for each feed from the title (rss-indiana.rss is the IU feed, for example).

Tuesday, June 24, 2008

Old Windows XP on New Dell Latitude

More adventures in system administration: I needed to install Windows XP on a new Dell Latitude laptop to support a demo. My old Windows XP installation CD crapped out, claiming it could not find any disk drives/devices.

The workaround (thanks to Rick McMullen) turned out to be to change from SATA AHCI to ATA in the BIOS. On the downside, it wiped out my old Vista installation, although switching back to AHCI seemed to recover it.

Wednesday, April 30, 2008

Useful TeraGrid REST Information Services

Courtesy of J. P. Navarro and Laura Pearlman. See for more. These aren't all strictly REST but close enough for my purposes. They return simple XML formats.

Here's how you can get a list of Grid services on Abe at NCSA.

Some other examples:

Some finer grained examples using XPath.

Give me all gridftp URLs in TeraGrid:[Type='gridftp']/Endpoint

Give me all gridftp URLs at NCSA:[SiteID='']//Service[Type='gridftp']/Endpoint

Give me the URLs of all advertised services at Big Red:[ResourceID='']//Service/Endpoint

Give me a little more information about all services at Big Red:[ResourceID='']//Service

Friday, April 04, 2008

Mildly Interesting KML+Google Maps Bug

Google map keys are of course tied to a particular base URL. You can get a key for http://localhost for development, but you may have noticed this isn't necessary: maps will load correctly in http://localhost/ for any map key.

However, you can't load KML into your maps on localhost unless you are using a key for http://localhost. That is, your map will load but the KML overlay wont' be added. That is,

geoXml=new GGeoXml("");

won't add the overlay to your map if you are using mismatched keys. To see this in inaction, try serving up the code sample
from your own web server (using your own key, of course). First try using localhost:8080 and then use This assumes the google key was generated for

Friday, March 28, 2008

Google's GData Java API

This has been out for a while, but I recently realized that Google's GData API provides a way to interact with all Google services (Doc, Calendar, YouTube, Blogger, Picasso, etc) through your own server side code. One nice application for Science Gateways is to write a Web Service that submits grid jobs, writes service state changes to Google Calendar, spits out result synopsis to a blog, and posts visualizations to Picasso or YouTube.

Below is some code liberally borrowed from Blogger, YouTube, and Calendar API gdata examples to show how simple this is. Some notes:

Code example below

import java.util.*;

//Google Calendar stuff

public class TestBlogger {
String clientID="XXXXX";
String developer_key="XXXXXXX";

String feedUrl="";
String scarFace5sec="";

GoogleService googleService;
YouTubeService youTubeService;
CalendarService calendarService;

public GoogleService getGoogleService() {
return googleService;
public void setGoogleService(GoogleService googleService) {

public YouTubeService getYouTubeService() {
return youTubeService;

public void setYouTubeService(YouTubeService youTubeService) {

public void setCalendarService(CalendarService calendarService) {

public CalendarService getCalendarService() {
return this.calendarService;

public TestBlogger() throws Exception {
//Replace these with real values.
String userName="";
String password= "qwerty";

//Log into blogger.
GoogleService myService=new GoogleService("blogger","");

//Log into YouTube
YouTubeService service = new YouTubeService(clientID, developer_key);

//Log into Calendar
CalendarService calService=new CalendarService(clientID);

public void writeToCalendar(String titleOfEvent,
String contentToPost) throws Exception {
URL postUrl =
new URL(""+userName+"/private/full");
CalendarEventEntry myEntry = new CalendarEventEntry();

myEntry.setTitle(new PlainTextConstruct(titleOfEvent));
myEntry.setContent(new PlainTextConstruct(contentToPost));

// DateTime startTime = DateTime.parseDateTime("2008-03-18T15:00:00-08:00");
System.out.println((new Date()).toString());
DateTime startTime = new DateTime(new Date());
// DateTime endTime = DateTime.parseDateTime("2008-03-18T17:00:00-08:00");
When eventTimes = new When();
// eventTimes.setEndTime(endTime);

// Send the request and receive the response:
CalendarEventEntry insertedEntry = calendarService.insert(postUrl, myEntry);

CalendarEventEntry myEntry2 = new CalendarEventEntry();
String now=(new Date()).toString();
myEntry2.setContent(new PlainTextConstruct("Test post at "+now));

// Send the request and receive the response:
CalendarEventEntry insertedEntry2 =calendarService.insert(postUrl, myEntry2);

public String getBloggerEntryString() throws Exception {
String returnString="";
URL feedUrl=new URL("");
Feed resultFeed = googleService.getFeed(feedUrl, Feed.class);

// Print the results
for (int i = 0; i < resultFeed.getEntries().size(); i++) { Entry entry = resultFeed.getEntries().get(i); returnString+=entry.getTitle().getPlainText()+"
return returnString;

public String getVideoFeedString() throws Exception {

String returnString;

VideoFeed videoFeed=youTubeService.getFeed(new URL(feedUrl),

VideoEntry videoEntry = youTubeService.getEntry(new URL(scarFace5sec),

returnString="Title: " + videoEntry.getTitle().getPlainText()+"
return returnString;

public Entry createPost(GoogleService myService, String blogID, String title,
String content, String authorName, String userName)
throws ServiceException, IOException {
// Create the entry to insert
Entry myEntry = new Entry();
myEntry.setTitle(new PlainTextConstruct(title));
myEntry.setContent(new PlainTextConstruct(content));
Person author = new Person(authorName, null, userName);

// Ask the service to insert the new entry
URL postUrl = new URL("" + blogID + "/posts/default");
return myService.insert(postUrl, myEntry);

public static void main(String[] args) throws Exception {
String blogId=1292929011;
String someName="test";

TestBlogger blog=new TestBlogger();
// System.out.println(blog.getBloggerEntryString());
// System.out.println(blog.getVideoFeedString());
String content="Some post test with the blogger Java api"+"
"Testing Blogger API",
"Marlon Pierce",
blog.writeToCalendar("Test Calendar Post",content);

Thursday, March 27, 2008

Broken TeraGrid RSS News Feeds

[Nagger's Update: the titles were fixed after I posted this. Hopefully reordering will be soon to follow.]

If you have a TeraGrid account, you probably have noticed the 6.02(10)^23 emails you get every day (Ranger is up! Ranger is down...). You can disable all of these by going to ' Even better, you can subscribe instead to RSS feeds and get these updates in a sensible fashion.

There are only two problems: the title fields in the feeds are broken and most of the feeds are in newest-entry-last order. The iGoogle RSS reader (for one) assumes that the first entry is the newest (not the oldest) entry.

Here is an example (to pick on the home team): I did notice that TACC had at least fixed their ordering, but most of the other centers have the same problem.

I reported these errors to the help desk like a good soldier, got my trouble ticket, and then...nothing. After a couple of weeks through some back channel emails, I found out that my ticket had been received and that they would fix it within the the year. I am not making this up. No one is willing to take five minutes to fix this simple problem. Probably they don't have any time left after all the meetings.

Friday, March 14, 2008

Testing Blogger API

Two Nice Papers by Jim Grey
SSH Empty Passphrases
Password Protected RSS/Atom Feeds
Get Off My Cloud
Counting Unique IPs in Apache Log Files
XFN Test
BBQ Flickr Mashup
Emacs for C#/.NET/Mono
Running GOAP on PolarGrid Testbed
Polar Grid Matlab Note
More with Shindig and Open Social
Quick Start with Open Social, Shindig
A Little More JavaScript and FaceBook Hacking
Quick Facebook JavaScript and PHP API Notes
Portals and Web Applications
Polar Grid URL
Some Project Zero Thoughts
More Mac OS X Emacs
Birdbath and Globus RSL
Installing and Configuring Globus Services on Mac OS X
Condor-G, Birdbath and TransferOutputRemaps
File Retrieval with BirdBath and Condor-G
Condor-G Plus TeraGrid Example
Returning Standard Output in Condor-G
Globus Client+MyProxy+Pacman on FC7 Notes
Title: Scarface in 5 seconds

Wednesday, March 12, 2008

Two Nice Papers by Jim Grey

Thanks to Shrideep for sending these:

G. Bell, J. Gray,  and A. Szalay, "Petascale Computational Systems:Balanced
CyberInfrastructure in a Data-Centric World," IEEE Computer, vol. 39, no 1,
pp. 110-112, Jan. 2006.

J. Gray, "Distributed Computing Economics", Computer Systems Theory,
Technology, and Applications, A Tribute to Roger Needham, A. Herbert and K.
Sparck Jones eds., Springer, 2004,pp 93-101

Friday, February 29, 2008

SSH Empty Passphrases

Of course you should never set up empty passphrases on ssh keys, but if you do and it doesn't work, check the permissions on your .ssh directory and .ssh/authorized_keys file. These should not be readable or writable by anyone but the owner.

Thursday, February 28, 2008

Password Protected RSS/Atom Feeds

HTTP Authenticated feeds don't seem to work with iGoogle or Internet Explorer's feed reader, so I looked for a couple of alternatives. These work:
  • Sage: a FireFox plugin. Works fine on both Mac and Windows, as long as you have FireFox.
  • NewsGator: works great on Windows Vista
I also looked into UniveRSS, a 3D (!?) RSS reader for Vista that is based on Windows Presentation Foundation. Um, no thanks. Keep It Simple, Stupid.

Monday, February 25, 2008

Get Off My Cloud

I'd like to put in a stake in the ground for anti-cloud computing. "Cloud computing" (i.e. Grid computing, only not gruesome) has a lot of buzz, and it is assumed that in the future we will all use computing and data storage as for-fee services virtually hosted on sites like Amazon. One brilliant guy compares it to the electrical power grid: Nicholas Carr, you owe Larry Smarr royalties.

In general, it is assumed that these online services will make the home computer become thinner and thinner until it no longer exists in its current form. In this vision, the next generation will be dominated by lightweight client devices (i.e. iPhones, Blackberries, Nokia N-series tablets, etc.), and most computing and data management will be performed by large online services. Existing service providing companies like Google will dominate, and Microsoft is desperately trying to get market share.

While this will happen, I don't think it is the whole story. The home PC isn't dead and (for many of us) will become a full fledged home appliance. Think of it as the long tail of cloud computing. Consider
  • There will be dozens of computing cores on relatively modest home computers within a decade (i.e. home computers will be mini parallel supercomputers).
  • Storage on these same machines will commonly be in the terabytes (already there on high end desktops).
  • High speed internet access will be available in most homes (probably already there for upper income households).
Actually, we are already there. The main thing that will change is the excessive parallel core horsepower that the next generation of PCs will have (and maybe we are there, too--check out this thing--72 cores on a desktop!). Major vendors are desperately trying to improve the software development environments to take advantage of these cores. Seems to me that Web servers are a natural fit for this.

So instead of everyone uploading photos and movies to Flickr and YouTube, why not provide "home editions" of these services that allow me to run my own web service? Or why not a "home edition" of in which I can serve up and share my music collection with my friends and family? Ideally, I would also be able to federate my servers with friends and family into peer-networks and other such topologies so that I could share information transparently with friends while maintaining fine-grained control of the data at the same time. Obviously there is a lot of file and music sharing software already--I am thinking more of adding the social networking capabilities provided by various Web 2.o sites to this.

Another possible application for the home supercomputer is managing MyHouse@HOME (a hypothetical project). In Tom Igoe's book, Making Things Talk, he introduces the concept of object-oriented hardware (or, if you prefer, service-oriented hardware). Ubiquitous micro-computers and sensors combined with simple network protocols allow you to build interesting home applications, such as twitter-style blogs for your house. This could allow you, for example, to monitor your house while you are away or keep an eye on elderly family and friends. Event detection on this sort of sensor data is an interesting problem. This sort of sensor data obviously should go through a central Web server hub under your control.

More importantly and generally, with a home computing appliance, I can control my own personal data and make available as I choose instead of trusting large central repositories. Family medical records, genetic testing results, financial records, etc could be maintained locally and accessed by outside clients in ways that the owner controls.

One may object that there are many security problems. For example, the home computer becomes a bigger honey pot of valuable information than it is already, so it becomes a target for both internet break-ins and physical theft. And you would lose most of your records in a house fire that melted your computer. Hard drive failures would be a disaster. But all of these are already problems to some degree. Secure, encrypted hard drives are already available technology (in laptops for example), so just physically stealing the machine won't expose you to unnecessary risk.

Home denial of service attacks would be a new problem. I can also see the need for a secure "Swiss Bank Account" for data on your hard drive: every night you can rsync your data to a secure site, where it will be stored in numbered accounts in encrypted form via public/private keys and retrievable only by you (i.e. malicious employees don't know which account belongs to whom and can't read the data anyway). Probably this already exists in the financial world. Amazon's S3 and Micorsoft's SkyDrive are early examples, but security and reliability will need to be greatly enhanced.

The compelling thing to me is that none of this is particularly new. Context-aware sensors, parallel computing techniques and algorithms, network computing, social networks, and so on are old topics in the academic world. The key new concept is accessibility: do-it-yourself enthusiasts will broaden participation in these fields and will push researchers to deepen it. This will create the anti-cloud. Or if you prefer, everyone can have their own cloud.

What is missing seems to be the higher level building block software for building home cloud services (i.e. the home edition of Microsoft, in particular, seems to be missing the point. They are trying desperately to transform into a service provider (e.g. to compete with Google on search), but they are always chasing the leader (witness various Live projects, Virtual Earth, Skyserver. etc). Instead, why not concentrate on their core competency: home and business software, only now "cloud-enabling"? Note "cloud-enabling" is not "cloud-enabled". The former is server side (making use of the fat computing appliance in the home office). The latter is the thin client to some remote cloud service.

I'll conclude with a poke at the HPC and Grid world: yes, we were there first, just like the Vikings were the first Europeans in North America.

I note that the smart guys like Ray Ozzie at MS were way ahead of me with Live Mesh, which is debuting today (April 23).

Sunday, February 24, 2008

Counting Unique IPs in Apache Log Files

A little Unix one-liner, posted to my online memory bank:

cat /var/apache/logs/access_log | grep whatever | cut -f1 -d' ' | sort -n | uniq | wc

Change "whatever" to the name of the relative path you are trying to count.

Thursday, February 21, 2008

Emacs for C#/.NET/Mono

Thought I'd give Mono a try with Mac OSX+Emacs. Of course Emacs won't have the nice formatting Lisp scripts for your code, but you can get them from here: sans instructions. The other link suggested by Mono ( was broken.

Here's how to get it to work:
  • Unzip the zip file from the link above.
  • Edit (or create) your $HOME/.emacs file and add the appropriate sections from zbrad.emac s. You could just add everything.
  • Add the line (add-to-list 'load-path "~/.emacs.d/site-lisp") at the beginning of your .emacs file.
  • Copy the "site-lisp" directory from the downloaded zip and there you go.
You should now be in business. I don't understand why Mr. Merrill didn't provide a README with his fine stuff. Gives emacs a bad name. Filling in these sorts of gaps made Tim O'Reilly famous.

Tuesday, February 19, 2008

Running GOAP on PolarGrid Testbed

These notes assume you have the sample data and a snapshot of the code. These are unpacked in /home/polargrid/PolarGRID and /home/polargrid/code, respectively.

Configuring GOAP

These notes are Make sure that matlab is in your path:
export PATH=/usr/local/matlab/bin:$PATH
if necessary. Cd into the /home/polargrid/code directory and edit/check the ParamMCRDS_IU.m file. Make sure these parameters are OK:

param.in_path = '/home/polargrid/PolarGRID/raw/';
param.out_path = '/home/polargrid/PGOut/';
param.ref_path = '/home/polargrid/PolarGRID/150MHz_PD03_PD10/';
param.pos_path = '/home/polargrid/PolarGRID/';
param.pos_name = 'MCRDS_20070917_ALL_pos.mat';
param.prefix = 'data.20070917';

param.in_path should point to the .raw data files.
param.out_path can point anywhere you like, as long as the directory exists.
param.ref_path should point to the directory with the MCRDS_reference_* files. param.pos_path should point to the directory that contains the param.pos_name file.

You may also want to fool around with these parameters.

param.type = 'MCRDS';
param.file_idx_start = 25;
%param.file_idx_stop = 99;
param.file_idx_stop = 26;

These correspond to the data files *.0025.raw to *.0099.raw in /home/polargrid/PolarGRID/raw/. The above just runs the code with 2 input files (should take about 3 minutes) instead of 75.

I also had to use these parameters below.

param.sched_type = 'local';
%param.sched_type = 'jobmanager';
param.sched_name = 'localhost';
%param.sched_name = '';

You can leave these as is for now. I also had to add the "exit" command to the end of the main script for force matlab to exit. Otherwise running the command script with the -r option (see below throws you back to the >> command prompt.

Running the Code
Do this from the /code/ directory. Run the command

matlab -nodisplay -nodesktop -r ParamMCRDS_IU

Don't use redirected standard inputs to read the matlab file (i.e. don't use "matlab < ParamMCRDS_IU.m") or you'll get

Warning: Type-ahead buffer overflow.
In distcomp.abstractjob.waitForState at 92
In StartScriptCluster at 28

See bottom of
Also, the code won't create the final jpegs correctly this way. So...don't do that.

Where's the Data?
The output files go to the directory you specified above (PGOut). The final jpegs are placed in


along with the header files (metadata, .txt) and .mat files.

Polar Grid Matlab Note

If you get the error

[matlab@polargrid003 ~]$ /usr/local/matlab/bin/matlab
Warning: Unable to open display , MATLAB is starting without a display.
You will not be able to display graphics on the screen.
License checkout failed.
License Manager Error -95
MATLAB is unable to connect to the license server.
Make sure you can resolve the hostname of your machine.
If you are unable to resolve the hostname, contact your System Administrator.

Troubleshoot this issue by visiting:

Diagnostic Information:
Feature: MATLAB
License path: /usr/local/matlab/etc/license.dat:/usr/local/matlab/etc/*.lic:
FLEXnet Licensing error: -95,378. System Error: 115

This just means that the matlab license server isn't running. Start it as the matlab user with the command

/usr/local/matlab/etc/lmstart -l /tmp/eeee

Monday, February 18, 2008

More with Shindig and Open Social

Get Going
  • After svn checkout, cd to $HOME/shindig/java/gadgets (or wherever you installed).

  • Run "mvn package" and then "mvn jetty:run-war" from this directory.

  • Point your browser at


    or similar. Note this is an "aggregating client container" for collecting and displaying gadgets. The gadgets are loaded as XML from a specified remote URL and arranged using a layout manager. However, these containers don't know anything about your social networks--they are just for presentation and organization of content. If you actually want to host your own social gadgets you will need to deploy them into the "gadget hosting container"(e.g. Orkut or some other app). This is available on your Shindig server at


  • Note the aggregating client container just shows how to add gadgets from multiple providers. Any social networking information is NOT controlled or accessible by these containers. To run your own gadget and manipulate social network information (such as a user's profile and list of friends), you will need to run your gadget in a container.

  • If you want to add your own sample HTML file for aggregating gadgets, you need to put these in $HOME/shindig/javascript/container/. The maven package command will move these files into java/gadgets/target/gadgets/files/container/myjunk.html.

    You'll then have to shutdown and restart jetty.

    This is a little non-intuitive directory layout but see java/gadgets/pom.xml.

  • The $HOME/shindig/javascript/README is a good place to start.

Hosting your own gadgets.
  • Place your widget code in


    or any subdirectory (like /examples/).

  • After compilation, these will be located in


    or any subdirectory (/examples/ in the shindig SVN).

  • These will be served up from


  • You can use the XML from the sample widget at

  • Just download or cut and past this XML file onto your local file system.

  • For some reason, you will need to change the top:

    <ModulePrefs title="Title of Your Application">
    <Require feature="opensocial-0.6"/>

    <ModulePrefs title="People API Howto">
    <Require feature="opensocial-samplecontainer"></Require>

    This seems to have been corrected in newer versions. Check the header of the sample gadget and use this as a guideline for your gadget.
  • Anyway, load the samplecontainer URL (http://localhost:8080/gadgets/files/samplecontainer/examples/myapp.xml) in your browser, and then load your gadget by pasting in the URL

  • If you edit your gadget in any way, you must (apparently) restart your Jetty server.

This gadget is now ready to be displayed in any gadget container.

Putting Your Gadget Into Your Display Container
In the last previous two steps, you a) displayed existing gadgets using the gadget display container and b) set up your own gadget hosting container. The documentation doesn't very clearly distinguish these, but you should see this by now. The display container can load any gadgets (including production gadgets available from iGoogle). The gadget hosting container, on the other hand, is used to run your gadgets with your social network data.

We can now combine the two. Edit any of the sample HTML files in java/target/gadgets/files/container, changing the URL to point to the gadget you made in the previous example. Restart your Jetty server and you should see it correctly displayed.

Similarly, you should be able (if your Jetty server's URL is publicly accessible) to add your social gadget to other display containers.

Sunday, February 17, 2008

Quick Start with Open Social, Shindig

OpenSocial is of course the Google-led consortium of competitors to FaceBook, but until very recently most of its efforts appeared to be vaporware. This is starting to change.

One of my immediate concerns about the OS API was how easy it would be to build an OS compliant container. Otherwise, we would have to rely upon existing containers (Orkut, anyone?) to host our OS-based applications. Fortunately, the Apache Shindig incubator project ( is something of an open source reference implementation for the OS API (written in Java with a Maven build no less).

Check it out with SVN and build from source with Maven (worked perfectly on version 628525). Run with Jetty server.

Point browser to http://localhost:8080/gadgets/files/container/sample1.html and look through other samples. You can also load these HTML files using file:// if you don't want to run the Jetty server.

Next time: can I use OS clients to query my LinkedIn profile?

Friday, February 15, 2008

A Little More JavaScript and FaceBook Hacking

Facebook's JavaScript example is pretty minimal, so I thought I'd do a little more hacking around. I'm not a great JS programmer, so I like to unroll the function definitions.

You should grab the very minimal JS API documentation from here: It is the "zipped archive" link near the bottom. This guide is very unfriendly at the current time, but in general you should see that the JS equivalents of the API list at replace the "." with a "_". So users.getInfo becomes users_getInfo(...).

Here's an example that uses the users_getInfo() method and some other stuff to get a user's profile information.

<!-- Output area to show the output from Facebook API -->
<textarea style="width:500px;height:300px;" id="_traceTextBox">
<script src="" type="text/javascript">
<script type="text/javascript">

// Create an ApiClient object, passing app's api key and
// a site relative url to xd_receiver.htm
var api = new FB.ApiClient('<your_key_here>', '/xd_receiver.htm', null);

// Get friends list
function getResults(result,exception) {
Debug.dump(result,'The stuff');

function getInfo(result,exception) {
Debug.dump(result,'The info');

// require user to login
api.requireLogin(function(exception) {
var myinfo='last_name,first_name,hometown_location,work_history,pic_small';


You need to update this stuff above to use your application's key, as indicated.

The output of this will be something like
you: 627774031

The stuff: {Array}

The info: {Array}
[0]: {Object}
first_name: Marlon
hometown_location: {Object}
last_name: Pierce
uid: 627774031
work_history: {Object}

friendsResult from batch execution: {Array}
notificationsResults from batch: undefined

I guessed the values for the myinfo array by looking at the big XML file example shown at That's the real URL for the tiny version of my FaceBook photo.

So in summary, you can use this approach to embed FaceBook profile information in your own web application. You'll of course have to do a bit of work to format all of this stuff. Note also that you can substitute another user's numeric ID in the users_getInfo() call (ie that of one of your friends). I used this to spy on Dave De Roure, who spends a lot more time on FaceBook than I do. And I won't even start on Dan Katz.

FQL and JS
Finally, note that all of the FaceBook API is basically a set of wrappers around their SQL like FQL query language. So you can, if you prefer, make custom methods out of FQL query strings. For example

var myQuery='SELECT name FROM user WHERE uid='+api.get_session().uid;

will execute the indicated query string from JavaScript and pass it to your (developer defined) function getFQLResponse().

To list information about groups you are a member of, use the following. You can replace your uid with another UID.

In the second line, 18629081888 is the GID for the very inactive OGF Web 2.0 group.

Thursday, February 14, 2008

Quick Facebook JavaScript and PHP API Notes

Took a quick look at and found a few obfuscations. To get started, you need to do these things (described a little too briefly in the link above).
  • Register for an application key at (Don't hit enter yet)
  • Cllick the "Optional Fields" drop down and
    • Provide a callback URL (that is, the full URL of your HTML page containing the Facebook JavaScript application).
    • Choose "IFrame" as the canvas page URL.
When it is working, this little app just prints out your friend's Facebook IDs (numbers) onto the screen. When you load your page (, you will be redirected to Facebook to login and then will go back to your URL.

Also took a quick tour of the PHP examples ("footprints" and some hello-world type stuff). Looks like the instructions for setting up the JS example assumed you were familiar with doing things the PHP way.

For footprints, you will need a web server (apache), PHP, and MySQL. Get, install, and start these in the usual ways. Hints: for Macs, get the nice disk image from MySQL. Also, read the PHP installation instructions before installing Apache, since PHP will require some non-default configuration options (particularly DSO support).

The web server provides the callback URL as well (""), which will need to be on a world accessible Apache (or other) server. Dump your application in htdocs/footprints and make sure paths are correct (index.php needs to point correctly to facebook.php). Note also that will be taken, so make a different name. The facebook app name does not have to match your callback app's name.

Create the MySQL database. The footprints PHP scripts don't do this for you.
  1. Edit config.php. You will need the root password for your db.
  2. From the command line, log into your db with /usr/local/mysql/bin/mysql -u root -p and provide the root password at the prompt. You will then need to create and switch to the footprints database.
  3. mysql> CREATE DATABASE footprints;
  4. mysql> USE footprints
  5. mysql>CREATE TABLE `footprints` ( `from` int(11) NOT NULL default '0', `to` int(11) NOT NULL default '0', `time` int(11) NOT NULL default '0', KEY `from` (`from`), KEY `to` (`to`) );
Step 5 is just a cut and paste of the config.php file from footprints, but note that they omitted the final ";" on the statement.

When it is working, this little application allows you to step on your friends. It gives you a little text area for typing names that will be autocompleted as you type. Note this must run inside Facebook, unlike the previous example.

When setting up your PHP app, you'll notice you can choose between FBML and IFrames. FBML will (as you might have guessed) a nice FaceBook styled application and some extra goodies. The IFrame option will produce a pretty bare bones footprint application, but obviously you have more power to customize the styles.

Hosting Your Services
Here are some instructions on using Amazon Web Services to host your application.

Tuesday, February 12, 2008

Portals and Web Applications

There are many tools for building web applications and portals, so it is useful try to classify these a bit. The field also suffers from overlapping terminology ("framework" providing the worst example) and overlapping functionality. I'll take an informal stab at trying to organize these things.

Programming Tools and Languages: this is the simplest group (hopefully) to define. PHP, PERL/CGI, Java Server Pages and Servlets are all examples. One could argue that JSP is more of a development framework (below), but these aren't sharp definitions. I'll err on the side of usage--most people don't write JSP using beautiful MVC architected code.

Goodies and Widgets: these are useful little or medium-sized extensions that can be easily plugged into several different development environments. PEAR/PHP and Java Tag Libraries seem to be best examples on the server side. Various JavaScript libraries (such as YUI) are good examples of client-side applications. These can be embedded into several different toolkits and frameworks. These are distinct from development framework modules (i.e. Ruby on Rails modules) since they are meant to work with both Programming Tools (above) and more specific, derived Development Frameworks (below). Thus one can put useful tags into JSP as well as JSF.

Development Frameworks: Frameworks are typically one level of abstraction above the common programming language tools. Java is full of these: Struts, JSF, Velocity, etc. Ruby on Rails is another famous example. DWR, Google Web Toolkit, and other tools also fall in here. ASP.NET and various compatible, open source implementations like Mono go here.

Frameworks typically encode a development methodology (MVC) and provide an extensibility framework so that all developers create code more or less in the same way. Frameworks thus allow you to develop reusable and useful little modules that can be shared with other developers using the same framework.

A framework can be used to build standalone applications, but it can also be used to develop a rich, self contained component that can be deployed into a container. In contrast, a container (after installation) comes up immediately with useful tools such as login modules, layout managers, and some user tools (say, a calendar or a blog authoring tool). In this example, one may use JSF (a framework) to build a web calendar or a web email application, but one would also typically want to embed these into a more comprehensive container (a portlet container in the Java world).

Application Containers: Joomla (php), Zope (python), Mambo, and Drupal (php) are examples. Java JSR 168 containers and various similar efforts also fit here: GridSphere, LifeRay, uPortal, Jetspeed2, etc. Sakai is another example--it supports the Java standard but also has its own component model. Interestingly, the non-Java containers mentioned above all began life as content management systems.

Arguably only Java's portlet standard really takes the container philosophy completely seriously. Joomla et al, really don't care about portability of codes across container vendors, although they do obviously care about shareable community developed modules. All Java portlet container providers (except Pluto, the reference implementation) don't just give you a container for managing your own portal components--they also give you lots of built in functionality and other goodies (calendars, chat tools, document managers, etc). LifeRay and Sakai are good examples of this.

Related Things: Web-based containers for content and document management, blogging, wikis, and learning management are related closely to portals. There are many different content management systems. Learning management varies from Moodle to Blackboard. Blogging is often provided as an online service (such as Google's Blogspot), but you can also install and run your own blogging service. WordPress is a prominent example of blogging software. Many frameworks (such as Drupal) come with blogging modules. Wikis are similar to content management and document management systems. The best known wikis (MediaWiki and Twiki) are standalone tools, but many containers come with Wiki modules (Sakai for example).

Services for Portals: One of the dangers of portal tools at any level is that they can become monolithic. The danger here is that you will be stuck with inflexible technology and can't respond to changing requirements. The current Enterprise vs. Web 2.0 conflict provides a compelling example. Many famous Web 2.0 sites (flickr, youtube, etc) are Web Services as much as they are Web sites, so modern portals must be able to easily integrate (rather than compete) with these large third party services.

External services: there is a great temptation of course to bundle everything into a particular portal for simplicity of installation and maintenance. However, this is not a good plan in the long term since you will (for example) tie your managed content to a particular tool rather than a portable standard.

When choosing a framework or container, a good starting point in the evaluation process is to see if it supports clients to the following standard services.
  • Authentication and Authorization: these are often tied to LDAP, CAS, or (in the Grid world) MyProxy. Shibboleth is an authorization mechanism. The basic idea is that the portal needs to go to an external database to get the login information. This DB is independent of the portal and may have other clients.
  • RSS/Atom feeds are particularly important in Web 2.0 applications, since most of them publish syndication in some form or another. Many containers provide RSS readers. This is any easy way to incorporate external content (say, YouTube videos) into a portal.This is particularly important topic in modern education portals, as for example one may embed publication feeds from Connotea and CiteULike, slides from SlideShare or ImageLooop, SciVee or YouTube for scientific visualizations and online lectures, etc. As can be seen from the previous list, there are dedicated, high quality online services for many education and science portal content that can be incorporated into a portal pretty generically as RSS/Atom. It is a bad idea to reinvent this stuff or tie it too closely to your portal framework.
We may also turn this problem upside down: if you run a resource management portal,
you must be able to export that portal's content to other user interfaces such as iGoogle
and NetVibes that users are more likely to use every day.
  • Calendars: Collaborative calendars are obviously best extracted from portals since there are so many calendar clients. This is also an example of a reversible service: you typically want to synch your online calendar with your local PC or mobile device. A relevant standard is iCal.
  • WebDAV servers for content management. WebDAV is an extension of HTTP and is supported by many Web servers (with the right module, of course).
  • Grid and workflow services for running scientific applications and accessing data.
  • Digital libraries: access to Fedora, etc. Don't reinvent this or jackbooted librarians will be goose stepping into your cubicle.
It is of course inadequate to simply support such services through clients, as the implementation must be useful and enjoyable. On the downside, however, it is also hard to know when one has crossed the line that separates a rich, useful client from a full fledged internal service that can't be easily extracted from the framework. I certainly can think of several examples that I've written.

Social Networking: the other challenge facing education portals is how much to integrate with social networking sites such as FaceBook or LinkedIn.

Polar Grid URL

Using the power of my blog to shift google page ranks, I hereby declare that PolarGrid (aka Polar Grid) has the following official URL:

Damn polar coordinate system.

Sunday, January 20, 2008

Some Project Zero Thoughts

IBM's little Project Zero is their effort to provide a Java-based Web 2.0 framework/toolkit/whatever for building your own applications. Several things look promising, such as the list of core Web 2.0 services (commenting, tagging, rating, etc) at

But looking at it raises more concerns and confusions. I could just be wrong, but here are some criticisms.
  • They seem to be providing their own Web server container, which is a really bad idea. How do I port this to some other Web Server like Tomcat or Jetty?
  • I managed to stumble on a bug in their simple "to-do" list application--only the last item in the list gets deleted properly.
  • They rely on Groovy as their Java Web framework (you can also develop in PHP). Groovy may be nice, and it does sound sort of like "Ruby", but there are also a lot of JSP developers out there. Why ignore them? Having seen a lot of Java ideas come and go, I am really reluctant to embrace anything but the lowest common denominator technology from them.
  • The packaging of the tutorials is nice and easy to get going (looks like maven or ant under the hood of some scripts), but I have to critique them a bit on their documentation (or maybe I didn't read carefully enough) here: it was not clear until I dug a little more that the methods implemented in
    todos.groovy represent the universal Project Zero API for REST services: all services implement onList(), onCreate(), etc, and these are bound to various HTTP Methods (onList <--> GET). See think this is reasonable (after, REST is essentially a universal API, the equivalent of a one size fits all WSDL). I just wish the nice tables in the link above were shown in the tutorial.

More Mac OS X Emacs

There is a nice list of emacs for Mac OS X here:

I've been using the build from Porkrind (as I posted some time ago), but I have recently switched to EmacsApp. EmacsApp features much nicer integration with the Mac GUI widgets and supports both the classic Emacs keyborad shortcuts as well as standard Mac shortcuts (that is, paste can be either cntrl-y or apple-v). Moving text between Emacs and other Mac applications is much better implemented.

Friday, January 18, 2008

Birdbath and Globus RSL

The condor_submit globusrsl parameter's "real" name is GlobusRSL (use condor_q -l to see this). Thus to specific in your Birdbath Java client an RSL parameter such as maxWallTime, use

new ClassAdStructAttr("GlobusRSL",