Friday, February 29, 2008

SSH Empty Passphrases

Of course you should never set up empty passphrases on ssh keys, but if you do and it doesn't work, check the permissions on your .ssh directory and .ssh/authorized_keys file. These should not be readable or writable by anyone but the owner.

Thursday, February 28, 2008

Password Protected RSS/Atom Feeds

HTTP Authenticated feeds don't seem to work with iGoogle or Internet Explorer's feed reader, so I looked for a couple of alternatives. These work:
  • Sage: a FireFox plugin. Works fine on both Mac and Windows, as long as you have FireFox.
  • NewsGator: works great on Windows Vista
I also looked into UniveRSS, a 3D (!?) RSS reader for Vista that is based on Windows Presentation Foundation. Um, no thanks. Keep It Simple, Stupid.

Monday, February 25, 2008

Get Off My Cloud

I'd like to put in a stake in the ground for anti-cloud computing. "Cloud computing" (i.e. Grid computing, only not gruesome) has a lot of buzz, and it is assumed that in the future we will all use computing and data storage as for-fee services virtually hosted on sites like Amazon. One brilliant guy compares it to the electrical power grid: Nicholas Carr, you owe Larry Smarr royalties.

In general, it is assumed that these online services will make the home computer become thinner and thinner until it no longer exists in its current form. In this vision, the next generation will be dominated by lightweight client devices (i.e. iPhones, Blackberries, Nokia N-series tablets, etc.), and most computing and data management will be performed by large online services. Existing service providing companies like Google will dominate, and Microsoft is desperately trying to get market share.

While this will happen, I don't think it is the whole story. The home PC isn't dead and (for many of us) will become a full fledged home appliance. Think of it as the long tail of cloud computing. Consider
  • There will be dozens of computing cores on relatively modest home computers within a decade (i.e. home computers will be mini parallel supercomputers).
  • Storage on these same machines will commonly be in the terabytes (already there on high end desktops).
  • High speed internet access will be available in most homes (probably already there for upper income households).
Actually, we are already there. The main thing that will change is the excessive parallel core horsepower that the next generation of PCs will have (and maybe we are there, too--check out this thing--72 cores on a desktop!). Major vendors are desperately trying to improve the software development environments to take advantage of these cores. Seems to me that Web servers are a natural fit for this.

So instead of everyone uploading photos and movies to Flickr and YouTube, why not provide "home editions" of these services that allow me to run my own web service? Or why not a "home edition" of in which I can serve up and share my music collection with my friends and family? Ideally, I would also be able to federate my servers with friends and family into peer-networks and other such topologies so that I could share information transparently with friends while maintaining fine-grained control of the data at the same time. Obviously there is a lot of file and music sharing software already--I am thinking more of adding the social networking capabilities provided by various Web 2.o sites to this.

Another possible application for the home supercomputer is managing MyHouse@HOME (a hypothetical project). In Tom Igoe's book, Making Things Talk, he introduces the concept of object-oriented hardware (or, if you prefer, service-oriented hardware). Ubiquitous micro-computers and sensors combined with simple network protocols allow you to build interesting home applications, such as twitter-style blogs for your house. This could allow you, for example, to monitor your house while you are away or keep an eye on elderly family and friends. Event detection on this sort of sensor data is an interesting problem. This sort of sensor data obviously should go through a central Web server hub under your control.

More importantly and generally, with a home computing appliance, I can control my own personal data and make available as I choose instead of trusting large central repositories. Family medical records, genetic testing results, financial records, etc could be maintained locally and accessed by outside clients in ways that the owner controls.

One may object that there are many security problems. For example, the home computer becomes a bigger honey pot of valuable information than it is already, so it becomes a target for both internet break-ins and physical theft. And you would lose most of your records in a house fire that melted your computer. Hard drive failures would be a disaster. But all of these are already problems to some degree. Secure, encrypted hard drives are already available technology (in laptops for example), so just physically stealing the machine won't expose you to unnecessary risk.

Home denial of service attacks would be a new problem. I can also see the need for a secure "Swiss Bank Account" for data on your hard drive: every night you can rsync your data to a secure site, where it will be stored in numbered accounts in encrypted form via public/private keys and retrievable only by you (i.e. malicious employees don't know which account belongs to whom and can't read the data anyway). Probably this already exists in the financial world. Amazon's S3 and Micorsoft's SkyDrive are early examples, but security and reliability will need to be greatly enhanced.

The compelling thing to me is that none of this is particularly new. Context-aware sensors, parallel computing techniques and algorithms, network computing, social networks, and so on are old topics in the academic world. The key new concept is accessibility: do-it-yourself enthusiasts will broaden participation in these fields and will push researchers to deepen it. This will create the anti-cloud. Or if you prefer, everyone can have their own cloud.

What is missing seems to be the higher level building block software for building home cloud services (i.e. the home edition of Microsoft, in particular, seems to be missing the point. They are trying desperately to transform into a service provider (e.g. to compete with Google on search), but they are always chasing the leader (witness various Live projects, Virtual Earth, Skyserver. etc). Instead, why not concentrate on their core competency: home and business software, only now "cloud-enabling"? Note "cloud-enabling" is not "cloud-enabled". The former is server side (making use of the fat computing appliance in the home office). The latter is the thin client to some remote cloud service.

I'll conclude with a poke at the HPC and Grid world: yes, we were there first, just like the Vikings were the first Europeans in North America.

I note that the smart guys like Ray Ozzie at MS were way ahead of me with Live Mesh, which is debuting today (April 23).

Sunday, February 24, 2008

Counting Unique IPs in Apache Log Files

A little Unix one-liner, posted to my online memory bank:

cat /var/apache/logs/access_log | grep whatever | cut -f1 -d' ' | sort -n | uniq | wc

Change "whatever" to the name of the relative path you are trying to count.

Thursday, February 21, 2008

Emacs for C#/.NET/Mono

Thought I'd give Mono a try with Mac OSX+Emacs. Of course Emacs won't have the nice formatting Lisp scripts for your code, but you can get them from here: sans instructions. The other link suggested by Mono ( was broken.

Here's how to get it to work:
  • Unzip the zip file from the link above.
  • Edit (or create) your $HOME/.emacs file and add the appropriate sections from zbrad.emac s. You could just add everything.
  • Add the line (add-to-list 'load-path "~/.emacs.d/site-lisp") at the beginning of your .emacs file.
  • Copy the "site-lisp" directory from the downloaded zip and there you go.
You should now be in business. I don't understand why Mr. Merrill didn't provide a README with his fine stuff. Gives emacs a bad name. Filling in these sorts of gaps made Tim O'Reilly famous.

Tuesday, February 19, 2008

Running GOAP on PolarGrid Testbed

These notes assume you have the sample data and a snapshot of the code. These are unpacked in /home/polargrid/PolarGRID and /home/polargrid/code, respectively.

Configuring GOAP

These notes are Make sure that matlab is in your path:
export PATH=/usr/local/matlab/bin:$PATH
if necessary. Cd into the /home/polargrid/code directory and edit/check the ParamMCRDS_IU.m file. Make sure these parameters are OK:

param.in_path = '/home/polargrid/PolarGRID/raw/';
param.out_path = '/home/polargrid/PGOut/';
param.ref_path = '/home/polargrid/PolarGRID/150MHz_PD03_PD10/';
param.pos_path = '/home/polargrid/PolarGRID/';
param.pos_name = 'MCRDS_20070917_ALL_pos.mat';
param.prefix = 'data.20070917';

param.in_path should point to the .raw data files.
param.out_path can point anywhere you like, as long as the directory exists.
param.ref_path should point to the directory with the MCRDS_reference_* files. param.pos_path should point to the directory that contains the param.pos_name file.

You may also want to fool around with these parameters.

param.type = 'MCRDS';
param.file_idx_start = 25;
%param.file_idx_stop = 99;
param.file_idx_stop = 26;

These correspond to the data files *.0025.raw to *.0099.raw in /home/polargrid/PolarGRID/raw/. The above just runs the code with 2 input files (should take about 3 minutes) instead of 75.

I also had to use these parameters below.

param.sched_type = 'local';
%param.sched_type = 'jobmanager';
param.sched_name = 'localhost';
%param.sched_name = '';

You can leave these as is for now. I also had to add the "exit" command to the end of the main script for force matlab to exit. Otherwise running the command script with the -r option (see below throws you back to the >> command prompt.

Running the Code
Do this from the /code/ directory. Run the command

matlab -nodisplay -nodesktop -r ParamMCRDS_IU

Don't use redirected standard inputs to read the matlab file (i.e. don't use "matlab < ParamMCRDS_IU.m") or you'll get

Warning: Type-ahead buffer overflow.
In distcomp.abstractjob.waitForState at 92
In StartScriptCluster at 28

See bottom of
Also, the code won't create the final jpegs correctly this way. So...don't do that.

Where's the Data?
The output files go to the directory you specified above (PGOut). The final jpegs are placed in


along with the header files (metadata, .txt) and .mat files.

Polar Grid Matlab Note

If you get the error

[matlab@polargrid003 ~]$ /usr/local/matlab/bin/matlab
Warning: Unable to open display , MATLAB is starting without a display.
You will not be able to display graphics on the screen.
License checkout failed.
License Manager Error -95
MATLAB is unable to connect to the license server.
Make sure you can resolve the hostname of your machine.
If you are unable to resolve the hostname, contact your System Administrator.

Troubleshoot this issue by visiting:

Diagnostic Information:
Feature: MATLAB
License path: /usr/local/matlab/etc/license.dat:/usr/local/matlab/etc/*.lic:
FLEXnet Licensing error: -95,378. System Error: 115

This just means that the matlab license server isn't running. Start it as the matlab user with the command

/usr/local/matlab/etc/lmstart -l /tmp/eeee

Monday, February 18, 2008

More with Shindig and Open Social

Get Going
  • After svn checkout, cd to $HOME/shindig/java/gadgets (or wherever you installed).

  • Run "mvn package" and then "mvn jetty:run-war" from this directory.

  • Point your browser at


    or similar. Note this is an "aggregating client container" for collecting and displaying gadgets. The gadgets are loaded as XML from a specified remote URL and arranged using a layout manager. However, these containers don't know anything about your social networks--they are just for presentation and organization of content. If you actually want to host your own social gadgets you will need to deploy them into the "gadget hosting container"(e.g. Orkut or some other app). This is available on your Shindig server at


  • Note the aggregating client container just shows how to add gadgets from multiple providers. Any social networking information is NOT controlled or accessible by these containers. To run your own gadget and manipulate social network information (such as a user's profile and list of friends), you will need to run your gadget in a container.

  • If you want to add your own sample HTML file for aggregating gadgets, you need to put these in $HOME/shindig/javascript/container/. The maven package command will move these files into java/gadgets/target/gadgets/files/container/myjunk.html.

    You'll then have to shutdown and restart jetty.

    This is a little non-intuitive directory layout but see java/gadgets/pom.xml.

  • The $HOME/shindig/javascript/README is a good place to start.

Hosting your own gadgets.
  • Place your widget code in


    or any subdirectory (like /examples/).

  • After compilation, these will be located in


    or any subdirectory (/examples/ in the shindig SVN).

  • These will be served up from


  • You can use the XML from the sample widget at

  • Just download or cut and past this XML file onto your local file system.

  • For some reason, you will need to change the top:

    <ModulePrefs title="Title of Your Application">
    <Require feature="opensocial-0.6"/>

    <ModulePrefs title="People API Howto">
    <Require feature="opensocial-samplecontainer"></Require>

    This seems to have been corrected in newer versions. Check the header of the sample gadget and use this as a guideline for your gadget.
  • Anyway, load the samplecontainer URL (http://localhost:8080/gadgets/files/samplecontainer/examples/myapp.xml) in your browser, and then load your gadget by pasting in the URL

  • If you edit your gadget in any way, you must (apparently) restart your Jetty server.

This gadget is now ready to be displayed in any gadget container.

Putting Your Gadget Into Your Display Container
In the last previous two steps, you a) displayed existing gadgets using the gadget display container and b) set up your own gadget hosting container. The documentation doesn't very clearly distinguish these, but you should see this by now. The display container can load any gadgets (including production gadgets available from iGoogle). The gadget hosting container, on the other hand, is used to run your gadgets with your social network data.

We can now combine the two. Edit any of the sample HTML files in java/target/gadgets/files/container, changing the URL to point to the gadget you made in the previous example. Restart your Jetty server and you should see it correctly displayed.

Similarly, you should be able (if your Jetty server's URL is publicly accessible) to add your social gadget to other display containers.

Sunday, February 17, 2008

Quick Start with Open Social, Shindig

OpenSocial is of course the Google-led consortium of competitors to FaceBook, but until very recently most of its efforts appeared to be vaporware. This is starting to change.

One of my immediate concerns about the OS API was how easy it would be to build an OS compliant container. Otherwise, we would have to rely upon existing containers (Orkut, anyone?) to host our OS-based applications. Fortunately, the Apache Shindig incubator project ( is something of an open source reference implementation for the OS API (written in Java with a Maven build no less).

Check it out with SVN and build from source with Maven (worked perfectly on version 628525). Run with Jetty server.

Point browser to http://localhost:8080/gadgets/files/container/sample1.html and look through other samples. You can also load these HTML files using file:// if you don't want to run the Jetty server.

Next time: can I use OS clients to query my LinkedIn profile?

Friday, February 15, 2008

A Little More JavaScript and FaceBook Hacking

Facebook's JavaScript example is pretty minimal, so I thought I'd do a little more hacking around. I'm not a great JS programmer, so I like to unroll the function definitions.

You should grab the very minimal JS API documentation from here: It is the "zipped archive" link near the bottom. This guide is very unfriendly at the current time, but in general you should see that the JS equivalents of the API list at replace the "." with a "_". So users.getInfo becomes users_getInfo(...).

Here's an example that uses the users_getInfo() method and some other stuff to get a user's profile information.

<!-- Output area to show the output from Facebook API -->
<textarea style="width:500px;height:300px;" id="_traceTextBox">
<script src="" type="text/javascript">
<script type="text/javascript">

// Create an ApiClient object, passing app's api key and
// a site relative url to xd_receiver.htm
var api = new FB.ApiClient('<your_key_here>', '/xd_receiver.htm', null);

// Get friends list
function getResults(result,exception) {
Debug.dump(result,'The stuff');

function getInfo(result,exception) {
Debug.dump(result,'The info');

// require user to login
api.requireLogin(function(exception) {
var myinfo='last_name,first_name,hometown_location,work_history,pic_small';


You need to update this stuff above to use your application's key, as indicated.

The output of this will be something like
you: 627774031

The stuff: {Array}

The info: {Array}
[0]: {Object}
first_name: Marlon
hometown_location: {Object}
last_name: Pierce
uid: 627774031
work_history: {Object}

friendsResult from batch execution: {Array}
notificationsResults from batch: undefined

I guessed the values for the myinfo array by looking at the big XML file example shown at That's the real URL for the tiny version of my FaceBook photo.

So in summary, you can use this approach to embed FaceBook profile information in your own web application. You'll of course have to do a bit of work to format all of this stuff. Note also that you can substitute another user's numeric ID in the users_getInfo() call (ie that of one of your friends). I used this to spy on Dave De Roure, who spends a lot more time on FaceBook than I do. And I won't even start on Dan Katz.

FQL and JS
Finally, note that all of the FaceBook API is basically a set of wrappers around their SQL like FQL query language. So you can, if you prefer, make custom methods out of FQL query strings. For example

var myQuery='SELECT name FROM user WHERE uid='+api.get_session().uid;

will execute the indicated query string from JavaScript and pass it to your (developer defined) function getFQLResponse().

To list information about groups you are a member of, use the following. You can replace your uid with another UID.

In the second line, 18629081888 is the GID for the very inactive OGF Web 2.0 group.

Thursday, February 14, 2008

Quick Facebook JavaScript and PHP API Notes

Took a quick look at and found a few obfuscations. To get started, you need to do these things (described a little too briefly in the link above).
  • Register for an application key at (Don't hit enter yet)
  • Cllick the "Optional Fields" drop down and
    • Provide a callback URL (that is, the full URL of your HTML page containing the Facebook JavaScript application).
    • Choose "IFrame" as the canvas page URL.
When it is working, this little app just prints out your friend's Facebook IDs (numbers) onto the screen. When you load your page (, you will be redirected to Facebook to login and then will go back to your URL.

Also took a quick tour of the PHP examples ("footprints" and some hello-world type stuff). Looks like the instructions for setting up the JS example assumed you were familiar with doing things the PHP way.

For footprints, you will need a web server (apache), PHP, and MySQL. Get, install, and start these in the usual ways. Hints: for Macs, get the nice disk image from MySQL. Also, read the PHP installation instructions before installing Apache, since PHP will require some non-default configuration options (particularly DSO support).

The web server provides the callback URL as well (""), which will need to be on a world accessible Apache (or other) server. Dump your application in htdocs/footprints and make sure paths are correct (index.php needs to point correctly to facebook.php). Note also that will be taken, so make a different name. The facebook app name does not have to match your callback app's name.

Create the MySQL database. The footprints PHP scripts don't do this for you.
  1. Edit config.php. You will need the root password for your db.
  2. From the command line, log into your db with /usr/local/mysql/bin/mysql -u root -p and provide the root password at the prompt. You will then need to create and switch to the footprints database.
  3. mysql> CREATE DATABASE footprints;
  4. mysql> USE footprints
  5. mysql>CREATE TABLE `footprints` ( `from` int(11) NOT NULL default '0', `to` int(11) NOT NULL default '0', `time` int(11) NOT NULL default '0', KEY `from` (`from`), KEY `to` (`to`) );
Step 5 is just a cut and paste of the config.php file from footprints, but note that they omitted the final ";" on the statement.

When it is working, this little application allows you to step on your friends. It gives you a little text area for typing names that will be autocompleted as you type. Note this must run inside Facebook, unlike the previous example.

When setting up your PHP app, you'll notice you can choose between FBML and IFrames. FBML will (as you might have guessed) a nice FaceBook styled application and some extra goodies. The IFrame option will produce a pretty bare bones footprint application, but obviously you have more power to customize the styles.

Hosting Your Services
Here are some instructions on using Amazon Web Services to host your application.

Tuesday, February 12, 2008

Portals and Web Applications

There are many tools for building web applications and portals, so it is useful try to classify these a bit. The field also suffers from overlapping terminology ("framework" providing the worst example) and overlapping functionality. I'll take an informal stab at trying to organize these things.

Programming Tools and Languages: this is the simplest group (hopefully) to define. PHP, PERL/CGI, Java Server Pages and Servlets are all examples. One could argue that JSP is more of a development framework (below), but these aren't sharp definitions. I'll err on the side of usage--most people don't write JSP using beautiful MVC architected code.

Goodies and Widgets: these are useful little or medium-sized extensions that can be easily plugged into several different development environments. PEAR/PHP and Java Tag Libraries seem to be best examples on the server side. Various JavaScript libraries (such as YUI) are good examples of client-side applications. These can be embedded into several different toolkits and frameworks. These are distinct from development framework modules (i.e. Ruby on Rails modules) since they are meant to work with both Programming Tools (above) and more specific, derived Development Frameworks (below). Thus one can put useful tags into JSP as well as JSF.

Development Frameworks: Frameworks are typically one level of abstraction above the common programming language tools. Java is full of these: Struts, JSF, Velocity, etc. Ruby on Rails is another famous example. DWR, Google Web Toolkit, and other tools also fall in here. ASP.NET and various compatible, open source implementations like Mono go here.

Frameworks typically encode a development methodology (MVC) and provide an extensibility framework so that all developers create code more or less in the same way. Frameworks thus allow you to develop reusable and useful little modules that can be shared with other developers using the same framework.

A framework can be used to build standalone applications, but it can also be used to develop a rich, self contained component that can be deployed into a container. In contrast, a container (after installation) comes up immediately with useful tools such as login modules, layout managers, and some user tools (say, a calendar or a blog authoring tool). In this example, one may use JSF (a framework) to build a web calendar or a web email application, but one would also typically want to embed these into a more comprehensive container (a portlet container in the Java world).

Application Containers: Joomla (php), Zope (python), Mambo, and Drupal (php) are examples. Java JSR 168 containers and various similar efforts also fit here: GridSphere, LifeRay, uPortal, Jetspeed2, etc. Sakai is another example--it supports the Java standard but also has its own component model. Interestingly, the non-Java containers mentioned above all began life as content management systems.

Arguably only Java's portlet standard really takes the container philosophy completely seriously. Joomla et al, really don't care about portability of codes across container vendors, although they do obviously care about shareable community developed modules. All Java portlet container providers (except Pluto, the reference implementation) don't just give you a container for managing your own portal components--they also give you lots of built in functionality and other goodies (calendars, chat tools, document managers, etc). LifeRay and Sakai are good examples of this.

Related Things: Web-based containers for content and document management, blogging, wikis, and learning management are related closely to portals. There are many different content management systems. Learning management varies from Moodle to Blackboard. Blogging is often provided as an online service (such as Google's Blogspot), but you can also install and run your own blogging service. WordPress is a prominent example of blogging software. Many frameworks (such as Drupal) come with blogging modules. Wikis are similar to content management and document management systems. The best known wikis (MediaWiki and Twiki) are standalone tools, but many containers come with Wiki modules (Sakai for example).

Services for Portals: One of the dangers of portal tools at any level is that they can become monolithic. The danger here is that you will be stuck with inflexible technology and can't respond to changing requirements. The current Enterprise vs. Web 2.0 conflict provides a compelling example. Many famous Web 2.0 sites (flickr, youtube, etc) are Web Services as much as they are Web sites, so modern portals must be able to easily integrate (rather than compete) with these large third party services.

External services: there is a great temptation of course to bundle everything into a particular portal for simplicity of installation and maintenance. However, this is not a good plan in the long term since you will (for example) tie your managed content to a particular tool rather than a portable standard.

When choosing a framework or container, a good starting point in the evaluation process is to see if it supports clients to the following standard services.
  • Authentication and Authorization: these are often tied to LDAP, CAS, or (in the Grid world) MyProxy. Shibboleth is an authorization mechanism. The basic idea is that the portal needs to go to an external database to get the login information. This DB is independent of the portal and may have other clients.
  • RSS/Atom feeds are particularly important in Web 2.0 applications, since most of them publish syndication in some form or another. Many containers provide RSS readers. This is any easy way to incorporate external content (say, YouTube videos) into a portal.This is particularly important topic in modern education portals, as for example one may embed publication feeds from Connotea and CiteULike, slides from SlideShare or ImageLooop, SciVee or YouTube for scientific visualizations and online lectures, etc. As can be seen from the previous list, there are dedicated, high quality online services for many education and science portal content that can be incorporated into a portal pretty generically as RSS/Atom. It is a bad idea to reinvent this stuff or tie it too closely to your portal framework.
We may also turn this problem upside down: if you run a resource management portal,
you must be able to export that portal's content to other user interfaces such as iGoogle
and NetVibes that users are more likely to use every day.
  • Calendars: Collaborative calendars are obviously best extracted from portals since there are so many calendar clients. This is also an example of a reversible service: you typically want to synch your online calendar with your local PC or mobile device. A relevant standard is iCal.
  • WebDAV servers for content management. WebDAV is an extension of HTTP and is supported by many Web servers (with the right module, of course).
  • Grid and workflow services for running scientific applications and accessing data.
  • Digital libraries: access to Fedora, etc. Don't reinvent this or jackbooted librarians will be goose stepping into your cubicle.
It is of course inadequate to simply support such services through clients, as the implementation must be useful and enjoyable. On the downside, however, it is also hard to know when one has crossed the line that separates a rich, useful client from a full fledged internal service that can't be easily extracted from the framework. I certainly can think of several examples that I've written.

Social Networking: the other challenge facing education portals is how much to integrate with social networking sites such as FaceBook or LinkedIn.

Polar Grid URL

Using the power of my blog to shift google page ranks, I hereby declare that PolarGrid (aka Polar Grid) has the following official URL:

Damn polar coordinate system.