Wednesday, December 30, 2009

Quick Guide to Using Google's OpenID

These notes are based on some work done by Jun Ji in our group.  He's been converting the QuakeSim project's portlets into Google gadgets as a prelude to Open Social integration, as discussed in previous posts.  One problem is that the gadgets need to identify and authenticate the user. We don't have excessive security requirements.  We just need to save the users' interactions with the web interface as persistent sessions ("projects") so that they can be accessed again in later sessions.

This identification step is actually pretty simple to do with OpenID and openid4java, but you may not guess this from Google's documentation (see for example http://code.google.com/apis/apps/sso/openid_reference_implementation.html).   To start, you will need to download the openid4java tar file (see http://code.google.com/p/openid4java/) and unpack it. The openid4java jars need to go in your webapps's WEB-INF/lib directory, as usual.

Next, you need to copy the provided samples consumer_redirect.jsp and consumer_returnurl.jsp to your webapp's directory. They are in ./samples/simple-openid/src/main/webapp/ of the openid4java download.   You will need to modify them a little: consumer_redirect.jsp for example has a variable returnToUrl that needs to be changed to use your webapp's name. For our other modifications to these codes, see the source code at source forge: http://crisisgrid.svn.sourceforge.net/viewvc/crisisgrid/QuakeSim2/portlets_dev/Disloc3/src/main/webapp/


This completes the setup.  You can now make simple login page like the one below (call it index.jsp or whatever).  You need to do the following:
  1. See if a logout action has been requested.  If so, clean up relevant OpenID session variables.
  2. See if the openid attribute has been set.  If so, give the user an option of logging out or navigating on to other pages.
  3. Otherwise, redirect the user to the Google OpenID login page.  
Here's what the JSP page looks like:
<%@ page session="true" %>
<html>
<body>
<%
    if (request.getParameter("logout")!=null)
    {
        session.removeAttribute("openid");
        session.removeAttribute("openid-claimed");
%>
    Logged out!<p>
<%
    }
    if (session.getAttribute("openid")==null) {
%>
<form method="POST" action="consumer_redirect.jsp">
<strong><br>QuakeSim2 OpenID login test page:<br><br>
</strong>
<input type="hidden" name="openid" value="https://www.google.com/accounts/o8/id" size="60"/><br>
<input type="submit" value="log in into google"/>
</form>
<%
} else {
%>

Logged in as <%= session.getAttribute("openid") %><p>
email address <%= session.getAttribute("email")%><p>
<a href="?logout=true">Log out</a>

<% } %>

</body>
</html>

Note the form action sends the user to the consumer_redirect.jsp page we took from openid4java.  The hidden parameter openid is sent to consumer_redirect.jsp and tells it to redirect to the Google login page. The consumer_returnurl.jsp page acts like a callback page and tells the Google login form where to redirect the browser request if login was successful.  Your future requests will be filtered through this servlet, which can enforce verification.

Finally, note that several session variables will be set. Here are some sample name-value pairs (just use session.getAttributeNames() in index.jsp and list):

openid-disco OpenID2 OP-endpoint:https://www.google.com/accounts/o8/ud ClaimedID:null Delegate:null
email marpierc AT gmail.com
openid-claimed https://www.google.com/accounts/o8/id?id=ABCdefg-hIJKLMNop
openid https://www.google.com/accounts/o8/id?id=ABCdefg-hIJKLMNop


You may want to use these in other applications.  We use the email attribute to identify the user for database purposes.  You may also want to use these variables to associate a successful OpenID login with a legacy user entry in your database.

Monday, December 14, 2009

Getting Started with Friend Connect

This blog post will cover two simple capabilities of Google Friend Connect: creating a social network and rating items in a table.  First, however, let's provide some motivation for science portals.

We've developed the QuakeSim portal for several years to support earthquake modeling research and education. A typical scenario, common in many other gateways, is for a user to set up and launch an application through a Web interface.  Web services are needed to both run the modeling code and preserve metadata about the run: the input parameters, the time the code was executed, the URLs of all the input, output, and error files, and so on.  We organize these metadata fields as projects owned by a particular user.  The metadata is modeled with JavaBeans and stored in object databases, although these are irrelevant technical details here.  It is also possible, and appropriate, to model the metadata as Atom/RSS, for example.

Let's look first at creating a little GFC social network of people who may be interested in looking at the metadata feeds.  You will need to create a Google account if you don't have one and register a URL with GFC.  Just use your development machine's URL, as you would with Google Maps.  Note the Site ID that you are assigned (it will be in the GFC URL of the projects you register).  You'll need to use this in your GFC applications.

To get started, you can generate code for lots of GFC applications.  Just save this generated HTML/JavaScript in files that can be loaded from your Web server.   If you want to use these as standalone pages, be sure to add <html> and <body> tags to the beginning and end of the document.  You can also embed the generated code into existing HTML pages.  To wrap the code as a gadget, just paste it in the CDATA section of an "html" gadget.

First, now, we need a GFC code snippet to manage logins and mini-social networks.  The GFC way for doing this is shown below:

<html>
<body>
  <!-- Include the Google Friend Connect javascript library. -->
  <script type="text/javascript" src="http://www.google.com/friendconnect/script/friendconnect.js"></script>
  <!-- Define the div tag where the gadget will be inserted. -->
  <div id="div-4719709228063302739" style="width:276px;border:1px solid #cccccc;"></div>
  <!-- Render the gadget into a div. -->
<script type="text/javascript">
var skin = {};
skin['BORDER_COLOR'] = '#cccccc';
skin['ENDCAP_BG_COLOR'] = '#e0ecff';
skin['ENDCAP_TEXT_COLOR'] = '#333333';
skin['ENDCAP_LINK_COLOR'] = '#0000cc';
skin['ALTERNATE_BG_COLOR'] = '#ffffff';
skin['CONTENT_BG_COLOR'] = '#ffffff';
skin['CONTENT_LINK_COLOR'] = '#0000cc';
skin['CONTENT_TEXT_COLOR'] = '#333333';
skin['CONTENT_SECONDARY_LINK_COLOR'] = '#7777cc';
skin['CONTENT_SECONDARY_TEXT_COLOR'] = '#666666';
skin['CONTENT_HEADLINE_COLOR'] = '#333333';
skin['NUMBER_ROWS'] = '4';
google.friendconnect.container.setParentUrl('/gadgets-repo/' /* location of rpc_relay.html and canvas.html */);
google.friendconnect.container.renderMembersGadget(
 { id: 'div-4719709228063302739',
   site: '16707400350554686557' },
  skin);
</script>
</body>
</html>

The site ID in the above example is one of my site IDs.  You would need to replace this with your site ID.

This is just generated code from the GFC site.  I only added the (necessary) HTML and BODY tags so that it would work as a standalone  page. Load this new HTML page in your browser and use it to sign in and add people to your network.  If you are already signed in to a Google site (say to iGoogle) on the same browser, then the GFC widget above should detect this through cookies. Note that this network should apply only to a specific site ID, so you can create multiple social networks.  One difficulty here of course is that all of this is external to your legacy portal's authentication system.

Our next example is a rating widget that can be embedded into an existing page.  Recall our motivating problem was an archive list of project metadata associated with a particular user's previous code submissions.   In QuakeSim, we render this information using JSF datatables populated with JavaBeans pulled from an object database, but this detail is irrelevant here.  In the browser, the generated page would look something like the HTML below:

<html>
<body>
<!-- Include the Google Friend Connect javascript library. -->
<script type="text/javascript" src="http://www.google.com/friendconnect/script/friendconnect.js"></script>
<!-- Define the div tag where the gadget will be inserted. -->
<table border="1">
  <tr>
     <th>Project</th><th>ID</th><th>Date</th><th>Input File</th><th>Output File</th><th>Recommend</th>
  </tr>
  <tr>
     <td>
        Northridge
     </td>
     <td>
        123456789
     </td>
     <td>
        08:43 GMT, Dec 14, 2009
     </td>
     <td>
        <a href="http://www.quakesim.org/mpierce/northridge/input">Input</a>
     </td>
     <td>
        <a href="http://www.quakeims.org/mpierce/northridge/output">Output</a>
     </td>
     <td>
        <div id="div-1095870898733665271" style="width:100%;"></div>
     </td>
  </tr>
  <tr>
     <td>
        Plieto
     </td>
     <td>
        987654321
     </td>
     <td>
        10:12 GMT, Dec 12, 2009
     </td>
     <td>
        <a href="http://www.quakesim.org/mpierce/plieto/input">Input</a>
     </td>
     <td>
        <a href="http://www.quakeims.org/mpierce/plieto/output">Output</a>
     </td>
     <td>
        <div id="div-123456789" style="width:100%;"></div>
     </td>
  </tr>
</table>

<script type="text/javascript">
var skin = {};
skin['HEIGHT'] = '21';
skin['BUTTON_STYLE'] = 'compact';
skin['BUTTON_TEXT'] = 'Recommend it!';
skin['BUTTON_ICON'] = 'default';
google.friendconnect.container.setParentUrl('/gadgets-repo/' /* location of rpc_relay.html and canvas.html */);
google.friendconnect.container.renderOpenSocialGadget(
 { id: 'div-1095870898733665271',
   url:'http://www.google.com/friendconnect/gadgets/recommended_pages.xml',
   height: 21,
   site: '16707400350554686557',
   'view-params':{"pageUrl":"http://www.quakesim.org/users/mpierce/northridge","pageTitle":"Northridge","docId":"recommendedPages"}
 },
  skin);

google.friendconnect.container.renderOpenSocialGadget(
 { id: 'div-123456789',
   url:'http://www.google.com/friendconnect/gadgets/recommended_pages.xml',
   height: 21,
   site: '16707400350554686557',
   'view-params':{"pageUrl":"http://www.quakesim.org/users/mpierce/plieto","pageTitle":"Plieto","docId":"recommendedPages"}
 },
  skin);
</script>
</body>
</html>


The main thing to notice is that each project has been given a URL.  I use http://www.quakesim.org/users/mpierce as the base--see the pageUrl parameter in the code above. This assumes a REST style programming model for the metadata services, which unfortunately isn't implemented yet (but now I have some motivation to do it...).  The javascript at the end is a slightly modified version of generated GFC code.  The values for the view-params (pageURL, pageTitle) are generated and should be taken from entries in your project database.


Each entry in the table will have a button with "Recommend It" in the final column.  You can get a list of recommended projects in a page like the one below (again, this is generated by an online GFC tool):

<html>
<body>
<!-- Include the Google Friend Connect javascript library. -->
<script type="text/javascript" src="http://www.google.com/friendconnect/script/friendconnect.js"></script>
<!-- Define the div tag where the gadget will be inserted. -->
<div id="div-6585877907190446397" style="width:300px;border:1px solid #cccccc;"></div>
<!-- Render the gadget into a div. -->
<script type="text/javascript">
var skin = {};
skin['BORDER_COLOR'] = '#cccccc';
skin['ENDCAP_BG_COLOR'] = '#e0ecff';
skin['ENDCAP_TEXT_COLOR'] = '#333333';
skin['ENDCAP_LINK_COLOR'] = '#0000cc';
skin['ALTERNATE_BG_COLOR'] = '#ffffff';
skin['CONTENT_BG_COLOR'] = '#ffffff';
skin['CONTENT_LINK_COLOR'] = '#0000cc';
skin['CONTENT_TEXT_COLOR'] = '#333333';
skin['CONTENT_SECONDARY_LINK_COLOR'] = '#7777cc';
skin['CONTENT_SECONDARY_TEXT_COLOR'] = '#666666';
skin['CONTENT_HEADLINE_COLOR'] = '#333333';
skin['HEADER_TEXT'] = 'Recommended stories';
skin['RECOMMENDATIONS_PER_PAGE'] = '5';
google.friendconnect.container.setParentUrl('/gadgets-repo/' /* location of rpc_relay.html and canvas.html */);
google.friendconnect.container.renderOpenSocialGadget(
 { id: 'div-6585877907190446397',
   url:'http://www.google.com/friendconnect/gadgets/recommended_pages.xml',
   site: '16707400350554686557',
   'view-params':{"docId":"recommendedPages"}
 },
  skin);
</script>
</body>
</html>

Integration Thoughts: the main issue I see in the above example is integrating the GFC authentication and authorization system with your portal. We'll look at this next.  Also, one of the strengths of GFC is the simplicity of the JavaScript code exposed to the end developer.  Gateways have long had a problem integrating in programming-literate computational scientists who just didn't care to learn Enterprise Java and similar things. To retain this advantage of GFC, we would probably need to consider building our own higher level gadget that queries our project database and is rendered with google.friendconnect.container.renderOpenSocialGadget() GFC call.

Thursday, December 10, 2009

Building Simple Social Network Applications with Google Friend Connect

Within the academic community, there has been a push by various groups to provide a "Facebook for Scientists", a "Facebook for Scholars", or similar things.  This is to sell you something, of course. I am skeptical of this approach, since we already have Facebook, and I don't want to join and contribute to yet another social network, especially if I don't know its long term sustainability.

Let's try to follow a systematic approach.  As a starting point, I would like to point you to a nice submission by Roger Curry, Cameron Kiddle and Rob Simmonds to our recent GCE09 workshop. The authors looked at using Facebook, Ning, and Elgg as three social networking platforms.  You know Facebook, and you probably know that Ning is a social networking hosting service.  Elgg was new to me: it is a downloadable software package, like Drupal or Mambo.  Unlike Drupal, however, Elgg is designed from the beginning to be a social networking platform rather than a content management system. 

Curry, Kiddle, and Simmonds's paper presents three detailed case studies.  I would add some additional remarks on requirements.  First, a new social networking site should be compatible with existing social networks and related standards.  I should be able to import in my existing profile and social networking information.  I should also be able to reuse my identity from another site.  I don't want to remember yet another password.  Similarly, I should be able to export my social networking information.  I don't want to get locked in. OpenID, OpenSocial, OAuth, and related standards may make it easier to share one's identity and information across social networking services.

Second, development flexibility is important.  Of the three social networks Curry et al reviewed, Ning's hosted service was generally the least favored. Part of this was because of the advertisements, but it was also noted that access to Ning source code was limited.  If you are considering a hosted service, you need also to consider the development framework that you will be required to use, its strengths, and its weaknesses.

Third, I want to use the best available implementation of a capability, which may not be the one that comes in the software. For example, there are many wikis and calendaring tools.  I want the ability to easily import these tools into a social networking framework.  I don't want to rely upon the wiki or calendar plugin that comes with some particular framework.

For these reasons, I have become very interested in Google Friend Connect (GFC) as a way to build social networking science gateways (and credit goes to Raminder Singh and Gerald Guo in our lab for introducing this to me).  Gadgets are particularly interesting in that they are a client-side technology: any Web development framework can be used to develop gadgets.  We have developed an open source gadget container (see http://www.collab-ogce.org/ogce/index.php/OGCE_Gadget_Container) that can be used to host your own gadgets.   We discuss these things in more detail and provide an implementation case study in our own workshop paper.

I have advocated using Open Social and Google Gadgets for some time (see for example http://grids.ucs.indiana.edu/ptliupages/publications/INGRIDFinal.pdf), but the JavaScript API is not particularly simple, and of course Facebook (the biggest US social networking site) doesn't participate in Open Social.  Thus, while Open Social provides a nice and well documented interface, it is likely that the social network information that it can access is lower quality than Facebook's (at least for US users).

GFC solves those two problems. First, the developer API, although still in JavaScript, is much simpler than Open Social's. The GFC site provides a number of basic capabilities (chat, comment, rating, polling, etc) that have generated HTML and JavaScript code.  You can build up more complicated applications by cutting and pasting these capabilities into a single mash-up, add them to an existing HTML page, or make them into a gadget.  Second, GFC applications are linked to Google's single sign-on system.  Google may not have the social networking depth of Facebook, but they do have untold millions of accounts through GMail, Groups, and other popular online tools.  This allows you to create miniature social networks for a particular GFC application out of your gmail contact list.  Each GFC application is associated with a particular URL (and generated site ID), so you can create multiple networks out of the pool of Google accounts.

In subsequent posts, I'll look at GFC in a little more detail to examine its potential for science gateway social networks.

Thursday, December 03, 2009

Mass Imports of Maven Repositories into Archiva

Credit goes to Fugang Wang for this crafty trick.  Archiva is a Web-based system for managing your Maven repositories.  Unfortunately the Web interface only allows you to upload one jar at a time, and you also need to provide metadata (artifactId, version, group, etc).  This isn't very useful if you need to import a few dozen or more jars.

Here is how to import an existing local repository (that is, your $HOME/.m2 directory) into Archiva:

1. Create a new named repository, say cyberaide. Then you'll get a directory 'cyberaide' under the archiva installation directory.
2. Move the whole repo directory under the newly created directory.
3. Click the 'Repository scanning' button for your repository. This will 'repair' the artifacts db so the newly added artifacts would show up.

Thursday, July 23, 2009

Karajan and GridFTP

I owe big thanks to Mihael Hategan for some help here.

Karajan is a workflow system that is part of the Java COG Kit and also powers the Swift system. You can get it in the 4.1.5 version of the COG Kit, but this version is out of date, so it is better to get the code from the SourceForge SVN:

View it: http://cogkit.svn.sourceforge.net/viewvc/cogkit/trunk/current/src/cog/

Get it: svn co https://cogkit.svn.sourceforge.net/svnroot/cogkit/trunk/current/src/cog

Compile with ant. Here's an example script that will run gridftp on a remote host and return the file listing and metadata to your screen. Run it with $COG_INSTALL_PATH/bin/cog-workflow gridftpExample.xml:

<project>
<include file="cogkit.xml"/>

<set name="l">
<file:list dir="." host="gridftp-hg.ncsa.teragrid.org" provider="gridftp"/>
</set>
<for name="file" in="{l}">
<set names="type,permissions,size,modified">
<file:info file="{file}"/>
</set>
<print message="{file} Type:{type} Permissions:{permissions} Size:{size} Modified:{modified}"/>
</for>
</project>


Mihael actually added some of these fields on my request (they are part of the underlying code).

Karajan also has its own scripting language that is much less awkward than XML for expressing the scripts:

import("sys.k")
import("task.k")

element(niceType, [type]
if (
type == FILETYPE:FILE, "File"
type == FILETYPE:DIRECTORY, "Directory"
type == FILETYPE:SOFTLINK, "Softlink"
type == FILETYPE:DEVICE, "Device"
"Unknown"
)
)

files := file:list(dir=".",host="gridftp-hg.ncsa.teragrid.org",provider="GT2")

for(f, files,
[type, perm, size, modified] := file:info(f)
print("name: {f}, type: ", niceType(type),
" ({type}), size: {size}, modified: {modified}, permissions: {perm}")
)

Tuesday, July 07, 2009

Blogger, Google, and Bing

I noticed my inflammatory post on how to use Gaussian on the TeraGrid (http://communitygrids.blogspot.com/2009/07/running-gaussian-on-big-red.html) is the #3 match if you google "gaussian teragrid", but it doesn't make the Top 50 if you use Bing. I suppose Google is weighting the search results to direct you to Blogger/Blogspot content.

Monday, July 06, 2009

Running Gaussian on TeraGrid

These are notes for running Gaussian serially on the TeraGrid. I'll look at IU's BigRed and NCSA's Abe, Mercury, and Cobalt. Surprisingly this wasn't documented anywhere that I could find with Google (I had to resort to the Help Desk, which was helpful but this is missing the point). And of course running things on each machine requires a different incantation. You may wonder why the TeraGrid doesn't make its environments more consistent for these common applications. I suggest posting hypotheses on this subject as comments.

IU's Big Red
Gaussian is a famous piece of Quantum Chemistry software. I'll assume you have an input file, your_input.inp.

1. Request to be added to the Gaussian group. I found this through non-standard routes (thanks to Ray Sheppard) but try contacting the TeraGrid help desk. You have to do this before you can proceed.

1.5 Add gaussian to your .soft file. For more on the SoftEnv, see http://www.teragrid.org/userinfo/jobs/environment.php

My .soft file looks like this:
more ~/.soft
#
# This is the .soft file.
# It is used to customize your environment by setting up environment
# variables such as PATH and MANPATH.
# To learn what can be in this file, use 'man softenv'.
#
#
@bigred
@teragrid-basic
@globus-4.0
@teragrid-dev
+gaussian
----------------------
Either type "resoft" or just exit and login again.

2. Make a Loadleveler script like the one below:
---------------
# @ output=stdout.txt
# @ error=stderr.txt
# @ wall_clock_limit=5:00:00
# @ account_no=YOUR_ACCOUNT_NUMBER
# @ queue=default

export g03root=/N/soft/linux-sles9-ppc64/gaussian/g03-d.02
. $g03root/g03/bsd/g03.profile


g03 $HOME/your_input.inp $HOME/your_output.out
---------------

A real script would move stuff in and out of scratch space and stuff like that (Gaussian creates very large files while you run it), but I intend to automate this through Globus+Condor-g. Also this only will submit serial jobs.

3. Submit with llsubmit, track with llq. The IU Knowledge Base has a nice comparison of Loadleveler and PBS commands and directives here: http://kb.iu.edu/data/axpz.html

NCSA's Abe

Unlike the other machines, you definitely do not want "+gaussian" in your .soft file on abe. This will produce the following error:

g03: error while loading shared libraries: /usr/local/intel/mkl/10.0.3.020/lib/em64t/libmkl.so: invalid ELF header

Make a PBS script like the one below. Again this is only a serial job.
[mpierce@honest2 ~]$ more gaussTest.pbs
#PBS -o stdout.txt
#PBS -e stderr.txt
#PBS -A YOUR_ACCOUNT_NUMBER
#PBS -q normal
#PBS -l walltime=05:00:00

soft add +intel-mkl
setenv g03root /usr/apps/chemistry/gaussian/g03
source $g03root/g03/bsd/g03.login
g03 $HOME/your_input.inp $HOME/yourOutput.out
----------------------

Alternatively, I could have added intel-mkl to my .soft file. I had to get the NCSA Help Desk to tell me this (thanks go to John Estabrook @ NCSA for a quick and accurate response to my ticket). They pointed me in the right direction, but why not just have this documented?

NCSA also has a command-line tool, /usr/local/bin/qg03, that will make and submit a better PBS script than the one above.

globusrun -o -r grid-abe.ncsa.teragrid.org/jobmanager-pbs '&(executable=/u/ncsa/mpierce/gaussian.abe.ncsa.sh)(arguments=/u/ncsa/mpierce/input.in /u/ncsa/mpierce/mepjunk12.out)(project=YOUR-TG-ACCOUNT)(queue=debug)(host_types=himem)(host_xcount=1)(xcount=8)'

or

globusrun -o -r grid-abe.ncsa.teragrid.org/jobmanager-pbs '&(executable=/u/ncsa/mpierce/gaussian.abe.ncsa.sh)(arguments=/u/ncsa/mpierce/input.in /u/ncsa/mpierce/mepjunk12.out)(project=YOUR-TG-ACCOUNT)(queue=debug)(count=1)(hostCount=8)(minMemory=16000)'

I think these are equivalent.

NCSA's Mercury

This still works but who knows how long it will be before the system/application admins break it. As with Abe, best to at least use the qg03 command (it should be in your path) to generate a sample PBS script.

1. Add "+gaussian" to your .soft file (in $HOME) as above.

2. Make a PBS script like the one below. Again this is only a serial job.
[mpierce@honest2 ~]$ more gaussTest.pbs
#PBS -o stdout.txt
#PBS -e stderr.txt
#PBS -A YOUR_ACCOUNT_NUMBER
#PBS -l walltime=05:00:00

g03 $HOME/your_input.inp $HOME/yourOutput.out
----------------------

3. To invoke this with Globus clients or equivalent, you should use a command like

globusrun -o -r grid-abe.ncsa.teragrid.org/jobmanager-pbs '&(executable=/u/ncsa/mpierce/gaussian.cobalt.ncsa.sh)(arguments=/u/ncsa/mpierce/input.in /u/ncsa/mpierce/mepjunk12.out)(project=YOUR_ACCOUNT)(queue=debug)(minMemory=1600)'

or

globusrun -o -r grid-abe.ncsa.teragrid.org/jobmanager-pbs '&(executable=/u/ncsa/mpierce/gaussian.mercury.ncsa.sh)(arguments=/u/ncsa/mpierce/input.in /u/ncsa/mpierce/mepjunk12.out)(project=YOUR_ACCOUNT)(queue=debug)(host_types=himem)(host_xcount=1)(xcount=1)'

NCSA's Cobalt
This has changed and so previous instructions (identical to running Gaussian on Mercury above) no longer work. *DELETED TIRADE*. Setting +gaussian in your .soft no longer seems to work. Use the following minimal scriptlet

#!/bin/csh
setenv g03root /usr/apps/chemistry/gaussian/G03/pp5_e01
source $g03root/g03/bsd/g03.login
$g03root/g03/g03 $1 $2

You can invoke this with a globus run command like the one above.

PSC's Pople
You have to fill out a PDF form and mail or fax it to PSC to get access to Gaussian. I did not pursue this.

Friday, July 03, 2009

Error in gsi-ssh and ssh on Mac 10.5: percent_expand: NULL replacement

I found this error while trying to log onto the TeraGrid ("Tara Grid") with gsissh (google "teragrid single sign on" for context).

percent_expand: NULL replacement

Apparently this is a well-known bug with Mac 10.5's ssh. The workaround is to append "-i ~/.ssh/id_rsa" or "-i ~/.ssh/id_dsa" to the command line, like so:

gsissh login-hg.ncsa.teragrid.org -i ~/.ssh/id_rsa

I got this from http://www.nabble.com/ssh%3A-percent_expand%3A-NULL-replacement-to13439207.html

Tuesday, June 30, 2009

Some CML Fun

These are some notes on some initial work to create a chemical properties pipeline. End result will be calculated properties in an RDF triple store that can be exported using ORE/Atom. If I don't write this stuff down, I'll forget it.

Step 0: get Jumbo CML converter tools from SourceForge. Peter Murray Rust was kind enough to write a little README, now in the SVN. I did an SVN checkout rather than download the tagged Jumbo versio 5.4 or 5.5.

svn co https://cml.svn.sourceforge.net/svnroot/cml/jumbo-converter/trunk jumbo-converter

Step 0.5: Compile as an executable jar. I used "mvn assembly:assembly". See http://www.springone2gx.com/blog/scott_leberknight/2008/06/creating_executable_jars_using_the_maven_assembly_plugin.html


Step 0.75: Run with sample data using the command (must set classpath):

java -jar target/jumbo-converters-jar-with-dependencies.jar -converter org.xmlcml.cml.converters.molecule.pubchem.PubchemXML2CMLConverter
-sd examples/input -odir ../output -is pubchem.xml -os pubchem.cml


Step 1: get XML descriptions of molecules from Pubchem. I'm not a chemist, so I did a structure similarity search on caffeine, applied the Rule of Five, and got 1298 matches. That's pretty good.


Step 2: Unfortunately the PubChem download gives everything in one big XML file, but Jumbo's converter expects each molecule to be in a separate file. I googled around and made a little Perl script to do this:

#!/usr/bin/perl

use File::Basename;

$file = @ARGV[0];

open(_FH, "< $file") or die "Unable to open file: $file\n";

$count=0;
$max_records=1;
$files_counter=0;

while(<_FH>)
{
if($count == 0)
{

my @suffix=qw(.pubchem.xml);
my ($basefilename, $path, $suffix)=fileparse($file,@suffix);
$filename=$basefilename . "_part_" . $files_counter . $suffix;
open(FH2, "> $filename") or die "Unable to open file: $filenam
e\n";
$count++;
print FH2 "\<\?xml version=\"1.0\"\?\>\n";
}

if (grep /<\/PC-Compound>/, $_ )
{
print FH2 $_;
$count++;
}
elsif (grep /<PC-Compound>/, $_ )
{
print FH2 "<PC-Compound xmlns=\"http://www.ncbi.nlm.nih.gov\"
xmlns:xs=\"http://www.w3.org/2001/XMLSchema-instance\" xs:schemaLocation=\"http:
//www.ncbi.nlm.nih.gov ftp://ftp.ncbi.nlm.nih.gov/pubchem/specifications/pubchem
.xsd\">\n";

}
else {
print FH2 $_;
}

if ($count == $max_records + 1)
{
$count = 0;
$files_counter++;
close(FH2);
}
}

I stole a lot of this from http://www.mysysad.com/2008/01/parse-xml-records-with-perl-script.html


Step 2.5: Rerun the converter command on the new files. Jumbo was able to handle most but barfed on a few, so I'll have to follow up on this.

Step 3: Go to lunch. Next need PMR and Co. to decide on the best way to change the CML into Gaussian input files.

Step 4: Realize you downloaded a bunch of 2D structures, so repeat steps 0-3 with 3D structures.

Step 5: Make Gaussian input files from CML with

java -jar target/jumbo-converters-jar-with-dependencies.jar -converter org.xmlcml.cml.converters.compchem.gaussian.input.CML2GaussianInputConverter -sd examples/output -odir ../gaussian -is pubchem.cml -os pubchem.gauss

Current SVN of jumbo-converters won't compile without first checking out and compiling also cmlxom and jumbo5 from SVN (they are peers in the directory structure).

Step 6. Run Gaussian on TeraGrid machines. See http://communitygrids.blogspot.com/2009/07/running-gaussian-on-big-red.html. Welcome to 2009!

Step 7. Get the data back and convert the Gaussian standard output to CML. I used the command

java -jar target/jumbo-converters-jar-with-dependencies.jar -converter org.xmlcml.cml.converters.compchem.gaussian.log.GaussianLog2CMLConverter -sd examples/gaussOut -odir ../gaussOut -is out -os cml

I put the Gaussian output files in the examples/gaussOut directory.

Step 8. Convert the output CML to RDF:

java -jar target/jumbo-converters-jar-with-dependencies.jar -converter org.xmlcml.cml.converters.compchem.gaussian.GaussianCML2OWLRDFConverter -sd examples/gaussOut -odir ../gaussOut -is cml -os rdf

Wednesday, June 24, 2009

Jobs at the Pervasive Technology Institute

We are looking to fill two positions. Please see the full ad below.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The following positions based at IUB and reporting to Marlon Pierce, Assistant Director, Community Grid Labs, is being posted externally. If interested, you must apply online at http://www.jobs.indiana.edu/. Search for job listing 401 and refer to positions number #00034420 and #00034425. See also https://ola.indiana.edu/joblisting/index.cfm?jlnum=401&search=2

Principal Software Research Engineers - PAE4IT (Two positions available)

Description and Responsibilities:

Indiana University is looking for experienced individuals to fill multiple positions in Cloud computing , Grid computing, and message-oriented middleware development. The positions call for expertise in Web service software development, experience developing Java Messaging Service (JMS) applications and related Enterprise event bus technologies, experience with distributed systems, sensor webs, and/or audio-video systems research and advanced development.
Working independently, the incumbent provides expertise in the evaluation, development, implementation and deployment of specific Cloud Computing, Message-Oriented Middleware, and Grid-related technologies in conjunction with Pervasive Technology Institute and Research Technology staff and collaborators. Investigate research problems in distributed computing systems, develop software using best software engineering practices, publish peer-reviewed research articles in workshops and journals, evaluate existing technologies, supervise staff and graduate students, and oversee all development aspects of Cloud and Grid scientific data and application management. Work with scientific users to develop Cloud and Grid systems. Collaborate with key academic researchers and open source user community including faculty, scientific researchers, and grant stakeholders.
Indiana University's Pervasive Technology was recently awarded a 5 year extension to its core funding from the Lilly Endowment, following its highly successful initial 7 years (as the Pervasive Technologies Laboratories). PTI will be located in Indiana University's new state of the art Research Incubator facility. See http://www.pervasive.iu.edu/ for more information. To date, the PTI laboratories have published more than 880 peer-reviewed papers, have won over $180,000,000 in grants over and above their core funding, and have released and maintain numerous open source software packages. PTI researchers work closely with Indiana University's Research Technology group, who maintain IU's world class cyberinfrastructure, http://racinfo.indiana.edu/cyberinfrastructure/.
Qualifications:

Master's degree in Computer Science or relate technical discipline (such as physics, informatics, engineering) is required. PhD preferred. Ph. D. incumbents must have demonstrated (through peer-reviewed publications and software products) a primary research focus on one or more of the following: a) distributed, parallel, Grid and/or Cloud computing; b) high performance, distributed messaging systems; c) audio/video collaboration systems; and d) security in distributed systems. Incumbents with Master's degree must have five to seven years experience in software development and deployment in the appropriate field or fields.

Requirements include a proven ability to undertake and succeed in innovative research projects, and excellent communication skill (covering scientific article writing, presentation skills, and the ability to work with and lead teams). Experience developing software systems using standard software engineering practices (such as source code management software, build and test systems, etc). For Cloud and Grid incumbents, experience developing Web Services, REST services, scientific workflows, and distributed Web applications on Unix/Linux/Mac OSX and/or Windows. Experience developing applications on Grids using software such as Globus or Condor. Experience developing scientific and research Cloud applications using Amazon Web Services, Microsoft Azure, Google AppEngine, and related systems. Experience developing data-parallel scientific, machine learning, and/or information retrieval algorithms using systems such as Apache Hadoop and Microsoft Dryad. For audio/video stream incumbents, demonstrated knowledge and implementation of standards as well as innovative systems. For message-oriented middleware incumbents, demonstrated sophisticated experience with standard MOM software. MOM incumbents should have experience both with core MOM development (efficient routing, security, reliable delivery) as well as applications.
Ability to foster and maintain collaborations with internal and external academic researchers through a balanced, service-based approach required. Demonstrated ability to learn new technologies and tools. Demonstrated initiative in evaluating and adopting new technologies. Organizational and leadership skills required. Able to operate effectively in a complex and dynamic environment with the capability of functioning as a project leader. Excellent oral and written communication skills. Demonstrated conceptual, analytical, and logical abilities. Ability to interact successfully with staff and faculty in extended consultations; demonstrate enthusiasm for new technologies; and deliver excellent outcomes under high pressure and short deadlines. Excellent communication skills required to interact with people of varying levels of knowledge.

Indiana University is an Affirmative Action/Equal Employment institution.

Friday, April 24, 2009

New Pervasive Technology Institutes Web Site

I'm doing my part for page ranking: http://www.pervasive.iu.edu/. This is the Community Grids Lab's new grandparent organization. We remain the same, just have a slightly different container org chart.

Thursday, April 16, 2009

MS Word's Formatting Palette on Mac

To get this, just click "View->Formatting Palette". The pop-up has everything you need (fonts, bullets, pre-defined styles, etc) for formatting your text.

I post this because I'm surprised that more people (Mac users anyway) apparently don't use this.

Monday, April 06, 2009

Old Dog Learns Old Trick

It's never too late. I needed to add Tomcat's FORM authentication to a webapp. The steps for doing this are surprisingly under-documented. Here are they are:

1. Write a login form page (login.jsp). The form action should point to "j_security_check". The input parameters for the form are "j_username" and "j_password". There are plenty of examples of this.


2. Add the following to your web.xml:

<login-config>
<auth-method>FORM</auth-method>
<realm-name>Example Form-Based Authentication Area</realm-name>
<form-login-config>
<form-login-page>/login.jsp</form-login-page>
</form-login-config>
</login-config>

I took this from the jsp-examples/security examples that come with Tomcat.

If you stop here, install your webapp, and try to login by going directly to login.jsp, it will not work. You will get an error like

The requested resource (/mywebapp/j_security_check) is not available

3. You need to also add the following to your webapp's web.xml file (taken from Tomcat's jsp-examples/ again):

<security-constraint>
<display-name>Example Security Constraint</display-name>
<web-resource-collection>
<web-resource-name>Protected Area</web-resource-name>
<!-- Define the context-relative URL(s) to be protected -->
<url-pattern>/*</url-pattern>
<!-- If you list http methods, only those methods are protected -->
<http-method>DELETE</http-method>
<http-method>GET</http-method>
<http-method>POST</http-method>
<http-method>PUT</http-method>
</web-resource-collection>
<auth-constraint>
<!-- Anyone with one of the listed roles may access this area -->
<role-name>tomcat</role-name>
<role-name>role1</role-name>
</auth-constraint>
</security-constraint>


This should work. The dirty trick is that you have to let Tomcat's security constraints redirect you to your login page. You can't point your browser to this page directly.

Friday, February 20, 2009

Word Cloud for My Blog

From Wordle, courtesy of Yu "Don't call me Marie on your stupid blog" Ma.

Friday, February 06, 2009

Some Xen command line stuff

Here's a Xen command line cheat sheet. We'll assume Xen is installed.

1. Make sure you've booted from the Xen kernel.

2. To start the Xen daemon, use "xend restart".

3. To install a new image, use virt-install --paravirt --name vc8 -r 500 -f /var/lib/xen/images/vc5.img --file-size 8 -l http://fedora.fastsoft.net/pub/linux/fedora/linux/releases/8/Fedora/i386/os --nographics

The virt-install man page has more examples and an explanation of the above. The above command installs Fedora 8 on the image. You should be prompted to install the OS in the VM after the download. This can obviously take a while, so you'll want to clone these images.

4. To list vms, you can use "virsh list" or "xm list".

5. To start your VM, use "virsh start vc8". The image "vc8" was the image created above.

6. To connect to the above VM, use "xm console vc8". Replace vc8 with your VM name. This will boot the operating system, which should get you to a login prompt.

Thursday, February 05, 2009

GRAM Job submission failed because data transfer to the server failed (error code 10)

Problem: I had a working pre-Web service GRAM "fork" job manager but then needed to use LSF job manager for submissions to the scheduler on a cluster. The LSF job manager was not built when we deployed globus initially, which is unusual.

The LSF job manager was built with the commands

% gpt-build globus_gram_job_manager_setup_lsf-1.17.tar.gz
% ./setup-globus-gram-job-manager-lsf

However the command line tests didn't work. For example, the command

globusrun -o -r my.secret.machine/jobmanager-lsf '&(executable=/bin/date)'

threw the error

GRAM Job submission failed because data transfer to the server failed (error code 10)

This is unfortunately an all-purpose Globus error. You will sometimes see it associated with problems in the grid-mapfile, but again my fork jobmanager worked fine, so I had a different bug.

Unfortunately nothing useful turned up in the gsi-gatekeeper.log, even after I turned up the logging level.

Solution: the problem turned out to be that the LSF job manager files were not given the correct permissions during the deployment. These should be 755 (group and world readable and executable). Find them with a command like

find $GLOBUS_LOCATION -name "*lsf*"

I then made the changes manually, but you may also do some "find|xargs" trick.

Globus error 123 and Condor_g

Thanks to Stu Martin and Todd Tannenbaum for independently providing the solution below.

Problem: Condor-G submission to a Globus pre-webservice GRAM failed and throws an error
 "Globus error 123 (could not write the job state file)"
This is described in more detail at http://www-unix.globus.org/mail_archive/discuss/2002/11/msg00131.html. You can reproduce it with this globusrun command:

globusrun -r my.secret.machine/jobmanager '&(executable=/bin/date)(save_state=yes)'

Solution: Edit $GLOBUS_LOCATION/etc/globus-job-manager.conf and change the value of the -state-file-dir to point to a local, non-NFS file system. For example:

-state-file-dir /usr/local/gram_job_state

Also, set permissions on this directory:

chmod ogu+rwxt /usr/local/gram_job_state

myproxy-logon error

The problem: my favorite myproxy-logon command suddenly stopped working and I got the error barf below

Failed to receive credentials.
Error authenticating: GSS Major Status: Authentication Failed
GSS Minor Status Error Chain:
globus_gss_assist: Error during context initialization
globus_gsi_gssapi: Unable to verify remote side's credentials
globus_gsi_gssapi: Unable to verify remote side's credentials: Couldn't verify the remote certificate
OpenSSL Error: s3_pkt.c:1052: in library: SSL routines, function SSL3_READ_BYTES: sslv3 alert bad certificate SSL alert number 42

I tried the usual soluiton: made sure my certificates were correct in $HOME/.globus/certificates and /etc/grid-security/certificates but still got the error.

Solution: I had a proxy certificate from another (non-overlapping) Grid sitting around in the default X509 location (/tmp/x509up_nnn). I deleted this and myproxy-logon worked again. Apparently the myproxy-logon command tries to use this credential to authenticate to the MyProxy server. The errors are what you expect since MyProxy from Grid #1 doesn't trust the CA of Grid #2.

Delete the old credential and things should work again.

Wednesday, January 14, 2009

Simple OAuth Client and Server Examples

The OAuth Google code (http://code.google.com/p/oauth/) is woefully undocumented, so I put together below a zeroeth order approximation to how things actually work. To use this, you need both the Open Social Java implementation (for the client) and the OAuth Java implementation. Check them both out from the Google Code SVNs as described in earlier posts.

In the examples below, we will use OAuth's two-legged authentication. This is the server-to-server authentication used by OpenSocial gadgets. It is appropriate when no human intervention is needed (or is possible) in the authentication process. Also, we are going to just look at shared secret key-style authentication, using the HMAC-SH1 algorithm and symmetric key encryption. The OAuth code will also support public/private asymmetric keys.

Finally, note that the steps for generating the OAuth consumer key and consumer secret are out of scope. We will just use the key and secret from the earlier OpenSocial gadget experiments. Generally, shared secrets will need to be communicated through a separate registration process. Presumably, the server would use the consumer key sent by the client to look up the secret key on the server side from a database, but check the OAuth specification for details.

Server Code
We'll implement a dummy server as a JSP. You can put this in an Apache Tomcat server under a webapp called OAuthTest. You will also need to put several jars in webapp/OAuthTest/WEB-INF/lib:

ls apache-tomcat-5.5.27/webapps/OAuthTest/WEB-INF/lib/
commons-codec-1.3.jar jetty-6.1.11.jar
commons-httpclient-3.1.jar jetty-util-6.1.11.jar
commons-logging-1.1.jar junit.jar
httpclient-4.0-beta1.jar oauth-core-20090108.jar
httpcore-4.0-beta2.jar

The above jars are all from OAuth/java/lib, plus the oauth-core jar that you generated when you compiled things (see previous post--use "mvn clean install").

Here at last is the actual JSP for the dummy service. I call this OAuthTest.jsp.

--------------------

<%
//Presumably this should actually be looked up for a given key.  Use a real key.
String consumerSecret="123121212yourkeyhere18918";

//Presumably the key is sent by the client. This is part of the URL, after all.  Use a real key.
String consumerKey="orkut.com:1231212122";

//Construct the message object. Use null for the URL and let the code construct it.
OAuthMessage message=OAuthServlet.getMessage(request,null);

//Construct an accessor and a consumer
OAuthConsumer consumer=new OAuthConsumer(null, consumerKey, consumerSecret, null);
OAuthAccessor accessor=new OAuthAccessor(consumer);

//Now validate. Weirdly, validator has a void return type. It throws exceptions
//if there are problems.
SimpleOAuthValidator validator=new SimpleOAuthValidator();
validator.validateMessage(message,accessor);

//Now what? Generate some JSON here for example.
System.out.println("It must have worked"); %>


-------------------

Client Code

To build an OAuth client, we can start from our earlier OpenSocial client. For convenience, we will leave in the OpenSocialUrl and OpenSocialHttpRequest classes, which help construct and execute the REST invocation, but one could easily eliminate this and use the standard java.net classes that underly these two.

Here's the client code. Save it in a file called MyOAuthClient.java in OpenSocial's java/samples directory.
---------------------
import org.opensocial.data.*;
import org.opensocial.client.*;
import net.oauth.*;
import java.util.*;

public class MyOAuthClient {

public static void main(String[] args) {
MyOAuthClient mosc=new MyOAuthClient();
}

public MyOAuthClient() {
String REST_BASE_URI=
"http://localhost:8080/OAuthTest/OAuthTest.jsp";
String CONSUMER_SECRET=
"uynAeXiWTisflWX99KU1D2q5";
String CONSUMER_KEY=
"orkut.com:623061448914";
// String VIEWER_ID=
// "03067092798963641994";
String VIEWER_ID="08354253340777199997";

try {
OpenSocialUrl requestUrl = new OpenSocialUrl(REST_BASE_URI);
OpenSocialHttpRequest request=new OpenSocialHttpRequest(requestUrl);
requestUrl.addQueryStringParameter("xoauth_requestor_id", VIEWER_ID);
requestUrl.addQueryStringParameter("st", "");

String requestMethod=request.getMethod();
String postBody = request.getPostBody();

OAuthMessage message =
new OAuthMessage(requestMethod, requestUrl.toString(), null);

OAuthConsumer consumer =
new OAuthConsumer(null, CONSUMER_KEY, CONSUMER_SECRET, null);
consumer.setProperty(OAuth.OAUTH_SIGNATURE_METHOD, OAuth.HMAC_SHA1);

OAuthAccessor accessor = new OAuthAccessor(consumer);
accessor.accessToken = "";
message.addRequiredParameters(accessor);

for (Map.Entry p : message.getParameters()) {
if (!p.getKey().equals(postBody)) {
requestUrl.addQueryStringParameter(
OAuth.percentEncode(p.getKey()),
OAuth.percentEncode(p.getValue()));
}
}

//Take a look at the signed URL
System.out.println("Signed REST URL: "+requestUrl.toString());

//Done with signing. Now back to OpenSocialBatch's submitRest()
//Finally, get the response. This is the meat of the getHttpResponse
//method, without the error checking.
request.execute();
}

catch(Exception ex) {
ex.printStackTrace();
}
}
}
------------------
Compile this from the OpenSocial SVN checkout's java directory with the command ant compile-samples.

Now set your classpath and execute. The following UNIX commands will do the trick:
export CP=`echo $HOME/opensocial-java-client-read-only/java/lib/*.jar | tr ' ' ':'`

export CP=`echo $HOME/opensocial-java-client-read-only/java/dist/*.jar | tr ' ' ':'`:$CP


java -classpath $CP:/Users/marlonpierce/opensocial-java-client-read-only/java/samples/bin/ MyOAuthClient


Check the output of your Tomcat server's catalina.out to see if it worked.

Obviously this has all been a simple exercise to see how to get things going and needs a lot of work to make it a real service.


Monday, January 12, 2009

Quick Note on Google Code OAuth SVN Checkout

I'm looking at the Google OAuth code (http://code.google.com/p/oauth). It's pretty light on instructions. To compile the Java checkout from SVN, you also need to check out the Maven repositories:

mkdir oauth-fun; cd oauth-fun
svn checkout http://oauth.googlecode.com/svn/code/java
svn checkout http://oauth.googlecode.com/svn/code/maven

You compile with

cd java (assuming you are in oauth-fun)
mvn clean install

The problem is that my checkout today didn't compile. I had to add the maven repository to the top level pom.xml (the one in oauth-fun/java) like so:

<repositories>
<repository>
<id>oauth-local</id>
<url>file:../maven</url>
</repository>
</repositories>

Sunday, January 11, 2009

Roll Your Own REST with Open Social Clients

As in the last few posts, I've been looking at the OpenSocial REST/RPC client code. This can be used to access (for example) Orkut as a back-end service. The Java clients are here: http://code.google.com/p/opensocial-java-client/. The general (programming-language neutral) API is documented here: http://code.google.com/apis/orkut/docs/rest/developers_guide_protocol.html

The Java client examples show how to do simple stuff like access a person's profile data and his/her friends' information. The primary higher level API for this is the OpenSocialClient.java class. The problem is that this class only implements a subset of the possible REST/RPC commands at this time. Peeling back the covers a little, however, you can see that this class is really just constructing REST URLs or RPC commands using lower level classes, so you can in principle do this yourself and bypass OpenSocialClient entirely.

I decided to try this out. The code below uses the REST style invocations to duplicate the DisplayProfileData.java sample code. It turned out to be a little convoluted to do this because the OpenSocialClient class itself gets passed to lower level classes (particularly OpenSocialRequstSigner.java), which is not helpful. Hopefully the developers will reorganize this in the future.

Anyway, on to the code. I think it can be modified to construct any OpenSocial REST request, including PUT and DELETE (which are not currently in OpenSocialClient), but that is a test for another day. I don't claim this is beautiful. It is just my attempt to unravel everything.

I used Revision 33 of the SVN checkout. The OAuth snapshot jar in lib can cause problems if you use a different version. See comments below.

----------------

import org.opensocial.data.*;
import org.opensocial.client.*;
import net.oauth.*;
import java.util.*;

public class MyOpenSocialClient {

public static void main(String[] args) {
MyOpenSocialClient mosc=new MyOpenSocialClient();
}

public MyOpenSocialClient() {
String REST_BASE_URI=
"http://sandbox.orkut.com/social/rest";
String CONSUMER_SECRET=
"uynAeXiWTisflWX99KU1D2q5";
String CONSUMER_KEY=
"orkut.com:623061448914";
// String VIEWER_ID=
// "03067092798963641994";
String VIEWER_ID="08354253340777199997";

try {
//Construct the request object
OpenSocialRequest req=new OpenSocialRequest("people","people.get");
req.addParameter("groupId","@self");
req.addParameter("userId","08354253340777199997");

//Construct the request URL from the request object
//This is the OpenSocialBatch.submitRest() method.
OpenSocialUrl requestUrl = new OpenSocialUrl(REST_BASE_URI);
requestUrl.addPathComponent(req.getRestPathComponent());
if (req.getParameter("userId") != null) {
requestUrl.addPathComponent(req.getParameter("userId"));
}
if (req.getParameter("groupId") != null) {
requestUrl.addPathComponent(req.getParameter("groupId"));
}
System.out.println("This is the REST URL: "+requestUrl.toString());

//Now construct and sign the request
//This is the OpenSocialRequestSigner.signRequest() code.
OpenSocialHttpRequest request = new OpenSocialHttpRequest(requestUrl);
requestUrl.addQueryStringParameter("xoauth_requestor_id", VIEWER_ID);
requestUrl.addQueryStringParameter("st", "");

String postBody = request.getPostBody();
System.out.println("Post bod: "+postBody);
String requestMethod = request.getMethod();
// OpenSocialUrl requestUrl = request.getUrl();

OAuthMessage message =
new OAuthMessage(requestMethod, requestUrl.toString(), null);

if (postBody != null) {
message.addParameter(postBody, "");
}

OAuthConsumer consumer =
new OAuthConsumer(null, CONSUMER_KEY, CONSUMER_SECRET, null);
consumer.setProperty(OAuth.OAUTH_SIGNATURE_METHOD, OAuth.HMAC_SHA1);

OAuthAccessor accessor = new OAuthAccessor(consumer);
accessor.accessToken = "";

message.addRequiredParameters(accessor);

//VERSION WARNINIG: This works with SVN version 33 and oauth-20081115.jar but other
//versions may have problems.
for (Map.Entry p : message.getParameters()) {
if (!p.getKey().equals(postBody)) {
requestUrl.addQueryStringParameter(
OAuth.percentEncode(p.getKey()),
OAuth.percentEncode(p.getValue()));
}
}

//Take a look at the signed URL
System.out.println("Signed REST URL: "+requestUrl.toString());

//Done with signing. Now back to OpenSocialBatch's submitRest()
//Finally, get the response. This is the meat of the getHttpResponse
//method, without the error checking.
request.execute();
String responseString=request.getResponseString();
OpenSocialResponse resp=OpenSocialJsonParser.getResponse(responseString, req.getId());
//A little clunky--have to add this to the OpenSocialResponse
//object's "items" map with the "people" key.
resp.addItem("people",responseString);

System.out.println("Response: "+responseString);

//Now cast the response into an OpenSocialPerson object
OpenSocialPerson person=resp.getItemAsPerson("people");

System.out.println("----------");

// Output the name and ID of the requested person
System.out.println("Info. for " + person.getDisplayName());
System.out.println("ID: " + person.getId());

// Retrieve individual fields using the getField method; fields may be
// complex (objects or arrays) or simple (Strings), which you can
// determine by querying the object using the isComplex method.
// The thumbnail URL should be a simple field, so we'll output
// the value as a String
OpenSocialField thumbnailUrlField = person.getField("thumbnailUrl");
if (!thumbnailUrlField.isComplex()) {
System.out.println("Thumbnail URL: " +
thumbnailUrlField.getStringValue());
}

System.out.println("----------");

} catch (Exception e) {
System.out.println("Request failed:" );
e.printStackTrace();
}
}
}

Friday, January 09, 2009

OpenSocial REST and WGET: Not So Much

The Open Social REST/RPC documentation (http://code.google.com/apis/orkut/docs/rest/developers_guide_protocol.html) says vaguely that you can play with the examples using the UNIX wget command, but I don't see it.

The problem is that the URLs have to be signed and carry proper security information. By fooling around with the example code, you can see that the OpenSocialClient class's fetchPerson() method is really just constructing a URL like the one below:

http://sandbox.orkut.com/social/rest/people/08354253340777199997/@self

The number 083542533407771999 is my orkut user ID. You can't just "wget" this URL or put it in your browser, however. Doing so will return the error

HTTP request sent, awaiting response... 401 The request did not have a proper security token nor oauth message and unauthenticated requests are not allowed

Fooling around with the client codes (see previous post) will reveal the actual, signed URL used in the REST operation:

http://sandbox.orkut.com/social/rest/people/08354253340777199
997/@self?oauth_consumer_key=orkut.com%3A623061448914&oauth_nonce=1231537930162003000&oauth_tim
estamp=1231537930&oauth_signature=0h%2FU49KtBplnmnc%2BhDKsDxFPR9k%3D&oauth_signature_method=HMA
C-SHA1&oauth_token=&xoauth_requestor_id=03067092798963641994&oauth_version=1.0

This URL is good for only one invocation (time-stamped).

To reproduce this, make the following change to DisplayProfileInfo.java:

// c.setProperty(OpenSocialClient.Properties.RPC_ENDPOINT,
// "http://sandbox.orkut.com/social/rpc");
c.setProperty(OpenSocialClient.Properties.REST_BASE_URI,
"http://sandbox.orkut.com/social/rest");

and then add the following line to OpenSocialBatch.java's submitRest() method.

OpenSocialRequestSigner.signRequest(request, client);
System.out.println("This is the post-signed REST url: " + request.getUrl().toString());

More soon. It turns out the the base OpenSocialClient.java class is actually just a wrapper around the other classes (OpenSocialRequest and so on). So although the client does not apparently support write operations, you can work around this by constructing the appropriate REST POST operation.

Thursday, January 08, 2009

OpenSocial REST API Is Here

I just noticed the Google-led OpenSocial REST/RPC APIs are finally available. These allow you to build Java, PHP, Python, and Ruby clients that can access and manipulate your social network data.

Blog announcement: http://opensocialapis.blogspot.com/

Java code: http://code.google.com/p/opensocial-java-client/

Orkut Sandbox information: http://code.google.com/apis/orkut/docs/rest/developers_guide_protocol.html

Client Examples
As you might expect, you can use Orkut's sandbox as your "backend" service. The Java code and examples are pretty simple but work. As a bonus, here is a recursive method that you can use in the DisplayProfileData.java example to display all information about a particular person. It will also work in DisplayFriends.java. You'll need to add imports for OpenSocialObject and OpenSocialField.

protected void processFields(OpenSocialObject osObj) throws Exception {
//See all fields
String[] fields=osObj.fieldNames();
for(String fieldName : fields) {
OpenSocialField afield=osObj.getField(fieldName);
if(!afield.isComplex()) {
System.out.println(fieldName+" "+afield.getStringValue());
}
else {
System.out.println(fieldName+" is complex:" );
Collection complexValues=afield.getValues();
for(OpenSocialObject compVal : complexValues) {
processFields(compVal);
}
}
}

Call it anywhere after creating the sample person, like so:

OpenSocialPerson person = c.fetchPerson("03067092798963641994");
...
processFields(person);

You may also want to change the Person ID in the above line. You get this ID when you add sample.xml social gadget to your sandbox. Mine, for example, is 08354253340777199997. This ID does not need to match the viewer ID discussed below.

Here is the sample output:
156-56-104-143:java mpierce$ java -classpath samples/bin:$CP DisplayProfileData
----------
Info. for API DWH
ID: 03067092798963641994
----------
Printing fields
----------
photos is complex:
value http://www.orkut.com/img/i_nophoto64.gif
type thumbnail
thumbnailUrl http://www.orkut.com/img/i_nophoto64.gif
name is complex:
givenName API
familyName DWH
id 03067092798963641994
isOwner false
isViewer true
------Done---------
Thumbnail URL: http://www.orkut.com/img/i_nophoto64.gif
----------

Modifying the Backend
Here's were the problems start. The sample clients above include a default view ID, consumer secret, and consumer key as part of the OAuth infrastructure:

c.setProperty(OpenSocialClient.Properties.CONSUMER_SECRET,
"uynAeXiWTisflWX99KU1D2q5");
c.setProperty(OpenSocialClient.Properties.CONSUMER_KEY,
"orkut.com:623061448914");
c.setProperty(OpenSocialClient.Properties.VIEWER_ID,
"03067092798963641994");

Of course you will want to replace these with your own values. To do this, you will need to deploy a gadget and verify that you own it. The steps are simple:
  1. Write a gadget. You can use http://opensocial-resources.googlecode.com/svn/samples/rest_rpc/sample.xml for example. I'm not sure if the gadget in anyway affects the REST/RPC server side calls that you can use.
  2. Get an Orkut sandbox account and log into sandbox.orkut.com.
  3. Install your application into your Orkut account.
  4. Verify that you own the gadget: https://www.google.com/gadgets/directory/verify
This last step is the problem. It involves two steps. First you provide the URL for your gadget and submit. You will receive a long signature string, which you need to add to your gadget. Replace the one in the sample.xml if you are using a copy of that. Save your gadget and click the verification web site button to complete the process. Presumably this will give you the consumer key and consumer secret that you need, but I have not gotten this to work for some reason.
See
http://groups.google.com/group/opensocial-orkut/browse_thread/thread/43640406754ab7f6