The current stable release of Apache Archiva (1.3.6) has a serious, known security flaw: http://cxsecurity.com/issue/WLB-2014010087. I've reported this to Apache security and to the Archiva mailing lists. If you are running Archiva as a privileged user, this would allow the remote attacker to gain access to your entire machine.
This is from the "right in front of our noses" department. XSEDE has excellent (and pithy) instructions for installing MyProxy and GSI-SSH clients at the bottom of https://www.xsede.org/accessing-resources ("Command Line GSI-SSH Tools").
Reader Beware: this post contains some XML markup that did not survive the cut and paste. I'll have to clean it up later. Google provides a full REST API for many of its services. While you could use their programming APIs, it’s quick and fun to make your requests directly with the “curl” command. These are notes on how to do this. I’m interested in querying multiple online spreadsheets in Google Drive along with public websites to find intersections of researchers who have large computing allocations, are interested in workflows, who have existing advanced support allocations, and who are attending various conferences. This is a good overview: https://developers.google.com/gdata/articles/using_cURL. Other useful links:
The response will be in ATOM/XML with an entry for every spreadsheet you own. You’ll probably want to save to a file. Passing through tidy will format it nicely as described in https://developers.google.com/gdata/articles/using_cURL. An example entry is shown below. The key in the URLs is a unique ID for the spreadsheet.
Formatting: The UNIX tidy command is nice for formatting. The more powerful xmllint command can also be used.
Note the trailing “-” after “--format” $AUTH and $KEY are environment variables set using values discussed above. The xmllint command can also do more powerful parsing, but this doesn’t seem to be working well on my Mac.
The “alternate” link above with type “text/html” is a link suitable for display in a browser (that is, how you usually view it). The “id” URL and also the “self” link are ATOM/XML feeds for the specific spreadsheet. If you just want to return this particular entry as a response, GET the “self” URL.
The atom+xml type URLs in the response are useful for various operations. In particular, the listfeed URL will allow us to interact with the content of a specific worksheet within a spreadsheet.
Use the worksheet feed URL to get the worksheet’s contents
The entire spreadsheet will be returned in ATOM/XML format. Each worksheet row will be an Atom feed entry with the worksheet columns as child XML tags. For example, the above spreadsheet has columns listing the owner of the allocation, the requested allocation, the approved allocation, and so on. There is one row for each allocation holder. An example entry in the response to the curl command above is below:
Sorting, filtering, and other operations are performed by appending the operation to the end of the feedlist URL. See https://developers.google.com/google-apps/spreadsheets/. In summary, the following operation keywords are supported (append as ?keyword=keyvalue).
orderby: orders the response alphabetically or numerically with the supplied column. For example, to list the largest approved allocations first, use ?orderby=approved.
reverse: reverses the order of the response. To sort the response with the largest response last, use ?orderby=approved&reverse=true.
sq: this is a structured query operation. To return a list of all responses with the “dependent” column value > 50, use ?sq=dependent>50. The format for the query language is described in https://developers.google.com/chart/interactive/docs/querylanguage. The queries must be URL encoded; the link provides a little tool for doing this.
Since the data in the spreadsheet cells often includes commas, TSV is the better choice. Unfortunately the export feature doesn’t work with the ordering and structured query parameters. On the other hand, TSV and CSV is easier to work with using standard UNIX command line tools than XML.
Sorting and Extracting Information from CSV and TSV Output
We’ll stick with CSV since we don’t really care too much about the document structure and just want to extract some output. For extremely large outputs, this may not scale (the server may do a better job). Let’s sort the allocations by the last field in the output (which indicates the percentage of the allocation projected to involve dependent jobs:
See http://stackoverflow.com/questions/1037365/unix-sort-with-tab-delimiter for the tip about sorting tab-deliminated files. We can also sort on multiple columns. To first sort by the dependent job percentage and then sort by awarded allocation size so that we can find out who the power users with dependent jobs, use
Let’s now clean this up, keeping only the last name, first name, and dependency percentages and removing the duplicated names. The latter come from users who have made multiple allocation requests.
The tail +2 removes the annoying header line. The above curl filters out users with >75% requirement for dependent jobs. The ORS in the last awk command separates the output with tabs rather than new lines, which will be useful later in constructing input arrays. We could have done a similar things by piping through xargs, but xargs doesn’t give us control over the separator. This is a problem for last names with spaces.
Here’s another useful one. After filtering by dependency > 25%, it keeps only the comma-separated last and first names. We then switch the order to Firstname Lastname and separate entries with a tab.
You can use a similar approach with CSV output using the following command. Note that the comma in the “name” field between last name and the first name causes problems for sort, so we just treat this as a new column.
We’re now ready to use the last names to search conference proceedings. Annoyingly, the XSEDE 2013 proceedings are not immediately obvious. The conference site uses JavaScript and
tags to populate the presentation topics, so this won’t be preserved if we curl the page. The proceedings are also in ACM’s Digital Libraries, but I can’t find a standalone “Table of Contents” page. I did find a curl-able version of this by viewing one of the papers, switching from the tabbed view to the static table view, and selecting the “Table of Contents” link. The URL is
I’m keeping only the last names, and the sort|uniq removes duplicates. Be careful with this: the ACM site will block your IP address if you make too many requests.
We can also search across two Google Spreadsheet accounts. The following command will return the first and last names of everyone receiving ECSS support:
