rc98.net

Code4Lib 2010

2010-02-22T10:41:57-05:00

Solr blackbelt preconf

Use "raw" LocalParams for facet queries. From SolrQuerySyntax:

raw term query, useful for avoiding query parser escaping madness when drilling into facets via fq parameters.
&fq={!raw f=field_name}crazy+\"field+value

Also, edismax still looks good.

Decentralized Energy

2009-11-06T14:44:24-05:00

Al Gore said it Wednesday night on the Daily Show (clip if you're in the US):

Al: Well the problem is that some very powerful corporations do make their money from the old ways of burning coal and oil.

Jon: That's the thing. Who? Give me a name.

Al: You mentioned one. Exxon Mobil.

Jon: So give them, say, "You've got the solar thing. You're done. Have the sun! It's yours!"

Al: Because if we switch over to renewable energy then all the money they're making from the old, dirty fuels, they're not gonna make as much anymore, and that's why they're fighting against it. ... A lot of these new technologies are not ones that are naturally controlled by one or a few big corporations.

Jon: Well, they will be.

Al: But solar panels, the sun shines everywhere, and you can put solar panels on the roofs of homes and buildings, and it's widely dispersed. Same with wind power, same with a lot of the new approaches, and so the older companies that established control of the old patterns, they don't want to change.

That's why the big companies push nuclear, clean coal, and natural gas as the solutions in our clean, green energy future. They've established control over these resources, and they would take the sun if they could. Just like Burns.

Free Software and Kittehs

2009-11-04T12:18:32-05:00

Seeing Jesper's post on the kitten he found reminded me of Karen's "free as in free kittens" analogy. Makes me wonder if some free software developers like to care for little things that might not otherwise make it in this cruel world.

Access 2009

2009-10-03T12:13:23-05:00

Hackfest

Cory Doctorow spoke with us about copyright. Holding action is beneficial as the technology gets better. Get your law schools involved in showing cover images and other copyright issues -- a consortium can save over paying Bowker. As an artist, charge for the bits that aren't easily copied: the performance. Ebooks as a privacy risk? Do those companies keep patron reading lists?

Linked data in your OPAC/ILS. Pulling linked data into your system versus exposing your data as linked.

Day 1

Cory Doctorow spoke with us about copyright. Again. Some more depth this time, and better stories. Backchannel discussion led me to this post of his on Mark Twain.

Richard spoke about http://data.vancouver.ca and the need to free our data. How do we connect to superpatrons who might do mashups. Right now there's a lot of value just in releasing your data. APIs versus raw data == control versus unimagined uses.

Dan Chudnov talked about the Library of Congress. His team was the first at the LoC to release software under an opensource license. The vision of incorporating digitally rests on linked data. http://id.loc.gov

Paul Pound talked about Islandora Drupal and Fedora work.

Roy Tennant spoke about OCLC Product Works (http://oclc.org/productworks/). Mike Rylander spoke about open source, cloud computing, and Evergreen as PaaS.

Stevan Harnad spoke from far away about Open Access and the evidence showing the real value of mandates. The mandate should be to deposit all articles in the institutional repository immediately upon acceptance for publication.

Day 2

Peter Rukavina spoke about digital and real data.

Cary Gordon did a birds-eye of Drupal in libraries.

Good thunder talks. Schools using Drupal. Bess on GIS at UVA. Canadensys and flat plants. Libraryh3lp on Google Android.

Hackfest talks. Human-readable bibliography in Evergreen. Linked data in catalogs.

Dorothea Salo spoke about data and our researchers, and how they dig around in the sand to build castles of knowledge. We made this mess and we need to clean it up. These Taylorist processes are not serving us well. IRs present everything as a PDF paper.

Roy Tennant spoke about the Hathi Trust workflow and outlook. He also played a video of Roy Tennant coding that featured Stan Rogers's "White Collar Holler".

Bess Sadler spoke about Blacklight and heterogeneous collections. Librarians write acceptance tests for the search interface.

Andrew Nagy described the progression of computing toward smaller and faster devices.

Day 3

Mark Leggot showed off UPEI's fancy Drupal/Fedora research support project.

Cathy Hartman and Mark Phillips spoke about UNT's Texas History Portal.
Pulling in objects from all over Texas and beyond. Nice ability to view newspapers. http://beta.texashistory.unt.edu

After the break, Jan Dawson gave a quick tour of roller derby.

Gwendolyn MacNairn spoke about Zotero easing the burden of citations.

William J. Turkel gave a talk titled "> hacking as a way of knowing".
He spoke of the need for a rich backchannel, for casual participation. He admitted he was preaching to the choir on many points. Imagine a history spray that produced a cloud of the past that you could feel. Like aquanet lets you touch the 80s.

Declan Fleming won the laptop and Mark Leggot wrapped it up.

A bit about Linked Open Data

2009-08-18T09:24:21-05:00

From TBL's page on Linked Data (the springboard):

The Semantic Web isn't just about putting data on the web. It is about making links, so that a person or machine can explore the web of data. With linked data, when you have some of it, you can find other, related, data.

...

Use URIs as names for things

Use HTTP URIs so that people can look up those names.

When someone looks up a URI, provide useful information, using the standards (RDF, SPARQL)

Include links to other URIs. so that they can discover more things.

From the RDF Primer (visit the EM ex.):

The Resource Description Framework (RDF) is a language for representing information about resources in the World Wide Web.

...

Unlike triples, which are intended as a shorthand notation, RDF/XML is the normative syntax for writing RDF.

See the known world of linked data in the graph at linkeddata.org. Resources of note are AudioScrobbler, DBpedia, Freebase, and US Census Data (working SPARQL endpoint).

And in libraries: id.loc.gov and LIBRIS.

LOD is rough around the edges, full of broken links and draft papers. Nonetheless, it's a beginning.

Debian on the ideapad S10e

2010-02-11T21:16:33-05:00

Getting wireless working with kernel 2.6.30

I upgraded the kernel and lost wireless. Thought I had it working with the b43-fwcutter package. Turns out that doesn't support the BCM4312 in this guy. What I needed was broadcom-sta-modules-2.6-686. Then I needed to rmmod b43 and modprobe wl. Oh, and also modprobe lib80211_crypt_tkip. But not in that order. So altogether it goes:

# apt-get install broadcom-sta-modules-2.6-686
...
# modprobe -r b43
# modprobe lib80211_crypt_tkip
# modprobe wl

Problem was I had to do the modprobe business every time I rebooted. I added b43 to the /etc/modprobe.d/blacklist.conf and added lib80211_crypt_tkip and wl to /etc/modules, but ifconfig still revealed no eth1 interface at boot. I ran depmod. I removed the b43-fwcutter package. Nothing changed. I even tried removing b43.ko from the kernel drivers. Thereafter, modprobe -r b43 would complain that it wasn't found, as it should, but no amount of modprobing wl or otherwise would bring up the eth1 interface until I returned b43.ko to its rightful place.

Finally, this Ubuntu forum prompted me to try the following for /etc/modprobe.d/wl.conf:

install wl /sbin/modprobe -r b43; /sbin/modprobe lib80211_crypt_tkip; /sbin/modprobe --ignore-install wl

That made modprobe wl work to bring up the eth1 interface. I put wl back in /etc/modules (having removed it by then) and wireless finally worked again at boot.

Update 2010-02-11: There's no broadcom-sta-modules for 2.6.32, so I installed broadcom-sta-source, and /usr/share/doc/broadcom-sta-source/README.Debian told me I should use b43-fwcutter instead. Once I fetched and extracted the firmware and ran modprobe b43, however, I started getting b43-phy0 ERROR: Fatal DMA error messages, so I went back to broadcom-sta and ran the module-assistant commands suggested in that README. The broadcom-sta-common package created /etc/modprobe.d/broadcom-sta-common.conf with the following:

$ cat /etc/modprobe.d/broadcom-sta-common.conf
# wl module from Broadcom conflicts with ssb
# We must blacklist the following modules:
blacklist b44
blacklist b43legacy
blacklist b43
blacklist ssb
install wl /sbin/modprobe lib80211_crypt_tkip; /sbin/modprobe --ignore-install wl $CMDLINE_OPTS
$

Except I inserted the bit about lib80211_crypt_tkip. It didn't work before I did that.

A couple of thoughts on Wolfram Alpha

2009-03-10T13:49:31-05:00

In his article on Wolfram Alpha, Spivack says the following:

The Big Deal is that Wolfram Alpha doesn't merely look up the answers like Google does, it computes them using at least some level of domain understanding and reasoning, plus vast amounts of data about the topic being asked about.

This is untrue. Google doesn't merely look up answers. A search for "philadelphia pizza" (sans quotes) returns a Google Local result set. Simple calculation and exchange rates can also be done. There's even a list of such features that demonstrate computation and domain understanding.

I'm not saying Google is perfect, but it's done some things right. It first built up huge indexes of information, then added clever ways beyond a simple search (and at this point even "simple" searches in Google are actually anything but). There may be some tricks to be cribbed from Wolfram Alpha and START and other Emerac wannabes, but Google, with its massive data stores and computational power, is well situated to do so.

How I do backups

2009-05-05T14:50:34-05:00

The server, we'll call it BOB, has two disks of the same size, and each night one gets rsynced to the other. Its /etc/fstab looks like this:

/dev/xvda       /       ext3    defaults                0 1
/dev/xvdb       none    swap    sw                      0 0
proc            /proc   proc    defaults                0 0
/dev/xvdc       /backup ext3    noauto                  0 0

The root crontab includes the lines:

30 3 * * *  /root/nightly-rsync
12 4 * * *  /root/nightly-rdiff

With /root/nightly-rsync containing the following:

mount -v /backup
/etc/init.d/apache2 stop
/etc/init.d/postgresql-8.3 stop
rsync -vaxE --del / /backup/ 
/etc/init.d/postgresql-8.3 start 
/etc/init.d/apache2 start 
umount -v /backup

Later in the night, the synced backup gets rdiff-backuped to another machine, which we'll call SUE. The /root/.ssh/config file on BOB contains the following:

host sue-bob hostname sue user bob identityfile /root/.ssh/backup_id_rsa compression yes protocol 2

A user "bob" has been created on SUE and ssh-keygen run to create a passwordless keypair for that user. The backup_id_rsa file on BOB is the created private key, with the public key added to the /home/bob/.ssh/authorized_keys file on SUE. The following lines are in /root/excluded-backup:

/backup/backup
/backup/dev
/backup/mnt
/backup/proc
/backup/sys
/backup/tmp
/backup/var/tmp

/root/nightly-rdiff looks like this:

mount -v /backup
rdiff-backup -v5 --exclude-globbing-filelist /root/excluded-backup /backup sue-bob::backup
umount -v /backup
rdiff-backup -v5 --remove-older-than 2M --force sue-bob::backup

An incremental backup of BOB going back 2 months will be in /home/bob/backup on SUE. Update 2009-05-05: Added --force to the remove-older-than line to remove multiple old backups when necessary.

Some final notes: Make sure to make nightly-rdiff and nightly-rsync executable. Also, both BOB and SUE must have rdiff-backup installed. Oh, and you'll want to ssh sue-bob once to add SUE's fingerprint to the known_hosts file. When I'm feeling fairly certain the backups are working, I'll lower the verbosity.

I was influenced on the rsync end by jwz's PSA on backups and on the rdiff-backup end by unattended rdiff.

Carl Malamud for head of the GPO

2009-03-08T16:48:13-05:00

Carl Malamud wants to run the GPO. I can't imagine a better person for the job.

Code4Lib 2009

2009-03-01T14:37:57-05:00

What follows is my sparse notes (increasingly so as the conference went on) from code4lib 2009.

Linked data preconf

rdf+xml == teh suxxor (lbjay). see ntriples with http://www.mindswap.org/2002/rdfconvert/. dchud: eliminate custom software. cache linked data so your links don't break. distribute and decentralize the data.

Keynote: Stefano Mazzochi

Reduction of marginal costs with each transition to a new technology. The marginal cost is the cost to create the product again.

speech: synchronous, no lasting record
cave drawings: can't take it with you, marginal cost is 1
clay tablets: portable, heavy, marginal cost is 1
fiber inscribing: portable, light, marginal cost is 1
fiber printing: marginal cost < 1, but still > 0 (production, consumption, storage, organization)
electronic publishing: asynchronous and synchronous, marginal cost ~ 0

Electronic publishing presents a degraded consumption experience. We lack the hundreds of years of experience of the paper publishing industry. Batteries are needed.

Libraries versus museums of books. Managing physical non-unique copies of books will be non-economical. With nearly infinite storage space, can we justify filtering? Is metadata still needed? From books of pages to journals of articles to networks of relational assertion (web of data). Questions to answer from freebase. A little reminiscent of EMERAC. David Huynh has been working on "sliding" -- moving from one dataset to another. Freebase currently tries to fit all sets, so doesn't display for a particular set well.

Acre is a javascript framework for writing freebase apps. Example app: http://genderizer.freebaseapps.com

Why libraries should embrace linked data

Stories from Libris, the National Library of Sweden. Started building a new user interface in December 2007. Design a library catalog within the context of the web, and ask what it could become. APIs suck, too much control in the hands of the producer (dependent on the producer's world view), but we have created some APIs anyway. Linked open data turns the web into an API (Corey Harper).

Jangle

Jangle core sends AtomPub to clients, speaks to connectors via JSON.

RESTafarian

Looked at the Sinatra Ruby framework. Java debugging with services depending on services depending on services. Ugh. You start developing for 20 years ahead of you and it ends up being totally overengineered. We got sucked into every standard. Sword, METS. I need a METS profile to copy a PDF from here to there?

Dashboard Initiative

Hard to build in delivery of numbers when you're working on delivery of services. Dashboard of a car gives overview of current flows. It's a simple way to provide library data flow to managers and administrators.

Open up your repository with a SWORD

What is a repository? OAI-PMH co-evolved with these repository systems.

Djatoka

JPEG2000

Lightning Talks

tinyurl.com/XCwebcast. Scriblio at Collingswood Public. enjoysthin.gs. zoia and foaf.

Biblios

No Galen Charlton, but Josh Ferraro gave a good overview. Chris Catalfo came next with a more "in the guts" view (including some couchdb goodness).

FRBR

All about relationships.

Solr

Erik Hatcher catches us up on developments in Solr.

Ian Davis

If you love your data, set it free.

New England Code4Lib

2009-01-12T11:44:47-05:00

First, Jay Luker talked about CiteULike and the Pylons stuff he's written to access their API.

People from WGBH talked about media metadata and using the LCSH dump from LCSH.info. They have a typeahead prompt for users to add subjects that encourages the use of LCSH, but doesn't limit to it.

Jodi Schneider talked about Zotero and the possibilities of the semantic web. Discussion landed on the positives and negatives of an in-browser tool. Casey Bisson made the point that it lives in the browser because our resources are inaccessible from the web at large. ~~Michael Klein said the ideal would be an in-browser tool with shared data storage, which is what Zotero Commons will be.~~ Update 2008-12-10 from Michael Klein: "Correction: What I want is a browser tool for scraping/harvesting (like we have now), but a cloud-based browser interface to read, manage, format, and export records. So you need the plugin to create your research pile, but you can do your research, cite your references, and create your bibliography from any browser, with or without the plugin."

Casey Bisson and Kelly talked about Scriblio and http://archives.colby-sawyer.edu. Scriblio will merge records as they're loaded in. Also looked at http://beyondbrownpaper.plymouth.edu and http://collingswoodlib.org/.

Michael Klein talked about BPL Digital Initiatives, including XForms and MODS. Includes a browsable thesaurus resting on Solr. Also includes 1300 lines of XML pain.

Just before lunch, Jay Luker did a quick overview of unAPI. Casey brought up the point that unAPI creates a separate URL for a resource, against the objectives of the microformat people.

After lunch, Sands Fish spoke about Simile, RDF, and ontologies. Now looking at author citation metadata for Longwell. Facade is Future-proofing Architectural Computer-Aided DEsign. Timeline is the more popular one (sometimes called Simile, but that's the name of the whole project). Exhibit is another one, written entirely in JavaScript, using the in-browser database and JSON (or a close relation to JSON). The Simile funding has ended, but the project code lives on at http://code.google.com/p/simile-widgets/.

I tried to help Jay install FBO, which led to a quick temporary fix for loading MARCXML data and a couple of other changes, but he had some Solr issues that got in the way of things. Afterward, we went to Bukowski's.

Update 2008-12-11: New England Code4Lib can be found at http://ne.code4lib.org.

Installing Debian from a USB key

2009-05-18T12:33:24-05:00

Not wanting to use up another CD, I decided to try to install Debian with a USB key. I followed the instructions at the Debian testing docs and downloaded boot.img.gz and the iso from the links at debian-installer. Update 2005-15-09: The daily build hd-media builds are erroring out right now, so I had to download boot.img.gz and an iso from the lenny page.
Here are the commands:

root@stubberud:~# zcat /home/gsf/boot.img.gz > /dev/sda
root@stubberud:~# mkdir /media/usbkey
root@stubberud:~# mount -t vfat /dev/sda /media/usbkey/
root@stubberud:~# cp /home/gsf/debian-LennyBeta2-amd64-netinst.iso \
/media/usbkey/
root@stubberud:~# umount /media/usbkey/

Popped it in and it worked!

Mail setup

2009-01-12T11:44:47-05:00

Important bits for .muttrc (need public-key crypto set up for ssh):

set folder=imap://gsf@breaksalot.org/
set tunnel="ssh -q breaksalot.org /usr/sbin/dovecot --exec-mail imap"

my_hdr From: Gabriel Farrell <gsf@breaksalot.org>
set envelope_from=yes

mailboxes /var/mail/gsf + +archives +debian +django +dspace-tech

folder-hook . set record="+archives"
folder-hook . save-hook . "+archives"
folder-hook . set postponed="+drafts"

set sort=threads
set move=no
set delete=yes
set fast_reply=yes
set confirmappend=no
set mailcap_path=~/.mutt/mailcap
set smart_wrap
unset markers
set simple_search="~f %s | ~s %s | ~C %s"
set certificate_file=~/.mutt_certificates

set index_format="%4C %Z %{%b %d} %-15.15F (%4l) %s"
set alias_file=~/.mutt/aliases

# simpler headers
ignore *
unignore from cc to date subject

Install Postfix as "Satellite system" and set "breaksalot.org" as the SMTP relay host. Defaults on everything else.

Add the following to the bottom of /etc/postfix/main.cf:

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous

Create /etc/postfix/sasl_passwd with the following line (substitute "passwd"):

breaksalot.org gsf:passwd

Run postmap /etc/postfix/sasl_passwd, then /etc/init.d/postfix reload.

On one laptop I was getting the following error in /var/log/mail.log when trying to send:

(SASL authentication failed; cannot authenticate to server 
breaksalot.org[207.192.69.239]: no mechanism available)

That problem was solved by installing libsasl2-modules.

Guido on importing in Python

2009-01-12T11:44:47-05:00

From http://groups.google.com/group/django-developers/msg/8991d2a2d6d87873:

I'm totally in favor of dropping the packages from the path; but I personally have grown very fond of Google's style convention of always importing modules, never classes or functions.

Transferring files between Debian and a Nokia 6555 cell phone

2009-01-27T18:11:02-05:00

After some frustration I got some great help from HanishKVC on getting things going. It seems a lot depends on gnome, and figuring out how to do bluetooth in XFCE, etc., proves difficult.

First I enabled bluetooth on my phone (via Settings -> Connectivity) and enabled it on my laptop:

magnum:~# echo enable > /proc/acpi/ibm/bluetooth

I was able to see the phone:

gsf@magnum:~$ hcitool scan
Scanning ...
        00:1B:AF:D5:CA:1C       Nokia 6555b
gsf@magnum:~$

I installed obexftp and tried to check some files, but failed:

gsf@magnum:~$ obexftp -b 00:1B:AF:D5:CA:1C -l
Browsing 00:1B:AF:D5:CA:1C ...
Channel: 10
Connecting...failed: connect
Still trying to connect
Connecting...failed: connect
Still trying to connect
Connecting...failed: connect
Still trying to connect
gsf@magnum:~$

Each time, the phone asked for a passcode/passkey and I put in the one I had in /etc/bluetooth/hcid.conf ("1234"), but it failed to pair. I was finally able to get past it by building passkey-agent (note that on a really stripped-down machine I had to install libdbus-1-dev for the make to work):

magnum:~# cd /usr/share/doc/bluez-utils/examples/
magnum:/usr/share/doc/bluez-utils/examples# gunzip passkey-agent.c.gz 
magnum:/usr/share/doc/bluez-utils/examples# make
cc `pkg-config --libs --cflags dbus-1` -DDBUS_API_SUBJECT_TO_CHANGE
-DVERSION="\"unknown\"" -o passkey-agent passkey-agent.c 
magnum:/usr/share/doc/bluez-utils/examples# mv passkey-agent /usr/bin/
magnum:/usr/share/doc/bluez-utils/examples#

I also had to set a default key:

gsf@magnum:~$ passkey-agent --default 1234 &
[1] 18890
gsf@magnum:~$

When I tried again, the passcode worked:

gsf@magnum:~$ obexftp -b 00:1B:AF:D5:CA:1C -l
Browsing 00:1B:AF:D5:CA:1C ...
Channel: 10
Connecting...Passkey request for device 00:1B:AF:D5:CA:1C
done
Receiving "(null)"...\<?xml version="1.0"?>
<!DOCTYPE folder-listing SYSTEM "obex-folder-listing.dtd"
 [ <!ATTLIST folder mem-type CDATA #IMPLIED> ]>
<folder-listing version="1.0">
    <folder name="Graphics" created="20060101T000000" user-perm="R" mem-type="DEV"/>
    <folder name="Themes" created="20060101T000000" user-perm="R" mem-type="DEV"/>  
    <folder name="Tones" created="20060101T000000" user-perm="R" mem-type="DEV"/>   
    <folder name="Video clips" created="20060101T000000" user-perm="R" mem-type="DEV"/>
    <folder name="Music files" created="20060101T000000" user-perm="R" mem-type="DEV"/>
    <folder name="Images" created="20070101T000000" user-perm="R" mem-type="DEV"/>  
    <folder name="Audio" created="20070101T000000" user-perm="R" mem-type="DEV"/>   
    <folder name="Received files" created="20070101T000000" user-perm="R" mem-type="DEV"/>
</folder-listing>
done
Disconnecting...done
gsf@magnum:~$

Once the device is paired, I no longer need the passkey-agent to be running to connect.

To pull down multiple files from my phone, I use a bash command such as this:

gsf@magnum:~$ for a in `seq 10 38`; do obexftp -b 00:1B:AF:D5:CA:1C -g \
Images/Image0$a.jpg; done

My Django and WSGI Setup

2009-01-12T11:44:47-05:00

The info at both modwsgi's "Integration With Django" and Django's "How to use django with mod_wsgi" are useful, but I'm hoping to cut to the chase here with a short and sweet example of how things work for me. I'll just go ahead and leave in the name of the project (lisinfo) the setup is for, but you should get the idea about where to change things. Okay, I'll start cutting and chasing now, then.

First, my Apache configuration, which, this being a Debian box, lives at /etc/apache2/sites-available/lisinfo.breaksalot.org:

<VirtualHost *>
    ServerName lisinfo.breaksalot.org
    ServerAdmin "the comptroller, and Miss Piggy"

    WSGIDaemonProcess lisinfo
    WSGIProcessGroup lisinfo

    WSGIScriptAlias / /var/www/lisinfo/django.wsgi
    <Directory /var/www/lisinfo/>
        Order deny,allow
        Allow from all
    </Directory>
    Alias /media /var/www/lisinfo/media/
    <Directory /var/www/lisinfo/media>
        Order deny,allow
        Allow from all
    </Directory>
    Alias /admin/media /usr/local/django/django/contrib/admin/media/
    <Directory /usr/local/django/django/contrib/admin/media>
        Order deny,allow
        Allow from all
    </Directory>
</VirtualHost>

The WSGIDaemonProcess and WSGIProcessGroup bits allow me to touch django.wsgi (the insides of which we'll get to in a sec) to kill and restart any processes associated with lisinfo, so code changes can be made visible without restarting Apache. This feature of the mod_wsgi setup is probably the one that makes the most positive difference for me. It makes developing almost as fast as with the built-in server. There's documentation on this stuff at modwsgi Configuration Directives if you're interested.

The django.wsgi script lives at /var/www/lisinfo/django.wsgi, the root of the lisinfo project, along with settings.py, urls.py, etc. It's short and sweet:

import os, sys

# path is the parent directory of this script ('/var/www' in this case)
path = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

# we check for path because we're told to at the tail end of
# http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIReloadMechanism 
if path not in sys.path:
    sys.path.append(path)

os.environ['DJANGO_SETTINGS_MODULE'] = 'lisinfo.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

Update 2008-08-01: The steps below are no longer necessary as of r8015 and r8032. Also, those changes have obviated the need to put your non-root script alias in the urls.py. Many thanks to Malcolm Tredinnick for the fix.

Okay, so far, so good, huh? That's all you need to do if your WSGIScriptAlias is at the document root, i.e. just '/'. If it has a non-root path then you need to apply the patch I submitted to ticket #3414, which looks like this:

Index: django/core/handlers/wsgi.py
===================================================================
--- django/core/handlers/wsgi.py        (revision 7534)
+++ django/core/handlers/wsgi.py        (working copy)
@@ -75,7 +75,7 @@
 class WSGIRequest(http.HttpRequest):
     def __init__(self, environ):
         self.environ = environ
-        self.path = force_unicode(environ['PATH_INFO'])
+        self.path = force_unicode(environ['SCRIPT_NAME'] + environ['PATH_INFO'])
         self.META = environ
         self.method = environ['REQUEST_METHOD'].upper()

That seems to make it work for me. I hope I haven't missed anything. Happy hacking!

Update 2008-08-26: Some looking around at things today taught me some new stuff. Graham Dumpleton actually recommends MPM Worker over Prefork, as noted in this informative thread. Also, I wanted to make a note here that on one of our servers the setting of maximum-requests=10000 on the WSGIDaemonProcess, as suggested in the Defining Process Groups section in the Configuration Guidelines, has helped some memory issues.

Solr init script

2009-01-12T11:44:47-05:00

I cobbled together the following init script for Solr, using the Jetty it comes with, with a little help from various other init.d scripts and Pete's example:

#!/bin/sh -e

# Starts, stops, and restarts solr

SOLR_DIR="/usr/local/solr/example"
JAVA_OPTIONS="-Xmx1024m -DSTOP.PORT=8079 -DSTOP.KEY=stopkey -jar start.jar"
LOG_FILE="/var/log/solr.log"
JAVA="/usr/bin/java"

case $1 in
    start)
        echo "Starting Solr"
        cd $SOLR_DIR
        $JAVA $JAVA_OPTIONS 2> $LOG_FILE &
        ;;
    stop)
        echo "Stopping Solr"
        cd $SOLR_DIR
        $JAVA $JAVA_OPTIONS --stop
        ;;
    restart)
        $0 stop
        sleep 1
        $0 start
        ;;
    *)
        echo "Usage: $0 {start|stop|restart}" >&2
        exit 1
        ;;
esac

Autocomplete and history in the Python interpreter

2009-08-08T09:53:39-05:00

I ran across this page of tips for the interactive Python interpreter the other day. It's part of the official Python documentation, but kinda buried in searches for "pythonrc" and "python interpreter", maybe because the example uses ".pystartup" for the name of the configuration file (a practice I'll follow as well, seeing as pythonrc is used for other things). I'm rambling on about it here so I don't lose or forget about it down the road. I realize ipython does everything that's described therein, but, frankly, it's too flashy for me, and I like being able to copy and paste into my docstrings for doctests.

I've been using the 'tab: complete' mapping for a few days, and I like it. It's kind of a bummer to have to hit the spacebar four times for each indent (and I agree with the commentary at the bottom that the interpreter should suggest proper indentation), but it's not a big deal for me. The default, mapping autocomplete to the Esc key, just feels weird. I tried '`: complete' (mapping the backtick to autocomplete), and that works, but tab-completion is so ingrained in me that I don't really want to train myself on another key (though I have adapted to Ctrl-P in vim).

Ooh, and a saved history between sessions is really the only other thing I like about ipython. Glad to see it can be done in the boring old, plain old interactive interpreter session.

My .pystartup, then, looks much like the example in the Python docs:

# Add auto-completion and a stored history file of commands to your Python
# interactive interpreter. Requires Python 2.0+, readline. Autocomplete is
# bound to the Esc key by default (you can change it - see readline docs).
#
# Store the file in ~/.pystartup, and set an environment variable to point
# to it:  "export PYTHONSTARTUP=/home/gsf/.pystartup" in bash.
#
# Note that PYTHONSTARTUP does *not* expand "~", so you have to put in the
# full path to your home directory.

import atexit
import os
import readline
import rlcompleter
import sys

# change autocomplete to tab
readline.parse_and_bind("tab: complete")

historyPath = os.path.expanduser("~/.pyhistory")

def save_history(historyPath=historyPath):
    import readline
    readline.write_history_file(historyPath)

if os.path.exists(historyPath):
    readline.read_history_file(historyPath)

atexit.register(save_history)

# anything not deleted (sys and os) will remain in the interpreter session
del atexit, readline, rlcompleter, save_history, historyPath

Wrapper Class for Paramiko

2009-01-12T11:44:47-05:00

I used paramiko for a project a while ago and wrote the following, called sftp_wrapper.py, to help with things:

'''
sftp_wrapper.py is a wrapper class for the sftp portion of the 
paramiko library.
'''

import paramiko

class Session(object):
    def __init__(self, hostname, username, password):
        self.tunnel = paramiko.Transport((hostname, 22))
        hostkeys = paramiko.util.load_host_keys('known_hosts')
        hostkey = hostkeys[hostname]['ssh-rsa']
        self.tunnel.connect(username=username, password=password,
            hostkey=hostkey)
        self.sftp = paramiko.SFTPClient.from_transport(self.tunnel)

    def open(self, filename, mode='r'):
        return self.sftp.open(filename, mode=mode)

    def chdir(self, dir):
        self.sftp.chdir(dir)

    def get(self, remote, local=None):
        if not local:
            local = remote
        self.sftp.get(remote, local)

    def put(self, local, remote=None):
        if not remote:
            remote = local
        self.sftp.put(local, remote)

    def listdir(self, remote):
        return self.sftp.listdir(remote)

    def remove(self, remote):
        self.sftp.remove(remote)

    def close(self):
        self.tunnel.close()

Code4lib Conference 2008 Report

2009-01-12T11:44:47-05:00

The 2008 code4lib conference, also known as code4libcon2008, took place February 25-28, 2008, in Portland, Oregon.

Links of interest:

(Notes to be added later.)

Kobold Chieftain Talk

2009-01-12T11:44:47-05:00

I gave a lightning talk at code4libcon2008 with Mark A. Matienzo. It was really just a bunch of links (some images), tenuously strung together:

Single Transferable Vote

2009-01-12T11:44:47-05:00

Voting's on my mind again because of some recent topics in #code4lib and on the mailing list, so I looked into Borda Count, etc. This time I landed on STV. I like that IRV is merely STV with a single winner. So a system that supports STV should support IRV without any modification.

On the software side of things, OpenSTV looks pretty cool in that it's a desktop app written in Python. I'm thinking it would be great to have a portable Django STV app, for which I might steal some of the OpenSTV code (with attribution, of course).

Also, the home page of OpenSTV currently links to Issue 24 of Voting matters, in which the creator of OpenSTV has articles on STV in Northern Ireland and the Droop quota. I've printed it out for my bus ride to NYC.

Conversation with father

2009-01-12T11:44:47-05:00

The "Daddy, are you a terrorist?" poster is a nice little slice of a moment.

Single-pass multiple replace

2009-01-12T11:44:47-05:00

I came across the desire, once again, to replace multiple patterns in a single pass, and feeling something just missing from or put too complicated in both the solution at ASPN and the one in the print version of the Python Cookbook, I put my version of it here:

import re

text = 'i ran from the bob.  after all, he\'s a large mother.'
subs = {
    'bob': 'panda',
    'mother': 'bear',
}

compiled = re.compile('|'.join(map(re.escape, subs)))
def lookup(match):
    return subs[match.group(0)]
new_text = compiled.sub(lookup, text)

It could only get simpler if I replaced lookup with a lambda, but I'm not a fan. I don't mind using re directly, but it is a bit hard on the eyes. Unnecessarily so if I might use it more than once in the same project. For that situation, I'd use the closure version from the second edition of the Python Cookbook. It reads as follows:

def make_xlat(*args, **kwds):
    adict = dict(*args, **kwds)
    rx = re.compile('|'.join(map(re.escape, adict)))
    def one_xlat(match):
        return adict[match.group(0)]
    def xlat(text):
        return rx.sub(one_xlat, text)
    return xlat

As described in the book, a translater for a particular dictionary can then be created with a simple translate = make_xlat(adict). Finally, I prefer the class version in the book to the one at ASPN because it's a "drop-in replacement" for the closure, and because the names make more sense to me. Here it is:

class make_xlat:
    def __init__(self, *args, **kwds):
        self.adict = dict(*args, **kwds)
        self.rx = self.make_rx( )
    def make_rx(self):
        return re.compile('|'.join(map(re.escape, self.adict)))
    def one_xlat(self, match):
        return self.adict[match.group(0)]
    def __call__(self, text):
        return self.rx.sub(self.one_xlat, text)

This recipe is a great example of the easy adaptability of Python for similar, but subtly different purposes. It's also a great example of just how educational each one of the recipes in the Python Cookbook can be. Damn that's a fine tome.

Mirroring a Google Code Subversion Repository for Trac

2009-01-12T11:44:47-05:00

Some of us working on the FBO/Helios project couldn't live with the ticket tracking and source browsing at Google Code, so I decided to set up a Trac instance for the project on my server at fruct.us. Problem is, Trac can't access remote repositories, so I had to mirror the one at Google over to fruct.us.

I looked at both svk and svnsync, and decided svnsync was the tool for the job. I created a repository at /home/svn/fbo-sync, set up the hook scripts, and ran svnsync as described in the SVN book at red-bean. The svnsync commands looked like this:

$ svnsync init file:///home/svn/fbo-sync http://fac-back-opac.googlecode.com/svn
$ svnsync sync file:///home/svn/fbo-sync

I watched as each revision from the repo at Google Code was checked into fbo-sync. To keep the two in sync, it would have been nice to add a post-commit hook to the Google Code repo that syncs the two whenever there's a commit. Since I don't have access to the hook scripts at Google, I set up a cron job on fruct.us to sync every five minutes:

# crontab for svn
# m h  dom mon dow   command
*/5 * * * * /usr/bin/svnsync --non-interactive sync file:///home/svn/fbo-sync

That's it for the syncing.

The trac setup was pretty standard. I pointed my trac install at the fbo-sync repo and it treated it as it would any other SVN repository. The tricky bit was getting the trac-post-commit-hook script to work. I put the shell script suggested in the comment section of that file (well, a slightly altered version -- see the ticket I filed) at /home/svn/fbo-sync/hooks/post-commit, made it executable, and ran a test commit with a refs #1 in the commit message. Nothing happened in the trac instance. Then I remembered to include the #!/bin/bash at the top of the hook script. I ran another test commit and it worked. I noticed, however, that the user "svn", under whom svnsync ran, was getting credit for the commit. I banged my head on the keyboard until I realized that svnsync makes the commit as "svn", then changes the revision properties to reflect the source repository. How could I tell the post-commit hook to wait until after the revision properties had been modified?

I couldn't, of course. A post-commit hook follows a commit, and must execute before the properties are modified. My first attempt at a solution was to remove the hook and instead parse the committed revision numbers from the output of the svnsync script. I replaced the svnsync line in svn's crontab with a pointer to the following script:

#!/bin/bash

SVN_OUT=`svnsync sync file:///home/svn/fbo-sync`

if [ -n "$SVN_OUT" ]; then
    REV=`echo $SVN_OUT | grep -o '[0-9]*' -m 1`
    echo 'trac-post-commit-hook processing ...'
    /home/svn/trac-post-commit-hook -p '/home/trac/sites/fbo' -r $REV
    echo 'done.'
fi

This worked, but I really don't like the thought of parsing output scripts as an API. So fragile. What if more than one commit occurs in a five-minute interval? I could have extended it to handle that case, but that way lies madness. Instead, I decided to use a post-revprop-change hook on the repository. I wrote the following, at /home/svn/fbo-sync/hooks/post-revprop-change, for testing:

#!/bin/bash

REPOS="$1"
REV="$2"
USER="$3"
PROPNAME="$4"
ACTION="$5"

echo "repos:" $REPOS "rev:" $REV "user:" $USER "propname:" $PROPNAME \
        "action:" $ACTION >> /home/svn/svnrevprop.log

svnrevprop.log looked like this:

repos: /home/svn/fbo-sync rev: 0 user: svn propname: svn:sync-lock action: A
repos: /home/svn/fbo-sync rev: 0 user: svn propname: svn:sync-lock action: D
repos: /home/svn/fbo-sync rev: 0 user: svn propname: svn:sync-lock action: A
repos: /home/svn/fbo-sync rev: 0 user: svn propname: svn:sync-currently-copying action: A
repos: /home/svn/fbo-sync rev: 96 user: svn propname: svn:log action: M
repos: /home/svn/fbo-sync rev: 96 user: svn propname: svn:author action: M
repos: /home/svn/fbo-sync rev: 96 user: svn propname: svn:date action: M
repos: /home/svn/fbo-sync rev: 0 user: svn propname: svn:sync-last-merged-rev action: M
repos: /home/svn/fbo-sync rev: 0 user: svn propname: svn:sync-currently-copying action: D
repos: /home/svn/fbo-sync rev: 0 user: svn propname: svn:sync-lock action: D

The first two lines of that example are the result of an svnsync without a revision update. The rest describe svnsync's processing of revision 96. Of importance is the order in which properties on that revision are modified. Further tests showed that they always occur in the same order (log, author, date). I replaced the post-revprop-change script above with the following:

#!/bin/bash

TRAC_ENV="/home/trac/sites/fbo"
REV="$2"
PROPNAME="$4"

if [ $PROPNAME = "svn:date" ]; then
    /usr/bin/python /home/svn/trac-post-commit-hook -p "$TRAC_ENV" -r "$REV"
fi

This script executes trac-post-commit-hook after each svn:date revprop modification. Because that is the last change svnsync makes on each revision, the trac instance is updated as expected, crediting the user who made the original commit to the repository at Google Code.

This is also fragile. If the order of the revprop changes were to switch, it would break. I'm trying to imagine what the ideal solution would be, but nothing comes to mind. In any case, the post-revprop-change script solves the problem at present.

p.s. At some point in my testing, I ran into the "Failed to get lock on destination repos" error after an aborted sync. I followed the advice in the "using svnsync" article at Paul's Journal and ran:

$ svn propdelete svn:sync-lock --revprop -r 0 file:///home/svn/fbo-sync

That worked for me as it did for him. Many thanks.

Releasing the Law to the Public

2009-01-12T11:44:47-05:00

Brett brought Public.Resource.Org's press release to my attention. Notable for both the challenge to the Westlaw and LexisNexis duopoly as well as the new Creative Commons mark.

WoW and SL

2009-01-12T11:44:47-05:00

I've been pondering the differences between WoW and SL, wondering what makes WoW more addictive for me, and today I ran across Casey's post on social networks. I like Stutzman's ego/object-centric dichotomy that Casey explores, but I think I'm disagreeing with him by saying both WoW and SL are "ego-centric" (I'm not sure which camp he puts WoW into). WoW is all about creating a kick-ass character, and I imagine that same sense of accomplishment in building a new person is what drives the hardcore SL fans. The two big differences I can identify are as follows:

WoW provides a framework, as Casey mentions, and I like the story it tells and the mythology behind it. I think part of the reason I escape into games is that life has too many choices, and I like being in a place where your objectives (quests) are listed out for you. (Consequently, I've been writing down more lists for my day-to-day stuff, and hey, they're useful!)
I've always preferred human vs. machine games over human vs. human. WoW provides both, and the human vs. machine one incorporates cooperation as a nice bonus. In other words, I could play WoW without the social aspect and still enjoy it. SL, on the other hand, has a little bit of the "beat the machine" in figuring out how to make objects and program them to do stuff, but if I'm going to be programming I'd rather be getting paid to do it. I'd rather be creating and fixing stuff that will be of use to people outside of SL, which is still a bigger community than inside, even considering SL's impressive numbers. I'd rather be working on something open source that's not constrained to a particular company's framework. And I'd rather be doing it in Python.

But I digress. Looking at Casey's post again, it occurs to me that the way I play WoW isn't really an ego-centric social experience. In fact, it's not social at all. Okay, maybe it's a little social, like when I get help on a quest, but my goal isn't "network-(re)establishment". I'm there to beat the game. As long as I feel there's more of the game to beat, the "What's next?" is answered. For some people, WoW is more social, and more like SL, but not for me.

And that might be the reason why I'm not a big fan of SL or social networking sites in general. While I appreciate the networks I have in my life (my neighbors, colleagues, friends, and family), I'm not one who looks forward to reestablishing those networks in a bunch of walled-off environments like Friendster, MySpace, Facebook, Orkut, and Second Life that each offer some variation on connecting to others on line. What interest I do have only holds if the benefit of the connecting tool outweighs the chore of setting up and maintaining a new network.

This reestablishment weariness isn't only true for ego-centric social networking sites. It's kind of a bummer having to reconnect with everyone to share bookmarks on del.icio.us and images on Flickr, as well. Which brings me to the article I read last night that's obviously affecting my train of thought here. Google is pushing something they're calling "OpenSocial", which seems to be some kind of standard or set of standards for social-networking site APIs. Sounds like a good idea. Standards are good.

Standards are great. Standards give me independence and control. That's another reason I don't get into the social-networking sites. I know how to make my own website, send email, and instant relay chat, and I don't have to rely on any one company or organization to continue doing those things. If OpenSocial lets me take my networks with me (maybe in cooperation with something like OpenID), then I'm all for it.

So Long, and Thanks for All the Edits

2009-01-12T11:44:47-05:00

It dawned on me some morning in one of the last few years that Wikipedia is now and may forever be the closest thing we'll get to a real Hitchhiker's Guide to the Galaxy. Today that was confirmed by one Oliver Lardner, who, in the comment sent along with his $50 donation to Wikipedia, said, simply, "don't panic." I happened upon it while glancing through the list of publicly viewable recent contributions, a list full of hope and humor written in myriad languages by people around the world.

Access 2007

2009-01-12T11:44:47-05:00

Hackfest was great. It seemed more productive this year than last, both personally and for others I talked to. Part of my productivity was due to my intent to work on a project I suggested: Refactoring Fac-Back-OPAC. We got a new branch created and did some code cleanup and merging with Helios, and I signed Birkin, Harish, and Mark up as committers on the code repository. I didn't take any photos, but I did one small video.

From "Observing Student Researchers in their Native Habitat": "Students do not know the difference between electronic resources and what's in the OPAC.... That's the biggest disconnect that we found. If you have a federated searching tool, put it up front."

Koha Demo

2009-01-12T11:44:47-05:00

Koha initially developed by Katipo Communications for Horowhenua Library Trust in New Zealand in 2000. Joshua Ferraro was at Nelsonville Public Library when it was adopted there in 2001.

Demonstration of OPAC at Nelsonville -- good searching capabilities (try "it" and "o") -- great sorting features -- "refine your search" faceting.
Everything is over the web. Not just OPAC, but all staff modules.

NEU Grand Library has image maps to books in the library.

Koha allows any encoding, but uses unicode internally. See http://opac.liblime.com "change language" dropdown for translation examples.

KohaZOOM OPAC demo at http://zoomopac.liblime.com/.

Koha's OPAC is just a Z39.50 client, and the backend is a Z39.50 server, which is why it's easy to integrate with Masterkey.

We might have difficulty laying KohaZOOM over III because III's Z39.50 server is slow and unstable. Most likely we'll need to pull everything out of the catalog and drop it in an indexer -- probably either Zebra or Solr.

We tossed around ideas for global updates. Cataloging interface fills in field defaults upon select.

Uses HTML::Template for templates.

Many thanks to Nicole Engard for putting the event together.

XML in Python

2009-01-12T11:44:47-05:00

After looking around (see links at bottom) I've come to the conclusion that, for now, the simplest, yet mostly successful, way to safely include input from various places is with xml.sax.saxutils and encode('ascii', 'xmlcharrefreplace'), like so:

>>> from xml.sax import saxutils
>>> xml_snippet = u'ö < på > & бб лн'
>>> xml_snippet
u'\xf6 < p\xe5 > & \u0431\u0431 \u043b\u043d'
>>> saxutils.escape(xml_snippet)
u'\xf6 &lt; p\xe5 &gt; &amp; \u0431\u0431 \u043b\u043d'
>>> saxutils.escape(xml_snippet).encode('ascii', 'xmlcharrefreplace')
'&#246; &lt; p&#229; &gt; &amp; &#1073;&#1073; &#1083;&#1085;'

The conversion of > to > is not required, but I prefer it, so I'm glad it's the default in saxutils. Also, we could encode to iso-8859-1 instead of ascii, but, as Mr. Lundh points out in [3], it's still safer to use ascii.

links

[1] http://www.xml.com/pub/a/2005/06/15/py-xml.html [2] http://www.diveintopython.org/xml_processing/unicode.html [3] http://online.effbot.org/2003_10_01_archive.htm#20031016 [4] http://boodebr.org/main/python/all-about-python-and-unicode

Quick Commenting of Blocks in Vim

2009-01-12T11:44:47-05:00

For a long time I have struggled with vim to comment blocks for any language or configuration file that only provides single line commenting (for me, usually python). Today a light turned on somewhere (and I got some help in code4lib from decasm), and I realized that to comment 4 lines all I have to do is

0 Ctrl-V jjj I # Ctrl-C

The Ctrl-C could be Esc, of course, but I'm used to Ctrl-C by now and there's an odd pause when I use Esc. To uncomment that text, do

0 Ctrl-V jjj d

The d could be substituted by an x, of course.

Ah, Vim. A minute to learn, a lifetime to master.

Update: As noted in my unanswered message on the vim-dev list (maybe "bug or a feature" sounded too flippant), Ctrl-C no longer works here. Too bad, as it's about the only time I hit Esc in vim.

SRW installation on the DSpace server

2009-01-12T11:44:47-05:00

Went to the OCLC Research SRW Server 2.0 Installation Instructions and downloaded SRW.zip and unpacked it in /usr/local/, then

# cd /usr/local/SRW-2.0/
# chown -R dspace:dspace .
# chmod -R g+w .

Then as user dspace I edited conf/SRWServer.props to change db.DSpace.home=d:/dspace/ to db.DSpace.home=/dspace/ and the db.DSpace.configuration line to db.DSpace.configuration=config/DSpace.SRWDatabase.props

Then I copied conf/DSpace.SRWDatabase.props to /dspace/config. Edited lines in that to brand the site and set administrator and added this important line:

indexSynonym.cql.serverChoice=default

That line sets the default query (e.g. query=horses) to search the "default" field in lucene, which is the same one that's searched in the "simple search" on the DSpace site.

Then I ran

$ ../apache-tomcat-5.5.17/bin/shutdown.sh
$ ant deploy
$ rm -Rf ../apache-tomcat-5.5.17/webapps/SRW
$ ../apache-tomcat-5.5.17/bin/startup.sh

Then later (much later) I figured out I had to edit /usr/local/SRW-2.0/build.properties and change the beginning of it to read:

# specify the name of the webapp.  Some sites use SRW in lower case.
webappName=SRW

# if you want the "deploy" ant target to work, you need to specify the location
# of your tomcat server below

serverPath=/usr/local/apache-tomcat-5.5.17/

# if you want support for DSpace included within the SRW server, provide
# the path to the directory that has the dspace, lucene and postgresql jars.
# if this property is not defined, then DSpace support will not be included.

dspaceLib=/dspace/lib

Instructions to edit the build.properties file are hidden in the very beginning of the Installation Instructions.

Also, I noticed D:\Apache Tomcat 4.1\webapps was showing up in weird places in the filesystem -- found it in conf/server-config.wsdd and replaced it with /usr/local/apache-tomcat-5.5.17/webapps -- but they're still showing up.

DSpace in SVN

2009-01-12T11:44:47-05:00

My DSpace code is in an SVN repository for the library, signified here by $SVNHOME. Our live machine runs off the code in projects/dspace/trunk. Any modifications to our local DSpace instance are checked into trunk.

The official releases from dspace.org go in projects/dspace/branches/official. What I mean by this is that the "official" branch (as well as trunk) started as an exact copy of the dspace-1.4-source released by dspace.org. Each time a new version is released (e.g. 1.4.1, 1.4.2, etc.), I copy that release over whatever's in projects/dspace/branches/official and then remove extra files until the following returns nothing:

$ diff -r src/dspace-1.4.2-source/ svn/library/projects/dspace/branches/official/ | grep -v .svn$

Then I commit that as the update to the official branch.

The log looks like so:

gsf@manheim:~/svn/library/projects/dspace/branches/official$ svn log
------------------------------------------------------------------------
r171 | gsf24 | 2007-07-26 12:53:39 -0400 (Thu, 26 Jul 2007) | 1 line

official update 1.4.2-beta to 1.4.2
------------------------------------------------------------------------
r168 | gsf24 | 2007-05-08 18:28:03 -0400 (Tue, 08 May 2007) | 1 line

official update 1.4.1 to 1.4.2-beta
------------------------------------------------------------------------
r131 | gsf24 | 2007-01-24 17:11:07 -0500 (Wed, 24 Jan 2007) | 1 line

moved dspace/branches/dspace-official to dspace/branches/official
------------------------------------------------------------------------
r94 | gsf24 | 2007-01-19 14:52:27 -0500 (Fri, 19 Jan 2007) | 1 line

dspace-official update from 1.4 to 1.4.1
------------------------------------------------------------------------
r92 | gsf24 | 2007-01-19 12:27:02 -0500 (Fri, 19 Jan 2007) | 1 line

import dspace-1.4-source into branches/dspace-official
------------------------------------------------------------------------

To upgrade the trunk, which includes local changes, from 1.4.1 to 1.4.2-beta, for example, I would then execute the following from within a working copy of the trunk:

$ svn merge -r 167:168 $SVNHOME/projects/dspace/branches/official

svn st should then list a large number of modified files, and possibly a few conflicting ones. I go in and resolve the conflicts, commit that to trunk, and, ta-da, my local instance has been upgraded.

As an aside, this technique obviates the need for JSPs in jsp/local/. Local JSPs would just need to be modified upon upgrade to take advantage of UI advances anyway, so I prefer to edit the general JSPs and resolve conflicts.

Debian -- Missing Public Keys

2009-01-12T11:44:47-05:00

Recent aptitude update replied with

...
W: There are no public key available for the following key IDs:
A70DAF536070D3A1
W: You may want to run apt-get update to correct these problems

Checking the recent archives for the debian-user mailing list, I found this thread, with the easiest answer being

# gpg --recv-key A70DAF536070D3A1
# gpg --export A70DAF536070D3A1 | apt-key add -

I'm guessing that works because the key is in the key server (subkeys.pgp.net for me), but it hasn't been added to the debian-archive-keyring package yet. This email led me to that conclusion.

Update 2007-07-04: The recv-key line may include the keyserver, like so:

# gpg --keyserver keyring.debian.org --recv-key A70DAF536070D3A1

Access, Day Two

2009-10-15T14:29:19-05:00

Roy: two click solution (attributed to Jeremy Frumkin) -- one click to search, one to get. Easier to find the book Before Taliban through Google, but only available through "snippet view". NCSU Catalog Availability Prototype. Need to be ready for people coming from Google. Need to provide better discovery of open content (even Google doesn't do such a bang-up job). OpenURL resolvers are one step -- need to go further. Why take them to a catalog screen when it's available full-text. Why show them ILL, for crying out loud! Umlaut and GUF. Google-like experience. OpenURL resolver. Aggregators -- more important as IRs grow.

(Here's an idea: a centralized service that points at freely available full-text books. Users can add sources, of course. Maybe some added eye candy to pull people to the site -- an ajaxy reader or something. But the real value is a web service that libraries, etc. can point at to provide links for full-text entries in the catalog.)

Anne Christensen -- library chatbots. http://www.sub.uni-hamburg.de/ Stella.

Annette Bailey -- LibX. Firefox extension that integrates library resources into the browser.

Thunder talks

Ross Singer -- Umlaut and U. Link resolvers are a closed door leading to a room full of closed doors. Came from Communicat.
Mark Jordan -- Drupal Hacks. http://drupalib.interoperating.info/ Content Creation Kit -- create your own fields.
Peter Binkley -- faceted search with Solr. Bitsets.
William Oldfield -- http://etheses.uwaterloo.ca/
Dan Chudnov -- lots of talks. Dialtone. (todo: zeroconf)
Grant Gelinas-Brown -- Get over it, Fanboy. Get into something, roll up your sleeves, and do it.

Nora Young, Access to information in an age of social media

How do kids use the internet? MySpace.

Cynthia Carlton, eXtensible Catalog

Rochester, Mellon Grant. http://www.extensiblecatalog.info http:///www.library.rochester.edu/access

Update 2006-11-21: The only snippet I managed to catch from Day 3 was Cliff Lynch saying, "For the University of Michigan, Google is the first backup strategy ever implemented." Audio of all of the talks, as well as links to many of the slides, and results from the hackfest, are up at the University of Ottawa Access 2006 site. Also, I've put on line the presentation I gave at Drexel.

At Access

2009-10-15T14:29:19-05:00

9:15-10:00 -- Art Rhyno and Walter Lewis spoke on Knowledge Ontario, Ty Cobb, and Lucene. A highlight was the added value of user comments in records. Overall, nice connections drawn between local history (stories found only in archives when disparate records can be collated through metadata)

10:30-11:00 -- Tito Sierra spoke about using Endeca at NCSU Libraries Searching (and finding) is the unofficial theme of the conference.

11:00-12:00 -- Lightning Talks. Zebra used to allow searching the University of Victoria Library catalog during a four-day Voyager upgrade.
Lucene-ws -- wanted to do indexing in Perl -- the web service allowed the client to be written in whatever, as long as it could do atom (Brian Cassidy).
LibraryFind at OSU and caching federated searching -- PHP/MySQL for now, soon to be Ruby on Rails -- jaf and Mr. T (Jeremy Frumkin). Georgia PINES and Evergreen

Installing Debian on the Thinkpad X31

2009-01-12T11:44:47-05:00

My X31 is the 2672-RHU model. I'm using the Lite On external USB DVD R/RW I mention on x31. This thinkpad boots off this with no problems. The install disk I'm using is Debian Netinst 3.1r1a.

Basic Installation

I hit enter at the prompt, then on the install screens I

choose English
United States
Dvorak
hostname: magnum
domain: touretteskimo.com (not the real domain, but that doesn't matter)
Erase entire disk: IDE1 master (hda) - 40.0 GB IC25N040ATCS04-0
All files in one partition
Finish partitioning and write changes to disk
<Yes>
Now I wait for it to install the base system. While we wait, I'll mention that I was kinda thinking about dual-booting Debian and FreeBSD, but really I'm so deep into Debian at this point and I just don't want to bother learning another distribution/operating system when it seems there are better things I could be doing with my time. (What they are, I'm still figuring out.) Also, if I really want to try something else, that's the perfect time to check out this virtualization thing everyone's been talking about.
<Yes> to install the GRUB boot loader in the master boot record.
Let the cd eject, hit <Continue>
Watch it reboot.
At the "Welcome to your new Debian system! blah blah blah" page hit <Ok>
Yes to hardware clock set to GMT (even if it's not now, I'll do it later).
time zone: Eastern
set Root password: ******
full name for the new user: Gabriel Farrell
username: gsf
password: ******
Archive access method for apt: http (ethernet card already plugged in)
United States
ftp.debian.org
no proxy
wait for it to connect and download package lists, and install mdetect (it seems to be pausing to resolve the name -- maybe because of ipv6? okay, now it's stuck, waiting to download mdetect -- I try ^C, but it's a stubborn bastard. It gives up on mdetect and tries for read-edid. Now this is getting interesting. Can't connect to ftp.debian.org, even though it did a minute ago to download the package lists. I jump over to another virtual terminal with LeftAlt+RightArrow and ping ftp.debian.org and I get immediate pongs from debian-mirror.mirror.umn.edu. Maybe the mirror I got for mdetect and read-edid was busted. Doesn't matter, I'll be updating everything anyway.)
leave nothing checked in Debian software selection -- I'll do that later
wait while more core system stuff, like libc6, is downloaded and installed
I get a prompt for Configuring Exim v4 (exim4-config) and select local delivery only
root and postmaster mail recipient: gsf
at Thank you for choosing Debian! hit <Ok>
plopped down onto the terminal login, I log in and su to root (now comes the fun stuff)
first thing to do is vi /etc/apt/sources.list and add matching testing lines for each line in the file, except the #deb file:///cdrom/ sarge main, which can be deleted. It's important to leave the sources pointing at
stable because a few packages still reside only in stable, and upgrades are messier if you don't have that package list at your disposal. I could save myself some of these steps if I had a netinst cd for the testing distribution of Debian. I thought I had one lying around. Oh well.
# aptitude, then u, U, and g. I check out everything to be upgraded, select a couple in the recommended area -- most importantly
debian-archive-keyring, which will keep aptitude from thinking it's downloading unspecial packages. Hit g.
Default answers to everything
got errors about my locale not being set or something, and a bunch of packages didn't install (this is exciting!). Tried to do dpkg-reconfigure base-config but I was told it is broken, so I ran aptitude dist-upgrade, and had to say yes to downloading "unsafe" packages (since I haven't done an aptitude update with the new, safety-checking aptitude I just installed). Looks like everything installed okay this time. Along the way I was told to purge hotplug, so when it finished I did aptitude purge hotplug.
Ran aptitude update again, and all looked good. That's the end of the basic installation. I'll work on getting everything else and configuring stuff like X and alsa and bluetooth and etc. tomorrow.

Making it Fancy

Now it's time to amp up the system and get a bunch of toys. Oh wait, but first, a couple more little things.

Add blacklist ipv6 to the bottom of /etc/modprobe.d/blacklist to make the internet fast.
In /etc/ssh/sshd_config change PermitRootLogin to no. This shouldn't ever really be a problem, but better safe.

Okay, on to the fun stuff. Going into aptitude again, I hit u and select the following packages for install (in the order I think of them):

vim (can't really live without this one)
linux-image-2.6-xen-686 (need to update that kernel, and I'm thinking about virtualization)
gpm (handy for selecting text on the console)
screen (can't much live without this, either)
xorg (for gui stuff)
xfce4 (ditto)
lynx (I'm just more used to it than w3m)
firefox (unfortunately, can't really live without that. No dissing firefox, it's amazing, it's design is good in a lot of ways (extensions++), but it's just such a monster. What with the web becoming Your Networked Operating System these days, I feel the need for options. I guess it's just not indie enough for me.)
backup2l (been using this to back up to the external and it works great)
cplay (the best music app I've found)
tpb (for the fancy osd)
wireless-tools (no-string-attached goodness)
bluez-tools (ditto)
xfonts-terminus (better fonts for x terminals)
xli (for pictures)
firefox-webdeveloper (that's a good one)
firefox-dom-inspector (another good one)
firefox-sage (what the hell, we'll try that too)

Okay, I'm sure I've forgotten a bunch, but that's plenty enough for now. Hit g and select a couple more from the suggested area.

a2ps (make sure I can print things)
menu (keep menus in sync in xfce)
python-id3 (for cplay to see id3)
sox (also for cplay, for wav files)
mpg321 (cplay, mp3)
sudo (need this for the gui shutdown in xfce)
vim-doc (I utilize the vim docs often)

Okay, hit g again and watch it go. Went with all the defaults when prompted. (Prompted for X stuff twice -- maybe because of xorg and x11-common?) (No, make that thrice. Huh.)

tpb told me to put users in the nvram group, so after I quit aptitude I addgroup gsf nvram. Now lets reboot to see what this new kernel image does for us.

Oh wait, it's not here, in the boot menu -- just the old 2.4.27 one. What did I miss? Go back into aptitude and check out linux-image-2.6-xen-686. It's installed, and so is linux-image-2.6.17-2-xen-686. Quit aptitude and lynx /usr/share/doc/ and look at the linux-image directories. Nothing there. Hmm. /boot contains the 2.6.17 files, but /boot/grub/menu.lst wasn't updated.

Looking around the web I see Joey Hess talking about the need for a Debian-specific guide for Xen and Julien Danjou talking about installing xen. My conclusion: wait on Xen and devote some days and separate posts to it later. So I installed good old

linux-image-2.6-686

Looks like it updated /boot/grub/menu.lst. Quit aptitude and reboot again.

And we're up. Looks good. Oh, for wireless we also need

ieee80211-modules-2.6-686

and I also want

ntp (for keeping the clock right)
comixcursors (the cursor theme I can't live without -- see my page on
changing the mouse cursor)

Those installed without a hitch. I've noticed I keep hitting the Thinkpad back and forward buttons (by the arrow keys) to switch consoles, but they're not set up. I take a look at the thinkwiki help for setting the special keys to refresh my memory and add the following lines to /etc/console/boottime.kmap.gz:

keycode 158 = Decr_Console
keycode 159 = Incr_Console

Executing loadkeys /etc/console/boottime.kmap.gz shows it works.

While I'm at it, I'll vim ~/.Xmodmap as gsf and add

! Access IBM button
keycode 159 = XF86LaunchA
! Back button
keycode 234 = F19
! Forward button
keycode 233 = F20
! Fn button
keycode 227 = F35

Okay, let's try starting up X with startx (as gsf). Not a hitch! The display is good, we got the mouse cursor, the flashing xfce rodent, all right. First thing I do is some basic configuration. Move the clock to the upper right corner. Get rid of a couple separators and frames. For some silly reason the xterm icon is a speaker, so I'll change that. There's a bunch of other little configuration things that I do to xfce, but not worth mentioning. Easy enough to make it your own, however you want. Yay xfce. I will say I prefer the Prune style window manager and a blueish desktop. And for stuff like .Xmodmap and tpb to work you have to open a terminal and run

gsf@magnum:~$ xmodmap ~/.Xmodmap
gsf@magnum:~$ tpb -d

Oh, except tpb -d spits back

Unable to open device /dev/.static/dev/nvram: Permission denied

This happens because tpb was installed before we had moved to the new kernel and udev, so I do a aptitude reinstall tpb as root and rerun tpb -d as gsf and we're good to go. Also, open up Window Manager in the Settings Manager, hit the Keyboard tab, and change Next Workspace to the Forward button (should show up as F20 because of ~/.Xmodmap) and Previous Workspace to the Back button (F19). Um, also in Window Manager in the Advanced tab uncheck "Snap windows to screen border", "Display content of windows when resizing", and "Display content of windows when moving". The last two options are a big part of why I like xfce. It's so easy to enable wireframes for moving and resizing.

Day 3

Okay, so we've got xfce up and running and I can do most things that I like to do on my computer. I did an aptitude install subversion so I could check my articles out; now I'm editing this on the X31. Once I get markdown syntax highlighting for vim and my vimrc I get an even fuller editing experience. (More on that at the about page).

Okay, let's focus on getting it to hibernate. First thing I do is consult http://thinkwiki.org. Reading a page there called How to make ACPI work I decide to check out /boot/config-2.6.17-2-686 to see how the kernel's configured, and it looks like we've got ACPI support. I install acpid as they recommend, and get the following complaint:

Starting Advanced Configuration and Power Interface daemon: acpid: 
can't open /proc/acpi/event: Device or resource busy

I googled on that and found a thread on Powersaved complains acpid is not installed that suggested the use of lsof /proc/acpi/event. First time I've heard of lsof. Handy one, that. It told me

magnum:/home/gsf# lsof /proc/acpi/event
COMMAND    PID      USER   FD   TYPE DEVICE SIZE       NODE NAME
hald-addo 3906 haldaemon    4r   REG    0,3    0 4026532116 /proc/acpi/event

So it's a conflict with hald. But the very next google hit was Bug#380520: marked as done (acpid and hald conflict), which says it's fixed in the version of hal I've got. Hmm. Try a reboot.

Day 4

Okay, so day 3 was not so productive. We'll try to get more done today. On the reboot, acpid started up before hald, and no one complained, and now

magnum:/home/gsf# lsof /proc/acpi/event
COMMAND  PID USER   FD   TYPE DEVICE SIZE       NODE NAME
acpid   3947 root    3r   REG    0,3    0 4026532116 /proc/acpi/event

but still nothing happens when I hit the "little blue moon" suspend button. Well, not nothing. If run acpi_listen, then hit the blue moon, I get

gsf@magnum:~$ acpi_listen
button/sleep SLPB 00000080 00000009

So it hears the button, and knows what it is, but does nothing. Ah, well, there's the following line in /boot/config-2.6.17-2-686:

CONFIG_ACPI_IBM=m

So let's modprobe ibm_acpi. Hmm, no change. I'll stick it in /etc/modules anyway, just to be on the safe side. I found the acpi sleep command, echo mem > sys/power/state. It goes to sleep (little green moon lights up and all), so I know that's working. (I also did the radeonfb stuff that page suggests). I'm thinking I can put another script in /etc/acpi, but then I see the "if pidof powersaved" section at the top of /etc/acpi/powerbtn.sh, so I decide to install powersaved and see how that works. I would install tpctl, but it depends on thinkpad-source, and I don't want to build my own kernel here, having followed that path in the past and felt I was missing out on the easy upgrading involved in sticking with the stock Debian. After it's installed, I try a reboot.

Still no luck, but I notice I get complaints on tty1 about trying to suspend to disk and needing to set up a resume partition. I'm gonna fly by the seat of my pants and suspend to ram, so I edit /etc/powersave/events so that EVENT_BUTTON_SLEEP="suspend_to_ram". Now hitting the blue moon suspends to ram. Only problem is the screen stays lit, but I'll work on that another day.

Day 11

Okay, not really, but feels like it. Trying to get wireless to work, I modprobe ipw2100 and get:

FATAL: Error inserting ipw2100 (/lib/modules/2.6.17-2-686/kernel/drivers/net/
wireless/ipw2100.ko): Unknown symbol in module, or unknown parameter (see dmesg)

dmesg tells me

ipw2100: disagrees about version of symbol ieee80211_wx_get_encodeext
ipw2100: Unknown symbol ieee80211_wx_get_encodeext
ipw2100: disagrees about version of symbol ieee80211_wx_set_encode
ipw2100: Unknown symbol ieee80211_wx_set_encode
ipw2100: disagrees about version of symbol ieee80211_wx_get_encode
ipw2100: Unknown symbol ieee80211_wx_get_encode
ipw2100: disagrees about version of symbol ieee80211_txb_free
ipw2100: Unknown symbol ieee80211_txb_free
ipw2100: disagrees about version of symbol ieee80211_wx_set_encodeext
ipw2100: Unknown symbol ieee80211_wx_set_encodeext
ipw2100: disagrees about version of symbol ieee80211_wx_get_scan
ipw2100: Unknown symbol ieee80211_wx_get_scan
ipw2100: disagrees about version of symbol ieee80211_set_geo
ipw2100: Unknown symbol ieee80211_set_geo
ipw2100: disagrees about version of symbol ieee80211_rx
ipw2100: Unknown symbol ieee80211_rx
ipw2100: disagrees about version of symbol ieee80211_rx_mgt
ipw2100: Unknown symbol ieee80211_rx_mgt
ipw2100: disagrees about version of symbol free_ieee80211
ipw2100: Unknown symbol free_ieee80211
ipw2100: disagrees about version of symbol alloc_ieee80211
ipw2100: Unknown symbol alloc_ieee80211

I think maybe the xen kernel image I installed earlier might be the cause of these version disagreements, so I go into aptitude and purge

linux-image-2.6-xen-686
linux-image-2.6.17-2-xen-686
linux-modules-2.6.17-2-xen-686

Still getting the error, so try a reboot.

Back from reboot, still getting the error.
aptitude reinstall ieee80211-modules-2.6.17-2-686. Still getting the error. Check http://bugs.debian.org/ieee80211-modules-2.6.17-2-686, looks like a similar problem there with ipw3945. Hmm, I might have to compile the kernel. Let's try aptitude remove ieee80211-modules-2.6.17-2-686 -- same problem. Reboot.

That fixed it! I can now modprobe ipw2100. So we don't need the ieee80211-modules. But the wireless still isn't functioning. This post on ipw2100 in etch suggests I might need the firmware. I'm gonna assum the driver in the kernel is after v0.55. Problem now is, where to put it so that it's loaded, used to be you put it in /lib/firmware and hotplug would just see it, but I'm not sure about udev, which has supplanted hotplug. I'll just try untarring it in there, add ipw2100 to /etc/modules, and reboot.

An lsmod shows ipw2100 is installed, but nothing shows up as wireless when I iwconfig. Not until this long-overdue point am I reminded by Debian on a T41 to check lspci, which tells me

02:02.0 Ethernet controller: Atheros Communications, Inc. AR5212 802.11abg NIC (rev 01)

Right! Now I remember why I couldn't use ipw2100 the last time around -- my X31 has the Atheros chipset, not Intel. So I think MadWifi is my only choice. Before I forget, I remove ipw2100 from /etc/modules and remove the ipw2100 firmware from /lib/firmware. By comparing Installing a WLan-Card with Atheros Chipset under Debian Linux with the MadWifi Debian pages and the MadWifi First Timer's Guide I decided to aptitude install linux-headers-2.6-686, then, as user gsf, I

gsf@magnum:~/svn$ svn checkout http://svn.madwifi.org/trunk madwifi
gsf@magnum:~/svn$ cd madwifi/
gsf@magnum:~/svn/madwifi$ make
[bunch of output, no errors]
gsf@magnum:~/svn/madwifi$ su
Password:
magnum:/home/gsf/svn/madwifi# make install
magnum:/home/gsf/svn/madwifi# modprobe ath_pci
magnum:/home/gsf/svn/madwifi# iwconfig
lo        no wireless extensions.

eth0      no wireless extensions.

eth1      no wireless extensions.

irda0     no wireless extensions.

wifi0     no wireless extensions.

ath0      IEEE 802.11b  ESSID:""
          Mode:Managed  Channel:0  Access Point: Not-Associated
          Bit Rate:0 kb/s   Tx-Power:0 dBm   Sensitivity=0/3
          Retry:off   RTS thr:off   Fragment thr:off
          Encryption key:off
          Power Management:off
          Link Quality:0  Signal level:0  Noise level:0
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:0   Missed beacon:0

Success! I think. I added the following to /etc/network/interfaces:

iface ath0 inet dhcp

and ifup ath0 seems to be looking for connections. There's also this:

magnum:/home/gsf/svn/madwifi# modprobe wlan_scan_sta
magnum:/home/gsf/svn/madwifi# wlanconfig ath0 list scan
SSID            BSSID              CHAN RATE  S:N   INT CAPS
dragonfly       00:0f:8f:4f:da:70    1   54M 12:0   100 EPSs WME
dragonfly       00:0f:8f:38:7a:70    6   54M 27:0   100 EPSR WME
dragonfly       00:0f:8f:4f:e0:30    6   54M 13:0   100 EPSR WME
dragonfly       02:02:6b:d1:bd:3a    5   11M 23:0   100 I
airportthru     02:fa:5a:c9:fd:a9   10   11M 18:0   100 I
dragonfly       00:0f:8f:4f:db:e0   11   54M  6:0   100 EPSR WME

So it's seeing Drexel's networks, and I've got some signal strength (under S:N). Plus, the little green radio tower is blinking. I think it's working well enough for me. Yay! Let's add ath_pci and wlan_scan_sta to /etc/modules.

A couple of quick superficial things now. As root, update-alternatives --config x-cursor-theme and select white regular theme, because I like that better, and xfce doesn't switch it via the gui mouse cursor selection settings window. Next, aptitude install gdm and at the login screen (reboot or quit X and /etc/init.d/gdm start) set to log in automatically. Then, following the steps outlined in How normal users shut down Linux, I visudo and add

gsf     ALL=(root)  NOPASSWD:/usr/sbin/xfsm-shutdown-helper

and the change is immediately visible when I click the "log out" button.

Screen blanking

Okay, so I never got around to writing this part, but the answer lies, as expected, at thinkwiki.org.

Hype, for searching with Python

2009-01-12T11:44:47-05:00

So in setting up a search engine for my little artsy project I ran across Hype, a Python wrapper API for the Hyper Estraier full-text search system. Actually, I first ran across this blog post that, forgive the pun, hyped it up. It's a straightforward-enough API. The relevant sections of the code I put together for it looked like this:

...

db = hype.Database(indexPath)
for artFile in self.artCollect():
    print '%s found' % os.path.basename(artFile)
    self.index(artFile, db)
db.optimize()
db.close()

...

def index(artFile, db):
    # grab values and fill the index
    doc = hype.Document()
    doc['@uri'] = unicode(os.path.basename(artFile)[:-4])
    doc['@created'] = binding.article.created
    doc['@modified'] = binding.article.modified
    doc['@title'] = unicode(binding.article.title)
    doc.add_text(unicode(binding.article.title))
    doc['@description'] = unicode(binding.article.description)
    doc.add_text(unicode(binding.article.description))
    doc['@keyword'] = u''
    flag = 0
    for keyword in binding.article.keyword:
        if flag:
            doc['@keyword'] += u' '
        flag = 1
        doc['@keyword'] += unicode(keyword)
        doc.add_text(unicode(keyword))
    for content in binding.xml_xpath(u'/article/content//*'):
        doc.add_text(unicode(content))
    db.put_doc(doc)
    print '%s added to index' % os.path.basename(artFile)

I didn't get around to a search cgi, but it would look something like

db = hype.Document(indexPath)
# some regex madness i'll figure out later would go here
hits = db.search(unicode(query)).add(unicode(attBits)))
for hit in hits:
    out += '&lt;a href="%s"&gt;%s&lt;/a&gt;' % \
        (hit['@uri'], hit['@title'])

So the thing is that with Hype a document has short, non-tokenized attributes, assigned to variables with the "@" sign, and everything else is tokenized text (see the usage doc). This obviously mirrors the way most of us search: we go for the general and do a full-text for concepts or in recollection of a few words from an article or whatnot, or we go for the particular and look for a date range or something in the title, or we combine the two approaches. That model works fine most of the time, but it constrains in a way the PyLucene API doesn't (really it's the Lucene API, but since I'm using PyLucene I'll use that as the reference). In PyLucene I can choose whether or not to tokenize each field; I'm not forced into a number of non-tokenized attributes and one catch-all tokenized text field. Also, there's a clearer way to involve different tokenizers and analyzers in PyLucene, but I haven't really messed around with that yet so let's just focus on which fields can do what for now.

The truth is that even though I'm using PyLucene I'll still create a catch-all tokenized text field, and most everything else about these article files can be put into non-tokenized attribute-like fields. The exception is the "description" field. I know it's an edge case, so it shouldn't be of much concern, but say someone wants to search for something that only appears in the description of an article? With Hype, that means searching for it as an attribute, and that means it's not tokenized, which sucks because the description has sentences with punctuation and such that screw with STRINC searches (I guess part of the problem is also Hyper Estraier's limited attribute searching, but that's also due to the attributes-and-text document model).

Don't get me wrong, I'm not bashing Hyper Estraier or Hype, and the API presented could be the perfect fit for a lot of projects, but for me the extra bit of control that the PyLucene API presents is something that I can't go without. Now, if only Lucene weren't written in Java, and PyLucene didn't require an old, old version of GCJ and the new, new Python 2.4 (well it's not quite so new but it's still not the default in Debian) in order to run. Can't we please get a decent, native Python text indexer/searcher? Maybe I'll resurrect Lupy and turn it into the fastest, rockingest one ever. Guess I better start doing some research on how these things really work.

Keep Your Stuff Safe

2009-01-12T11:44:47-05:00

How many people in the world store the cardboard boxes and foam packaging from the things they buy, like home stereo components and appliances, so they can protect the things in their original boxes when they're moved to a new home? How much space is taken up by those boxes, worldwide?

First world problem, and I've got plenty in my basement. Some of them I kept for the things I thought I might return. I give thanks to the great magnetic force of consumption that gathers unnecessary things into one's purview. After the rapture, serene giants will make love on our mountains of fluff.

Printer setup

2009-01-12T11:44:47-05:00

Steps for setting up my HP DeskJet 400 attached via the parallel port to my IBM Thinkpad X31 running Debian GNU/Linux and CUPS:

# aptitude install cupsys foomatic-db-hpijs (Note: hplip does not appear to be necessary, and currently in unstable has circular dependency problems with hplip-data. cupsys-bsd and foomatic-etc. also not necessary.)
Need the following modules: parport, lp.
Open up http://localhost:631/admin in a web browser and put in "root" and root password.
- Give whatever name, location, description.
- Device: Parallel Port #1.
- Make: HP.
- Model: HP DeskJet 400 Foomatic/hpijs (recommended) - HPLIP 0.9.10 (en)
- At printer page, click on "Configure Printer" button.
- Have a black cartridge, so change "Printout Mode" to "Draft Grayscale (Black cartridge)". (Or use Normal Grayscale, doesn't seem to make a difference.) Color printout mode without a color cartridge results in garbled output.

That's it. Don't know if the printer's just old, but if I put more than one sheet in the paper tray it eats them all at once. So, one page at a time. Also, I keep putting the scratch paper in the wrong way, so it prints on the used side.

Kernel compile

2009-01-12T11:44:47-05:00

2.6.16

`aptitude install linux-source-2.6.16`
Untarred linux-source.tar.bz2 in /usr/src/
Copied .config from /usr/src/kernel-source-2.6.11 to /usr/src/linux-source-2.6.16
Checked out the Linux Kernel Configuration Archive and consulted the trustworthy GNU/Linux on an IBM ThinkPad X31
Made some minor tweaks (found I could best tweak by looking at the .config file in one terminal while going through the `make menuconfig` selections in another)
- Made ext2 support a module, since I'm running ext3 (`CONFIG_EXT2_FS=m`)
- No rom or cram support, since I'm not using initrd (`# CONFIG_ROMFS_FS is not set` ... `# CONFIG_CRAMFS is not set`)
- Got rid of all usb network adapters except `CONFIG_USB_MON=y`, which were compiled as modules for some unknown reason
- Considered dropping apm for acpi, but decided to wait. If it ain't broke don't fix it, right? I mean, the tradeoffs are about equal, so it's just not worth it at this point. Also selected `CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y`.
- Added `CONFIG_PRINTER=m`, which I needed
- Annoyance: `CONFIG_RAMFS=y` shows up in .config, and there's no listing for it in menuconfig. Does that mean it's necessary for something else I selected, or just forgotten in the Kconfig?
Left madwifi and thinkpad modules' source in /usr/src/

Ran the following:

# make-kpkg clean modules_clean
# make-kpkg --revision=magnum.0.7 kernel_image
# make-kpkg modules_image
# dpkg -i linux-image-2.6.16_magnum.0.7_i386.deb
# dpkg -i madwifi-module-2.6.16_20050124-1-onoe+magnum.0.7_i386.deb
# dpkg -i thinkpad-modules-2.6.16_5.9-2.1+magnum.0.7_i386.deb

Reboot
Had to go into modconf and install the `ath_pci` module to get wireless working, same for apm and some others.
That's it for now! Here's my kernel .config file.

Addendum: Thinkpad modules won't install. Found this bug. Don't really need them, anyway.

Dude, Where's my Briefcase?

2009-01-12T11:44:47-05:00

I always loved seeing this silly waste of taxpayer money on the train. Thankfully, the MTA has put it up on their website for us as well. I guess it's not a total waste if the absurdity of it all brightens up my day a little.

Kucinich at AMI

2009-01-12T11:44:47-05:00

Just watched some video at the American Monetary Institute (AMI) from the recent Monetary Reform Conference. Dennis Kucinich gave the keynote. I agree with most everything he says, and he knows his stuff. Here's a quote:

What else is this war, but a transfer of the wealth of this nation upward, to the arms manufacturers, to the Halliburtons.

He refers to Henry George twice in his talk, and to Progress and Poverty in particular. Here's another good quote (at minute 15:30):

In 1901, the person who was the best Mayor of Cleveland, Tom L. Johnson, a friend of or, rather, a disciple of Henry George, wrote, "I believe in public ownership of all public service facilities: the waterworks, the parks, and schools (and on and on), because if you do not own them they will, in time, own you. They will corrupt your institutions, rule your politics, and, finally, destroy your liberty."

He knows some good Latin, like novus ordo seclorum, "a new order follows" (5:30) and, from the Canterbury Tales, radix malorum est cupiditas, "money is the root of all evil" (27:33) (I checked the accuracy of the latter Latin phrase at The Route of All Evil. Ah, the internet, how I love thy ways.). Probably it's his Catholic upbringing.

This page on Tom L. Johnson says he was buried next to Henry George in Brooklyn. A quick look on findagrave.com shows that they're buried at Green-Wood Cemetary, a scarce 30 minutes from here by bike. Maybe it'd be a nice thing to do on a Sunday before I leave for Philadelphia.

I also watched the video of Dave Kelley's talk. He was Kucinich's "Issue Consultant", according to the recent issue of GroundSwell I got in the mail (that's where I found out about AMI and its conference). Kelley mentions When Corporations Rule the World, by David C. Korten, as a must-read, as well as Wealth and Democracy. Maybe I'll check them out.

Debian on the Thinkpad X31

2009-01-12T11:44:47-05:00

This page is where I'll either put, or link to, lessons learned in the installation and maintenance of Debian GNU/Linux on my Thinkpad X31. One about the mouse cursor has already been put up, and there are many more to come.

Every six months or so I feel the need to wipe the slate clean and reinstall everything. The feeling's coming on again, and I promise to use this page to keep a better record of things than I have in the past. All I have to do is first burn some photos and a few valuable configuration files to a CD using the Lite-On external DVD R/RW drive I picked up.

Update 2006-05-16: Still haven't reinstalled everything, but decided to recompile the kernel.

Update 2006-05-17: And now the printer is working.

Update 2006-07-03: Picked up a Venus DS3 enclosure and hard drive from Newegg.

Update 2006-10-03: See the install notes from the reinstall.

About this site

2009-01-12T11:44:47-05:00

So I'll continue to update this page as I make changes to the site, but I'll say right now that I've designed it for a selfish purpose: I can't remember anything. This especially applies to computer problems I've solved, as the answers are usually arrived at after hours of late-night head pounding. It's a record of my thoughts, then, and a few pointers to information I appreciate.

Some thoughts on the site, as I'm working on it, probably not terribly interesting to anyone else:

Keep html files in the root directory whenever possible to make for short URLs.
Leave xml and their respective html files intermixed in the same directories -- makes for much easier file management later on (don't need to worry that xml files from this directory create html files for this other directory, but these xml files create html files for yet another directory, etc.).
Depends on the following odd modules for python: amara, PyLucene, and (optionally) aspell-pythonPyEnchant (because I didn't want to mess with system calls to build the aspell dictionary).

Update 2007-11-19: So this is all horribly out of date. XML files are in their own directory. Amara is no longer used. Neither is PyLucene or PyEnchant. Ugh. I'll say more when I do something interesting.

About me

2009-01-12T11:44:47-05:00

I'm Gabriel Sean Farrell and I live in Philadelphia (previously, Brooklyn). This is subject to change, and will be updated accordingly. Please see my resume (PDF). You can also get it in LaTeX, with a corresponding style file, if you're into that sort of thing.

I've put up a picture of myself.

You can reach me at gsf at this domain.

Freshly shorn

2009-01-12T11:44:47-05:00

I just cut my hair, and I figured now's a good time to put a photo of me on the site:

Smock smock smock

2009-01-12T11:44:47-05:00

Don't knock my smock, or I'll clean your clock.

Changing the mouse cursor in X (xfce) with update-alternatives

2009-01-12T11:44:47-05:00

So I've been using xfce for a while and I like it, but I wanted a different look for my mouse cursor, so on xfce-look I found ComixCursors (CC), which I thought looked nice. Added bonus, they're already packaged and available in the regular Debian apt-get. Only problem is I don't have a nice, easy way to change the mouse look like on gnome or kde. The easiest I had found was this tip on changing the mouse cursor, but that proved to be not very robust, because when I upgraded xfce earlier tonight I was back to the ugly old cursor. Well, after some frustration, I figured out by looking at the README for crystalcursors (cc) that I should be using update-alternatives. cc is more nicely packaged than CC, though. Whereas CC just plops a bunch of mouse icons in /usr/lib/X11/icons/, cc updates the alternatives for you, and provides the proper .theme files in /usr/share/crystalcursors/, which look something like:

[Icon Theme]
Inherits = crystalwhite

So in order to make CC work I created /etc/X11/cursors/ComixCursors-White-Regular.theme and filled it with

[Icon Theme]
Inherits = ComixCursors-White-Regular

I could have done this for all of the CC cursors, but I decided that white regular is my favorite, so why bother with the rest. Then I looked at the present alternatives for x-cursor-theme by entering

# update-alternatives --display x-cursor-theme

which spit the following back at me:

x-cursor-theme - status is auto.
 link currently points to /etc/X11/cursors/core.theme
/etc/X11/cursors/core.theme - priority 30
/etc/X11/cursors/redglass.theme - priority 20
/etc/X11/cursors/whiteglass.theme - priority 20
/etc/X11/cursors/handhelds.theme - priority 20

I decided to go ahead and give the white CC the same priority as the other non-core themes, so I updated update-alternatives with

# update-alternatives --install /etc/alternatives/x-cursor-theme \
x-cursor-theme /etc/X11/cursors/ComixCursors-White-Regular.theme 20

then I ran

# update-alternatives --config x-cursor-theme

and chose the number next to /etc/X11/cursors/ComixCursors-White-Regular.theme, and things still didn't work, but that was because the index.theme in /usr/lib/X11/icons/default/ had disappeared in the midst of my tinkering, so I had to link it back to the x-cursor-theme with

# ln -s /etc/alternatives/x-cursor-theme \
/usr/lib/X11/icons/default/index.theme

and after that, success! The moral of the story is that update-alternatives is a bit confusing at first. I learned to follow the usage message it spits out at you instead of the man page, because they conflict slightly. Once you get the hang of it, however, it's a fun and powerful tool.

One last note is that, as cool as update-alternatives is, xfce should really have an easy, gui way to change the mouse cursor. Sounds like a good project.

Update 2006-04-17: This solution isn't robust either, I guess, because a recent upgrade has borked it back to the ugly old core cursors. I think it has something to do with the last X.Org upgrade. More to come as I figure it out.

Update 2006-04-20: Something about that upgrade, and xserver-xorg, and how it removed xserver-common, messed a bunch of stuff up. I think it might have something to do with this bug report on the package. Anyway, after some freaking out on my part and installing and uninstalling stuff in aptitude, the ComixCursors are back. Yay. I think after the next clean-slate reinstall I'm going to go back to testing. Unstable is too stressful.

Update 2006-06-17: On a recent upgrade, the ComixCursors package installed all of the various ComixCursors .theme files in /etc/X11/cursors/. Sweet. The package maintainer must've received the brain signals I transmitted.

xhtml done right?

2009-01-12T11:44:47-05:00

Update 20060418: Okay, so a lot of the stuff written below is no longer true. See the updates at the bottom of the article.

This page, and pages like it on this website, are served up by Apache as application/xhtml+xml in Firefox, but as text/html for Internet Explorer because IE can't handle the xhtml. The reasons for doing so are, um, well, let's see, this guy has some good reasons for serving xhtml as application/xhtml+xml. Others say application/xhtml+xml should die [well, that link's down already. I'll send him an email.], or just outright no to xhtml. Well, one neat thing you can do is include inline svg:

Look at the source and you'll see the xml. With a graphic that large, though, it makes for a big old clunky xhtml page. Also, the w3c validator doesn't like it (check by clicking the XHTML link at the bottom of the page). Better to put it in an object tag, like as follows:

  <object data="/img/cecily1.svg" type="image/svg+xml" width="750" height="600"></object>

IE doesn't seem to like svg in any shape, even when pulled in with an object tag, so I'm not sure if that's a good idea either. But I digress. For better or worse (and with an eye toward the ever-glimmering future), I decided to serve it up with Apache's content negotiation. Oh, and one more reason to serve application/xhtml+xml is that Firefox tells me if I've missed a closing tag.

A quick google search on some of this page's keywords should bring up most of the pages I looked at. I got the most help from the official w3c page on serving via content negotiation, but it took a little experimentation. First, as per the w3c advice, all of my files end in .xhtml, and each has a symbolic link from a .html file of the same name. In the apache available-sites file, in the directory section, I needed the MultiViews and FollowSymLinks options. That works as long as you point at http://un.ctuo.us/xhtml (without the .html, and you let Apache decide which way to send it), or at http://un.ctuo.us/index, but if it's just http://un.ctuo.us/ then Firefox defaults (in error, in my opinion, but I'm not sure if it's Firefox's fault) to the .html, so I had to add "DirectoryIndex index" to the Apache config. That works! So my Directory section looks like this:

<Directory /var/www/un.ctuo.us/>
    Options Indexes FollowSymLinks MultiViews
    DirectoryIndex index
    AllowOverride None
    Order allow,deny
    allow from all
</Directory>

Update 20060403: As per these Apache performance tuning tips, I've changed

  DirectoryIndex index

  DirectoryIndex index.xhtml index.html

I also played with the creation of type-map files, as it suggests, but those extra files further cluttered up my directories that are already full of redundant files (more about that later in about this site) and as far as I can tell I'd have to end the URLs on my site in ".var" for that to work (ugly!), so it's out. While I might take a performance hit for using MultiViews, it's a simpler, more elegant solution.

Update 20060411: Oops, just noticed that

  DirectoryIndex index.xhtml index.html

doesn't take advantage of content negotiation. Internet Explorer isn't served the .html page automatically when visiting http://un.ctuo.us/, so it's back to

  DirectoryIndex index

No performance tuning for my Apache, then. Oh well, I tried.

Update 20060418: I've just realized that Google doesn't recognize the file format of my site, as you can see by the un.ctuo.us search results. Looks like Google doesn't recognize sites sent as application/xhtml+xml, and apparently it's not smart enough to content negotiate with Apache. I'll have to think about this one and get back to you.

Update 20060418: Okay, so after the trouble with Google I flirted with the idea (after looking at Tampering around with XSLT) of sending pages as XML with links to the XSLT stylesheets I'm using, but then I realized I'd have a lot of the same problems. I did some research and found the very informative Perils of using XHTML properly, and I'm convinced to just serve everything up as HTML 4 Strict. The back end will still be XML, as will be described in the page about this site, but all the browsers will see is straight HTML. No MultiViews, no content negotiation, just HTML.

Another lesson I just learned (from this Ampersand for URLs post) is that in order for these pages to run through the XSLT correctly all of the ampersands in my hrefs need to be referred to by the character entity "&". So, for example, instead of

  <a href="http://www.google.com/search?hl=en&amp;lr=&amp;q=un.ctuo.us&amp;btnG=Search">

I need

  <a href="http://www.google.com/search?hl=en&amp;amp;lr=&amp;amp;q=un.ctuo.us&amp;amp;btnG=Search">

Fun!

So now that these pages are HTML (I learned a little from HTML and XSLT) the SVG obviously no longer works, and it messes up validation like crazy, but I'm going to leave it in anyway, for posterity, and that others may benefit, I'm not sure how.

Update 20060420: On second thought (or, like, seventh thought), pages will be XHTML 1.0 Strict, but served up as text/html. Seems to be the best compromise. I guess that's why everyone else does it that way.