The Cron of Tab

June 19, 2008

Damn, setting up crons was supposed to be a walk in the park, so much so that I didn’t even budget mental energy for it! Maybe that was the problem…

Anyways, lessons learned from setting up simple cron jobs.

  1. How-Tos are good, but man is better.
  2. crontab -e will either load your current crontab, or load a blank template for you.
  3. crontab -r will kill your current crontab. Which is why it’s nice to keep the output of crontab -l, which lists your crontab jobs, in a backup file. Because unless you like setting up crons, you don’t want to be left high and dry without a backup.
  4. try running your jobs before putting them in the crontab. Or, if you’re like me, do crontab -l, cut and paste, and figure out why /usr/local/binruby is not the command you want (/usr/local/bin/ruby was what I was looking for).
  5. to make sure your jobs are running, tail -f /var/log/syslog. Note that this doesn’t tell you if they’re crap or not.
  6. Or, append your output to a log, and check to see that the log is growing.

I’m still kind of cruxing about how I’m going to run this on a deployed app. I think I’m going to have to get ops to add deploy_user, and add the crons on deploy_user’s behalf. Of course, I’ll worry about that when I can actually smoothly install mod_rails on my semi jacked up box. I must be the only tard in the universe who screwed up a mod_rails install, but more on that tomorrow.

Tomorrow (er, 1 week later)

In the mad rush to launch (more later), I forgot to update this page, which is bad because this is my scratchpad that has more than once saved me from repeating a painful process. Summary: anyone installing mod rails should take 4 minutes and view the railscast. I was missing the following in my conf file:

LoadModule passenger_module /usr/local/lib/ruby/gems/1.8/gems/passenger-1.0.5/ext/apache2/
RailsSpawnServer /usr/local/lib/ruby/gems/1.8/gems/passenger-1.0.5/bin/passenger-spawn-server
RailsRuby /usr/local/bin/ruby

Metrics Part IV: RRDTool on Ubuntu 7.04 (Feisty)

June 19, 2008

The production instance of this metrics server is going to run on Ubuntu Feisty, which comes installed with rrdtool 1.2, but the ruby bindings I want (need) to use bind to 1.3. So this is how to install rrdtool on Feisty.

(1) download the source:

curl > rrdtool.tar.gz

then follow the instructions rrdbuild page : basically,


sudo make

sudo make install

However, in order to get configure to complete, I needed to install a couple of dependencies, pango and xml-2. I like configure, it’s very good about telling you what is missing and where to get it. And the rrdbuild page is also great at specifying exactly how to install the missing packages.

I figured I was up and running at that point. I built the ruby bindings from {src dir}/bindings/ruby, but when I tried to run the files I had been running on my Mac, I got:

/usr/local/lib/ruby/site_ruby/1.8/x86_64-linux/ cannot open shared object file: No such file or directory - /usr/local/lib/ruby/site_ruby/1.8/x86_64-linux/ (LoadError)


How can a file that exists, /usr/local/lib/ruby/site_ruby/1.8/x86_64-linux/, not be found? Was it a permissions thing? I tried building as sudo, same thing. Then I made sure that the file actually existed, just to check my head. Yes, it’s there. Yes, I’m getting the same error. Wait. Could it be the ? I try a

ldconfig -v | grep rrd

and only get Hmmm. OK, desperate times…I edit /etc/, and add


to the path. That worked. The ops team is not going to be super thrilled about the amount of jackassery it took to get this up and running, which is why I’m documenting it here. Because they’ll make me maintain it. Just like I would if I were in their shoes 🙂

Metrics Fast ‘n Easy, part III: accessing Rails goodies outside of a Rails app

June 18, 2008

The basic architecture of this (very simple) metrics gathering and display application is:

  1. Scripts run as crons from within the rails directory. Every minute, the cron wakes up and checks the database for the last time run and the poll interval. If they are
  2. They update RRD files and generate .pngs that reside in the rails /public dir.
  3. they update the latest value, and the last time run.

The scripts and the rails app intersect at two points:

  • the interval and last time polled
  • the generated PNG file.

I may choose to store and display the last collected value, but that will happen after getting feedback from my customers (the development, operations, and product teams).

Since the script has to check the interval and the last time polled, it needs access to the database. I naturally wanted to use the ActiveRecord classes that I use in the rails app, I also wanted access to rails environment variables, like RAILS_ROOT.

By requiring environment.rb:

require File.dirname(__FILE__) + '/../../config/environment.rb'

I was able to get rails like behavior into my scripts.

Metrics Fast ‘n Easy, Part II: actually using RRDTool from Ruby

June 14, 2008

Continued from part I:

Now that the RRD bundle is installed in Ruby’s default load path, I require RRD and access the convenience methods. The methods basically pass all parameters in as strings, which is fine, but I don’t like thinking of time and values as strings if I can avoid it. So I wrote a wrapper class that allows me to pass in values as typed options, and then casts them to the internal strings.

A couple of notes about creating, updating, and rendering RRD graphs using the built in Ruby binding.

Creating an RRD graph

At create time, the first parameter is the name of the file, minus the .rrb extension (create will puke if you specify the extension). The start time is expressed in seconds, my code below passed it in as a Time object and converts it to seconds. The step time is the minimal amount of time an update can occur at — in other words, if your step is 50 seconds and you try to update at 10 seconds, you get an error.

The DS option defines a dataset as follows:

DS:[name]:[graph type]:[min time to show an error condition]:[min value or unknown]:[max value or unknown]. More explanation of the suitable graph types is found here.

"--start", "#{@start.to_i}",
"--step", "#{@step}",

In the example above, I only create a single dataset. You can create 1..N, although I’m sure N has an upper limit, I haven’t found it specified anywhere. Also, I believe the data set is restricted to < 19 characters in length. The RRA section syntax is as follows:
RRA:AVERAGE | MIN | MAX | LAST:xff:steps:rows
where the collapsing is done by averaging values, or min/maxing values, the xff value specified limits unknown values from being collapsed by establishing a max ratio of unknown values to known values. The steps value specifies the number of datapoints collapsed, and the rows value specifies how many collapsed datapoints to keep. So RRA is where you really get a chance to limit the size of the RRD file.

Updating an RRD Graph

Once the graph has been created, it exists as the file you specified using the name parameter above. You update it with time:value statements: in the code below, I’m updating an array of time:value statements:
# simple update of multiple values
def update(times, values)

for i in 0..times.length-1


In the code above, as for all RRD operations, you specify the name of the RRD file you want to operate on in the first parameter.

Note that you cannot update a graph with a time less than it’s start time or a time that is less then the last time + the step time specified at creation.

Displaying an RRD Graph:

Graph display is the most complex operation with RRD. I’m not going to go into all of the details: some really good examples are found here.

I’ve taken the simplest approach to displaying a graph:

"--title", title,
"--start", start.to_i.to_s,
"--end", finish.to_i.to_s,
"--imgformat", "PNG",

Unlike the update method, the name of the actual desired graph is the first parameter, not the name of the RRD file. The RRD file to load is specified in the DEF line. You can specify multiple DEF values to display dataset from different RRD graphs. You will need to specify the way you want each dataset rendered: in the above example, I define a value a with the DEF statement that I reference in the following LINE statement:

In order to render data, you will need to specify how you want to display it with( as a line, as area under a line, as a tick mark, etc). More details about how to define data sets, including creating datasets via the CDEF statement, are found in the graph data documentation. Details about how to display data are in the rrdgraph method documentation. The format of the DEF, CDEF, LINE statements is RPN, i.

Make sure to specify start and end in a way that shows values as you would like to see them, i.e. make sure your latest value is in the specified start and end range.

Metrics fast ‘n easy (?) with RRDTool and Ruby: Part 1: Installing RRDTool and ruby bindings on a Mac

June 10, 2008

I’m trying to set up a simple heads up display of key information about our running system as we move towards releasing our first product. This display is grouped into themes, each theme contains 1..N graphs of relevant system data.

As a manager this is not my full time job, so I didn’t to get into navel gazing mode wrt the  ‘ultimate’  stats monitoring system, especially since that wheel has already been invented. I settled on using RRDTool to display statistics, mainly because RRDTool has pretty advanced, robust ways to age data out, while keeping the database down to a finite size. It also generates PNG files, which I could update and throw up on a page as needed. There would be a need to track application specific metadata (like what each graph maps to), but no need to handle storing and retrieving metrics data, which gets me (and the poor bastard that inherits this tool) out of a world of pain.

It’s been years since I’ve set up RRDTool, and that was on a Red Hat system. Now I’m a Mac fanboy, and so here are my notes on how to set up RRDTool with Ruby binding on Mac OSX Tiger.

RRDTool can be installed via mac ports which makes pulling in the dependencies ‘not my problem’– the best kind of problem to have :):

sudo port install rrdtool

To build the ruby binding, you need to get the source from here. The ruby binding is in {src folder}/bindings/ruby. In order to get the generated Makefile to point to the installed rrd libs, you need to change the following line in extconf.rb:




Then run

ruby extconf.rb

to generate the makefile, then



make install

to respectively build and then install the RRD.bundle to /opt/local/lib/ruby/site_ruby/1.8/i686-darwin8.9.1

To confirm that things have run successfully:

ruby test.rb should generate a test.png file.

Sh*t keeps breakin’

May 28, 2008

So the day after my post about the bike, I was climbing a steep hill in a high-ish gear and felt/heard a sharp SPRANG! from my rear wheel. I’ve had that loving feeling before, and sure enough, I had snapped a spoke. Fortunately only 1/2 mile from home. I took the wheel in to get the spoke replaced and the wheel trued up, and the bike shop guy called back to let me know my rear rim was cracking all over the place. The cracks are hairline now, but the wheel is eventual toast.

Spoke breaks really piss me off, because as an ex-extremely mediocre racer, I have clinical data that proves that not only am I much less powerful than I once was, but even when I was at my best I was pretty weak. Like my VO2 max topped out at 60, which is OK but hardly ‘genetic enduro mutant’ like. And I have no fast twitch to speak of, my vertical jump is 6 inches on a good day. Yes, I’m a slow guy with no endurance. So WTF on the whole spoke snapping thing? It’s not like I was pulling a (pre doping?) Vino and shattering the spent lead group of Tour elites with a blistering attack. I was climbing a freaking short not so steep hill on my way back home from work

As for the rim, well, I’m no Alberto Contador. The extra weight in my belly, fueled by late nights of coding and the incredible amount of yummy sugar snacks at the office, may be aerodynamically efficient, but it does nothing good for the power to weight ratio, and I’m afraid that my non endomorphic physique contributed to the rim fatigue.

No worries, don’t panic, said the bike shop guy. But I would start looking for a new wheel if I were you.

The thing is, the wheel is a 24 spoke Bontrager, and they dont even make those anymore. So I think it’s time to upgrade to a new set of wheels, something bomber that wasn’t made for an anorexic little grimpeur.  While a nice pair of Ksyriums would be great, I’m value-drooling over a set of Neuvation wheels, having read the reviews that rate them very high. In any case a wheelset is the best upgrade you can make on a bike, and I’m very close to springing for the rear wheel at least….

Since the rear rim is cracking slowly, I’m just going to keep an eye on it and keep riding. But the god of equipment upgrades wasn’t done with me. The next time I went out, I was just finishing my Mercer Island 2 lap AM special when I clicked out of my pedal, tried to click back in, and couldn’t get in. I rode swearing the whole time up the 14% grade to my house with one leg. Thank god for granny gears! When I took a look at the pedal, the retaining spring had cracked. This is the second pair of eggbeater pedals I’ve trashed, and I was done with the brand. That day I picked up a pair of Speedplay light actions. Installing the cleats was somewhat labor intensive, but straightforward. I was worried about finding the tiny little pedals with my feet, but had talked to enough friends that rode them to be assured that this wouldn’t be an issue. Sure enough, they felt completely natural. The best thing you can say about a pedal or a saddle is that you didn’t notice it, and these fit the bill. I’ll weigh in on their durability as time goes by.

Hopefully I can motivate to get out of bed by 5, on the road by 5:30 to do a long-ish Issaquah loop. Work is super busy these days, but a 5:30 – 8:30 session on the bike does wonders for my temperament.

WSDL Code Generation Support in Maven

May 28, 2008

Full disclosure: I once wrote SOAP services, I even wrote WSDL for them. Actually, it was while writing WSDL that I decided that SOAP was really complicated for something I didn’t want to spend too much time thinking about. That lingering ‘smell’ and the resulting server load when we were servicing 1000s of fine grained RPC requests was about the time I began to really hate SOAP.

Fast forward to now, and I’m a huge fan of infinite resources and finite actions, aka REST. Why? My experiences with well designed REST interfaces is that they become intuitive once you know the domain model that the service is exposing. You ask for domain information in a kind of WYSIWIG (what you SAY is what you get) way, and that’s that. Yes, you can still maim your server with fine grained calls, but that is an orthogonal issue.

While I think that most of the rest of the world feels the same way about REST vs SOAP, i.e. Amazon AWS serves 85% of their requests over REST, that (sadly) is not the case. This weekend, I found myself in the unfortunate position of having to make a bunch of SOAP calls to a service that publishes documents. While most services circa 2008 would use RSS/Atom to do this, these guys haven’t quite made the conceptual leap. Maybe they’re still trying to figure out the whole series of tubes thing, I know I am.

The thing is that I’ve killed the brain cells that were responsible for all things SOAP. I mean it was more a case of ‘put them out of their misery with a few stiff pints’, but death is death, and that information had left my building. So I amped up the metal tunes and googled away for a while.

My (very modest) goals for the weekend, given that it was really nice out and I wanted to play with my kids and ride my bike:

(1) generate client code from the company’s WSDL file.

(2) do this as part of a build process.

Pretty basic stuff, I needed to find a Java client generation tool and hook it into Maven. I figured that lots of people had done this already and I would be standing on the shoulders of giants. Well, that turned out to be the case, sort of…

One problem I have with WS* in general is that there are so many different tools claiming to do the same thing. In the case of WS*, there are many different tools released by Apache that all generate clients from WSDL using their own baked versions of wsdl2java, a tool used to generate the server proxies from WSDL.

Muse: “The Apache Muse Project is a Java-based implementation of the WS-ResourceFramework (WSRF), WS-BaseNotification (WSN), and WS-DistributedManagement (WSDM) specifications. It is a framework upon which users can build web service interfaces for manageable resources without having to implement all of the “plumbing” described by the aforementioned standards.” WOW. Still reeling from all the T&FLAs in that sentence. I think Muse might be helpful, but I don’t know how much I end up getting ‘for free’.

Axis: Apache Axis is an implementation of the SOAP (“Simple Object Access Protocol”) submission to W3C. Hey, that sounds pretty straightforward. But wait, that doesn’t really tell me much! Do they handle client and server side dev? Also, what is the difference between Axis and Axis2?

CXF: Apache CXF is an open source services framework. CXF helps you build and develop services using frontend programming APIs, like JAX-WS. These services can speak a variety of protocols such as SOAP, XML/HTTP, RESTful HTTP, or CORBA and work over a variety of transports such as HTTP, JMS or JBI. Hmmm, again with the TLAs. Also, I don’t need the swiss army knife here, I’m not writing a multi protocol service, I’m writing a dumb as crap service consumer.

All three frameworks above claim to have something called wsdl2java, which does what it says. Lacking any other differentiation points, I decided to try them all. I was taking the “I don’t want to know sh*t about the underlying technology approach”. In other words, I really didn’t care about WSDL, I just wanted to generate some damn code and be done with it.

Try #1: Muse: whoops, it didn’t like my WSDL because there were 2 port elements:

~/muse/bin/ -wsdl newsdatasecure.asmx.xml -proxy
java.lang.RuntimeException: [ID = 'OnePortPerWSDL'] The resource inspector can only process WSDLs with one port element. The WSDL can have multiple portTypes, but only one concrete service and port.

Wait a minute. WTF is a port element? Damn, I need to read up on WSDL. (Later….) OK, here are my notes.

Next Up: Apache CXF or Axis? Who’s got more Maven Love?

So it was pretty obvious that short of hacking the provider WSDL, I wasn’t going to get the love I needed from Muse. That left two choices, CXF, which felt very framework-wonkish, and Axis/Axis2, which felt very vague. I decided to step back a little bit and see what kind of support both approaches had in the build world. After all, I wanted to integrate client side codegen into a build process, where WSDL changes would immediately show up as compilation failures.

This was the quintessential Yak Shaving exercise. Which might be a great way to spend a cold, blustery day, but not a sunny (in Seattle!?!) 75 degrees day. Turns out CXF and Axis2 have maven plugins.

Axis2 has the wsdl2code plugin. CXF has a wsdl2java plugin. At this point I was thinking I was out the door. My previous maven experiences with integrating plugins were largely a cut, paste, and go exercises, using or other maven POM file syntax I could get from the documentation.

I tried CXF and quite simply couldn’t get it to work. Specifically mvn generate-sources, or any of the variants proposed in various bug lists didn’t work.

My next option was Axis2. I got it working, but only after some effort detailed below.

The wsdl2code plugin from Axis2 claims to generate code as long as you simply follow the plugin instructions and insert the following plugin config information into your POM:


Unfortunately, that isnt valid POM schema. You need to embed the configuration element in the execution element:


That at least produces a valid POM. However, when I ran
mvn clean install

I recieved the following error:

java.lang.NoClassDefFoundError: javax/wsdl/WSDLException
at org.apache.axis2.maven2.wsdl2code.WSDL2CodeMojo.execute(
at org.apache.maven.plugin.DefaultPluginManager.executeMojo(
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalWithLifecycle(
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(
at org.apache.maven.DefaultMaven.doExecute(
at org.apache.maven.DefaultMaven.execute(
at org.apache.maven.cli.MavenCli.main(
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
at java.lang.reflect.Method.invoke(
at org.codehaus.classworlds.Launcher.launchEnhanced(
at org.codehaus.classworlds.Launcher.launch(
at org.codehaus.classworlds.Launcher.mainWithExitCode(
at org.codehaus.classworlds.Launcher.main(

CRAP! You mean the plugin is puking because it can’t resolve a dependency? Christ, isnt this the stuff that maven is supposed to resolve?

I did some more thread googling, and found here that I needed to add the following dependency as well as change the groupID and add a specific version number:



Now I had WSDL code generated in Maven, but that code didn’t actually compile…hmmm. No good.

My last gasp effort to generate some client stubs was to use Axis, which has noted here, has now been replaced with Axis2.

However, desperation leads to an open attitude, and when I downloaded axis-1.4, and did the following:

java -cp ./lib/axis.jar:./lib/commons-discovery-0.2.jar:./lib/commons-logging-1.0.4.jar:./lib/saaj.jar:./lib/jaxrpc.jar:
./lib/wsdl4j-1.5.1.jar \
org.apache.axis.wsdl.WSDL2Java ~/foo.wsdl

I generated WSDL that actually compiled, and ran. Moral of the story? Stay the F*ck away from SOAP, it’s 2008 after all!

WSDL element summary

May 27, 2008

This is more of a “notes to self” post than anything else. Here is the basic summary of WSDL’s most important elements:

A WSDL file consists of an abstract part and a concrete part. The abstract part lays out all the interfaces, the methods that make up those interfaces, the parameters and return types of those methods. The concrete part specifies how those abstract parts are actually implemented, and where you can find them.

‘Abstract’ Elements

described in bottom – up order:

  • <types> contains all data types used in all method calls described in the WSDL.
  • <message> defines a one way message, i.e. an RPC call w/o a return.
  • <part> a parameter or a return value in a message
  • <operation> combines multiple messages to form a complete message, i.e. a request/response sequence.
  • <portType> a sequence of 1..N operations that describes an interface

‘Concrete’ Elements

described in bottom up order:

  • <binding> the wire protocol and protocol specific details of the a portType element. Contains SOAP specific protocol mappings to the operations defined in portType. Can describe the type of binding, i.e. RPC or Document. I can’t tell which one is more heavily used or which one is valid.
  • <port> the actual location of each binding specified in the WSDL.
  • <service> wraps the specified locations (<port> elements ) of the bindings specified in the WSDL.

Wow. On the plus side, I remember what a port is now. On the minus side, I have to remember what a port is now.

My bike setup: (not so) initial impressions

May 20, 2008

A while ago I had written about a good bike setup for me that allowed me to keep my shifting at the brake levers and use the aero bars. Last month, after a particularly brutal winter that delayed the onset of spring riding, I found a pair of deda akros bullhorns and strapped them on. The brake levers are at least 4 inches further forward than they used to be:  I needed to lengthen my brake lines to account for the increased reach. And the ride and handling is completely different, I’m thrown forward much more and feel like I need to  be conscious of steering with my hips because a little motion with my arms does a lot more.

But, all in all, the adjustment has gone well. This year, due to patellar tendinosis and work, I haven’t gotten in the miles I would have liked to, and I’m in not so stellar shape. But I’m clocking the  same speed on my Mercer Island 2 lap morning ride as I did when I was much leaner and riding with standard road bars.

The only issue has been saddle discomfort. I’m making an effort to stay in the aero bars more,  that puts a lot more pressure on my, well, my ‘taint’, for lack of a better term. Even though I’ve jammed the saddle all the way forward, I’m riding on the nose, and what I’ve got has got nowhere to go. I’m looking for some relief, doing online saddle reviews, which is by definition impossible. But I think I’ve found a winner  in the Koobi Au Enduro, a saddle that keeps the split going all the way down the nose of the saddle, which just sounds wonderful to me right now.

Now that I’ve adjusted to the increased reach and saddle fore-aft position,  I feel a lot more powerful on flats an slight uphills, even with a super weak left knee. In fact, riding is just about the only thing I can do, because it is in a single plane, unlike soccer.  I’m still in denial about soccer being hard on the knees, but this injury is primarily due to trying to follow some 20 year old kid around on the field. I’m just glad my soccer team has morphed into more of a drinking team, because that doesn’t require young, strong knees.

I’m still adjusting to the increased stretch forward, I’ve felt some back pain but that could be partially the result of my body compensating for the knee, as well as a bad posture at work. The other day I did a May Valley ride to Issaquah and felt some back pain, but was able to stretch it out and have it completely dissapear. So maybe the morning sessions on the TRX have not been completely in vain!

I’m not an inventor, I just play one on TV!

May 11, 2008

I’ve been way too busy doing actual, real work to blog these days. The Co. is in heads down mode, we’re cranking along at mach 10 trying to make a product deadline. So as a dev manager I’ve been split in 30 different directions, none of which points to this blog.

The other day I was discussing a way to get a sequence ID back from Postgres with one of the senior devs on the team, who was pretty convinced it couldn’t be done. I was pretty convinced it could, but I didn’t really know why. On my bike ride home fighting headwinds and rain on I-90, I realized it was because I had actually tried to do it once before, in slightly different circumstances, and was actually told it wasn’t possible. However, in typical Arun fashion, I had forgotten reality and imagineered a successful outcome, in which I had triumphed over adversity. I tend to do this a lot, particularly on hard crack climbs or brutal bike rides but that is a survival mechanism worth another post.

The realization that I had been down this road before was a bummer, because the other solution was to make one call to the sequence, and insert the record using the result of that call. This, in the middle of a headwind and rain filled commute, really killed whatever buzz I had left that day. However, some quick googling showed that Postgres has extended SQL by adding a ‘returning’ clause to insert statements. So, victory was snatched from the jaws of defeat, at least for that day, along with a sobering realization that it might be good to google myself every once in a while just to make sure I havent asked the same question before. Hmmm, OK, I’m apparently old enough to start repeating myself professionally. Is that an accomplishment? Still not sure.

This brings up the other thing I found out about myself — a patent I had actually filed in 2002 has gone through. I’m not sure if this is something to be proud about given the unclear benefits and/or validity of software patents. The actual patent has to do with modeling mathematical operations in XML — an idea that I actually feel kind of embarrassed about because it’s pretty basic in nature, nothing super clever/elegant about doing basic math in XML, but it was an attempt to solve a use case that had to do with allowing network operators to mod the raw statistics and assemble meaningful graphs, while persisting the mod operations in XML. Well, the next time I have an idea, I guess I’d better make sure I haven’t had it before. Because apparently that happens 🙂

In any case, I think it’s telling that I find out about things I’ve done and forgotten about on the internet. It has in effect become an extension of my brain — the destination of a lot of pointers that allows my local wetware to focus more on knowing how to find information than on retaining the actual information itself. Damn. Here’s hoping the power never goes out.