A big day here at Evri, as we have taken the password protection off and opened ourselves up to the world. When we started looking at the problem of managing information on the web almost 3 years ago as a tiny research team of 2, our main goal was to make processing information easier for ourselves. We were stuck in an endless loop of keyword search -> sift through results -> alter keyword search -> forget what we were looking for in the first place.

Fast forward to now and Evri is an incredibly talented team who have gone far beyond the initial proof of concept prototypes and have delivered an intuitive and easy to use site that lets you find the content you want to find about the things you care about. Along the way I’ve been exposed to the real problems and solutions inherent in making a real product from a raw prototype, and I’ve got to say it’s been a great ride so far, and with the open beta we have just crossed the starting line…now it’s real!

Instead of writing about what we do, which is best summed up here, I encourage you to visit the site and poke around. If you have a blog, try installing the widget — note that my blog, which is hosted by wordpress, cannot run the widget, but this is a general wordpress problem, and there are known work-arounds that we are investigating. Stay Tuned!

As noted in other posts, I’m writing a monitoring application. Because this is at best a part time effort that needs to be done quickly, I’ve made some technology choices that emphasize rapid development: the app is a Rails app, and I’m storing statistics with RRD.

I really like RRD, as I’ve mentioned before, it gets me out of the business of drawing graphs and storing data, both of which are hard problems I’d rather not solve. But I was having some issues with it when I would try to store values.

How I thought RRD worked

It seemed pretty simple. I thought I would create an RRD file (wow, lots of parameters, wonder what they mean?), then update it whenever I had a number. And the values would be graphed. And all would be good. But when I did all of the above, I noticed that the values being graphed were not the values I was storing. Hmmm. Time to figure out what some of those parameters mean.

How RRD actually works

Pretty well, actually, because it was designed by some smart people to store numbers that came in at any time, and average those numbers across a create time defined interval. Well, that’s one of the ways RRD works. It can also store counters, store the results of methods applied to raw values, and store the derivative value of the line being graphed. I was using it in the simplest case, to store a value.

What I didn’t realize is  that if my values were updated  between interval boundaries (known as ’step values’), they would be averaged across that interval. If the values were updated outside of the specified ‘heartbeat’ value, RRD would store an ‘unknown’ value. A good explanation of how this works is found here ( in an SNMP monitoring solution).

That is actually the way graphing in a loosely coupled environment_should_ work. The reason that I was seeing strange numbers was because my insertions were falling within the same interval boundary. Which may be rational, but doesn’t jibe with then numbers I’m trying to (a) display and (b) alert on.

How I got my app to work with RRD

The key for this app is that it is expecting an average value across a time interval. So in essence I have to make sure that only one data point is inserted per interval. I do this by munging the time of insertion in the RRD graph (I still keep the original insert time for purposes of reporting).

I insert the data point at the end of the interval, so if I have a 5 minute interval and I receive and update at 21:52:34, my actual insertion is at 21:55:00. The next value will be inserted at 22:00:00. If the interval was 1 minute, I would have inserted at 21:53:00, and the next value would be inserted at 21:54:00.

More fun with RRD

I’m sure that more fun awaits. I have not hit the point where round robin averaging kicks in, and my ‘default’ values are based on my current (mis)understanding of RRD. I’ll update this post so I don’t repeat history.

That’s right Web Servicification. Servicifying, just like the subtle art of Strategery, is an oft derided but subtly powerful part of my toolkit.

Or at least it is now. I have been working on a monitoring application so that we could figure out when things were going pear shaped — as opposed to finding out after the fact. This monitoring application was somewhat novel in that it successfully got me out of doing hard things of little quantifiable value and let me focus on doing easy things of much greater value (see above goal). Using RRD is a good example of outsourcing a hard problem  — what to do with all that data ?!?– to something that handled it for me. Not an especially hard leap to make, thanks to lots of SysAdmins who feel exactly the same way, but still, I’m really happy that I’m not collapsing data to keep my disk footprint somewhat finite.

This whole strategy of ‘doing more with less effort’ is really fun, I’m searching for something non geeky to try it in.  If I get the same efficiency boost in my personal life I’ll have enough time to write a best seller, become a kickboxing champion, or both.

In the context of my (geeky) monitoring app, Web Servicification is something else that gets me out of a couple of fairly hard to do things. Wait, let me back up a bit. Web Servicification with ActiveResource gets me out of a couple of fairly hard to do things.

The first is writing a web service. Go ahead and sneer at the difficulties of writing an XML consuming service, but until you’ve rolled your own in Java, you just don’t have that Juan Valdez “I wrote this one bean at a time” feeling.

The second hard thing this gets me out of is writing monitors for a bunch of heterogenous systems. We’ve got some things implemented in Java, some in Ruby, some in Perl, etc. I either slap a bunch of web interfaces on all of those systems, then write some centralized code to poll those interfaces, or I slap a web interface on my monitor app and let other people figure out which statistics are meaningful to them, and how often they should be updated.

A Web Service by Default

ActiveResource comes more or less enabled by default in Rails 2.0. Every method in a default generated controller can be accessed either by the UI or an REST action. Here is an example controller generated for one of my resources.

# GET /samples
# GET /samples.xml
def index
…
respond_to do |format|
format.html  #index.html.erb
format.xml  { render ml => @monitor_instances }
end
end

# GET /monitor_instances/foo
# GET /samples/1
# GET /samples/1.xml
def show
end

# GET /samples/new
# GET /samples/new.xml
def new
end

# GET /samples/1/edit
def edit
end

# POST /samples
# POST /samples.xml
def create
end

# PUT /samples/1
# PUT /samples/1.xml
def update
end

# DELETE /samples/1
# DELETE /samples/1.xml
def destroy
end

Notice that the standard REST verbs are implemented in the same methods that handle rails application page requests. That’s pretty cool, and it means that you’ve got basic CRUD from the get go. The secret is in the render method (in bold above), which returns either a page or XML content depending on the requested format. If the request ends in xml, it’s assumed to be REST, otherwise it’s assumed to be a standard page request.

Routing for both REST and page based requests is provided in routes.rb:

map.resources :monitor_instances

provides routing access to the default methods defined above.

Accessing the Default Web Service

ActiveResource::Base is the class that abstracts the wire format and provides basic CRUD access to the resource. To access the MonitorInstance objects defined above, I could do the following:

class MonitorInstance < ActiveResource::Base
# define what you need in here

end

The ActiveResource based MonitorInstance acts similarly to an ActiveRecord based MonitorInstance:

monitor_instance = MonitorInstance.create(:name=>monitor_name,:monitor_instance_id=>parent_monitor.id,:frequency_id=>frequency.id,:status_id=>@status_by_name['good'].id,:monitor_type_id=>@monitor_type_by_name["stand_alone"].id)

creates a monitor with the parameters as specified above.

MonitorInstance.delete(monitor.id) OR

monitor_instance.destroy

removes the monitor instance.

monitor_instance = MonitorInstance.find(1) finds me the monitor instance with an ID of 1.

removes that Monitor. So far, so good.

Find (not by ID)

What if I want to find something by a secondary attribute, like name? The default rails app expects qualifying parameters to be passed in a params hash:

monitor = MonitorInstance.find(:first,:params=>{:name=>monitor_name})

My MonitorInstances can be nested under other MonitorInstances. In the Rails app model, each MonitorInstance model specifies that it  belongs_to :monitor_instance.

This doesn’t quite have a corollary in the ActiveResource world. ActiveResource is concerned with abstracting basic access of web based resources, and that associations are not available via that abstraction layer. When I want to find a nested MonitorInstance, I do the following:

def get_monitor(monitor_name,parent_name = nil)

if(parent_name != nil)
parent = get_monitor(parent_name)
@logger.debug(“finding first instance of monitor #{monitor_name} under #{parent_name}”)
monitor = MonitorInstance.find(:first,:params=>{:name=>monitor_name,:monitor_instance_id=>parent.id})
else
@logger.debug(“finding first instance of monitor #{monitor_name}”)
monitor = MonitorInstance.find(:first,:params=>{:name=>monitor_name})
end

…

end

So I need to first get the parent resource, then make a request with the parent ID in the params hash, as indicated by the bolded text above.

Updating — Avoid Non Writeable Parameters!

It was hard to find any updating doc that didn’t just say “to update, just invoke the ActiveResource-derived object save method”. Which sounds great in theory, but didn’t work, because the default implementation of save POSTS all attributes, even those that are considered immutable, to the web service endpoint. For instance, my MonitorInstance class has an id field that is immutable. That field is posted with all other (mutable) fields. There is a method in ActiveResource to remove all immutable/protected attributes, but that method calls an undefined logger object to notify you that you are trying to modify an immuatble attribute, and an exception is raised.

To get around this, I stripped the immutable attribute — the id — out of the incoming params hash of the controller update method (in the Rails app) — see the bolded text below:

class StatisticsController

…

# PUT /statistics/1
# PUT /statistics/1.xml
def update
@statistic = Statistic.find(params[:id])

if(params[:statistic])
logger.debug(params[:statistic].inspect)
if(params[:statistic][:id] != nil)
logger.debug(“removing ID from input params!”)
params[:statistic].delete(:id)
end
end
respond_to do |format|
if @statistic.update_attributes(params[:statistic])

…

end

…

end

….

end

Nested Resources

The StatisticsController above handles all posts to Statistics resources, which are 1..N measurements associated with a monitor. In order to enforce that kind of scoping in the request path, I need to update the monitor_instances routes to scope the statistics routes:

#map.resources :statistics

map.resources :monitor_instances, :has_many => [:statistics]

In the statistics controller, I now need to always be aware of the ‘owner’ MonitorInstance. I do this by adding a before_filter, a method that gets invoked prior to every method being called:

before_filter :find_monitor_instance

This before_filter corresponds to the find_monitor_instance method, which returns the appropriate MonitorInstance:

private

def find_monitor_instance
@monitor_instance = MonitorInstance.find(params[:monitor_instance_id])
end

Now I have an attribute that I can refer to in my controller. Note that in all of the controller methods that handle both REST and page requests, I need to scope my model requests/updates with the @monitor_instance variable:

def index
@statistics = Statistic.find(:all,:conditions=>{:monitor_instance_id=>@monitor_instance.id})

respond_to do |format|
format.html # index.html.erb
format.xml  { render ml => @statistics }
end
end

I can also take advantage of RAILS path freebies. For instance, after a create request for statistic, I redirect to the appropriate monitor_instance scoped path like this:

if @statistic.save
flash[:notice] = ‘Statistic was successfully created.’
format.html { redirect_to(monitor_instance_statistic_path(@monitor_instance,@statistic)) }
format.xml  { render ml => @statistic.to_xml, :status => :created, :location => monitor_instance_statistic_path(@monitor_instance,@statistic) }
else

monitor_instance_statistic_path generates a path that looks like {path to server}/monitor_instances/1/statistics/3.html or .xml depending on the requested output format.

Some Helpful Links:

ActiveResource RDoc

REST + ActiveResource

Comments from this Railscast

It’s July 2008, and I’m looking around wondering how many people think search is broken. Actually, I’m searching around wondering how many people think search is broken. And, using this method, I’ve been able to deduce the following (from the first page of results, no less):

1. search is broken because publishers have to put additional metadata/markup on the pages/sites they want found, in addition to sitemaps, no-follow, robots.txt, etc.
2. search is broken because when I want to find something, I have to look in so many different places.
3. search is broken because there needs to be a human editorial overlay in order to achieve useful result precision.
4. search is broken because search applications are not telepathic, i.e. they cannot perceive context and other subtle metadata in a user search request.

All of the above is more or less accurate (though IMO #1 is kind of whiny and #4 is unrealistic), but at the same time there is good evidence that search actually does work for most people most of the time. And search, as a knowledge acquisition paradigm, has become incredibly ingrained in people’s usage patterns. At Evri the #1-with-a-bullet question we always get asked is “where is search?”

I dont think that a yes/no answer to the question ‘Is Search Broken’ does that question, or the technology behind it, any justice at all. I can say that from the perspective of a software engineer, the fact that I can ask a question at any time of the day and get an answer back in hundreds of milliseconds (COMCASTically of course) is just amazing. The relative precision and especially the recall is mind blowing. The rumor/FUD about Google infrastructure is scary and exciting at the same time. The fact that TF/IDF works as well as it does proves that Simplicity does equal Elegance.

When I take a more philosophical viewpoint of search, I have to say that the way I take search for granted and have completely outsourced my long term memory is very scary. I have voluntarily ceded control of information in my head and traded it for the ability to retrieve that information. Which gives me a lot more apparent bandwidth, as long as there is a computer nearby

Still, while the implementation, and more importantly, the functionality of search is something so powerful that I cannot function without it, I do see some challenges ahead for the traditional search model:

1. Content and the traffic generated by users wanting to access that content are growing: specifically, traffic is expected to grow at 46 percent annually between now and 2012, while the amount of content available continues to skyrocket upward.
2. Content is morphing from text based documents to include videos and audio. Search is not keeping up. Currently, most video/audio content is not considered ’searchable’, other than by associated metadata. There are some attempts to change this, i.e. delve networks for video search, but by and large search cannot — in it’s current incarnation – treat video, audio, or image data as equivalent content to documents.
3. People are turning to less machine driven means of finding out information — Mahalo is an example of editorialized search, and Wikipedia, while not exactly a search engine, can be used like one if you use Firefox.

That last two points taken together are pretty interesting, because in this age of massive document recall, people are veering towards precision, and precision across media types. People want content — video, text, image — to be fused together into a single result page, not a result set. The Lance Armstrong Wikipedia Page and the Lance Armstrong Mahalo result page provide a much more readable information set than the Lance Armstrong Google result page.

Implicit in those edited result sets is that the information is one click closer — the salient facts about Lance are front and center, not (just) in the first returned document. Search has made us very good at ’search, inspect, reject, repeat’, in which we sift through keyword results and painstakingly evaluate the returned documents like ancient priests must have sifted through tea leaves, or entrails, or whatever their search engine result set was. Edited page result sets present that information in a page view that we can actually browse.

I don’t think editorialized pages are the right answer, mainly because they cannot possibly scale, and the amount of human effort required to keep them up to date is massive. I think the next internet scale ‘killer app’ must provide complete fusion of search results across media types into a single, focused page per entity, that changes in response to real world events and doesn’t rely on an army of editors to keep it up to date. That’s definitely a holy grail, but one worth shooting for if we want to have any hope of keeping the internet as useful — if not more useful — than it is today.

Whew! Its been an exciting couple of weeks as we’ve gone through our first launch, but it’s official: Evri has a real beta release, and we’re telling the world.

It’s been two and a half years since I started down this road, building our very first, very crude prototype in a couple of weeks, and walking into the first demo knowing that we were onto something big, even if we didn’t know what that really meant. I consider myself very lucky to have had the opportunity to dive headfirst into such an engrossing, challenging space over the last couple of years, it’s been a great ride with a great team!

Neil’s blog post says it much better than I could ever rehash it (but I’ll try anyway): we are building a data graph of the web, a set of people, places, things, concepts, connected by specific relationships harvested from relevant content. The company motto: “search less. understand more.” is our way of saying that we have something truly disruptive: this is not search gone to eleven, or Wikipedia on steroids, but something much more powerful that gets around the current, completely disjoint user experience of “search, then browse, then search some more, then forget what you were originally searching for”.

Check it out! Sign up, and, most importantly, tell us what you think, so we can make it (even) better.

Damn, setting up crons was supposed to be a walk in the park, so much so that I didn’t even budget mental energy for it! Maybe that was the problem…

Anyways, lessons learned from setting up simple cron jobs.

1. How-Tos are good, but man is better.
2. crontab -e will either load your current crontab, or load a blank template for you.
3. crontab -r will kill your current crontab. Which is why it’s nice to keep the output of crontab -l, which lists your crontab jobs, in a backup file. Because unless you like setting up crons, you don’t want to be left high and dry without a backup.
4. try running your jobs before putting them in the crontab. Or, if you’re like me, do crontab -l, cut and paste, and figure out why /usr/local/binruby is not the command you want (/usr/local/bin/ruby was what I was looking for).
5. to make sure your jobs are running, tail -f /var/log/syslog. Note that this doesn’t tell you if they’re crap or not.
6. Or, append your output to a log, and check to see that the log is growing.

I’m still kind of cruxing about how I’m going to run this on a deployed app. I think I’m going to have to get ops to add deploy_user, and add the crons on deploy_user’s behalf. Of course, I’ll worry about that when I can actually smoothly install mod_rails on my semi jacked up box. I must be the only tard in the universe who screwed up a mod_rails install, but more on that tomorrow.

Tomorrow (er, 1 week later)

In the mad rush to launch (more later), I forgot to update this page, which is bad because this is my scratchpad that has more than once saved me from repeating a painful process. Summary: anyone installing mod rails should take 4 minutes and view the railscast. I was missing the following in my conf file:


LoadModule passenger_module /usr/local/lib/ruby/gems/1.8/gems/passenger-1.0.5/ext/apache2/mod_passenger.so
RailsSpawnServer /usr/local/lib/ruby/gems/1.8/gems/passenger-1.0.5/bin/passenger-spawn-server
RailsRuby /usr/local/bin/ruby


The production instance of this metrics server is going to run on Ubuntu Feisty, which comes installed with rrdtool 1.2, but the ruby bindings I want (need) to use bind to 1.3. So this is how to install rrdtool on Feisty.

(1) download the source:

curl http://oss.oetiker.ch/rrdtool/pub/rrdtool.tar.gz > rrdtool.tar.gz

then follow the instructions rrdbuild page : basically,

./configure

sudo make

sudo make install

However, in order to get configure to complete, I needed to install a couple of dependencies, pango and xml-2. I like configure, it’s very good about telling you what is missing and where to get it. And the rrdbuild page is also great at specifying exactly how to install the missing packages.

I figured I was up and running at that point. I built the ruby bindings from {src dir}/bindings/ruby, but when I tried to run the files I had been running on my Mac, I got:

/usr/local/lib/ruby/site_ruby/1.8/x86_64-linux/RRD.so: librrd.so.4: cannot open shared object file: No such file or directory - /usr/local/lib/ruby/site_ruby/1.8/x86_64-linux/RRD.so (LoadError)


NOT COOL.

How can a file that exists, /usr/local/lib/ruby/site_ruby/1.8/x86_64-linux/RRD.so, not be found? Was it a permissions thing? I tried building as sudo, same thing. Then I made sure that the file actually existed, just to check my head. Yes, it’s there. Yes, I’m getting the same error. Wait. Could it be the librrd.so ? I try a

ldconfig -v | grep rrd

and only get librrd.so.2. Hmmm. OK, desperate times…I edit /etc/ld.so.conf.d/local-installs.conf, and add

/usr/local/rrdtool-1.3.0/lib


to the path. That worked. The ops team is not going to be super thrilled about the amount of jackassery it took to get this up and running, which is why I’m documenting it here. Because they’ll make me maintain it. Just like I would if I were in their shoes

The basic architecture of this (very simple) metrics gathering and display application is:

1. Scripts run as crons from within the rails directory. Every minute, the cron wakes up and checks the database for the last time run and the poll interval. If they are
2. They update RRD files and generate .pngs that reside in the rails /public dir.
3. they update the latest value, and the last time run.

The scripts and the rails app intersect at two points:

• the interval and last time polled
• the generated PNG file.

I may choose to store and display the last collected value, but that will happen after getting feedback from my customers (the development, operations, and product teams).

Since the script has to check the interval and the last time polled, it needs access to the database. I naturally wanted to use the ActiveRecord classes that I use in the rails app, I also wanted access to rails environment variables, like RAILS_ROOT.

By requiring environment.rb:

require File.dirname(__FILE__) + '/../../config/environment.rb'

I was able to get rails like behavior into my scripts.

Continued from part I:

Now that the RRD bundle is installed in Ruby’s default load path, I require RRD and access the convenience methods. The methods basically pass all parameters in as strings, which is fine, but I don’t like thinking of time and values as strings if I can avoid it. So I wrote a wrapper class that allows me to pass in values as typed options, and then casts them to the internal strings.

A couple of notes about creating, updating, and rendering RRD graphs using the built in Ruby binding.

Creating an RRD graph

At create time, the first parameter is the name of the file, minus the .rrb extension (create will puke if you specify the extension). The start time is expressed in seconds, my code below passed it in as a Time object and converts it to seconds. The step time is the minimal amount of time an update can occur at — in other words, if your step is 50 seconds and you try to update at 10 seconds, you get an error.

The DS option defines a dataset as follows:

DS:[name]:[graph type]:[min time to show an error condition]:[min value or unknown]:[max value or unknown]. More explanation of the suitable graph types is found here.

RRD.create(
name,
"--start", "#{@start.to_i}",
"--step", "#{@step}",
"DS:#{@dataset}:#{@type}:#{@heartbeat}:#{@min}:#{@max}",
"RRA:#{@collapse_method}:#{@xff}:#{@collapse_steps}:#{@collapse_rows}")


In the example above, I only create a single dataset. You can create 1..N, although I’m sure N has an upper limit, I haven’t found it specified anywhere. Also, I believe the data set is restricted to < 19 characters in length. The RRA section syntax is as follows:
RRA:AVERAGE | MIN | MAX | LAST:xff:steps:rows
where the collapsing is done by averaging values, or min/maxing values, the xff value specified limits unknown values from being collapsed by establishing a max ratio of unknown values to known values. The steps value specifies the number of datapoints collapsed, and the rows value specifies how many collapsed datapoints to keep. So RRA is where you really get a chance to limit the size of the RRD file.

Updating an RRD Graph

Once the graph has been created, it exists as the file you specified using the name parameter above. You update it with time:value statements: in the code below, I’m updating an array of time:value statements:
# simple update of multiple values
def update(times, values)

for i in 0..times.length-1

RRD.update(@name,”#{times[i].to_i}:#{values[i]}”)
end
end

In the code above, as for all RRD operations, you specify the name of the RRD file you want to operate on in the first parameter.

Note that you cannot update a graph with a time less than it’s start time or a time that is less then the last time + the step time specified at creation.

Displaying an RRD Graph:

Graph display is the most complex operation with RRD. I’m not going to go into all of the details: some really good examples are found here.

I’ve taken the simplest approach to displaying a graph:

RRD.graph(
renderedFile,
"--title", title,
"--start", start.to_i.to_s,
"--end", finish.to_i.to_s,
"--interlace",
"--imgformat", "PNG",
"--width=#{width}",
"DEF:a=#{@name}:#{@dataset}:AVERAGE",
"LINE1:a#0022e9:#{@dataset}")


Unlike the update method, the name of the actual desired graph is the first parameter, not the name of the RRD file. The RRD file to load is specified in the DEF line. You can specify multiple DEF values to display dataset from different RRD graphs. You will need to specify the way you want each dataset rendered: in the above example, I define a value a with the DEF statement that I reference in the following LINE statement:

In order to render data, you will need to specify how you want to display it with( as a line, as area under a line, as a tick mark, etc). More details about how to define data sets, including creating datasets via the CDEF statement, are found in the graph data documentation. Details about how to display data are in the rrdgraph method documentation. The format of the DEF, CDEF, LINE statements is RPN, i.

Make sure to specify start and end in a way that shows values as you would like to see them, i.e. make sure your latest value is in the specified start and end range.

I’m trying to set up a simple heads up display of key information about our running system as we move towards releasing our first product. This display is grouped into themes, each theme contains 1..N graphs of relevant system data.

As a manager this is not my full time job, so I didn’t to get into navel gazing mode wrt the  ‘ultimate’  stats monitoring system, especially since that wheel has already been invented. I settled on using RRDTool to display statistics, mainly because RRDTool has pretty advanced, robust ways to age data out, while keeping the database down to a finite size. It also generates PNG files, which I could update and throw up on a page as needed. There would be a need to track application specific metadata (like what each graph maps to), but no need to handle storing and retrieving metrics data, which gets me (and the poor bastard that inherits this tool) out of a world of pain.

It’s been years since I’ve set up RRDTool, and that was on a Red Hat system. Now I’m a Mac fanboy, and so here are my notes on how to set up RRDTool with Ruby binding on Mac OSX Tiger.

RRDTool can be installed via mac ports which makes pulling in the dependencies ‘not my problem’– the best kind of problem to have :

sudo port install rrdtool

To build the ruby binding, you need to get the source from here. The ruby binding is in {src folder}/bindings/ruby. In order to get the generated Makefile to point to the installed rrd libs, you need to change the following line in extconf.rb:

dir_config("rrd","../../src","../../src/.libs")

to

dir_config("rrd","../../src","/opt/local/lib")

Then run

ruby extconf.rb

to generate the makefile, then

make

and

make install

to respectively build and then install the RRD.bundle to /opt/local/lib/ruby/site_ruby/1.8/i686-darwin8.9.1

To confirm that things have run successfully:

ruby test.rb should generate a test.png file.