has_many :through

Hello, New York

2008-09-22T19:38:03-07:00

By way of the obligatory sorry-I-haven't-blogged-much-lately apologette, I should say that life has been busy this summer. The second biggest deal for me was, get this, moving to New York City. Yes, I know I just moved to a new place in San Francisco, and no, I haven't given up on my favorite city. It's a temporary move and I should be back in SF by the end of the year. What happened is that Pivotal Labs has opened an office in Manhattan, and I've come out to work on client projects and help get the office going. The takeaway on that is that Pivotal Labs is available for projects in New York, and we're also looking to hire top-notch Rails developers to work with us here.

The word I have to pick to describe New York is vital. That's true of the city in general, but from what I've heard it's also true of the Ruby development community. I am looking forward to getting to hang with folks here and see what I can learn from the other coast. There's a lot of Ruby developer events around here. I'm going to start with the nyc.rb hackfest tomorrow.

While I'm out here on the East Coast, I'm going to hit up some conferences. In November I'll be attending RubyConf 2008 in Orlando. Then later that month I'll be at the Voices That Matter: Professional Ruby Conference, giving a talk entitled "Ruby: Fragile or Agile?"

In the mean time, where's the best pizza in New York?

Sorting things out

2008-08-17T10:49:28-07:00

I recently packed up everything I own and moved. I'd lived in my old place for about nine years and I have the packrat gene on both sides of the family tree, so I had a lot of crap to sort through to figure out what to move and what to trash, as well as which box what should go in. Now that I'm here in the new place, I've had to sort through the remaining stuff to figure out where it all goes. So you might appreciate that sorting has been on my mind a lot lately. (See? It's a topical tie-in. I don't do those often, so I hope it wasn't too awkward.)

I've also been working on an application that does a lot of sorting to prioritize tasks in a workflow. These items often need to be sorted based on multiple criteria, such as how long an application has been waiting for approval, how many times a customer has been called recently, etc. We also have to sort names that have anywhere from two to four components (Hispanic names can have both paternal and maternal names instead of just a surname).

Ruby enumerables can be sorted using the #sort method, which orders contents using the <=> trinary comparison operator. It's pretty simple to override <=> if instances of a class are only ever sorted one way, but when you get to a situation where you need to sort things in different ways depending on the context, you need a more flexible solution

The next obvious thing to try is passing a block to the #sort method. This works great, but it has two drawbacks. One, if the sort criteria are complex, you can end up with some ugly looking code. And B, it's not very DRY for re-using sort criteria.

The approach I came up with for my application makes use of the sorting properties of Ruby arrays. Ruby arrays sort quite nicely using the <=> operator. <=> compares two arrays by comparing all their values until a non-equal result is found for any pairings of values in the two arrays. Here are some example comparisons:

>> [1, 2, 3] <=> [1, 2, 3]
=> 0
>> [0, 2, 3] <=> [1, 2, 3]
=> -1
>> [1, 2, 3] <=> [0, 2, 3]
=> 1
>> [1, 2, [4, 5]] <=> [1, 2, [6, 7]]
=> -1
>> [1, 2, [4, 5]] <=> [1, 2, [2, 3]]
=> 1

There are some things to watch out for, like don't have nil anywhere in the array contents, and shorter arrays sort less than longer arrays where all else is equal. But this is a great feature to build upon.

Based on the array sorting functionality, I defined a concept I call a sorter. A sorter is an array that holds multiple values to use to sort the collection by in order of precedence. For example, if you want to sort people by name using the precedence paternal, maternal, first, middle, you can construct an array of those values and sort the list using that.

class Person < ActiveRecord::Base
  def name_sorter
    [(paternal_name || ""), (maternal_name || ""), (first_name || ""), (middle_name || "")]
  end
end

@people = Person.find(:all).sort_by { |p| p.name_sorter }

Notice I'm making sure that I never have a nil as a value in the sorter, cause that will make things blow up like the ending of a Die Hard movie. If you want to avoid that verbiage, set up the model so that the names default to empty strings. Also notice that I'm using the #sort_by method, instead of the #sort method with a 2-arg block. Actually, let's look at all our options. The following three forms will give equivalent results:

@people = Person.find(:all).sort_by { |p| p.name_sorter }
@people = Person.find(:all).sort_by(&:name_sorter)
@people = Person.find(:all).sort { |a,b| a.name_sorter <=> b.name_sorter }

The first line is the one I'd use in most situations. The second line is slightly more compact, but the symbol-to-proc coercion should be avoided in any code where you care about performance (at least until Ruby 1.9 is ready for prime time). The third line is usually going to be a bad choice, because a sorter array would be created for every time its person was compared, which will be roughly ln(n) times if you are willing to believe my naive assumptions about Ruby's sorting algorithm. If you don't want to trust naiveté, check out the simple performance comparison I ran. This benchmark sorts an array of 60 items using a sorter on 4 values.

                user     system      total        real
sort        6.430000   0.000000   6.430000 (  6.466645)
sort_by     1.510000   0.000000   1.510000 (  1.509017)

That kind of difference is pretty significant to me. Of course, if your sorter can be cached so you can reuse it for multiple comparisons, a #sort might be better. As always, measure it if you care and you're not sure.

One last thing. What if we just rolled the sort by hand using a block? (The code below assumes names will default to an empty string.)

@people = Person.find(:all).sort do |a,b|
  if    0 != (comp = a.last_name <=> b.last_name)
    comp
  elsif 0 != (comp = a.maternal_name <=> b.maternal_name)
    comp
  elsif 0 != (comp = a.first_name <=> b.first_name)
    comp
  else
    a.middle_name <=> b.middle_name
  end
end

That's fairly ugly and it's actually slower than the #sort_by approach too.

And because I don't believe in secret benchmarks, here is a gist of the benchmark code I used.

Now if you'll excuse me, I have a lot of empty cardboard boxes to toss in the recycling.

Extra geeky: the recursive lambda

2008-06-20T11:06:10-07:00

I'm not sure where I first heard that you could do a recursive lamdba in Ruby, but it's been simmering on the back burner of my brain for a while. I've just never had a reason to use one, until now...

I wanted to process the Rails request params, which is a hash of strings and hashes of strings and hashes of strings and hashes... you get the idea. The need was to strip all the accent marks from user input throughout the application. Here's what I came up with:

class ApplicationController < ActionController::Base
  before_filter :strip_accents

  protected
  def strip_accents
    thunk = lambda do |key,value|
      case value
        when String then value.remove_accents!
        when Hash   then value.each(&thunk)
      end
    end
    params.each(&thunk)
  end
end

That's all completely clear, right? The filter enumerates the top-level hash using the &/to_proc operator to coerce the lambda to a block for the #each method. #each passes the key and value to the lambda, which either removes the accents from a string, or recursively enumerates the contents of a nested hash.

I think it's totally cool that you can do this in Ruby. Everyone thinks that Ruby is just an object-oriented language, but I like to think of it as the love-child of Smalltalk and LISP (with Miss Perl as the nanny).

An extra special case

2008-06-15T10:31:11-07:00

A couple of months ago I ran into a weird issue in my current Rails project that made no sense at all. All we did was add a lock_version field to a model to enable optimistic locking and suddenly things started breaking in a big way. After a bit of digging we found it was because ActiveRecord wasn't properly quoting a table name when updating a record with optimistic locking. I submitted a patch for that issue (so it's fixed in Rails 2.1), but lately I've seen a few similar bugs having to do with table name quoting in various circumstances. The amusing thing to me is that all of these bugs have one thing in common: they were uncovered by creating a model named Reference.

At first I thought this was a pretty big coincidence, but after just a moment's thought it seemed pretty obvious. ActiveRecord pluralizes model names to form conventional table names, and references is a reserved keyword in MYSQL. I guess Reference is a word that makes a good model name, especially if you're building a big data graph and can't think of a more specific relationship name, and it's about the only noun that pluralizes into a reserved keyword that anyone would ever use. In our case, we could have done a rename refactoring to change the model class name to CharacterReference. Instead we used an override and changed the table name to t_references, since that seemed like the least effort for a temporary workaround until the fix got released with Rails 2.1.

All these various issues with table name quoting are indeed bugs in ActiveRecord and should be reported and fixed. (There's also a major reworking of the internals of ActiveRecord in progress that should deal with virtually all of these issues in one fell swoop.) But in the mean time, you might want to avoid using model names that generate SQL reserved words, or just override the table name to something else.

me me me!

2008-06-06T09:57:02-07:00

I'm finally recovered from RailsConf. Wish I could say the same thing for my car, which seems to have gone into a major funk from being ignored so much. Anyway, it's been cool reading all the aftermath posts and seeing people's thoughts and analysis of the conf. MagLev seems to be getting a lot of the attention, and I'm eager to see how things develop with it.

If you missed me at the conf, or saw me but just can't get enough of me, here's some links to a couple interviews people did with me there.

Gregg Pollack did some very nice video spots with many of the conf speakers. His RailsConf in 36 Minutes video is great. If you don't want to watch the whole thing (but why wouldn't you?), he posted the interview with me separately too.

Fabio Akita did some great podcast interviews at RailsConf. You can grab mine directly too.

As I said in my chat with Fabio, the next thing I'm going to do is release the Teldra code on GitHub. Stay tuned, it shouldn't be too long now.

Happy Friday!

The Great Test Framework Dance-off

2008-06-01T15:19:55-07:00

Got my RailsConf talk done yesterday. Seems to have gone over well, though my slides didn't show color well in the brightly lit room so they turned the lights down all the way to see them better and I ended up doing the talk as a ghost story for half of it. Guess I have to revise my rules for slide creation to account for ambient light.

Anyway, I have a PDF of the slides posted: The Great Test Framework Dance-off. I've also opened up the example code and tests on github as the teldra_prime project. That project is based on the application that runs this blog, but you should probably avoid deploying it to run your own blog. I'll be releasing the Teldra blog software as its own project very soon, and that will be the project where I continue development and people can contribute changes.

Also, it turns out I have a craaaaazy fan. Here's a picture of us at the end of the talk, me posing with his ballpoint tattoo of me on his bicep. Yes, somebody drew picture of me on his body! That so beats people introducing themselves at the urinal!

UPDATE: Alright, it turns out there is a way to significantly simplify the be_sorted custom matcher I showed as an example for extending RSpec. David Chelimsky sent me this alternate implementation that uses the simple_matcher helper:

def be_sorted
 return simple_matcher("a sorted list") do |actual|
   actual.sort == actual
 end
end

That doesn't have all the power of the example I gave, but it is pretty much equivalent in operation to the test/unit example I gave and said was so much easier to write than one in RSpec. Nice one, David.

Quick RailsConf Update

2008-05-31T08:02:41-07:00

DrNic complained about the lack of blogging from RailsConf, so here's a quick little update with some highlights. It's still early here and I have to get to breakfast soon, so I have to be brief.

First impression: lots of people. Second impression: "they want us all to fit in there?"

I called it exactly right. Joel Spolsky tried to pull a Ze Frank and did some random crazy presentation in his keynote, but it fell pretty flat. It was really polished and fairly amusing, but I was often insulted by either his blatantly sexist attempts at humor or his estimation of the audience's intelligence. It was nearly entirely content-free, and while he tried to develop a theme about the importance of esthetics, he never went anywhere with it. If you missed it, count yourself lucky.

On the other hand, David Heinemeier Hanson did a keynote where he transformed himself from a technologist into a life coach. His talk on The Great Surplus was interesting, if only to get a glimpse of the world from his perspective. I think most of what he said was right on, especially the bit about how everyone should code less and sleep more.

One of his points was his expectation about how Rails might lose its productivity advantage. He said there were three options: 1) Mainstream tech copies Rails' good points, 2) A radically new tech outdoes Rails, and 3) Rails becomes mainstream so there is no longer an advantage. But I think there is a 4th option. It's what happened to Smalltalk. I'm talking about C++. We Smalltalkers used to think the advantages of our language were so significant that it would take over the world. We had a huge productivity advantage over C coders. Then C++ came along and gave C coders just enough to let them improve their productivity and their ability to write larger more complex systems. It still wasn't as good as Smalltalk, but it was better than C, and much more accessible to most programmers than Smalltalk. C++ eventually sucked up all the oxygen and Smalltalk is now only a language for hobbyists and the occasional programming god. I think this is the most likely threat to the Rails surplus, that C# or Scala or something can do a good enough job that people can double their productivity with far less of a change in mindset or tools, and eventually no one will care about the ten times (or whatever) productivity of Rails. "Good enough is good enough."

Last bit before I gotta run. I've heard rumors about MagLev for a while, and the early announcement last month got me really excited. The demo and discussion yesterday by Avi Bryant and Bob Walker was one of those jaw-droppers that had everyone in the room freaking out. What they showed was pretty spectacular, though I always like to keep in mind Lansford's Corollary to Clarke's Third Law: "Any sufficiently advanced technology is indistinguishable from a rigged demo." The staggering performance boost together with the scalability and seamless integration of persistence could be a serious game-changer. I think the GemStone OODB technology beats the pants off of the ORM approach for most web applications, and if they can pull this off it's going to have a huge impact on how I write my software.

I'm giving my talk today at 4:25pm. Yes, I still freak out about giving talks. As critical as I can be of others, I'm hardest on myself. I hope I don't give myself too lousy a review when I'm done, heh.

See you at RailsConf

2008-05-27T23:17:00-07:00

Seems like only last week that I was in Portland for RailsConf, but it was a whole year ago. That means it's time to do it again. I'm looking forward to the conference and getting to soak in all the Railsness. The program looks excellent this year, though I doubt Joel Spolsky can top Ze Frank or _why.

I always like to meet my readers, so please do say hi. If you want to know where to find me, I've got a few fixed points on my schedule:

7:30pm Thursday - Pivotal Labs BoF. We were going to talk about our open source projects, but instead we're going to talk about the agile development process and Rails.
12:30pm Friday - Book Signing. I've got a recipe in Mike Clark's new Advanced Rails Recipes book, and a bunch of us contributors will be signing copies. Look for us in the Powell's Books booth. I'm pretty sure Mike is buying me some Scotch for this, so make it worth his while.
4:25pm Saturday - The Great Test Framework Dance-off. I'm doing a talk comparing the most popular test frameworks. I don't know if it will be Adam Keyes level great, but I promise there won't be karaoke.
9:00pm-Midnight Saturday - Pivotal Labs Beer Night. Free drinks, food and pool. At the Rock Bottom Brewery, 206 SW Morrison St.

Also we're raffling off a Nintendo Wii in our swiipstakes on Saturday. Come to the BoF on Thursday to get one of our shirts, or if you can't be there you can make nice with a pivot later to scam one if we have extras. If a pivot spots you in a shirt over the next three days you can get a raffle ticket. (Wear the shirt while presenting a talk to get two tickets!) Then come to the beer night for the Wii raffle (must be present to win).

A simple alternative to namespaced models

2008-05-06T08:17:09-07:00

A project I'm working on now is up to 57 model classes and is still growing. That's a lot of classes - welcome to domain modeling. In my opinion, the number of classes is a fair tradeoff that keeps each class simple enough to understand. In some ways it moves complexity out of the model class internals into the inheritance hierarchy, which is an important part of object-oriented design. I've worked on projects with many more model classes than that too. (Financial applications seem to require a lot of classes to model the complicated workflow and permission systems.)

The place where it starts to get hard to manage is when I look at the file system and see so many files in one directory. My brain usually starts to overload when I see more than a dozen or so classes in a directory. My first inclination is to throw some related class files into a subdirectory. The problem is that the standard way to do that in Rails is to put those models classes in a namespace (module). Rails used to have big problems with namespaced models, mainly with the dependency auto-loading code that finds class files based on the model class name. Most of those problems have been fixed, but there are still some usability issues with namespaced models.

The first problem, and the biggest one, is foxy fixtures. I like foxy fixtures a lot and the feature makes fixtures much easier to work with. But namespaced models just don't work with foxy fixtures. To get them to sort of work, you have to insert calls to Fixtures.identify anywhere you'd use a normal association. You also can't use the fixture helpers in your tests, so you have to do an explicit find, again using Fixtures.identify. It's pretty ugly, and it's actually worse than it used to be with pre-foxy fixtures.

The other problem is setting up associations. You have to use the :class_name option to tell the association how to find the model class. Again, it's a bit ugly, but at least it works well once you tell the association which class to use.

There's also an issue with STI and polymorphic class names being saved un-namespaced, but it looks like that's been fixed on edge now.

So what's a developer to do? The workaround is really quite simple. Don't use namespaces.

By default, Rails looks for models only in the app/models directory, and in subdirectories of app/models for namespaced models. But if you just want to put the model class files in a subdirectory, all you have to do is add that subdirectory to the load paths. If you do that, you don't need to put the model class in a namespace.

In your environment.rb file you'll find a commented out line with an example of setting config.load_paths (in the initializer run block). Uncomment and adjust that line, or if it's not there, just add the following line.

config.load_paths += %W( #{RAILS_ROOT}/app/models/role #{RAILS_ROOT}/app/models/user )

That incantation adds the role and user subdirectories to the load paths collection, letting Rails find the various role and user class files. No namespacing required, and it works with or without inheritance.

UPDATE: It's true, there are no original ideas. Boy meets girl, boy loses girl, boy gets girl back. Man against nature, man against man, man against himself. Same thing for blog posts. Chris Wanstrath blogged this up last year in a post on errtheblog. Still, good ideas like good stories are worth a retelling now and then...

Symbols are not pretty strings

2008-04-19T09:25:51-07:00

Symbols are one of the basic features of Ruby that give it that certain charm we all love. They aren't unique to Ruby (look at Smalltalk or Lisp), but they are a fundamental piece of the language. I'm not going to review what symbols are in this article since there are plenty of other explanations a short google away. However, I do want to say a few words about what I consider a common misuse of symbols.

The way I see it, symbols are great for naming things in your code, but bad for using as domain data. Over the last year or so I've seen a growing number of cases where symbols are used as an alternate syntax for plain old strings. I guess some people like to see :thing instead of "thing" in their code. Well, I don't like it. Sure you get to save a character, but at what cost? Won't somebody think of the children?

My rule of thumb is that if you want an object where all you care about is its identity, then use a symbol. If what you care about is how it's spelled, use a string. If you just want to compare a thing to another thing to see if they are the same thing, go symbol. If you just need to output it in some way (puts, to_s, interpolated value, etc.), it's string all the way. In particular, symbols are awesome as keys in hashes, names of states, and names of methods.

A great example of this anti-pattern is the newish (but pre-sexy) way to write migrations. Consider the old-school style (this is still how schema dump works):

create_table "tags" do |t|
  t.column "name",           :string
  t.column "taggings_count", :integer, :default => 0, :null => false
  t.column "created_at",     :datetime
end

You can see that the names of the table and fields are strings, while the names of the field options are symbols. This makes total sense to me. The migration code uses the table and field names only to generate the DDL statement that creates the table - basically a big, formatted print. On the other hand, that code uses option names as keys to look up values in a hash, a proper use of symbols. Something to notice: table/field names usually occur just once in a migration, while options can occur repeatedly (e.g. maybe all your fields are set to :null => false).

Now, the new way:

create_table :tags do |t|
  t.column :name,           :string
  t.column :taggings_count, :integer, :default => 0, :null => false
  t.column :created_at,     :datetime
end

In this example, all names are symbols. While the code does what you'd want it to, I find it a bit harder to read. I have to take time to think about what is a code name versus what is a data name, whereas in the previous example it's obvious just by looking. Even with the new sexy migrations you have the same problem, just not as much. Hmmm, can I really call sexy migrations new when they've been around for a year already?

The cost of a colon

I wailed about the cost before, but what was I talking about? Besides the cognitive cost of muddling code and data names, there are some costs in resource usage. The two basic resource costs are memory and processing cycles, and using symbols hits you in both places.

Let's start off with memory. Every symbol is a memory leak. I don't know who said that first, but it's a great way to think about symbols. Every symbol lives forever, whether created as a literal in source code or programatically with #intern or its alias #to_sym. Ruby's garbage collector will never reclaim symbols, since they have to stay around in the symbol table. (There are ways to avoid that situation by having the symbol table hang onto symbols using weak references or table compaction, but so far Ruby doesn't do that.) On the other hand, strings are garbage collected as any other normal object. Symbols need to have an enduring identity that is re-used every time to ask for a symbol with the same spelling, but strings do not. So if your names are ephemeral, using strings instead of symbols will save you memory in the long run. On the other hand, if you are using the same name over and over, symbols can be a big win. Why allocate the string "integer" hundreds of times when you can use a single :integer symbol instead?

What about the processing cost? Fundamentally, every creation of a symbol involves doing something called interning a string. That process includes doing a string comparison to see if the symbol is a new one, and allocating an entry in the symbol table if it is, or finding and using the existing symbol with that spelling if it is not. This amount of overhead is not onerous for proper use of symbols. In fact, it saves you processing in the long run because after a symbol has been interned you can compare symbols with a simple integer comparison, which is much faster than doing a full string comparison. But if you are misusing symbols as pretty strings, you've paid the cost of interning for no benefit, and then you're wasting cycles for the sake of using a single colon instead of two quotation marks.

If you've ever written a program you've probably heard that "premature optimization is the root of all evil." Well, that doesn't mean you should ignore performance considerations entirely. As Ezra Zygmuntowicz says, "Postmature optimization is the root of all hosting bills." If cutting corners means you can get your code written faster, then by all means throw performance to the wind and write code at warp speed. But why incur a performance penalty when you get nothing for it? It's like putting snow chains on your tires in the summer.

Indifferent Access

But all is not joyful in Symbolville. As I said above, every symbol is a memory leak. Usually that's not a huge problem, but what happens when you lose control of creation of symbols? Consider how Rails manages request parameters. Request parameters are made available in the params hash, and you can access them using the param's name in the form of a symbol as the key. Take a look at a completely contrived example:

authenticate_user(params[:username], params[:password])

This is a pretty awesome way to get to the request parameters, but there's a problem. Think about what happens if your app gets a series of requests to the following URIs:

/books?title=Nine+Princes+in+Amber
/books?author=Roger+Zelazny
/books?genre=fantasy
/books?series=The+Chronicles+of+Amber
/books?this=1&is=2&a=3&surprise=4&attack=5

Things get off to a good start but quickly go awry. After those five requests, you'll have added nine symbols to the symbol table: :title, :author, :genre, :series, :this, :is, :a, :surprise, :attack. Every query param gets converted to a symbol! Hello, DOS attack! Any old dork can generate random URLs and use up your memory a few bytes at a time. This is what we call eventual certain doom.

Clearly, it's a bad idea to generate a symbol for every param name in every URL when you can't control what names you'll have to deal with. This is why we have HashWithIndifferentAccess. In most ways this object acts like a normal hash, but it doesn't distinguish between symbols and strings as keys, and stores keys internally as strings. Rails can store all the request params using string keys (that can be garbage collected after the request completes), but your controller gets to access them using symbols. You get the appearance of the efficiency of using symbols, with all of the overhead of both symbol interning and string comparisons at the same time! But on the other hand, you don't have to worry about a deluge of random query params using up all your memory.

HashWithIndifferentAccess is the sort of solution that drives me slightly insane. While I can understand the reasoning behind it and I haven't come up with a better solution that retains compatibility and usability as well, there's something about it that just smells wrong. It's the worst of both worlds, but probably the best way to go. Someday symbols will get garbage collected, then we'll all be happy we've been using symbols as keys all along. (Evan Phoenix says Rubinius will support some sort of reclamation of symbols sooner or later, so "someday" will probably arrive before the end of the year.)

Association names

ActiveRecord associations are created using macro-style class methods that specify the name of the association. For example:

class Post < ActiveRecord::Base
  belongs_to :user
  has_many :comments
end

Comparing this example to the migration example I used above, you might think I'd say symbols were the wrong choice here. After all, the names :user and :comments get transformed into class names User and Comment. That's the sort of thing you should use a string for, right? But in this case, I think symbols are a fine choice. That's because those names are also used to name association methods. The association macros define #user and #comments methods (and several others). Those names are in the world of source code instead of domain data, and that's a good use of symbols. Also, the symbols will be allocated anyway when the methods are defined, so there's no extraneous use of symbols. It's the right concept with no extra cost, so no worries.

Doing the right thing

When thinking about using a symbol or a string, I consider two things.

Conceptually, if it's a name for something in code, it should be a symbol. If it's domain data, it should be a string. If it's both, it's probably better to go with a symbol.

For efficiency, it's a matter of cost versus payoff. If a name is only going to be used once, a string will be more efficient. If it's going to be used many times, then the overhead of creating a symbol will be paid back in reuse of a single symbol instead of multiple strings and doing integer comparisons instead of string comparisons.

Sometimes, it's really easy to see whether you should use a symbol or a string. Other times, it's not so simple. Rules of thumb are great, but there's that gray area where names are used as both code names and domain data. That's when you start earning your keep as a programmer.