How dynamic finders work

— August 13, 2006 at 09:49 PDT


I got into a discussion on IRC last week talking about how ActiveRecord's magic find_by_whatever methods work. It's an interesting topic (at least to geeks like us) and probably worth an explanation.

As part of its magic, ActiveRecord gives you some convenience methods to find records by their attribute values. These methods are called dynamic finders. There are two forms, find_by_whatever, and find_or_create_by_whatever to create the record if you can't find it. (In edge Rails there is also find_or_initialize_by_whatever which instantiates the model but doesn't save it.) You can use one attribute or many. Here are some examples and their equivalents in standard finders.

User.find(:first, :conditions => ["name = ?", name])
User.find_by_name(name)

User.find(:all, :conditions => ["city = ?", city])
User.find_all_by_city(city)

User.find(:all, :conditions => ["street = ? AND city IN (?)", street, cities])
User.find_all_by_street_and_city(street, cities)

Notice how much shorter and easier to read the dynamic finders are than the standard finders. Pretty nice! There are some limitations, the most obvious being that you can only test based on equality, so you couldn't find all users who signed up more than 30 days ago.

Read on for an under-the-covers look at how this magic works.

ActiveRecord implements dynamic finders using a method_missing trick. If you're not familiar with how method_missing works, you can find a good description in Chapter 13 of David Black's Ruby for Rails. In brief, method_missing lets you trap when you send an object a message that it doesn't have a method for, then do something useful instead of just choking. In this case, method_missing lets ActiveRecord objects pretend they have all these find_by_whatever methods that don't really exist in code anywhere.

If you want to follow along in the code, look at ActiveRecord::Base, line 1090 (or search for "method_missing"). I'm going to simplify things by glossing over some details that don't matter much to understanding the basic operation.

First off, lets look at the top-level structure of method_missing.

def method_missing(method_id, *arguments)
  if match = /find_(all_by|by)_([_a-zA-Z]\w*)/.match(method_id.to_s)
    # find...
  elsif match = /find_or_create_by_([_a-zA-Z]\w*)/.match(method_id.to_s)
    # find_or_create...
  else
    super
  end
end

If you're like me, you might be a little tempted to refactor the method to move the guts of those cases into their own methods just so method_missing can look as clean in the real sources as my abbreviated example. Are you, just a little? Anyway, method_missing is called with a method_id symbol that is the name of the missing method, and a splat list of the arguments that were passed to that method. If all is well, those arguments should be the values that will get bound to the query's conditions.

There are then just three cases for what can be done with the method_id. First, it might be a find_by_whatever or find_all_by_whatever method. Second, it might be a find_or_create_by_whatever method. If it doesn't match either of those, then it's not our problem so we do a super to pass handling of the missing method on up the superclass chain.

Drilling down to see what happens in the first (find only) case, we see this (cleaned up a bit):

finder = determine_finder(match)

attribute_names = extract_attribute_names_from_match(match)
super unless all_attributes_exists?(attribute_names)

conditions = construct_conditions_from_arguments(attribute_names, arguments)

case extra_options = arguments[attribute_names.size]
  when nil
    options = { :conditions => conditions }
    send(finder, options)
  when Hash
    # deal with extra options
  else
    # deprecated extra options API
end

First, we determine whether we're dealing with a method name starting with find_by or a find_all_by. The "finder" method is an internal method of ActiveRecord::Base that doesn't appear in the public API. For find_by it is find_initial, which corresponds to sending find(:first) to the model class. For find_all_by it is find_every, which likewise corresponds to find(:all).

Next, we use extract_attribute_names_from_match to parse the rest of the missing method name and extract the names of the attributes we want to search on. split('_and_') does what we want since attributes are delimited by "_and_". If unless all_attributes_exists? tells us that any of the attribute names don't match actual attributes of the model, we have to punt and use a super to make the missing method someone else's problem.

Once we know what attributes we're dealing with, we need to get the values we'll use to search for matching records. Say we are handling User.find_by_name_and_city("Josh", "San Francisco"). By this point we have "name" and "city" as the attribute names, so the next step is to generate :conditions => ["name = 'Josh' AND city = 'San Francisco'"]. The construct_conditions_from_arguments method creates the conditions string for us. It provides for three types of equality tests depending on the type of argument: IS for nil values, = for singular non-nil values, and IN for lists of values.

Looking back at the code for the find case we can see we're almost done. The next thing is to deal with any options in the call. Dynamic finders allow the usual find options. For example, you can sort the results using an :order option:

User.find_all_by_city("San Francisco", :order => "name")

I'm going to skip over the options processing since it's not all that interesting. You can explore that code on your own if you want the gory details. So, ignoring extra options, all we have left to do is to call the finder method with the generated conditions string as the only option.

send(finder, options)

For find_all_by_street_and_city("Lombard", "San Francisco"), that is equivalent to

find_every(:conditions => "street = 'Lombard' AND city = 'San Francisco'")

That's it for the basic find case. Now we can look at find_or_create.

attribute_names = extract_attribute_names_from_match(match)
super unless all_attributes_exists?(attribute_names)

options = { :conditions => construct_conditions_from_arguments(attribute_names, arguments) }
find_initial(options) || create(construct_attributes_from_arguments(attribute_names, arguments))

The find_or_create case looks much shorter than the find case. That might seem a bit odd, since it isn't just finding, it also has to create missing records. However, the find case had to support both "find one" and "find all", and also had to deal with potential extra options. But find_or_create doesn't support the "find all" case, and so needn't support any of the extra options that are used to process lists of multiple results.

This is an important point worth mentioning, as it isn't mentioned in the API documentation. find_or_create dynamic finders don't support options as find_by and find_all_by do. I suppose it's time for a doc patch.

Anyway, back to the code. We've seen the first part before (which suggests a potential refactoring). It's just like the code in the simple find case, and extracts the attribute names from the method name.

The rest looks somewhat similar to the find case code. We generate the conditions for the search query based on the attribute names and argument values. Note that we don't need to use send to use the finder method, since it will always be find_initial (remember, "find all" is not supported). The final bit is to notice if the search query returned nil, and if so create a new record using the supplied arguments. That is done by creating a hash that maps attribute names to argument values in order. The code for that is the clearest explanation:

attribute_names.each_with_index { |name, idx| attributes[name] = arguments[idx] }

The found record is returned, or if there was none found, the created record is returned.

And that's all there is.

Limitations

It's worth noting a couple limitations of dynamic finders.

Dynamic finders work only with model attributes, not with associations. So if you have a belongs_to :user association on an article, you have to write Article.find_by_user_id(user.id). I have an experimental patch that provides this functionality. I'm not entirely sure what to do with the patch. It works fine for simple belongs_to associations, but doesn't yet handle polymorphism. I'm looking into supporting polymorphic belongs_to associations, but the code is getting ugly enough that I'm not sure it's worth it. Feedback would be appreciated.

One more thing to note is that find_or_create has yet another limitation. While you can pass an array of values to find_all_by finders, you can't do so for find_or_create. It might not make sense to pass arrays to find_by (as opposed to find_all_by) either, but I can see both sides of the issue so I'll let it be for now.

Last words

I hope this spelunking expedition through the dynamic finder code has been useful. I find that I appreciate the magic of Rails even more when I understand how it works, and I also think I can make better decisions about when to rely on the magic and when to roll my own solution. This may also give you some ideas of how to use method_missing in your own code, though I'll caution against profligate use.

If you like dynamic finders, you should also take a look at Ezra Zygmuntowicz and Fabien Franzen's ez_where plugin, which lets you build queries with Ruby operators and also supports more than simple equality conditions.

9 commentsrails

Comments
  1. Bob Silva2006-08-13 10:29:01

    Nice writeup Josh. I'll be monitoring trac for that doc patch!

  2. Aitor2006-08-13 11:16:30

    Interesting post!

  3. Ted2006-08-13 12:10:58

    Another great one. I'm glad the new job isn't keeping you too busy to write :-)

  4. Robertas Aganauskas2006-08-13 13:09:18

    Will all this magic uncovered things became more clear to me. Thank You

  5. Will Fitzgerald2006-08-14 06:09:06

    Very nice writeup, Josh.

    I found myself wanting find_or_create_by to take initialization parameters, too, so that instead of:

    user = User.find_by_name("Josh")
    unless user
        user = User.new(:name => "Josh")
        user.city="San Francisco"
        user.save
    end
    

    I could write something like:

    user = User.find_or_create_by_name("Josh", :create_args => {:city => "SF"})
    

    Actually, the code I wrote is much uglier than the unless block above, but you'll discover that soon enough. I suppose this isn't so bad:

    user = User.find_by_name("Josh")
    user = User.create(:name => "Josh", :city => "San Francisco") unless user
    
  6. Josh Susser2006-08-14 07:53:50

    @Will: That's close to what find_or_initialize_by is useful for.

    user = User.find_or_initialize_by_name("Josh")
    user.city = "San Francisco"
    user.save
    

    Still not ideal for your use case, but it's close. I like your idea for passing a hash of initial attribute values. Or it could be done with a block.

    user = User.find_or_create_by_name("Josh") do |u|
      u.city = "San Francisco"
    end
    

    Actually, I think just passing a hash of all attributes might be simplest. Match on the ones named in the finder name, use the others to initialize the new object.

    attrs = {:name => "Josh", :city => "San Francisco"}
    user = User.find_or_create_by_name(attrs)
    user.city # => "San Francisco"
    

    That actually isn't too tricky a change to the code. I might have to give it a go.

  7. bgates2006-09-16 14:25:15

    I noticed an oddity when testing to make sure a dependent=>destroy association was working. If a user(id=3) is the only user who belongs_to a group(id=7) that gets destroyed,

    User.find_by_group_id(7)

    returns nil, but User.find(3) raises an ActiveRecord exception. Is there a reason for that?

  8. bgates2006-09-16 14:34:23

    Even better -

    User.find_by_id(3)

    returns nil, but User.find(3) raises an ActiveRecord exception.

  9. Katherine2006-10-07 14:35:25

    @bgates: It's a feature, not a bug. User.find(3) is only meant for use in cases where you know a user_id of 3 exists. If you don't, you're supposed to use User.find_by_id(3). I can't remember exactly why, but there's an explanation somewhere near the beginning of Agile Web Development with Rails. I don't have it in front of me or I'd give you the page number.

Sorry, comments for this article are closed.