CRUDdy searches

— June 30, 2006 at 19:58 PDT


An open question with the World of Resources paradigm is how to do searches. GET /people returns all people, GET /people/23 gets one person... but how do you get something in between? What we need is the equivalent of :conditions on an ActiveRecord find.

One approach is to use a GET request with conditions on the URL. So to find all males older than 25 years of age:

GET /people?gender=male&age=gt,25

DHH said this is a good way to go, but I wonder. One, this approach will eventually run into the URL length limit. But more importantly, it's not using the CRUD approach of reifying abstractions.

Instead, consider using a Search model. Create a search resource that holds all the options for the query. Then you can execute it whenever you want. POST /searches (post data contains query options) would redirect to GET /searches/42 or GET /people/?searches=42. You could even make it so that searches are unbound until execution, like blocks: GET /searches/42?on=people.

I need to try this out and see how it works, but I won't have a chance to do that for a few days.

Thoughts?

17 commentsrails

Comments
  1. Wilson Bilkovich2006-06-30 20:03:18

    Even cooler, GET /searches/42.xml ..for the results in XML format.

  2. Dr Nic2006-06-30 20:22:16

    I don't think we get much benefit from renaming the people resource as searches, just so we can indicate that we're going to return 0+ results, instead of 1 only.

    If we needed to support a super-generic, user-can-search-over-whole-resource url, then /people?gender=male&age=gt,25 should be ok.

    If our app has known subsets of a resource, e.g. male people or male's over 25, that it uses regularly and requires no UI forms (ie. its always 25, and never 24 or 26) then it might be an idea to create a wrapper model, called AdultMale. From this we'll get the free resource and all the CRUD that goes with it.

    But even here, what is the difference between the create/update/delete actions for the AdultMale or Searches models and the base Person model? None. This itself might give us a suggestion that wrapping up base Models/Resources with fancy names might be the wrong path for us to take.

    Cheers Nic

  3. Josh Susser2006-06-30 21:55:51

    @DrNic: I think putting the search params in the URL works fine, though as I mentioned there are potential limits in URL length to worry about. I had considered using a POST for search, but that seems like a significant departure from the GET verb standard. I agree that the search model is a bit weird, but I don't think it's any weirder than POST /sessions for logging in (as DHH described in his keynote). I'm not sold on the idea though, just throwing it against the wall to see if it sticks.

  4. Jonathan Broad2006-06-30 22:36:34

    I came to the same conclusion as you recently, Josh.

    I've come to think of a search as an event. This came about because an application I'm developing wraps a very sophisticated search engine that often consumes dozens of parameters. Also, the same search mechanics are applied to multiple models.

    The more I thought about it, the more important I thought it was to wrap this event in its own model. Search is really the center of our business (right next to finding--and then buying!). By capturing search activity as a first-order object, I can establish the search context of all sorts of other activity, such as putting items in a shopping cart. As it was, all this valuable data was just slipping inert into logs which were very hard to mine for useful info.

    Now, I just do a quickie insert of the pertinent info into our DB, then apply the search to the engine (totally different middleware). Boom! Instant 'search history'. To find out why someone did something 'downstream' in the app, looking up their most recent search is trivial.

    Also could be helpful in getting around Ajax+back button issues, although I haven't gone there yet.

    So I think it makes a lot of sense from the business side of things, in addition to being RESTful--provided your search really is more complicated than a few parameters. It's a fuzzy area, but an interesting one IMO.

  5. choonkeat2006-07-01 03:10:56

    I'm very much intrigued by "everything is CRUD" approach and think of your suggestion as very Trac-like (so the edgewall guys "got it" long ago?).

    This method also has the additional benefit of getting the developers to do The Right Thing (redirecting-after-post). Normally, lazy me would have rendered the output of the post there and then.

  6. Jay Levitt2006-07-01 07:35:30

    Maybe it's because I wasn't at RailsConf, but I'm just not drinking the everything-is-CRUD Kool-Aid. I think lots of things can be mapped nicely to CRUD, and for those, REST works great. But one of the nicest things about Rails is its ability to create powerful, DSL-like controllers, and limiting each controller to four actions feels like you're exposing too much infrastructure to the "UI" (which, in this case, is the routing engine).

    Yes, if there's a shared search infrastructure, then a separate search controller makes sense. But there are plenty of other actions that don't fit nicely into CRUD, and force-fitting them because it makes a pretty database picture seems like a bad trend. I like expressive controllers.

    This feels to me like a phase that we're going through, not a true design innovation. Maybe I'm alone in that.

  7. Josh Susser2006-07-01 08:32:53

    @Jay: I'm not a true CRUD-the-world believer yet. At this point I'm trying to explore the concept to see where it may have benefit. What I _do_ think is seriously cool is the idea of ActiveResource. When I first looked at Ruby on Rails last year, I immediately wanted something like ARes for the work I was doing at the time. If using CRUD-everywhere design enables building ARes-type services, then I think it's worth looking at. But stay tuned for more analysis, as I think the idea still needs work.

  8. Great idea2006-07-01 09:23:26

    Hi Josh,

    I too drank the CRUD coolaid at RailsConf, and I agree it should be your first thought towards everything.

    Having a search model object I think is actually a great idea. At first I thought you were crazy, but like all great ideas, that's how it starts :)

    Consider this: have you ever wanted to:

    • Know what your users are searching for?
    • Allow your users to save their search criteria for later use?
    • Define a "google alert" type feature in your app?

    If you wanted to do any of these, having a search model object would save your bacon. Imagine having an Alert object that has a 1-many relationship with your Search object.

    Sounds great to me. Thanks for giving me the idea!!

  9. Great idea x22006-07-01 10:02:34

    I liked this idea so much I had to blog about it myself. In the process, I found several new ideas on how to improve my user's experience once a Search object is added to my domain model.

    Here's a quote:

    As for my current baby, my Password Manager and AutoFiller for Mac OS X would benefit greatly from having a search box on the website. If I added a Search object to my domain, I could have realtime data on whether my users are finding what they want.

    Imagine how more effective I could be if I had a list of everything my users searched for and how many results they had. If they searched for "Keepass", they would currently have zero results because it is not mentioned in the manual. Once I saw this, I could add "How 1Passwd Goes Beyond KeePass" to my documentation.

    Thanks again Josh for the great idea. I wish I met you at RailsConf, but a cold/flu bug kept me in my hotel room most of the time. I'll buy you a beer at the next one.

  10. Peter2006-07-02 22:43:55

    choonkeat wrote:

    This method also has the additional benefit of getting the developers to do The Right Thing (redirecting-after-post). Normally, lazy me would have rendered the output of the post there and then.

    But what do you do when the post fails and you want to display the errors? Do you attach all the errors to the flash, redirect, check for errors and then render? With strick CRUD controllers how will I know where I came from and where to redirect to?

  11. peter2006-07-04 21:30:06

    I really like DHH's suggestion of something like

    GET http://www.domain.com/people?gender=male&age=gt,25

    It is easily bookmarked. If I use a search model then I'll have to save the searches forever (or at least a long while) if I want bookmarking to work (or work for a while).

    This issue really bothered me at first but then I realized I've seen some very long URLs. Google says the IE bookmark URL limit is 508 characters. The above url is only 50 characters. That means I have room for about 43 more search scope parameters. I think that will do just fine for most of my cases. Actually, I think that will do fine for all of my cases. And the DHH suggestion is simple.

    I think this is a case of use what is simple and works for 80% of the cases. For the other 20% with extra long search parameter strings or where a searches model is better then the issue deserves more attention.

  12. peter2006-07-06 18:34:03

    Am I the only one still thinking about this? :)

    I still think that DHH's suggestion is a good one. The idea of a scoped GET to /people really seems like what a search is all about.

    http://www.domain.com/people?gender=male&age=gt,25

    If the URL length limit is a problem then why not abuse POST a little more. POST will already be abused to fake PUT and DELETE to work around browser limitations. Why not also use POST to fake GET to work around a different browser limitation? It won't be possible to bookmark these searches like a normal GET but that may not be important. It would even be possible to use POST to fake GET only when the URL limit is excceded.

    I imagine, and I hope, that the changes to Rails are going to be very general and allow for more than two new verbs faked through POST.

  13. Josh Susser2006-07-06 23:22:23

    @peter: I don't want people to think I'm putting word's in DHH's mouth. That URL search form was my idea; DHH just responded that he thought it would work. He may have his own thoughts about other ways to handle search.

    I don't think abusing POST to simulate an extended GET is a good idea. The _method hack is a workaround that we can hope will go away when HTML and browsers begin to support PUT and DELETE methods. Perhaps the thing to do is to extend the semantics of POST on resource collection URLs. I guess that's what the semicolon is supposed to be for, right?

  14. Bosko2006-07-13 15:54:11

    There is more to the parametrized find story than just search.

    Admist my initial confusion, I've commented on this in DHH's blog (lost somewhere in the hundreds of comments), but I'll repeat the scenario I used there: a Site object which hasmany Pages. You can CRUD both, but you still need to be able to pull up "all Pages which belong to Site Foo". Now you can do this the "normal" way which is from the SitesController's "show" action, where you display Site properties, including all Pages belonging to it. But, what if you want SitesController's "show" action to only display the basic Site properties, and have a link to optionally display all Pages belonging to said Site, via let's say an Ajax call. Then you would probably shove a "listpages" action inside the SitesController, or a "find" action inside the PagesController which accepts a site_id as one of its argument(s). Suppose you go with the latter. Then you need a parametrized find in your controller, since creating a Search model on-the-fly simply does not make sense (it's wasteful and doesn't make sense for this problem domain).

    Basically what I'm getting to here is that find, in general, is an edge case. Search applications themselves can very well adopt the Search-CRUDable-entity approach, and that's great, but it doesn't always eliminate the need for being able to introduce more than just the basic CRUD actions, which accepts one or more parameters, in order to be able to accomodate UI requirements.

    I see the CRUD approach really - and DHH pointed this out in his talk - as a design paradigm, not a completely 100%-adoptable end-goal. You start with CRUD, it forces you to think about your domain, possibly leading to simplification, then you hit the front-end/view code and iterate within reason.

  15. Rich Collins2006-07-17 12:41:51

    Josh,

    I was wondering the same thing. We did take the plunge but only ended up getting our feet wet. I suppose a SearchPage model would implement fully granular CRUD, but I still have a hard time not prematurely optimizing at least a little bit.

  16. Bill Eisenhauer2006-07-29 07:22:30

    The discussion is fading on this topic, but I've just now discovered it.

    Its funny, I can see the trail DHH has been on. Over at XML.com, there are articles about creating URIs that correspond to resources and there are articles talking about responding based upon accept headers. There are even sample Python implementations.

    Further, there is a book called, "Ajax Patterns and Best Practices" which details something called the Permutations Pattern. It covers both topics. But as I move back to the topic at hand, he does briefly address searches.

    So consider these:

    http://mydomain.com/search/books http://mydomain.com/search/users

    Where they respectively are searches for a book or books or a user or users. What's not pictured is the query string that would be used to define the search.

    He terms these action URLs that are meant to generate a result set. PUT and DELETE should return errors for these URLs.

    Lastly, the format may also be requested. As in, format=xml may be appended at the end. This enables you to override the accept header with exactly what you want.

  17. Thomas Shelton2006-08-15 08:34:43

    I have been following the CRUD discussion since railsconf. Given a recent project, my interest in the search functionality has increased. After reading the post and comments above, it occurred to me that searching might be easier if considered a two-step process. Similar to preparing a SQL block for execution when calling multiple times. So you might POST to setup a search which could return a URL and then GET the results.

    Just a thought. I haven't implemented anything yet but it does appear to be a little cleaner and may also allow for some optimization.

Sorry, comments for this article are closed.