Agile git and the story branch pattern

— December 18, 2008 at 23:39 PST


I've been using git for source code management for over a year now and I'm totally hooked. I won't rave about all the usual reasons WhyGitIsBetterThanX since it's been done already. Instead, I'm going to share how I use git for easy agile development.

The basic idea is to never do anything in the master branch except use it to move changes between the remote repo and local branches. Keeping master clean takes very little effort and will save your bacon when you get into trouble. The example I'll use here is working on a story to render title text in a bold style on a page.

1. Locate story "titles are in bold text" in Pivotal Tracker and click Start.

What do you mean you're not using Pivotal Tracker?

2. Get going with a clean start.

$ git checkout master
$ git pull
$ rake db:migrate   # if there are new migrations
$ rake              # if you don't have a CI server

3. Make a branch to work in.

$ git checkout -b bold_titles

4. Test-drive the story.

Since you are committing locally, you can commit often. Sometimes I like to commit every time I get a test to pass, or after crucial steps in a refactoring.

$ git commit -am "Don't bold empty titles"

We should all know the advantages of frequent commits: when an experiment fails, it's easy to get back to a previous state where things worked; or when refactoring, you can see how things were before you started changing things around. With git you can also do things like cherry-pick commits, so if you want to grab a single test and its implementation and extract it as a patch or whatever, you can do that.

When you are done...

$ rake
$ git commit -am "Render titles with bold style"

5. Merge with master

When you're done with the story, you need to integrate your changes with the master branch. You'll have to fetch and merge the changes from origin/master into your branch.

a. Review changes

I like to review changes to make sure I didn't leave any debugging cruft in, and as a final check on code quality.

$ git diff master

b. Merge changes

$ git checkout master
$ git pull
$ git checkout bold_titles
$ git rebase master

The four commands above may seem a bit convoluted, but all you're really doing is merging updates from the remote repo to your story branch by way of the master branch. Now, if you have any merge conflicts, they are all in the story branch where you can work on resolving them in your normal work environment without disturbing the master branch.

If you took enough time resolving merge conflicts that someone else checked in changes to the remote repo, repeat this step until you're fully merged locally.

Also, notice I used rebase instead of merge to integrate the master branch changes, but that choice is up to you. Whether you use rebase or merge is a matter of convention for your project. Doing a rebase will replay your changes on top of the changes you just merged with. That should have a net effect of zero on the final product but has the advantage of keeping all your changes contiguous in the history, and won't generate a separate merge commit when you merge back to the master branch since the changes come in as a fast-forward.

You can also rebase with the -i (or --interactive) option to squash several (or all) of your commits into a single commit, or to organize your changes as a reduced number of commits. Frequent commits are great to manage your own work and give many checkpoints to roll back to if needed. But adding dozens of commits to the project for a simple feature can be overwhelming for others on your team, and can make it difficult to navigate or understand the project history. For example:

$ git rebase -i master

That will show you an $EDITOR session like this:

# Rebasing 1234abc..3333ccc onto 1234abc
#
# Commands:
#  pick = use commit
#  edit = use commit, but stop for amending
#  squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
pick 1111aaa Use <strong> tag to bold titles
pick 2222bbb Don't render empty titles as bold
pick 3333ccc Use font-weight property on h1.title instead of <strong>

The commits are listed in chronological order, oldest on top. If you want to combine several commits into one, edit the word pick at the start of the line to say squash, and that commit will be merged with the one above it. Since you are seeing all your commits and their messages, you can probably say something useful about the whole set as you package it for pushing to the shared repo. Sometimes you'll want to squash down to just one commit, others you may want to structure the package as a small number of related commits, say one commit for your feature addition, and another for a refactoring your feature relies on.

Important: Rebasing rewrites history and should never be done to reorder changes you have previously pushed to another repo or shared publicly.

c. Be green

Once done merging, make sure you're still green.

$ rake

d. Merge early, merge often

If the work on your branch takes long enough, you might want to perform a merge several times. As other people push to origin/master, you can pull those changes and integrate them into your branch incrementally, rather than waiting until the end when you're done. Whether to do this is a judgement call, and would depend on factors like the nature of the work you're doing the the kind of changes you'd be integrating. But one of the more valuable practices of agile development is continuous integration, and spending too much time accumulating change on a branch that is cut off from the rest of the development stream can eventually make for a difficult merge at the end of your story.

6. Push

Once you have integrated your changes with those from master, you can merge them back to the master branch and from there to the remote repo.

$ git checkout master
$ git merge bold_titles
$ git push

7. Cleanup

$ git branch -d bold_titles

Delete the story branch, click Finish on the story in Pivotal Tracker, then get up and stretch and get some water.

Caution

Probably the biggest gotcha particular to Rails development on more than one branch is dealing with database migrations. It's easy to get confused between the state of the database and which migrations are in what branch. I'll take one of two approaches, depending on how complicated things are. One is to rollback the migration on the current branch before switching to another branch that doesn't have that migration yet, then run pending migrations on the new branch after switching. But if you're going to be switching around branches with different migrations a lot, the best thing is to integrate the migration changes so they exist on both branches and it ceases to be an issue.

Discussion

Okay that's a lot of stuff going on. What's the point? Well, the point is dealing with the unintended. (I was going to say "unexpected", but if you're developing software or working on a team, you should expect crazy things to occur occasionally.) There is obviously overhead in managing multiple branches, and you may wonder what you're getting for the cost of that overhead. There are a couple things that come to mind that I'll mention here, but I'd like to hear more in the comments from people about what situations they deal with this way (or other ways).

The first thing this approach helps with is being able to make frequent commits without breaking the build or disrupting the work of teammates. Working with a centralized SCM system like Subversion can force you to choose between checkpointing your work and keeping the state of the project consistent. I know agile developers who swear that making frequent commits to the shared repo is the way to go, and just deal with having to keep the build green and functional on every commit. But I find that approach too constraining, and it can impose an uncomfortable style on how I do my work. I find being able to do frequent commits without worrying about disrupting my teammates' work with half-finished features lets me concentrate on getting things done, instead of thinking about how big a change I can make without messing up someone else's work. And I also don't have to worry about the converse either. Don't you hate pulling down changes that include a migration and wondering if the relevant model changes to let that migration do the right thing have been made yet?

But why the separate story branches? Isn't it just as convenient to do all your work in your local master branch and do your local commits there? Up to a point, sure. But keeping a separate branch for work means that you always have a good baseline around for reference, it lets you rebase to manage your changesets easily, and it makes dealing with merge conflicts a bit more sane.

And if you have to share or backup the state of your work in progress, it's very easy to push your local branch to a branch in the remote repo without disrupting anything.

There's also the advantage that it's easier to deal with being interrupted. Say you're working on a story that's going to take you all day and someone finds a nasty bug in production. Time to drop everything and get it fixed right away! If you have your work in progress on a branch, it's simple to do a checkpoint commit, switch back to master, create a new branch for the bugfix, fix the bug, push the fix, then get right back to where you were before.

Push

I've been using the story branch pattern for development for most of the last year, and have found it useful, convenient and a lifesaver when things get weird. I've also watched teammates that do their work in the master branch, and it is more work for them to deal with issues and avoid making unintended messes. Like all useful practices it takes a little bit of effort up front, but it saves a lot more effort when it really matters.

25 commentsagile, git

Comments
  1. Johannes2008-12-19 00:20:33

    Ah, good story. Would you mind to specifiy the directories your in, when you issue the commands?

  2. Kev Mears2008-12-19 03:19:33

    This sounds like a great improvement on the way we are currently doing things. We seem to have lots of little commits and merge commits, when we could really be making the commits work for us. Git is the first VCS that I've really got on with.

    Cheers for the article

    (and the link to Pivotal Tracker)

  3. Daniel2008-12-19 03:46:08

    Nice post. I'm just getting started with git and this looks a lot like the workflow I've been heading for.

    But as this shouldn't be a WhyGitIsBetterThanX-post, I think you could have rephrased the paragraph about centralized SCM/SVN a little. It's possible to use a workflow like this in SVN too - you can use a semi-private story branches in the main repository. That wouldn't disturb someone else and would allow frequent commits without breaking the build. (That said, I think it's both easier and more efficient in git.)

  4. Chris Redinger2008-12-19 06:36:09

    This is very nearly our development process that is working very well for us. The major difference is that when a developer is done with a task, he pushes that story branch up to the github repo, where it can be tested and "approved." Once approved, we merge it into master, where it is automatically deployed the next morning and the remote branch is deleted.

    IMHO, this has been a fantastic way of developing.

  5. Ryan Bates2008-12-19 09:28:16

    I'm usually a solo developer, but even so the advantages of making one branch per story are well worth it. I often have many branches at once because I'm waiting on something for each story. Whether it needs to be designed, checked, approved, etc. the ability to switch between branches has made my life much easier.

    That said, if I expect something to only take one or two commits I don't make a branch. Typos, small bug fixes, etc. I just commit right onto master for convenience.

    Thanks for the -i tip to squeeze the commits. I'll try that out.

  6. Yossef2008-12-19 11:10:56

    Steps 5b and 6 could be made a little easier. We have scripts for these common groups of commands. ReinH wrote a little something about it.

  7. Adam Keys2008-12-19 11:49:04

    I'm a big fan of this approach. Thanks for including the Git workflow to support this process; its sometimes hard to explain to those who only have a working understanding of Git.

  8. Nick Zadrozny2008-12-19 12:43:48

    Great writeup, Josh. I've been using the same set of git commands for quite some time as well. It's definitely nice being able to freely commit without worrying about breaking anyone else's build.

    In fact, I wrote two shell functions to handle rebasing and pushing. Maybe someone here will find them useful: http://gist.github.com/38105

  9. Josh Susser2008-12-19 13:45:49

    Johannes: None of these commands care what directory you are in, as long as you are somewhere in your git project. But I usually work at the root of my project, since the paths to files make more sense that way.

    Ryan: I'll also fudge sometimes and work on master if it's just a really little change, but now and then I get caught doing that. It's all about cost vs value. This has very little cost, so I don't mind paying it even for small changes.

    Yossef and Nick: Thanks for those links. I know some people like to reduce typing as much as possible, but I like to keep that level of stuff manual, since those scripts don't deal well with exceptions.

  10. Johannes2008-12-19 15:48:36

    Beeing in one directoy of a project works for nearly every scm. I meant, in which project you are.

    I'm working with mercurial - so I liked this "better than X" a lot.. :-)

  11. cogmios2008-12-19 19:19:19

    nice but still very primitive compared to ClearCase 7.1 UCM

  12. Don2008-12-19 22:09:42

    My favorite part of this workflow is the ability to sanity check what you're getting when you update your code from the repository. Most of my work is as a freelancer augmenting a larger team. Before I bother seeing if my new commit works with the latest code, I can see if the latest code needs a migration or a gem, and actually passes all its tests. So helpful when you're a remote developer.

  13. Steve C2008-12-20 09:18:18

    Thanks for the thorough post on Git, I'm getting a good idea of what people like about it.

    I would just point out that this part:

    "I know agile developers who swear that making frequent commits to the shared repo is the way to go, and just deal with having to keep the build green and functional on every commit. But I find that approach too constraining, and it can impose an uncomfortable style on how I do my work."

    is, rather than a different take on Agile, probably better read as discomfort with a practice widely understood to be an underpinning of Agile as actually practiced: continuous integration.

    The more I read about usages of Git the more I understand that what Git proponents really want is to go back to personal branches. "Back" implies a value judgment - but I don't discount the possibility that Agile proponents got it wrong.

    It's probably also true that for someone who values continuous integration - to be specific: the practice of slamming the team's code together early and often and seeing whether it works - Git looks like a bunch of extra steps, making someone like me wonder "why"?

    So here's my challenge to Git proponents: tell me Git is a bad idea for people who are happy with continuous integration. Or tell me what I'm missing that still makes it an advantage over, say, svn.

  14. Ted Young2008-12-20 11:09:04

    Not having used git, I appreciate the step-by-step narrative. However, I still don't "get" how git is helping here?

    Is it the frequent commits so you can rollback at any point? I get this for free (with a great UI) in IntelliJ IDEA's Local History. Once I have tests passing, I submit into the trunk to make sure that other folks won't be diverging too much from _my_ changes.

    I can also make separate changelists in Perforce, one for each "story", though they're all in the same "project", so you can get into trouble with one changelist interfering with another. If git isolates these different work areas, then that's a plus. Though I personally prefer not to have multiple changelists "in flight", it's nice to have isolation in case you need to do a bug fix before your story is ready to commit.

    Is it the update/pulling from the master? I can do that in Perforce from within IDEA, so again I'm not getting the advantage.

    With Perforce + IDEA, I seem to be able to do almost everything you're doing (except for the isolation of changelists), so I feel like I'm missing something more significant.

  15. Josh Susser2008-12-20 15:16:47

    Steve C: I do like continuous integration, and as I said I don't like accumulating too much change on a branch. Usually that only goes one way, but it's fine to push a checkpoint from your branch up to origin/master anytime you feel things are settled enough. Usually my stories don't take long enough that I have to worry about that.

    Ted: As I said at the start of the article, my goal wasn't to sell anyone on git. If you're happy with Perforce, that's great. The point was to describe a useful workflow for people who use git already.

  16. Brian2008-12-21 06:10:04

    Very timely article for me. Thanks!

  17. vicaya2008-12-22 10:37:06

    The merge/rebase steps can be simplified with:

    [working branch]$ git fetch [working branch]$ git rebase origin/master # or whatever upstream branch

    checkout involves a lot of unnecessary work space file changes if someone committed a huge series of changes upstream.

  18. vicaya2008-12-22 10:38:10

    reformatted:

     [working branch]$ git fetch
     [working branch]$ git rebase origin/master # or whatever upstream branch
    
  19. Eric Mill2008-12-22 19:03:46

    Awesome article Josh, thank you. I've been adjusting to git more gradually than some of my peers, and this advanced my education a couple steps towards the glorious future. I'm going to change the way I'm working in one of my client projects to use this model right away.

  20. Walter McGinnis2008-12-26 21:22:03

    On the Kete project (http://kete.net.nz/) we use the extension to the pattern that Chris Redinger outlines (with a few small differences because of the nature of our project) so that we can collaborate on or review branches before we merge them to master.

    We also use a naming convention for our branches so that others can quickly figure out what the work is:

    (bugfix, enhancement, or refinement) underscore (ticket number in lighthouse) underscore (2 or 3 words describing work)

    It's been really handy.

  21. Chad Woolley2009-01-02 00:46:52

    Very useful writeup Josh, thanks a lot.

    -- Chad

  22. Patrick Berkeley2009-01-03 20:10:16

    Thanks for this article on your everyday workflow using git; it's the most clear, complete guide I've found. Also, it answers a lot of the questions I've had about how to use git in collaboration with others (so far I've only used git as a solo developer).

  23. Chris Bailey2009-01-09 09:54:38

    Josh, good writeup, that's pretty much our exact workflow at DealBase, including use the hack & ship scripts from Rein, slightly tweaked.

    We have one twist. We too use Pivotal Tracker. I wrote a little GitHub post-receive hook for Tracker, so that our checkin message gets added to the story it was for automatically. All you have to do is add [Story######] to your commit message and it gets added as soon as you push to GitHub.

    To help with this, what I've found I do myself, is that I now name my branches with "Story123456" kind of suffix, so, in your example, it'd be "boldtitlesStory123456", where "123456" is replaced of course with the actual story ID from Tracker. That way when I'm doing my commits, I have the story ID handy, and don't have to go copy/paste it out of Tracker. Just one of those little things that makes the workflow a tiny bit faster.

    (Note, you can also put a state change into the commit message to update the state/status in Tracker, such as [Story123456 state:finished]).

  24. Harry Seldon2009-01-31 17:17:03

    Thanks Josh, that is a very detailed article about the development workflow. When dealing with Open Source Software (OSS) you have some extra steps to separate your code in branches for your production and for the OSS development. Here is an attempt at showing the workflow with git for an OSS dev (namely Typo), including plugins and submodules:
    http://harryseldon.thinkosphere.com/2009/01/14/git-and-rails-a-detailed-tutorial-including-plugins-submodules-development-and-production

  25. Mark2009-02-25 13:48:05

    Seems painful to me. But then again we use Accurev.

Sorry, comments for this article are closed.