The illustrious Chris Eppstein recently tweeted:
If some code should be refactored, stop what you are doing and refactor it.
I was about to respond, but realized I had more to say than would fit in a tweet. (Waiting for someone to fix that problem!) (Then I got distracted and didn't finish this article for a few days, oops.)
Now, Chris is really smart and probably doesn't mean exactly what he said, but it's easy to misinterpret his advice. I'll agree with him that you should be eager to refactor code when you discover the need. Don't let that technical debt accrue interest longer than necessary! However, you shouldn't be reckless about it.
Please keep this in mind:
Don't do a refactoring in the middle of making another change.
If you are working on a story and in the middle of making a code change when you discover the need to refactor something, make a note of it (I usually create a chore in Pivotal Tracker) and forget about it until you're done with the change in progress. After you complete the change, come back and do the refactoring. Make sure all tests are green before starting to refactor, and are green when done. The refactoring change should be a separate commit in git (separate checkin in SVN, etc).
OK, I'm pragmatic and realize that that approach doesn't work all the time, but it's a good ideal to shoot for. I considered discussing cases where it would be OK to refactor something in the middle of another change, but on second thought I think I'll leave that be for now. You'll learn that for yourself better than by following someone else's advice on the subject.