The Return of the Cucumber

It’s been about ten weeks since I wrote about Cucumber the first time. Since then, I’ve continued to use Cucumber and now seemed like a good time to update some thoughts on how and why it seems to be working for us.

The big headline, of course, is that I’m still using it after ten weeks. I’m pretty quick to abandon tools that aren’t pulling their weight, so just the fact that Cucumber is still in the toolbox means that despite the time that it takes to write Cucumber tests and step definitions, I’m finding the process of writing the tests and the tests themselves to be valuable.

Where I’m using it

Right now, I’ve been using Cucumber primarily on two different projects.

• An internal project that has a well-defined testing process, and 100% coverage for unit tests on models and for functional tests on controllers.
• A client project that we took over from the original coders, who wrote no tests at all. All new features come in with tests, but we haven’t

Cucumber plays out a little differently in each case.

How I’m Using it

I struggled a bit with how to integrate Cucumber into the testing process, and eventually came to the following rule of thumb:

Just as all lines of code should be in response to a failing unit or functional test, all new unit or functional tests should be in response to a failing line of Cucumber.

The process seems to go something like this:

1. In conjunction with a stakeholder of some kind, write the cucumber tests for the feature, usually, this will be multiple scenarios. At this stage, don’t worry about setting up data, and I think you’re better off focusing on making the test as human-readable as possible, but see below for some thoughts in that direction.
2. Start writing step definitions for the Cucumber tests, step by step. In many cases, you’ll find yourself adding some steps to create data (especially to the background step of the feature). Especially on a mature project, you’ll often be able to write several steps without needing new code in the app itself — “Given I am a logged in user”, “When I visit the project page”… in essence, the setup.
3. In the fullness of time, you’ll come to the meat of the feature, and you won’t be able to make the Cucumber line pass without writing application code. At this point, you drop into a normal TDD process, with the goal of moving the application forward enough to make this line of Cucumber succeed.
4. What happens here depends on a little bit on personal taste and the project team. For the internal project, which has a goal of 100% functional test coverage, I do write functional tests even if they overlap the Cucumber test strongly. On the legacy project, I do sometimes drop do the model/unit tests from the Cucumber test if there doesn’t seem to be a substantial difference between the acceptance and functional test.
5. When I think the regular TDD process has gotten me enough code to pass the Cucumber line, then I run Cucumber again to verify and move on to the next line.
6. This process has a lot going for it. The TDD part is always focused on a specific, definable goal — the next line of Cucumber test. The cycles of acceptance and regular testing are short enough that the two feed into each other (sometimes the Cucumber tests change in response to the shape of the code, and sometimes vice versa), and they work together to keep the code simple, complete, and covered.

Cucumber vs. Functional testing

The big question is to what extent the Cucumber tests duplicate or replace functional tests. At the moment, my answer is somewhat, but not completely. I see the Cucumber tests as being the way to go for testing a work flow of more than one one action, but I still find it a little easier to set up data in the functional test framework. That said, I don’t think that adding the functional tests has cost me that much more time, and there are still enough gaps in Cucumber/Webrat (most notably Ajax) that I’m reluctant to give up controller tests entirely. I expect my thinking on this will change as we have projects that had Cucumber from the very beginning.

General vs. Specific

I’m having a stylistic issue with the Cucumber tests between putting the specifics in the Cucumber or in the step definition. This came up most sharply in the legacy project. We were working on a feature that made a distinction between new users and old users. My first pass at the Cucumber test started off like this:

Given a user named "Noel Rappin" who has been a member for 5 months

With a step definition that was something like this:

Given /^A user named "(.*)" who has been a member for (.*) months$/ do |name, mo|
  @user = Factory.create(:user, :name => name,
      :created_at => mo.months.ago)
end

The idea was that “5 months” was just a stand in for “an old user”, but the client, quite understandably, wanted to know what the deal was with five months — the specificity of the rule seemed to imply that it was important functionally. Eventually, I changed the rule to be more like this:

Given /^A user who is not a new member$/ do
  @user = Factory.create(:user, :created_at => 5.months.ago)
end

Once I saw this issue, I started seeing it everywhere. The second form has the advantage of being more general and more requirements like, and the disadvantage of hiding a lot of magic detail in the step definition. The original version has the advantage of putting specific data front and center and the disadvantage of not making it clear which details are important.

I’m not aware that there’s a consensus on the style issue. My tendency, right now, is to favor the general form on the theory that it’s a good thing to make the Cucumber test as readable as possible and let the step definitions sort it out. On the other hand, this seems like a good way to get really messed up by having the step definition not quite match what you expect, and have that be hidden.