Article

A Real Testing Example

Adam Carnagey

Published on: July 17th, 2009

This post was previously on the Pathfinder Software site. Pathfinder Software changed its name to Orthogonal in 2016. Read more.

As sort-of promised in last week’s post, I’m going to work through a real-world test example, with an eye toward explaining how and why I tested the way I did. Hopefully, I’ll be able to do this at blog-post length. If not, well, there’s always next week.

This site, which was a legacy rescue, allows users to send messages to each other within the site without having to give away their other contact information. The problem is that nefarious spammer types were creating logins and immediately sending messages to large numbers of the user population, irritating them. After some deliberation, the client decided on a rate-limiting strategy, where a member could only send a certain number of messages in a day, and a new member could send even fewer messages a day. Messages above that point would require administrative action to unblock the user’s privileges.

Self-promotion alert: More details about Rails testing can be found at Rails Test Prescriptions, there’s a free getting started tutorial which contains an extensive section on Cucumber, and a nearly 300 pages and counting full book for $9. Thanks. Also, follow @railsrx on Twitter for testing tips and updates.

I started with Cucumber tests — the first two are representative of the initial batch. Everything you see here is actual code, only slightly tweaked to anonymize details of the site. But I’m trying not to present simplified “example” code. That said, this is the final state of the code, there were some intermediate steps and false trails that I’m sparing you from having to read about. I’m also focusing on the tests here, rather than the resulting application code.

Background:
  Given the user database is cleared
  Given a group of recipients.

Scenario: Normal User behavior
  Given I am a user who is not a new member
  When I try to send 5 messages in a day
  Then 5 messages are sent
  And the administrator does not get an email

Scenario: Excessive User behavior
  Given I am a user who is not a new member
  When I try to send 6 messages in a day
  Then 5 messages are sent
  And I am blocked from sending further messages
  And the administrator gets an email

This test uses an implicit style, so a lot of the details are in the step definitions. I start by writing the step definitions one by one until I get to one that I can’t make work without writing new code. As it happens, since not sending email is the current state of the system, the entire first scenario should work without any new code. It’s very important to include that scenario, though, to ensure that the new changes don’t break the basic behavior.

Here’s what the steps look like — the user who is not a new member defers to the boilerplate RESTful Authentication step for logging in:

Given /^I am a user who is not a new member$/ do
  @user = Factory.create(:user, :created_at => 5.months.ago)
    Given "I am logged in"
end

Given /^I am logged in$/ do
  visit "/login"
  fill_in("email", :with => @user.email)
  fill_in("password", :with => @user.password)
  click_button("Sign in")
end

Next up, sending messages. Since the point of Cucumber is to treat the application as a black box, the step definition uses Webrat to simulate number of posts to the create message RESTful action. (The @recipient is created in the background action, which I didn’t show here because it’s not very interesting.)

When /^I try to send (.*) message(.*) in a day$/ do |count, plural|
  count.to_i.times do
    visit(messages_path, :post, {:recipient => @recipient.id,
             :message => Factory.attributes_for(:message,
                                :sender => @user)})
  end
end

So this step definition allows you to match “send 3 messages” and “send 1 message” by adding that little group at the end of the message, that group needs to have an associated variable in the block argument list, but it’s just ignored. Somebody with better offhand regular expression skills could easily make it so the end of the expression only matches “message” or “messages”, but I can live with it being a little overly matchy for now.

Closing out the normal case, the step definition to test emails sent uses the excellent
email_spec plugin, which creates RSpec matchers and Cucumber step definitions for email testing. This one checks an all_emails method, filters out any methods sent to the administrator, and makes sure the right number exist.

Then /^(.*) message(.*) (is|are) sent$/ do |count, plural, verb|
  emails_out = all_emails.select do |e|
    !e.to.include?("[email protected]")
  end
  assert_equal(count.to_i, emails_out.size)
end

The administrator email step definition uses the email_spec steps directly — and yes, I could have included those steps explicitly in the Cucumber scenario. I chose not to, on the perhaps dubious grounds that I wanted to keep explicit string literals out of the scenario. But it doesn’t make that much difference either way.

Then /^the administrator (gets|does not get) an email$/ do |status|
  if status == "gets"
    Then '"[email protected]" should receive an email'
  else
    Then '"[email protected]" should not receive an email'
  end
end

Everything here passes for the normal case. So far, my main note is that all the step definitions are really simple — it’s almost impossible to misinterpret them.

At this point, I move on to the blocking case. There’s one more step definition to write. In actuality, the “then 5 messages are sent” step will fail, since the blocking isn’t in the code. So, I would normally jump to writing regular tests at that point, but since I’m here, I’ll present the last step definition.

Cucumber is a black box, therefore in order to detect if the user is blocked, we need to find some place in the application that will display it. After their initial notification, the user doesn’t see anything confirming their block status. But the administrator does, via an admin screen that actually hasn’t been written yet. So, the cucumber tests logs the user out, logs an admin in, and checks the admin page for a row associated with the user.

Then /^I am blocked from sending further messages$/ do
  @sender = @user
  visit "/logout"
  Given "I am a logged in administrator"
  @admin = @user
  visit path_to("the admin messaging page")
  assert_select("tr#?", dom_id(@sender, :blocked_row), :count => 1)
end

This step definition is a bit more complicated than the others, and it’s also taking it slightly on faith that the row with the correct DOM id will actually have the information the admin needs. (The tradeoff in view testing, as always is faith an flexibility vs. certainty and brittleness…)

Okay, there are genuinely failing Cucumber steps, it’s time to make them pass.

A couple of things to note:

After going back and forth on this some, I’ve decided that Cucumber tests don’t really replace controller tests, except in maybe the simplest cases. My rationale is that Cucumber and controller tests look at different error conditions, and that Cucumber tests are really (partially) replacing the hand-testing that I would be doing to verify that the feature is working. Cucumber is the 30,000 foot view of the application, I still feel the need to test from the ground.
This app was a legacy with effectively no tests, leading me to have to write some extra tests to cover the normal, non-blocking case. If the whole app was TDD, then that functionality would already have tests, but since I’m working on the new features, I need to go back and cover the base functionality.

The basic ideas is write the tests in the class where the code is going to go. I can guess that code will need to go in the MessageController, the Message model, and the User model. In fact, I’m going to need a new field in the user model, a string representing message sending status. (It’s a string and not a boolean, because I know from a later requirement that there will be more than two states). I actually will wait to write the migration, though, until a test compels it.

My preference is to start with the controller tests — it’s the easiest way into the system for me.

Here’s the batch of tests I wrote to cover the normal sending a message and not getting blocked functionality. These tests will probably look weird to just about everybody since I wrote them using a combination of Shoulda, Zebra, and Matchy. Zebra gives nice one-line tests, and Matchy gives a sort-of RSpec syntax that I sometimes like.

context "POST CREATE" do
  setup do
    ActionMailer::Base.deliveries.clear
    @recipient = Factory.create(:user)
    @user = login!
  end

  context "with a clean user" do
    setup do
      post :create, :recipient => @recipient.id,
          :message => Factory.attributes_for(:message,
                :sender_id => @user.id)
    end

    expect { @user.messages_sent.size.should == 1 }
    expect { @user.spam_message_count.should == 1 }
    expect { assigns(:message).should_not be_new_record }
    expect { @user.should_not be_message_sending_blocked }
    expect { assert_no_email_to_administrator }

    should "create and send" do
      assert_sent_email do |email|
        email.to.first == @recipient.email &&
        email.from.first == "[email protected]"
      end
    end

  end

### Outer context continues

What to say about these tests…

In controller tests, I organize contexts by action. The outer context setup clears the email and creates a couple of users — the login! method creates a factory user and simulates a login.
The internal context posts the message
The various expect blocks each resolve into a test that passes if the block returns true. The first one checks that a message has actually gone into the database (messages_sent is an association on user). The spam_message_count is what is used to determine if a user is blocked — that’s not going to pass yet, because the concept of a spam count is new to the app. The third tests that the message assigned in the controller is actually saved. The fourth checks the message_sending_blocked? method of a user, which is also going to need to be written. The last calls a helper method to determine if an email is sent to the administrator, which should only happen if the user is blocked.

Many of these tests pass as is. The ones that don’t are dependent on the new user features. Which means we need user tests.

I realize that some of you are starting to think that having all these tests is ridiculous overkill — I mean, you’ve got your Cucumber tests, you’ve got your controller tests, and now model tests. That’s, like, triple the work, isn’t it?

No, I don’t think it is. My rationale for doing all this testing goes something like this:

In my head — which is, admittedly a strange place to be — these are all different tests covering different aspects of the same feature. They could all fail independently of each other, at least in theory. Each of them is exercising a different part of the code, and crucially, each of them is responsible for potentially testing error conditions at in their own space.
It’s not that much extra time — I’m presenting the code in larger chunks than I actually wrote it (can you imagine how long this would be otherwise?). In the rhythm of testing, it’s a very short piece of test, followed by a very short bit of code, and the amount of tests grows over time.

Anyway, the user tests cover the case where a user has four messages, and then the transition to five messages. The setup is pretty similar (another reason why the extra tests don’t take as long to write as you might think)

context "rate limiting" do

  setup do
    ActionMailer::Base.deliveries.clear
    Timecop.freeze(Date.today)
    @user = Factory.create(:user, :created_at => 1.month.ago)
    4.times do |i|
      Message.new_from_params(
          Factory.attributes_for(:message,
              :created_at => i.hours.ago, :body => "Message #{i + 1}"),
          @user, Factory.create(:user)).save!
    end
  end

  should "correctly count messages" do
    @user.spam_message_count.should == 4
    @user.update_message_block_status.should be_nil
    @user.should be_able_to_send_messages
    assert_number_of_emails_to_administrator(0)
  end

  should "move user to blocked mode after fifth message" do
    Message.new_from_params(Factory.attributes_for(:message,
        :created_at => 10.minutes.ago), @user,
        Factory.create(:user)).save!
    @user.update_message_block_status.should be("blocked")
    @user.should_not be_able_to_send_messages
    assert_number_of_emails_to_administrator(1)
  end

## Outer context continues

Most of this should be clear given all my blabbering so far (and I have no real clear reason for abandoning Zebra in this section of tests). One thing is that the method update_message_block_status is actually responsible for changing the user status after a message is sent.

This is getting way long, so back next with with how this played out with new requirements, and a kind of interesting bug.

Quibbles with my testing style or process should go in the comments.