Thursday, December 21, 2006

Writing a test approach

http://discuss.joelonsoftware.com/default.asp?joel.3.429670.3

If this is the first time someone has asked you to produce one, it's kind of an odd exercise.

First, recognize that you have some pretty significant prerequisites, and second that you have to consider multiple audiences. But what it all comes down to is making sure that the application does what it's supposed to do.

The prerequisite comes from needing a good definition of "what it's supposed to do." I wish this were as easy as it sounds. Look at how much is written about whether/how to write specifications to see what you're up against.

Once you have a good spec, you have two questions to answer. It's really the same question answered for two audiences, but the answers can be pretty different: How do you ensure that it does the right thing; and how do you demonstrate to your client that it does the right thing?

Answering the first question will involve unit testing, regression testing (for upgrades), creation and execution of test scripts, etc. It's possible a large part of the execution will actually be done by the developers.

Answering the second question -- satisfying the client -- requires that you talk in terms of: user interaction, use cases, end-to-end scenarios, scalability/performance/load testing, support and training needs, etc.

Imagine you are presenting a sales pitch to a client, and you need to describe how you will demonstrate the value and correctness of your application. Write a narrative proposal, eg:

Working with end users and subject matter experts (name names if possible), we will define use cases that describe the expected functionality of the application. This process should take X hours of the users' time over the course of Y days.

These use cases will be executed and verified by someone, using volume of data extracted from source. Developers will also create automated scripts to be executed on toolname, simulating how_many concurrent users executing how_many transactions per second for how_long.

You're not trying to actually write the test scripts yet. Start out with the higher-level description of what types of testing you expect to do, and who is responsible for writing and executing each type. Theoretically you may not know the performance figures in that second paragraph, but I find it's better to put some numbers in front of the stakeholders and let them disagree.

No comments: