Incantations in High Elvish?

Great blog post about exploratory testing by James Marcus Bach, and why some people Just Don’t Get It.

It’s difficult for them because Factory School people, by the force of their creed, seek to minimize the role of humanness in any technical activity. They are radical mechanizers. They are looking for algorithms instead of heuristics. They want to focus on artifacts, not thoughts or feelings or activities. They need to deny the role and value of tacit knowledge and skill. Their theory of learning was state of the art in the 18th century: memorization and mimicry. Then, when they encounter ET, they look from something to memorize or mimic, and find nothing.

Those of us who study ET, when we try to share it, talk a lot about cognitive science, epistemology, and modern learning theory. We talk about the importance of practice. This sounds to the Factory Schoolers like incomprehensible new agey incantations in High Elvish. They suspect we are being deliberately obscure just to keep our clients confused and intimidated.

As I’ve explained in previous blog posts, I’ve always taken an exploratory approach to testing, even if what I did wasn’t formally identified as such. Trying to force testing into a purely mechanical script-based approach not only sucks all the fun out of testing, risking disillusionment and burnout, but makes the actual testing less effective.

And while we’re on the subject of old-school techniques, are these guys for real? “Unlike a traditional development process, ours establishes all the system’s requirements before a line of code is written“. Seriously, folks, does anyone still try to develop software that way in 2011? Sounds like a perfect way to implement what the client thought they wanted eighteen months ago.

Remember that old cartoon of the swing hanging from the branch of the tree?

This entry was posted in Testing & Software. Bookmark the permalink.

9 Responses to Incantations in High Elvish?

  1. PaulE says:

    Both “a purely mechanical script-based approach” to testing and establishing “all the system’s requirements before a line of code is written” are exercises in posterior covering in case the client screams “it doesn’t work” at some later point.

  2. Tim Hall says:

    Except in practice it results in a system that doesn’t actually do what the client wanted.

    I’ve spent enough time writing detailed test cases against a requirements specification that turned out have been based on a great many wrong assumptions. I suppose I did manage to cover my own backside even when it failed user acceptance testing.

  3. PaulE says:

    I thought I was agreeing with you on this. You are absolutely right about the wrong assumptions producing the wrong system. A clear case of “Garbage In Garbage Out”. The earlier in the process such a mistake is made, the worse the impact upon the project. All I was pointing out was how both methods described create large amounts of paperwork that probably wont be read again unless something is wrong. I suppose it is a bit cynical of me to then jump to the conclusion that that is the primary purpose of creating these formal documents. But it certainly feels that way a lot of the time.

    Also, by a “purely mechanical script-based approach” I assumed you meant a mechanistic approach to the selection of scenarios, e.g. going through the options sequentially. It does produce plenty of paperwork to show (development & testing can be a real tree killer – maybe that incantation should be in Old Entish !). Isn’t your argument for exploratory testing essentially saying that intelligently targeted tests are more efficient at finding bugs than a “brute force” method of going through all the options (not that that is ever actually possible for many systems)?

  4. Tim Hall says:

    Yes, you were agreeing with me – sorry if I didn’t make myself clear. And what you say is absolutely right.

    The “mechanical script-based approach” is the idea that you should write all the test cases in advance and merely run through them in auto-pilot when executing them. In practice I found detailed test scripts was little more than rewriting the functional spec in a different format – I could just as easily test from the functional spec itself.

    Experience taught me that a far efficient way of testing is to devise the test cases and execute them as a single activity, and a list of test cases is the end result of testing rather than the starting point.

    Once I start finding bugs it becomes clearer which features and functions need to be focussed on. Especially when, as you say, a brute force approach is impossible given the number of potential permutations.

    It’s also taught me that manually repeating the same test cases over and over on new releases as regression testing seldom uncovers new bugs, but they come crawling out the moment you try something subtly different from what you did last time.

  5. John Parr says:

    Hehe. You’ve reminded me of a bit of testing I did early in my career. I was handling inventory in the millions and turned up a couple of bugs as a result. The developer asked why I was using such large numbers and so I pointed out that the keyboard was equipped with a double zero key and I thought it would be fun to make use of it. (OK, I didn’t get out a lot – so sue me)

    Have a look at this site. They’ve extended the tree swing example to hilarious effect

  6. Tim Hall says:

    Indeed. Testing boundary conditions is so fundamental that any developer who takes exception is somewhat immature (I know, in an earlier life I was that developer).

    I’ve tested text fields by copying and pasting the entire text of Tolstoy’s “War and Peace” (available from Project Gutenburg). The system should not crash with overflow errors.

    And never say “Oh, but a real user would never try to do this”. It very often happens that a users will use features for completely different purposes from that for which they were designed.

  7. PaulE says:

    Talking of boundary conditions, a little amusing story …
    Many years ago in my first IT job, a colleague announced that he was now going to give a system his “special” test. Whereupon he launched a form and then just started hitting keys randomly on the keyboard. Very much in the way that a non-typist attempts to impersonate someone who can do 80 words per minute. After a couple of minutes he was suddenly back to typing against the system standard prompt – the program had crashed. He called over the developer who looked very concerned until he heard how it had happened. Then with a little smirk he said “let’s see you do that again” – knowing that an unrepeatable bug would simply be logged with “no further action possible”. My colleague calmly said “OK” and started it up for a re-run. After a minute or two of random bashing it crashed again. The developers face was an absolute picture !
    To give both guys credit, they went away from there and worked for a couple of days on both a less random test and a solution. But for a while after that, various people would leave odd looking contraptions on their keyboards to enter vast numbers of a character while they went to make a cup of tea.

  8. Tim Hall says:

    When you’ve got a web-based system which is supposed to word-wrap long blocks of test, lengthy strings of characters with no spaces can sometimes break the display in interesting ways.

  9. Michael Orton says:

    Can’t resist adding a tale of my own here.

    I’m not going to name the system, but a new specification has just been mandated by an international trade organisation which defines a file format which members will use to send invoices to each other.

    The gross value of an item is defined as being in a 15 digit field.
    Commission may be payable, and if so it would usually be at 9%.
    The commission value field has 11 digits.
    If VAT is to be charged you have to detail it in an extra record.
    This record has a field for the base value you are charging VAT on.
    The base value field has 13 digits.

    All these fields have an implied two decimal places.

    How many organisations need to bill for US$9,999,999,999,999.99?
    I did ask who in my company can sign for an invoice that big.

    The one thing which is probably going to keep all the plates spinning for a while is that the maximum value you can invoice on a single item is US$1,000,000. But you do not have to invoice in US$. You just have to say what currency you are invoicing in and what exchange rate you used.

    However if the exchange rate shifts by a certain proportion within a month, you can rebill asking for the effect of that change to be taken into account and if you do you wrap all your previous invoice items up into one big memo which must deal in the sum of those items. There could be 10,000 or so items.

    Now just to keep things interesting, this system was specified by accountants. Accountants require all values to be able to exist as real money at all times.
    So whenever you do a currency conversion you must round to two decimal places.
    When you convert back, you round again. I prefer my multiplication to be reflexive and commutative, but the basic principles of maths do not apply in the finance department.

    When you validate that you are billing the right value in the invoiced currency (probably US$) the value you send back must equal the number you were sent to the second decimal place. Even if you yourself worked in your own native currency (such as GB£) within your own accounting system.

    As a result a system which used to run entirely automatically now absorbs over 1 man-day a week just to get the central clearing house to accept our files.

    No IT staff were on the committee which agreed the specification.