Testing & Software Blog

The occasional thoughts of a freelance software tester, drawn from experience across the software development life-cycle.

Stop ISO 29119

So, there is a proposed ISO standard for software testing, ISO 29119, which is causing an awful lot of controversy in the testing world.

Stop 29119Just about every software testing professional with an online presence is concerned about ISO 29119′s likely impact on the profession. The consensus is that forcing a highly bureaucratic one-size-fits-all cookie-cutter approach to testing across the whole software industry is unlikely to result in higher quality software, but will almost certainly stifle innovation and inhibit exploration of new creative approaches.

Rob Lambert is just one of many with serious reservations, and James Christie has this to say:

I’m afraid my hackles rise when I see phrases like “one definitive standard” and “used within any software development life cycle”. It immediately triggers an adverse emotional reaction as I remember this rhyme from Lord of the Rings, about the One Ring that would give the holder power over all.

“One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie”

Unfortunately it’s not something that anyone can guarantee will just go away if people ignore it.

Naturally those whose businesses revolve around selling consultancy to middle-management are going to support the introduction of a standard. As will the certification mills. And don’t even mention lawyers. I’m sure we can all easily imagine technically-illiterate politicians demanding that ISO 29119 be mandatory for all government contracts. After all, everyone knows that those gargantuan government IT failures we keep hearing about in the media are entirely down to sloppy software testing and have nothing to do with reality-denying project management.

There is now a petition against it. If you think ISO 29119 is a bad thing, go and sign it.

But not everyone agrees with the petition. Although this ridiculous Godwinesque screed hardly helps the cause:

Their objection is that not everyone will agree with what the standard says: on that criterion nothing would ever be published. The real reason the book burners want to suppress it is that they don’t want there to be any standards at all. Effective, generic, documented systematic testing processes and methods impact their ability to depict testing as a mystic art and themselves as its gurus.

I would say that resorting to personal attacks of that nature is strong indicator for the bankruptcy of their argument.

Posted in Testing & Software | Tagged | 7 Comments

The Perils of Hiring for “Cultural Fit”

In a blog post entitled “Inside the Mirrortocracy“, Carlos Bueno skewers the notion of hiring for “cultural fit”, where a perfectly qualified candidate can be rejected purely for liking the wrong sports or having the wrong taste in music.

Call­ing it out and mak­ing fun of it is not en­ough. Whatev­er else one can say about the Mir­rortoc­ra­cy, it has the vir­tue of ac­tual­ly work­ing, in the sense that the lucky few who break in have a de­cent rate of suc­cess. Com­pared to what, well, that is careful­ly left un­as­ked. The col­later­al damage of “false negatives” is as large as it is in­visib­le. But it is dif­ficult to argue with suc­cess. It takes a humil­ity and generos­ity that must come from with­in. It can’t be for­ced on oth­ers, only en­couraged to de­velop.

Lest you get the wrong idea, I’m not mak­ing a moral case but a fair­ly amor­al one. It’s hard to argue against the fact that the Val­ley is un­fair­ly ex­clusiona­ry. This im­pl­ies that there is a large un­tap­ped talent pool to be de­veloped. Since the tech war boils down to a talent war, the com­pany that figures out how to get over it­self and tap that pool wins.

Yes, it’s probably far worse in Silicon Valley than perhaps it is in the rest of the world, but I’m sure there are plenty of other places in the world with similar problems. And as Carlos Bueno says, it results in a monoculture so limiting that those inside don’t even realise it.

Posted in Religion & Politics, Testing & Software | Tagged , | 3 Comments

Should you log every bug?

Are all bugs serious enough to log?

I found one during testing today that raised the question of the costs of raising, triaging, fixing and retesting a bug compared with the costs of just leaving it in the system.

It set off some interesting discussions on Twitter on the subject.

The bug in question was a drop-down list for an optional value in a create dialog in a relatively little-used part of the system. One of the values in the dropdown was not valid, and quite obviously not so, at least with the data in the test system. If you selected that bogus value and saved the record, the server-side code didn’t throw an error, but inserted null into the column.

Since neither the field nore the underlying column were mandatory, the bug didn’t result in any data corruption, and it did save the correct value to the database if you selected one of the legitimate, valid values.

From a user perspective, it’s more of an irritation than a major issue; it certainly doesn’t prevent you from using the feature in its intended way.

In the end I discussed it with the business analyst, and we agreed to log it in the bug tracking system, but assign a low severity.

Posted in Testing & Software | Tagged , | 2 Comments

Is This A Bug?

On the reset password function, Username is case-insensitive.

On the login function, Username is case-sensitive.

So if you try to log in with the exact same username as you used to reset the password (i.e, with the “wrong” capitalisation”), and the brand new password you just created, it’s rejected as “Invalid username and password”.

Logon and password reset features are basic functionality for all websites. But still some people get it wrong…

Posted in Testing & Software | 2 Comments

Will Everything Always Be Broken?

This piece entitled “Everything Is Broken” describes the dire state of internet security, with major security breaches every other day, with everyone from Russian criminals to The NSA making off with our data.

It was my exasperated acknowledgement that looking for good software to count on has been a losing battle. Written by people with either no time or no money, most software gets shipped the moment it works well enough to let someone go home and see their family. What we get is mostly terrible.

It paints a very depressing picture of an internet held together with sellotape and string, but do you know what it reminds me of?  Replace catastrophic data breaches with fatal accidents, and it reminds me of the railway industry when it had been around for about the same length of time as the internet has, somewhere in the middle of the 19th century.

If you read L.T.C.Rolt’s classic “Red For Danger”, the mid-Victorian railways suffered serious crashes on a regular basis. Primitive signalling systems were vulnerable to human error. Braking systems were crude and ineffective. And flimsy wooden carriages with gas lighting were reduced to matchwood in relatively low-speed collisions and often went up in flames.

But things got better. It took many years, but eventually a combination of legislation and market pressure saw safety taking a much higher priority, and serious crashes are now few and far between.

It’s anyone’s guess what the internet will look like in a century’s time. But it’s entirely possible that netizens of the 22nd century will look back at the data breaches and insecurity of today like we look upon 19th century industry.

Posted in Testing & Software, Uncategorized | Tagged | Comments Off

Always Back Up Your Data

It Bites have had to cancel their appearance at Prog Splash in Holmfirth because they’ve lost all their data:

It Bites have been forced to pull out of Saturday’s Holmfirth Prog Splash. Due to technical issues and a memory card malfunction John Beck’s entire It Bites catalogue of keyboard programming – some 30 years worth – has been lost.

Under normal circumstances in the time between now and the gig John would have worked day and night to reprogram it but with him committed to other projects this week the task has proved impossible. Therefore, rather than put on a sub standard show the band have taken the difficult decision to cancel the gig.

It Bites realise this impacts on both their fans and to other bands on the bill and sincerely apologise to all.

The moral to this is “always back up your data”. Or go back to the good old days of Hammond B3s and Mellotrons….

Posted in Music News, Testing & Software | Tagged | 3 Comments

Is this a Bug or an Issue?

SNCF Wide TrainThe French railways seem have have found a serious bug in integration testing. As reported in BBC news, French red faces over trains that are ‘too wide’

The error seems to have happened because the national rail operator RFF gave the wrong dimensions to train company SNCF.

Our correspondent says that they measured platforms built less than 30 years ago, overlooking the fact that many of France’s regional platforms were built more than 50 years ago when trains were a little slimmer.

This is a prime example of a bug which would have been an awful lot cheaper to fix had it been caught at the design phase of the project.

Posted in Testing & Software, Travel & Transport | Tagged | 4 Comments

Reality Avoidance

Good blog post on Waterfall, Reality Avoidance & People Who Say “No”

One of the problems some managers have with iterative software development is that, when it’s done well – seeking early and frequent feedback and acting on it, as opposed to just incrementally executing a waterfall plan – it reduces the scope for avoiding reality.

On a waterfall project, reality can be avoided for months or even years. The illusion of progress can be maintained, through form filling and the generation of reams of reports that nobody ever reads, right up until the point that software needs to be seen to be working.

If it were my money, this would scare the shit out of me – not knowing what my money’s been spent on until the last moment.

But I can see the attraction for managers. It’s not their money. And typically they get rewarded for this illusion of progress, which can go as far as pretending the software is ready the night before it’s deployed into a live environment.

I think most of us who have worked in the software industry for any length of time will be nodding at that one. Been there, done that, got the polo shirt.

The whole thing is well worth a read, with some real life war stories leading to the inevitable conclusion.

Managers need to be rewarded for testable achievements, and steered away from peddling illusions. The reason this doesn’t happen more often, I suspect, is because the value of illusion increases the further up the ranks you go. If a PM gets a pat on the back for saying “we’re on track”, the CTO gets a trip to Disneyland, and the CEO gets a new Mercedes. Hence, the delusion gets stronger as we go higher. People running governments tend to be the most delusional of all, such is their power and influence. This effect is what produces the sometimes gargantuan IT failures only governments seem capable of creating.

Indeed. Though quite how you can stop politicians behaving like politicians isn’t an easy problem to solve.

Posted in Testing & Software | Tagged | 1 Comment

Testing Challenges

I don’t often post about the testing I’m currently working on, but a current project has thrown up some interesting challenges that are worth commenting on.

The new feature I’m testing is a batch update process that replicates the functionality of some existing GUI forms, using another object in the system as the source data. The actual processing is quite complex with a lot of validation and data update rules. The GUI front-end hasn’t yet been built, so I’m starting out by testing the Oracle server-side procedures by calling them through TOAD, populating the temporary tables that drive them through some simple SQL scripts.

We’re not using the main system test environment with its strictly-controlled weekly builds, but a separate environment where the developer can apply changes as soon as required. This means there’s a much swifter turnaround than I’d become accustomed to when it comes to fixing bugs I raise. Sometimes I’ve found an issue, to see it fixed in minutes.

Because none of the code under test has reached the point of a formal build the bugs aren’t going into the bug tracking system. Initially we were just using an email thread between me, the developer and the business analyst to keep track of the bugs, but after a bit that started getting unwieldy so I started recorded them in a simple spreadsheet instead.

I’m still using an exploratory testing approach, with the added advantage that the persistent data in the “temporarily” tables can serve as a record of the test cases I’ve tested. As for the source of those test case, there are very detailed functional specifications, but I’m finding the best “oracle” is actually the equivalent GUI functionality, which I’ve tested and regression tested often enough to have memorised most of the validation rules. Indeed, a significant proportion of the bugs I’ve been finding have turned out to be specification issues rather than coding errors, or edge cases that weren’t covered by the new business rules.

It’s not the usual way I’ve been working on this project, which is very Waterfall, but I’m finding working closely with the developer and seeing bugs turned round far more rapidly is a very productive way of working. Of course, once the GUI front-end is built and it’s all included in the build and spun into the proper system test environment, I’m going to have to test it all over again. But at least we’ll have pre-emptively squashed most of the bugs, and it will take far fewer iterations for it to become stable.

Posted in Testing & Software | Tagged | Comments Off

Context Collapse

Interesting post on the Software Testing Club on the subject of Context Collapse.

I recently heard the term “context collapse” in a podcast discussing the possible flight of the younger audience from some social media applications. It is unclear who originally coined the term in the early 2000′s, which initially referred generically to the overlapping circles on social media leading to a poster’s inability to focus on a single audience. In the podcast, the meaning was more specifically defined to identify the clash of incompatible social circles: college acquaintances, close friends, family, and work connections (especially management). That incompatibility leads to an abandonment of the media or couching postings in coded terms that are (supposedly) only understood within a specific circle.

Yes, that’s exactly why I decided to leave Facebook. I didn’t realised there was actually a term for it. The post on STC goes on to describe another case of Context Collapse involving accessibility testing, which the team eventually dealt with by getting actual disabled people to test the product. It’s a very interesting read.

Posted in Social Media, Testing & Software | Tagged | Comments Off