[All Things Quality] Root Cause Analysis Template

Well there's your problem!

When something gets released to Production which adversely affects a customer, we sometimes write up a formal Root Cause Analysis to send to the customer. It basically explains what happened, and how we intend to prevent it from happening again.

Here's a template for what we send.

{Company Logo goes here}

Root Cause Analysis (INTERNAL)


Incident Date: {the date the incident occurred}
Root Cause Analysis Author: {the author of this RCA}
Client: {the client to whom this RCA is sent}
Product: {the product impacted by this incident}


Synopsis of events:
Include a summary paragraph describing what happened. copyrightjoestrazzere

Timeline:  
Describe the timeline of events – what happened, when. Note the time of the first action leading up to the incident, the time when it first impacted clients, the time when it was resolved, and the time of any other significant events.

Root cause:
Describe the root cause which led to the incident.

Resolution:
Describe the actions which resolved the incident.

Corrective Actions to Prevent Re-occurrence: 
Describe what is being done (or what will be done) to prevent similar incidents from occurring in the future.

And here's a doc file you can use:
https://docs.google.com/file/d/0B1X3i44SoVGlR2xxWUtjUlY1aHc/edit


Image courtesy of Just2shutter / FreeDigitalPhotos.net


This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.

[All Things Quality] New And Improved Bug Hunting!

Better Bug Hunting?

So you say there's nothing new in bug hunting?

Well then you haven't seen the new BUG-A-SALT salt-shooting shotgun. It's designed to take down flies, mosquitoes, and the occasional roach or wasp (if shot at point blank range) using ordinary table salt! This beauty fires about 50 shots before reloading and can be accurate up to 3 feet.

As the website proclaims:
These disease carrying gangs and their maggot counterparts must be exterminated at all costs. We the people must join forces to combat these $h*t breeding motherfu*ker$!
That's pretty much how I feel about software bugs, too.

http://bugasalt.com/


This article originally appeared in my blog: All Things Quality
My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
I like to lead, to test, and occasionally to write about leading and testing.
Find me at http://AllThingsQuality.com/.

[All Things Quality] Lessons Learned While Testing Loosely-Coupled Systems

Hang Loose!

In my company, many of the systems we build are loosely coupled with our customers' systems, systems from our division in India, or both.

Our loosely-coupled systems have some characteristics that can make them a challenge to develop and a challenge to test:
  • the interface is defined
  • but partly because these systems were written long ago with no current documentation, little or no knowledge exists about how the other end uses it, nor their dependencies

For some of our systems, the interface consists of one or more XML files that are passed between systems. You might think that it would be reasonable to assume that the other end is consuming and parsing the file as XML, but you would be wrong. In one case, they were consuming and parsing the file as a flat file. The program they used to ingest the XML file in this non-standard manner was written such that order and whitespace was critical! We had to revise our testing to take this into account.

In one of our systems, a flat file was delivered via a URL as the interface. While we originally thought the customers would be consuming the file as a browser would, we found that this wasn't actually the case. The customers had written a custom program which depended on fields in the HTTP header that often changed without affecting browsers. We had to add steps in our testing to specifically inspect the header fields to make sure they never changed.

Many of our systems pass flat files back and forth via FTP. When we test these systems, we must consider:

  • What if the file arrives early?
  • What if the file arrives late?
  • What if the file never arrives?
  • What if the file is corrupt?
  • What if the file is read-only?
  • What if the file is empty?
  • What if the file is much bigger than expected?
  • What if more than one file arrives?
  • What if you try to use the file while it is being written, but is not yet complete?
  • We often use a ".Done" file which is created only after the primary file is ready to be picked up, or indicate "done-ness" in some other way
  • What if the file contains a virus?

  • Lessons: copyrightjoestrazzere
    • Don't assume!
    • Document as many of the interface rules as you can
    • Define the interface as precisely as you can
    • There must be lots of defensive programming, which must be tested
    • You must plan to prepare and execute many negative cases
    • In future versions, prepare to detect ANY changes in the interface, not matter how trivial, no matter how seemingly-insignificant


    This article originally appeared in my blog: All Things Quality
    My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
    I like to lead, to test, and occasionally to write about leading and testing.
    Find me at http://AllThingsQuality.com/.

    [All Things Quality] What do I do? I meet!

    meetings - lots and lots of meetings!


    When people ask me what I do for a living, I tell them about computers, about software, about software bugs, about testing, and about management.

    But thinking about it recently, and looking at my Outlook Calendar, I could sum up the majority of what I do lately with two simple words:

    I MEET

    Just to give you an idea, here's a list of my current regularly-scheduled weekly meetings:
    Regularly Scheduled Weekly Meetings 
    Monday 
    1.0 Status Meeting with boss
    1.0 QA Team meeting
    1.0 Project Team Meeting - Project A 
    Tuesday 
    0.5 One-on-One Meeting with QAer 1
    1.0 Internal Client Opportunities Meeting
    1.0 QA Team Learning Session (bi-weekly)
    1.5 Senior Management Staff Meeting
    2.0 Project Team Meeting - Project B 
    Wednesday 
    0.5 One-on-One Meeting with QAer 2
    1.0 Production Control Board Meeting
    1.0 Project Team Meeting - Project C 
    Thursday 
    1.0 India Telephone Meeting
    0.5 One-on-One Meeting with QAer 3
    1.0 Project Team Meeting - Project D
    2.0 Project Team Meeting - Project B 
    Friday 
    0.5 One-on-One Meeting with QAer 4
    1.0 Project Team Meeting - Project E
    1.0 Project Team Meeting - Project F 
    Total = 18.5 hours

    And in addition to these weekly meetings there are many others that occur monthly, semi-annually, periodically, and sporadically. copyrightjoestrazzere

    Since I always like to be prepared, I also spend time before each of these meetings reviewing or preparing the agenda and background materials. I spend more time related to meeting than any other task.

    So next time someone asks "What do you do?", I'll probably arrange a meeting to tell them all about it.

    Image courtesy of sixninepixels / FreeDigitalPhotos.net


    This article originally appeared in my blog: All Things Quality
    My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
    I like to lead, to test, and occasionally to write about leading and testing.
    Find me at http://AllThingsQuality.com/.

    [All Things Quality] Is This Really "What we expect in 2013"?

    not actually me

    I was browsing through the January edition of SDTimes magazine (yes, the real dead-tree version), and found an article titled "What we expect in 2013". In it, the editors make a few predictions about what the new year will bring for the Software Development community. copyrightjoestrazzere

    Here's an excerpt that caught my eye:
    Development teams will focus on testing. Whether driven there by mobility or by the cloud, we will see decreased corporate and customer tolerance for buggy or insecure software. Developers will get more test training, more testers will be hired, and more outsourced testing services will be launched. It's about time.
    I've been reading similar predictions for years, and I've yet to see most of this happen.
    To me (except for the part about "more outsourced testing services will be launched" which has been happening for quite a long time), I think this is all wishful thinking.

    What about you? 

    Do you see actually any evidence that corporations or individuals are becoming less tolerant of bugs? 
    Are your Developers really getting more test training? 
    Are you hiring more testers?


    This article originally appeared in my blog: All Things Quality
    My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
    I like to lead, to test, and occasionally to write about leading and testing.
    Find me at http://AllThingsQuality.com/.

    [All Things Quality] Top Ten Search Terms in 2012

    Top Ten Terms

    As part of my review of All Things Quality for 2012, here were the top ten search terms folks used when they ended up on this blog: copyrightjoestrazzere
    • status report template
    • six
    • free kindle games
    • requirements
    • why software gets in trouble
    • nasdaq logo
    • quality interview questions
    • quality report template
    • all things quality
    • lark
    Ok, I'm pleased with "status report template", "requirements", and of course "all things quality".

    But, "six"? And "lark"? I'm guessing some folks didn't get what they were looking for.


    This article originally appeared in my blog: All Things Quality
    My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
    I like to lead, to test, and occasionally to write about leading and testing.
    Find me at http://AllThingsQuality.com/.

    [All Things Quality] Top Ten Countries Reading All Things Quality

    Top Ten Countries Reading All Things Quality

    As part of my review of All Things Quality for 2012, here were the top ten countries with the most visits to All Things Quality, according to Google Analytics: copyrightjoestrazzere

    United States
    India
    United Kingdom
    Canada
    Australia
    Germany
    France
    Romania
    Poland
    Phillippines

    Welcome, friends from around the world!

    welcome
    में आपका स्वागत है
    welcome
    welcome and bienvenue
    welcome
    willkommen
    bienvenue
    bine ai venit
    powitanie
    maligayang pagdating

    (I hope Google Translate got these right...)


    This article originally appeared in my blog: All Things Quality
    My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
    I like to lead, to test, and occasionally to write about leading and testing.
    Find me at http://AllThingsQuality.com/.

    [All Things Quality] Top Ten Blog Pages for 2012

    All Things Quality's Top 10 for 2012

    Aside from my home page, these were the top ten content pages for 2012.  They received the most visits according to Google Analytics:

    People in Testing / What People are Writing

    My pages of links to other testing-related blogs of interest and aggregation of people's most recent posts.
    Please continue to send me links for sites that you think should be added - your site, or sites that you read. Thanks! copyrightjoestrazzere

    My Status Report Template

    This is a spreadsheet template I use to communicate my an entire year's worth of status to my boss. I see that people find templates useful, and will try to post more this year.

    Testing Terms Glossary

    A list of terms in QA and Testing.  I originally started this many years ago so that I could copy and paste the definitions when people asked "What is...?"  I'm so happy that you find it useful.

    QA and Testing Interview Questions (And Some Answers)

    An oldie, but goodie. It's hard to be prepared for an interview.  I'm really happy if this list helps you.

    25 Things About Me

    You still like to read about me? Ok!

    Interview Questions You Should Ask

    I've written in the past about how to be prepared for interview questions that will be asked of you. But this time, I wrote about interview questions that you might want to ask. People seemed to like them.

    Issue Tracking Template

    People clearly like templates. I'm glad. I promise to post more of them.

    About This Blog

    Lots of new visitors on 2012, apparently wondering "what is this blog all about?" I hope you all found what you were looking for!

    Perhaps They Should Have Tested More - Nasdaq

    This was a post about NASDAQ's "difficulties" when Facebook went public. It was an interesting, and very public, demonstration of software bugs.

    Six Years!

    I wrote about my six-year anniversary at my current company. While I think a few readers enjoyed it, I suspect many landed here after searching for the word "six". Perhaps they were searching for a different 3-letter word?


    This article originally appeared in my blog: All Things Quality
    My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
    I like to lead, to test, and occasionally to write about leading and testing.
    Find me at http://AllThingsQuality.com/.

    [All Things Quality] Perhaps They Should Have Tested More in 2012

    2012 saw more "interesting" bugs show up in the news.  Perhaps these companies should have tested more?


    A system designed to help Mitt Romney volunteers record who voted, and to identify those who hadn't for follow-up, apparently failed miserably. It probably didn't cost Romney the election.
    Perhaps They Should Have Tested More - Romney Project Orca
    http://www.allthingsquality.com/2012/11/perhaps-they-should-have-tested-more.html

    When Apple released its new iOS 6 software update, they chose to replace Google Maps with Apple Maps. Hilarity, unhappy users, and a VP firing, ensued. copyrightjoestrazzere
    Perhaps They Should Have Tested More - Apple iOS 6 Maps
    http://www.allthingsquality.com/2012/09/perhaps-they-should-have-tested-more.html

    Knight Capital rolled out a change to its trading software - and in the process lost $440 million. Knight's logo refers to them as "The Science of Trading. The Standard of trust."
    Perhaps They Should Have Tested More - Knight Capital
    http://www.allthingsquality.com/2012/08/perhaps-they-should-have-tested-more.html

    When accounts were transferred from LaSalle Bank to Bank of America, a software bug allowed a man to withdraw $1,543,104 more than he owned. He gambled the money away.
    Perhaps They Should Have Tested More - Bank Of America
    http://www.allthingsquality.com/2012/06/perhaps-they-should-have-tested-more.html

    In a precursor of bugs to come, the Mitt Romney campaign released a new iPhone app titled "A Better Amercia". Hey, it was only a minor typo, right?
    Perhaps They Should Have Tested More - Mitt Romney Campaign
    http://www.allthingsquality.com/2012/05/perhaps-they-should-have-tested-more_30.html

    When Facebook went public in May, NASDAQ's IPO system went loopy even after "thousands of hours" of testing. Maybe thousands and one would have done the trick.
    Perhaps They Should Have Tested More - Nasdaq
    http://www.allthingsquality.com/2012/05/perhaps-they-should-have-tested-more.html

    On February 29th, Windows Azure experienced about 8 hours of downtime, demonstrating that Microsoft still hasn't quite got the hang of this whole Leap Day thing. Maybe next time.
    Perhaps They Should Have Tested More - Windows Azure
    http://www.allthingsquality.com/2012/03/perhaps-they-should-have-tested-more.html

    The Russian Mars probe Phobos-Grunt crashed back to Earth in January, after a programming error forced it into "safe mode". Oh well - a paltry budget of $165 million apparently doesn't allow for much testing.
    Perhaps They Should Have Tested More - Russian Space Agency
    http://www.allthingsquality.com/2012/02/perhaps-they-should-have-tested-more.html

    The company founder of Fisker Automotive personally called every owner to apologize for bugs in the software controls of their electric "Karma" sedan. It was the second time in two months Fisker cars were recalled. "Hello? Yes, this is Henrick again..." 
    Perhaps They Should Have Tested More - Fisker Automotive
    http://www.allthingsquality.com/2012/01/perhaps-they-should-have-tested-more.html


    This article originally appeared in my blog: All Things Quality
    My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
    I like to lead, to test, and occasionally to write about leading and testing.
    Find me at http://AllThingsQuality.com/.

    [All Things Quality] Perhaps They Should Have Tested More - Romney Project Orca

    Perhaps the Project Orca team should have tested more?

    Described as a "Killer What Project", a system designed to help volunteers record who voted, and to identify those who hadn’t for follow-up, apparently failed miserably. copyrightjoestrazzere
    • Designed to digitize and streamline the poll watchers' strike lists
    • Not only did Project Orca not improve the process, it may have actually hindered it. 
    • Checklist in instruction packet listed the same item twice, and missed a critically important step
    • Didn't auto-forward users who didn't know to add an S to the http:// protocol in the app's URL
    • PINs that were required to login and download the voter lists didn't work and couldn't be reset
    • In one area, the head of the Republican Party plus 10 volunteers were all locked out
    • All of Colorado had the wrong PINs
    • In North Carolina every pin was wrong and not fixed until 6PM
    • The system went down for a half hour during peak voting, but for hundreds or more, it never worked all day
    • The system projected every swing state as pink or red
    • As of 4 p.m., Orca still projected a Romney victory of somewhere between 290 and 300 electoral votes
    • An aide said Orca had pretty much crashed in the heat of the action
    • By pushing Orca into the most important swing states they drew 37,000 volunteers away from more battle-tested methods and left local campaign offices flying blind

    According to a Romney campaign release from just before the election:
    "Project ORCA is a massive undertaking – the Republican Party’s newest, unprecedented and most technologically advanced plan to win the 2012 presidential election.
    ...
    We have tweaked and improved Project ORCA throughout primary, so going into the general, we had several ideas and more time to incorporate those ideas into a system that would work nationally."

    Perhaps (with all those ideas and more time), they should have cast a vote or two for more testing?

    See also:

    This article originally appeared in my blog: All Things Quality
    My name is Joe Strazzere and I'm currently a Director of Quality Assurance.
    I like to lead, to test, and occasionally to write about leading and testing.
    Find me at http://AllThingsQuality.com/.