Test Automation Architectures

I recently received an email that held gold. Well gold in the way of interesting reading material, but gold nonetheless. It was a link to a PDF on "Test Automation Architectures" by Bret Pettichord, and linked off of the famous CSTER website.

The entire concept of testing and automation patterns is one that interests me, and I think is one that is ripe for some research and further investigation. Patterns in software development are common now with the Gang of Four's famous book, Design Patterns. But the development of niche patterns, in areas like testing and automation, are very interesting indeed. I am aware that work in this field is being distributed and collected by testing.com, and I think their work is fantastic.

The Absolute Need to Document

During my time in the field of QA and testing, I have learned to appreciate the need for documentation. As a student, it always seemed that commenting and documenting your code was a frivolous waste of time and energy. To be truthful, as a student, documentation was a waste of time and energy. Nobody will see your code ever again after it's turned in. Nobody will be inheriting your code later on, and have to struggle to figure out what the hell you were doing with your code. Also, most of my projects simply weren't large enough to warrant thorough documentation.

However, this lack of documenting creates bad habits that are passed on after one eventually graduates and enters the work force. The size and complexity of the programs one works on become much grander in scale, and yet the skill and desire for proper documentation is not present. This creates more problems than you can know, and has caused severe headaches and a lot of wasted time in my case.

If an alpha-geek super programmer makes you an amazing program that does exactly what you want, is completely localized, and has very few defects, I'm happy for you. If he or she didn't document it, you're completely screwed. When the program in question needs to be updated, added to, patched, or modified in any way (which is almost a certainty), you are dependent on the original programmer to be able to remember how they implemented the program the first time (assuming that the original programmer still works for you), or you must have another programmer waste considerable amounts of time understanding how the program works. If, however, that original programmer effectively documented the program in a way that made knowledge transfer possible, you can have another programmer pick up the slack with much less ramp-up time.

In this world where knowledge is king, there must be a way to transfer knowledge effectively, or it is essentially lost. Enforcing documentation is simply a smart way to protect your company assets. You cannot assume that the developers who know how something works will work for you eternally, and you must provide a way to communicate the internal workings of the program to others. Otherwise those that need to know how something works (QA, managers, other programmers) will waste the programmers time over and over again while they explain how their program works.

In addition, I've found that documenting helps me to focus on what I'm working on, and flesh out implementation details that I'd put off in the back of my mind. By having it down on paper, it's like I have a map of exactly what I need to create and implement. Documenting is about looking ahead, thinking ahead, and spending time now to save it in the future. So the next time a programmer tells you he doesn't need to document his program, tell him to suck it up and do it anyway, because documenting isn't about him or her - it's about everyone else. Spending a couple hours now to document a program will save you and others hours and hours later on.

Despair.com takes on Testing

I couldn't help but laugh at this picture. All comedy has its roots in tragedy.

The Importance of Test Design

"More than the act of testing, the act of designing tests is one of the
best bug preventers known. The thinking that must be done to create a useful test can discover and eliminate bugs before they are coded - indeed, test-design thinking can discover and eliminate bugs at every stage in the creation of software, from conception to specification, to design, coding and the rest."
-Boris Beizer, Software Testing Techniques

This is so true it's almost painful. I've seen projects where thousands of tests are run, but they aren't the right tests, and hence defects are missed. Creating massive automation infrastructures and purhasing expensive commercial testing programs (or "solutions", as they like to be called), is really a worthless exercise in time and money if you're not designing intelligent tests.

For example, lets say you have a function that takes in a string and returns the number of characters in that string. All the fancy test programs in the world aren't going to help you if your QA engineer doesn't test it with unicode characters, null strings, or strings with no null terminator.

Test design is absolutely one of the more interesting and creative aspects of QA. The simple process of coming up with possibile ways that one can break a component is a creative process on its own, but also one that requires technical expertise. How can you know the many ways to break a product on a Windows platform if you don't have indepth technical knowledge of Windows? It's almost a philosophical question - how can you know the things you don't know? There is only one answer: keep learning and expand your knowledge boundaries.

Defining Automation

I find the big problem with automation is that the word "automation" is generally used as some sort of all-encompassing silver bullet keyword. Cold feet? Try automation! Lost your sense of smell? Try automation! The very word seems to imply some sort of panacea for all your testing aches and pains.

I've found that automation works best when it is solving specific problems, with a narrow scope and focus. A good problem or process to automate would be one where you want to build your program, copy it to a server, run some sanity tests against it, and then email you to tell you it's done. A bad problem to attempt to automate would be trying to run all unit, functional, and system tests on your build, across multiple operating systems, programming languages, and is extensible enough to withstand future changes in requirements. In my opinion, this type of program is too large in scope to be reliable. I'm sure there are people (or companies, rather) that have attempted and succeeded at such a project, but an automation project of that size needs its own separate QA team.

What I'm trying to get at is when testing complex systems, don't use even more complex systems. Complexity will by nature introduce errors into the system. Keeping your testing systems simple will reduce the amount of errors within your test system. Obviously there is some loss of functionality, which is why I'm a fan of stringing together multiple simple systems in order to do more complex automation, when necessary. Modularity in automation is the key to success.

Here are a few good blog posts on the subject that cover different ways to view the same problem:

• Automation Testing versus Manual Testing Guidelines
• Tenet: If you are going to run a test more than once, it should be automated.

Open Source Chocolate Factory

I had a fantastic visit to the Scharffen Berger chocolate factory in Berkeley today. They were quite generous with the samples, and even more interesting, they showed us the exact process they use to make their chocolate, recipes and all. They even let us take pictures of their equipment inside the factory. As the tour guide (Mandy) told us, "We're the only chocolate factory in the US or Europe that not only allows you inside the factory, but we let you take pictures". It's like an open source chocolate factory.

Ok, not exactly. I can't start modifying their chocolate making process with my own beans and spices. But the transparency of their process and ingredients reminded me of open source software. The transparency builds trust with their clients, who are now more confident about exactly what they're eating. They are obviously confident that nobody will steal their processes, and for good reason. I may know now how they make chocolate, but investing the millions of dollars to copy them is not something I'm willing to spend the time and effort to do. As you can probably tell by now, I'm making parallels to software here.

PHP Static Analysis Tools

I did some searching today for PHP static analysis tools, and came across some interesting ones.

PHP-Front: Not yet even at v0.1 yet, but I await the release eagerly. You can download unstable releases if you want to test it out.

PHP-SAT: Made by the same people (person?) as PHP-Front, and also not yet even at v0.1 yet. You can download unstable releases if you want to test it out.

Pixy: This looks like an academic project, but at least they have something working! The analysis tool deals mainly with detecting XSS vulnerabilities.

Searching for PHP dynamic analysis tools did not turn out as fruitful.