Sunday, December 7, 2008
Measuring the Value of Testing
Showing this sort of result in testing is a much harder proposition. It's easy to say that you tested a feature or product that made $X over a time frame of Y years, but lets be honest here: being the product manager or lead developer and making the same statement definitely carries more weight when attached to a resume. The person generating the idea and the person originally implementing the idea will always be seen as being closer to the end result and positive cash flow than the person ensuring the value of the idea, and ensuring that the mass market can actually use your idea as expected. I'm not saying that this is "the way things should be", I'm just stating it's the way things are.
That being said, here's why I think demonstrating the value you've directly provided to a product is harder for testers: you can't show the diff between what the product would have made had the bugs you found been released to the public. That is an alternate reality we don't have access to; it's an A-B test that can't be run.
If it were possible to conclusively show that the company made $Z more because of the bugs that had been found, I think testing would be taken more seriously across the industry.
Friday, December 5, 2008
Testing in Layers
In my world, I use this idea in the following context:
- Test internally with engineering (QA)
- Test internally with non-engineering (Marketing, etc)
- Test by releasing to trusted users on our forums
- Test by sending the release to particular people who email into Support, whose problems may be fixed with this release
- Test by releasing to everyone on our forums
- Test by releasing to a small number of people who download through the website
The idea of quality feedback loops for your product will be saved for another blog post. Lets just say that twitter has been proving quite useful in getting product feedback lately.
Thursday, December 4, 2008
When has Outlook "Started"?
One particular measurement which we've been stuggling with for some time is simply known as "Outlook Startup Time". How do you know when Outlook has fully, really, finally, finished starting? Most importantly, when do our users think that Outlook has really "started"? This is an important question for us: if we're going to improve Outlook startup time with Xobni installed, we have to know what that means. Well here are a few ideas for measuring Outlook startup that we've implemented in a tool of ours:
- When the "Reading Pane" is visible and has text
- When the Xobni sidebar appears AND the Reading Pane is visible
- When the Application.Startup event fires in Outlook
- When Outlook has finished syncing with Exchange
- When you are able to move to the next mail and have it load within a certain period of time
- When the CPU usage drops back to a level on-par with usage before you started Outlook
Thursday, May 29, 2008
A Different Kind of QA: Calling all Engineers
It’s common for people to ask why a good engineer like myself would want to work in QA, especially when you have to fight the stigma’s of QA, namely:
1) You are in QA because you are not good enough for development
2) You are in QA as a stepping stone for development
3) You are in QA because you don’t like coding
My response to those statements: pish-posh. While these statements may apply to some people in the field, they certainly don’t apply to the people serious about QA. A good QA Engineer solves quality problems with an algorithmic intensity that rivals traditional programmers. They are a true hacker in the older sense of the word - they are here to find and exploit the problems in the system in any way possible.
Every problem has its boundaries. For most developers, the boundaries for implementing solutions are usually confined to one language, stack, or technology. The boundaries for problem solving in QA are generally much wider, simply because our solutions don’t have to be productized, exposed to the public, and aren’t necessarily even in the same language or stack.
This allows a much wider range of creative freedom when solving problems. Learning new languages and technologies becomes essential for your work. Having a large arsenal of tools to attack a problem becomes a necessary part of the job. This provides you with even more of a reason to learn about the latest and greatest in tech, which is something that appeals to all engineers alike.
At Xobni we approach QA differently than most. The people we look for are not here because they are not good enough for development. They are not here because they don’t like coding. The QA people here are expected to be at the top of their game. They are expected to build and create software that can topple the Jenga-like building blocks of our product. They are expected to be creative people who like to learn, explore, and exploit software.
That being said, Xobni is looking for a QA engineer! Check out the job post, and send resumes to ryan dot gerard at xobni.com if you think you can rock our world.
Tuesday, March 11, 2008
Productivity and Flow
I was thinking recently that if I could induce this state of mind more frequently, I could become a more productive person. After a short amount of googling, I discovered that there is an entire area of research in psychology devoted to this subject; it's known as "flow". One of the main researchers in this field is a Hungarian psychology professor named Mihaly Csikszentmihalyi. He's published a few books on the subject, and the one I picked up is called "Flow: The psychology of optimal experience".
The book itself was a little too self-helpy for my tastes (if you look carefully, you'll see that the subtitle of the book above is "Steps toward enhancing the quality of life"), however I found tidbits inside could have been taken out of any of the software management books I've read.
After skimming through the bits on how and why you desire happiness, I found the core of the book: the elements of flow experiences.
1. Engaging in a Challenging Activity
He explains that the activity you're engaged in has to be at the edge between skill and anxiety. Even if your activity is complex, if you're too familiar with it, it won't be considered "challenging" to your psyche. You have to find something that is within your reach to learn or finish, but isn't easy.
2. Merging of Action and Awareness
Your attention is completely devoted to the activity, such that you have no awareness of the outside world. It is intense concentration, but seems effortless when deeply involved.
3. Clear Goals and Feedback
This is fairly self-explanatory, and it is also where I started seeing parallels in software development and management. On the development side, having small coding goals that are constantly achieved and iterated on is how I think many productive people program. On the management side, providing clear feedback and goals to your employees is a staple of good management.
4. Concentration on the task at hand
This is probably obvious, however what I found interesting was that he believes only a select range of information can be allowed into your awareness when in this state. Irrelevant information in your mental activity can break your concentration, and hence your flow.
Relating this to back to software environments, he goes on to state that quiet environments are essential to keeping your concentration. Much has been written already about how loud environments are productivity killers, and this just provides more evidence to that.
5. The Paradox of Control
He says that the flow experience is strongly associated with a sense of control. This resonates strongly with programming in my experience. One of the psychological benefits of programming (in my non-expert opinion) is the sense of mastery and control one gains over the system you're programming against. "Hacking" (in the Paul Graham sense, not the Kevin Mitnik sense) is merely another way of asserting your control and power over the system, by finding a non-obvious or faster solution to a specific problem. It's a very primal feeling that I think many, if not most of us, desire.
Mihaly then writes that the "paradox" of control "...is that it is not possible to feel a sense of control unless one is willing to give up the safety of protective routines". In essence, your sense of control comes by putting yourself into situations where you actually have less control, since the unknowns are much greater than in situations that you've experienced before. As he writes, "Only when a doubtful outcome is at stake...can a person really know whether they are in control".
6. The Loss of Self-Consciousness
Losing your sense of self-consciousness is a phenomena typically talked about in association with meditation, or zen-like activites. This loss is typically accompanied by, "a feeling of union with the environment". Projected onto programmers, this environment you feel a union with is typically whatever framework, system, or specific program you're working in.
Mihaly explains that what is temporarily lost is not the sense of self, but the concept of the self. High-performing violinists are very aware of their fingers, as runners are aware of their breathing. They haven't lost their sense of self, but the boundaries for how they define the self have temporarily vanished. This can be a very liberating experience, as "...the feeling that the boundaries of our own being have been pushed forward".
7. The Transformation of Time
It is normal to emerge from a flow experience and see that hours have passed without your awareness. What you're measuring when in this activity is not time, but states or milestones. When programming intensely, it's not uncommon to think of your progress not in terms of minutes and hours, but in terms of functions written, functionality working, and pieces integrated. Your world turns into a state-driven world, and not a time-driven one.
For the skeptical types (like me), I want to say that these elements are conclusions drawn from many studies of people experiencing flow in many different types of activities. While this doesn't mean that the conclusions are true, it does have more credibility than just some quack spouting off what he thinks brings about flow experiences.
Hopefully this has provided you with some thought-food to chew on regarding your own productivity. I think the main take-aways from this for me are that to really engage deeply in an activity, one needs:
- A challenging task
- A quiet environment
- Clear Feedback (usually in the form of finished functions and functionality in what I'm writing)
- A clear mind
- Enough time set aside to engage deeply with the activity
Wednesday, January 23, 2008
The Great TODO List
I was discussing day-to-day planning tactics with some friends yesterday, and we found we all had different methods. I thought I'd share mine with you. I call it the Great TODO List. It's quite simple really. Anyone can start using this method immediately. The low-techness of it is astounding.
Step 1: Open Notepad
Step 2: Write down everything you need to do
Step 3: Put everything you need or want to do today at the top of the list
Amazing, isn't it?
Every day or so I go through the list and roughly prioritize what should be at the top that I may have forgotten about. As I go through the day, when I start to feel like I should be working on something else, I just consult the list, and pop from the top. Yes, the list is a stack.
The one downside to this method is that the list continually grows. I have stuff on my list from a few weeks back that I should still do at some point, but the likelihood of me doing that stuff is getting smaller and smaller by the day. The list needs love and pruning.
Tuesday, November 13, 2007
Quality and Perception
I've been thinking a lot more about product quality lately, in the general sense. This is probably because product quality will be much more on my shoulders than before, as I'm working with a relatively new product that has already been launched (in the form of a beta). I've been thinking about how one can attribute the status of "high quality" to any product, and I've realized that the only person who can bestow that status is the consumer.
Ultimately, no matter how much a product is tested and sent through the wringers of QA, the consumer is the one who decides whether your product is of high quality. It is the perception of quality that makes something of high quality. Everyone I know who has owned a BMW has moaned a little about how often it has to be taken into the shop for repairs. And yet BMW retains the status that their cars are of high quality, due to the fact that they are "german-engineered".
The perception of quality is powerful, and can directly contribute to product success. The first product put out by Xobni is called "Insight", and is a plug-in for Outlook that gives you a "people-centric" view of email. Email is data that is important to the consumer (vitally important to some), and anything that builds on top of that data must provide value while not corrupting or interfering with the basic tasks (emailing) in any way.
That is the quality challenge with this product, as I see it from a high-level. What is more important than verifying the product functionality is working as expected, is making sure that the consumers current tasks and environments aren't disturbed by the product.
That being said, now the question is, how can we affect the perception of quality to the user? This is a question I'll have to ponder more. The value we're building into Outlook will allow them to accomplish their tasks faster, and more efficiently. I think this increase in efficiency is the key to the quality perception for our product: the users are getting more done by using our product, without having their current email environment disturbed. This may sound quite basic and obvious, but I think it's good to reinforce these base points as to why we're building what we're building.
Interesting times are ahead.