Well it's a new start for me. I have a new job with Xobni as their Quality Jedi, and it's a nice change. The startup life has been something I've been talking about for a while, and the time for action presented itself in the form of a ripe opportunity.
I've been thinking a lot more about product quality lately, in the general sense. This is probably because product quality will be much more on my shoulders than before, as I'm working with a relatively new product that has already been launched (in the form of a beta). I've been thinking about how one can attribute the status of "high quality" to any product, and I've realized that the only person who can bestow that status is the consumer.
Ultimately, no matter how much a product is tested and sent through the wringers of QA, the consumer is the one who decides whether your product is of high quality. It is the perception of quality that makes something of high quality. Everyone I know who has owned a BMW has moaned a little about how often it has to be taken into the shop for repairs. And yet BMW retains the status that their cars are of high quality, due to the fact that they are "german-engineered".
The perception of quality is powerful, and can directly contribute to product success. The first product put out by Xobni is called "Insight", and is a plug-in for Outlook that gives you a "people-centric" view of email. Email is data that is important to the consumer (vitally important to some), and anything that builds on top of that data must provide value while not corrupting or interfering with the basic tasks (emailing) in any way.
That is the quality challenge with this product, as I see it from a high-level. What is more important than verifying the product functionality is working as expected, is making sure that the consumers current tasks and environments aren't disturbed by the product.
That being said, now the question is, how can we affect the perception of quality to the user? This is a question I'll have to ponder more. The value we're building into Outlook will allow them to accomplish their tasks faster, and more efficiently. I think this increase in efficiency is the key to the quality perception for our product: the users are getting more done by using our product, without having their current email environment disturbed. This may sound quite basic and obvious, but I think it's good to reinforce these base points as to why we're building what we're building.
Interesting times are ahead.
Tuesday, November 13, 2007
Thursday, August 23, 2007
GTAC!
Sorry for the lack of posts -- I've been out of town recently.
I'm currently at the Google Test Automation Conference, at Google's New York office. Thus far, it's been a great experience. There are two things I've learned about (this morning) that I wanted to blog about: how to design a conference, and Google's testing philosophy.
Allen Hutchinson, an engineering manager at Google, gave some good insight into how they designed the conference, and why they made the decisions they did. I pulled lots of good insights from this talk.
*Keep the conference to under 150 people. Sociological research has shown this is an upper bound (and a magic number) that allows a group to keep a sense of community
*Provide outlets for people to discuss the conference. Their high-profile testing blog is open for unmoderated comments on this conference, and a google group was created as well.
*Make the talks available online after the conference
*Try to decrease the amount of repeat speakers from past conferences. They want fresh blood and new ideas introduced in their conference
*Ask old speakers to help select new speakers - or just have people outside your organization help select the speakers
Allen also mentioned that they kept the conference to a single track so that you can see all the talks, but to be honest I rather like the multiple track system. It allows more speakers, and allows you to skip talks that do not interest you.
Pat Copeland, a Director in Google, then spoke about the testing philosophy that Google maintains, which I found quite interesting.
*Measure everything you can. More data allows for more analysis.
*Continuous builds and faster builds allow more time for testing
*Focus on improvement of the system, and not the bits and pieces
*Testing Goal: fully test a product in one day
Pat also mentioned some challenges that they face as an organization, which I think applies to pretty much everyone:
1. Simulating real-world load, scale, and chaos is difficult
2. Deciding what to measure is difficult
3. Complex failure scenarios are expensive to test
The talks (and more importantly, the people) have been quite interesting thus far. I'll hopefully have more to update after tomorrow.
I'm currently at the Google Test Automation Conference, at Google's New York office. Thus far, it's been a great experience. There are two things I've learned about (this morning) that I wanted to blog about: how to design a conference, and Google's testing philosophy.
Allen Hutchinson, an engineering manager at Google, gave some good insight into how they designed the conference, and why they made the decisions they did. I pulled lots of good insights from this talk.
*Keep the conference to under 150 people. Sociological research has shown this is an upper bound (and a magic number) that allows a group to keep a sense of community
*Provide outlets for people to discuss the conference. Their high-profile testing blog is open for unmoderated comments on this conference, and a google group was created as well.
*Make the talks available online after the conference
*Try to decrease the amount of repeat speakers from past conferences. They want fresh blood and new ideas introduced in their conference
*Ask old speakers to help select new speakers - or just have people outside your organization help select the speakers
Allen also mentioned that they kept the conference to a single track so that you can see all the talks, but to be honest I rather like the multiple track system. It allows more speakers, and allows you to skip talks that do not interest you.
Pat Copeland, a Director in Google, then spoke about the testing philosophy that Google maintains, which I found quite interesting.
*Measure everything you can. More data allows for more analysis.
*Continuous builds and faster builds allow more time for testing
*Focus on improvement of the system, and not the bits and pieces
*Testing Goal: fully test a product in one day
Pat also mentioned some challenges that they face as an organization, which I think applies to pretty much everyone:
1. Simulating real-world load, scale, and chaos is difficult
2. Deciding what to measure is difficult
3. Complex failure scenarios are expensive to test
The talks (and more importantly, the people) have been quite interesting thus far. I'll hopefully have more to update after tomorrow.
Tuesday, July 24, 2007
My Favorite Laws of Software Development
There was a great blog on the Laws of Software Development, and there were a few I saw as related to testing that I thought I should share.
Brooks' Law: Adding manpower to a late software project makes it later. Or testers, in this case.
Conway's Law: Any piece of software reflects the organizational structure that produced it. Good to keep in mind when testing.
Heisenbug Uncertainty Principle: Most production software bugs are soft: they go away when you look at them.
Hoare's Law of Large Programs: Inside every large problem is a small problem struggling to get out. Finding those small problems is worth searching for.
Lister's Law: People under time pressure don’t think faster. Or test faster.
Nathan's First Law: Software is a gas; it expands to fill its container.
Brooks' Law: Adding manpower to a late software project makes it later. Or testers, in this case.
Conway's Law: Any piece of software reflects the organizational structure that produced it. Good to keep in mind when testing.
Heisenbug Uncertainty Principle: Most production software bugs are soft: they go away when you look at them.
Hoare's Law of Large Programs: Inside every large problem is a small problem struggling to get out. Finding those small problems is worth searching for.
Lister's Law: People under time pressure don’t think faster. Or test faster.
Nathan's First Law: Software is a gas; it expands to fill its container.
Sunday, July 15, 2007
Scalable Test Selection
A few weeks ago I gave a presentation at the Google Scalability Conference on the idea of Scalable Test Selection (video can be found here). Now that the video is up, I thought I'd share with you all this idea developed within my team. I must preface this by saying that this was not my idea -- the credit belongs to Amit and Fiona.
As Cem Kaner has said before me, there is never enough time to do all the testing. Time, in essence, is your scarce resource that must be allocated intelligently. There are many test strategies one can choose from when testing a product, however the question must be raised: how do you know whether you're testing the right things, given the amount of time you have to test? This is an impossible question to answer, however there are situations where the question is pertinent. For instance: when a change is introduced to your product at the last minute before release, and you have to decide what to test, how do you choose? There are probably some intelligent guesses you can make on what to test based on what has changed, but how can you know that this change hasn't broken a distant dependency?
This is the situation where we believe Scalable Test Selection can help. Given a source code change, what test cases are associated with that source code? Essentially, how can we link test cases to source code using test artifacts?
We have identified (and presented on) three ways to associate test cases with source code:
What we're doing is applying simple data-mining techniques to the test data. There is much more to the idea that I'm not talking about (prioritization of test cases, implementation ideas, etc), however I hope you get the jist. I fully recommend you watch the video if this topic interests you, and feel free to email me if you want the slides :).
As Cem Kaner has said before me, there is never enough time to do all the testing. Time, in essence, is your scarce resource that must be allocated intelligently. There are many test strategies one can choose from when testing a product, however the question must be raised: how do you know whether you're testing the right things, given the amount of time you have to test? This is an impossible question to answer, however there are situations where the question is pertinent. For instance: when a change is introduced to your product at the last minute before release, and you have to decide what to test, how do you choose? There are probably some intelligent guesses you can make on what to test based on what has changed, but how can you know that this change hasn't broken a distant dependency?
This is the situation where we believe Scalable Test Selection can help. Given a source code change, what test cases are associated with that source code? Essentially, how can we link test cases to source code using test artifacts?
We have identified (and presented on) three ways to associate test cases with source code:
- Requirements: If source code is checked in to satisfy requirements, and test cases are checked in to satisfy requirements, then a connection can be made.
- Defects: A test case fails, which then is associated with a new defect. When source code is checked in to fix that defect, an association is made between the code and the test case
- Build Correlation: For a single build, you can associate a set of source code changes with a set of test case failures. Now iterate that over successive builds, and you have a large set of source code and test case associations
What we're doing is applying simple data-mining techniques to the test data. There is much more to the idea that I'm not talking about (prioritization of test cases, implementation ideas, etc), however I hope you get the jist. I fully recommend you watch the video if this topic interests you, and feel free to email me if you want the slides :).
Monday, July 9, 2007
CAST Conference
I just finished Day 1 of the AST Conference (CAST), and boy did I have a good time. The talks were good, but the people were better. There was definitely a general feeling of community there, which is something I haven't seen much at conferences I've been to lately. The crowd was relatively small (~180), but all good people.
Highlights for me:
* Hearing Lee Copeland talk about the role of QA as a services group (this makes almost too much sense to me)
* Talking to Jon Bach about the future of AST, CAST, and the other conferences that AST is hosting
* Watching a small group gather after my talk on Meta-Frameworks to share their experience with Meta-Frameworks
Anyway, now that I'm leaving tomorrow, I wish I was staying another day.
Highlights for me:
* Hearing Lee Copeland talk about the role of QA as a services group (this makes almost too much sense to me)
* Talking to Jon Bach about the future of AST, CAST, and the other conferences that AST is hosting
* Watching a small group gather after my talk on Meta-Frameworks to share their experience with Meta-Frameworks
Anyway, now that I'm leaving tomorrow, I wish I was staying another day.
Saturday, July 7, 2007
The Time Trade-Off
I started prepping for the CAST conference next week by reading up on some test patterns that the AST group has produced in the past. I was reading some great stuff this weekend by Cem Kaner on Scenario Testing, when I came across a fantastic quote:
It's so true, it's painful. Deciding on what to test is becoming increasingly important in my own work, as the amount of work stacks up, and the amount of time to test it decreases. It's an interesting balancing act.
"The fundamental challenge of all software testing is the time tradeoff. There is never enough time to do all of the testing, test planning, test documentation, test result reporting, and other test-related work that you rationally want to do. Any minute you spend on one task is a minute that cannot be spent on the other tasks. Once a program has become reasonably stable, you have the potential to put it through complex, challenging tests. It can take a lot of time to learn enough about the customers, the environment, the risks, the subject matter of the program, etc. in order to write truly challenging and informative tests."
Friday, June 29, 2007
An interesting bug in Java binarySearch
Josh Bloch, author of Effective Java Programming and one of the best in the biz, blogged last summer about an error in his own implementation of binary search within Sun's JDK. It turns out that the most straightforward implementation admits a flaw so subtle that the version in java.util.Arrays, in particular, lay dormant for almost a decade. It is one of the most interesting bugs I have seen in a long time, and in fact I didn't notice it myself. Simplified slightly from the original:
The culprit is sly: Operating on a large enough array, computing m = (i+j)/2 is subject to integer overflow, becoming negative due to wraparound, e.g.
Fortunately the midpoint can be computed safely via m = i + (j-i)/2. Already there are many blog posts on this blemish, but none concerning quality. What might a software engineer learn about testing from this long-hidden oversight?
1. A lesson from numerical analysis, that time-honored formulas aren't necessarily robust when copied to finite arithmetic.
2. Vary your test data. Surely the Sun test battery for binarySearch passed, again and again, on the same sets of arrays and keys. Fine for 9 years -- but static lists of tests become stale. Which arrays should we select, or search for? Read on...
3. Specialized conditions are required to pinpoint rare defects. This flaw cannot be exposed for sizes below 2^30, about 1 billion; no wonder nobody noticed. The truth is that the endpoints of anything almost always provide interesting insights. Here, searching far to the left always passes, and only searching far enough to the right can throw the Exception. Aha. This leads to a favorite maxim...
4. Design tests specific to the implementation. The greatest of the Java int primitives is M -- nothing to do with binary search -- instead, a boundary in the language itself! Therefore, an array of that length makes a worthwhile test case in Java (cf. Mathematica, Python). Something to remember when writing your own test suites.
Professor Kahan at Berkeley once summed up his lessons about testing (from the infamous Pentium divide glitch) into a fitting quote:
Wisely put, as are Josh's remarks about programming defensively, vigilance, code reviews, static analysis, solid proofs, and formal verification. (I would disagree with his view that there will always be bugs, but opposing formidable Dr. Bloch isn't really something one ought to do).
The paradox is that although testing can't exterminate our errors, it still needs to be done to provide corroborative evidence of correctness, even for well-proved algorithms! Later, I'll augment this post with test strategies which might have exposed the flaw. With test methods in mind, we might be able to expose ever more errors, and add to our QA repertoire.
An exercise: Using 32-bit two's compliment integer arithmetic, compute the average of i and j, correctly rounding to even. Test it with i and j at the extremes. My solution is not short.
// locate an index of key k in sorted array a[],
// or if k is not present, return -1
int binarySearch( int[] a, int k ){
int i = 0;
int j = a.length - 1;
while( i <= j ){
int m = (i + j) / 2;
if( k < a[m] ){ // left half
j = m - 1;
} else if( a[m] < k ){ // right half
i = m + 1;
} else { // found
return m;
}
}
return -1; // not present
}
The culprit is sly: Operating on a large enough array, computing m = (i+j)/2 is subject to integer overflow, becoming negative due to wraparound, e.g.
int M = 0x7fffffff; // 2^31 - 1
int[] a = new int[ M ];
a[ M - 1 ] = 1;
// the call...
int r = binarySearch( a, 1 );
// ...throws an ArrayIndexOutOfBoundsException !
Fortunately the midpoint can be computed safely via m = i + (j-i)/2. Already there are many blog posts on this blemish, but none concerning quality. What might a software engineer learn about testing from this long-hidden oversight?
1. A lesson from numerical analysis, that time-honored formulas aren't necessarily robust when copied to finite arithmetic.
2. Vary your test data. Surely the Sun test battery for binarySearch passed, again and again, on the same sets of arrays and keys. Fine for 9 years -- but static lists of tests become stale. Which arrays should we select, or search for? Read on...
3. Specialized conditions are required to pinpoint rare defects. This flaw cannot be exposed for sizes below 2^30, about 1 billion; no wonder nobody noticed. The truth is that the endpoints of anything almost always provide interesting insights. Here, searching far to the left always passes, and only searching far enough to the right can throw the Exception. Aha. This leads to a favorite maxim...
4. Design tests specific to the implementation. The greatest of the Java int primitives is M -- nothing to do with binary search -- instead, a boundary in the language itself! Therefore, an array of that length makes a worthwhile test case in Java (cf. Mathematica, Python). Something to remember when writing your own test suites.
Professor Kahan at Berkeley once summed up his lessons about testing (from the infamous Pentium divide glitch) into a fitting quote:
Seek singularities! There lie all the errors.Wisely put, as are Josh's remarks about programming defensively, vigilance, code reviews, static analysis, solid proofs, and formal verification. (I would disagree with his view that there will always be bugs, but opposing formidable Dr. Bloch isn't really something one ought to do).
The paradox is that although testing can't exterminate our errors, it still needs to be done to provide corroborative evidence of correctness, even for well-proved algorithms! Later, I'll augment this post with test strategies which might have exposed the flaw. With test methods in mind, we might be able to expose ever more errors, and add to our QA repertoire.
An exercise: Using 32-bit two's compliment integer arithmetic, compute the average of i and j, correctly rounding to even. Test it with i and j at the extremes. My solution is not short.
Tuesday, June 19, 2007
Ubuntu: A Fresh Perspective
I decided to replace the Fedora core 4 machine I had at home with the bizarrely named "Feisty Fawn" version (7.04) of Ubuntu, which has gotten oh so much attention as of late. Having never installed Ubuntu, I wanted to know what the big deal was.
There was only one disk to burn. That was a nice surprise. I started booting, and found that the CD I burned was actually a Live CD. It booted me right into Ubuntu. Naturally, I didn't read much about installing the distro, so I surmised for a minute that perhaps Ubuntu was just a Live CD. Not so -- after I actually looked at the items on the desktop, I found a single "Install" icon. Clicking this proceeded with the install of Ubuntu on my local hard drive.
The distro has so far proved to be very usable. There is an "Add/Remove Applications" feature which is remarkably like the "Add/Remove Programs" feature in Windows. That is very useful for users who grew up in a Windows world. My one complaint thus far is that I can't login as the root user. I'm sure that if I look around the net enough I can find a way around that :).
There was only one disk to burn. That was a nice surprise. I started booting, and found that the CD I burned was actually a Live CD. It booted me right into Ubuntu. Naturally, I didn't read much about installing the distro, so I surmised for a minute that perhaps Ubuntu was just a Live CD. Not so -- after I actually looked at the items on the desktop, I found a single "Install" icon. Clicking this proceeded with the install of Ubuntu on my local hard drive.
The distro has so far proved to be very usable. There is an "Add/Remove Applications" feature which is remarkably like the "Add/Remove Programs" feature in Windows. That is very useful for users who grew up in a Windows world. My one complaint thus far is that I can't login as the root user. I'm sure that if I look around the net enough I can find a way around that :).
Saturday, June 9, 2007
Concurrent Programming: The Next Best Thing
I was one of the lucky few to attend the Google Developer Day in San Jose last week. It was quite interesting. One of the best talks I went to concerned the Google backend infrastructure. One of the topics they've worked on extensively is storing and organizing massive amounts of information. This probably isn't a surprise to anyone, but it highlighted to me the coming importance of concurrent programming.
For a short one-line introduction to the subject: concurrent programming concerns the simultaneous execution of multiple interacting computational tasks. O'Reilly recently pointed out that as the processor speeds remain stable, but the number of processors inside a computer continue to increase. In 5-10 years, having a computer with 50-100 processors will probably not be unusual. The real question is: how do you take advantage of all those processors?
I think many consumers assume that all these extra processors in their computers are being leveraged. I would say they are most likely assuming incorrectly, or at least, they are being leveraged inefficiently. Concurrent programming languages (such as Haskell) allow multiple processors to execute tasks simultaneously, and coordinate tasks between them. Now that's some sexy programming.
It can also make good financial sense to move into this area of programming. In O'Reilly's article, "The Faint Signals of Concurrency", utilizing an 8 CPU machine for processing data is much cheaper than buying a cluster of 8 machines for processing data.
Haskell is already a "hip" programming language (stop laughing). Among nerds, it is the cool language that wears its sunglasses at night, listens to jazz, and can recite quotes from famous philosophers. I would say that this is a bleeding edge language, and really needs more investigation by me.
For a short one-line introduction to the subject: concurrent programming concerns the simultaneous execution of multiple interacting computational tasks. O'Reilly recently pointed out that as the processor speeds remain stable, but the number of processors inside a computer continue to increase. In 5-10 years, having a computer with 50-100 processors will probably not be unusual. The real question is: how do you take advantage of all those processors?
I think many consumers assume that all these extra processors in their computers are being leveraged. I would say they are most likely assuming incorrectly, or at least, they are being leveraged inefficiently. Concurrent programming languages (such as Haskell) allow multiple processors to execute tasks simultaneously, and coordinate tasks between them. Now that's some sexy programming.
It can also make good financial sense to move into this area of programming. In O'Reilly's article, "The Faint Signals of Concurrency", utilizing an 8 CPU machine for processing data is much cheaper than buying a cluster of 8 machines for processing data.
Haskell is already a "hip" programming language (stop laughing). Among nerds, it is the cool language that wears its sunglasses at night, listens to jazz, and can recite quotes from famous philosophers. I would say that this is a bleeding edge language, and really needs more investigation by me.
Tuesday, May 22, 2007
User Expectations
Bill Higgins has recently written a fantastic post titled, "The Uncanny Valley of User Interface Design". He discusses how people have a basic mental model of how things should appear, and when they differ from that mental model, we reject them.
He has a great example of this concerning the lack of an explosion of Java GUI apps:
He's definitely on to something. This got me thinking about user expectations, and how yet another thing a pursuer of quality must keep in mind is this difficult to quantity quality. The sort of je ne sais quoi of a web or desktop app that maybe meets all requirements, but just doesn't feel right. Sorry, was that french phrase a bit too pretentious? I couldn't find a better phrase to use. It had a sort of je ne sais quoi about it.
He has a great example of this concerning the lack of an explosion of Java GUI apps:
"My own experience with using Java GUI apps of the late 1990s was that they were slow and they looked and behaved weirdly vs. standard Windows (or Mac or Linux) applications. That’s because they weren’t true Windows/Mac/Linux apps. They were Java Swing apps which emulated Windows/Mac/Linux apps. Despite the herculean efforts of the Swing designers and implementers, they couldn’t escape the Uncanny Valley of emulated user interfaces."
He's definitely on to something. This got me thinking about user expectations, and how yet another thing a pursuer of quality must keep in mind is this difficult to quantity quality. The sort of je ne sais quoi of a web or desktop app that maybe meets all requirements, but just doesn't feel right. Sorry, was that french phrase a bit too pretentious? I couldn't find a better phrase to use. It had a sort of je ne sais quoi about it.
Sunday, April 29, 2007
High Performance MySQL
Databases have recently become interesting to me. Previously, they've always seemed like these massive beasts that provide such simple and basic functionality, I didn't understand how companies such as Oracle could make billions selling databases. Yes, it was a dirty word in my vocabularly. And yes, I'm now a reborn DBA in training.
Recently I've been working on a small PHP project reporting statistics on some data from our network. The small project has been collecting for some time now, and has begun to attract some attention. It's also started to slow down considerably. I read a few MySQL performance articles and blog entries, which had quite a few tips and tricks, but in reality I was adding indicies and making config changes without really knowing how to measure the performance differences, or how the change was really affecting my web app. It was blind performance tuning.
And then I picked up "High Performance MySQL". This book is changing my life one page at a time. I've noticed that most of the articles and blogs around MySQL on the web (at least, the ones I could find) were centered around the basic stuff: installation, basic configuration, and cookbook recipes. This book goes to the next level and answers many questions that I've previously had, but didn't have an aggregated place to find.
The key to that last statement is aggregated. I'm sure there are articles out there that explain the differences between InnoDB, and MyISAM, various ways to benchmark your queries to measure performance differences, or methods to cache your queries to improve performance, but having them all aggregated in one place provides context that has proved valuable.
If you're looking for the next level of intelligence on using MySQL in real-world environments, this book is highly recommended.
Tuesday, April 24, 2007
Testing Old Versions of IE in Linux
As much as I wish people would move off their old versions of IE and Firefox as fast as possible, it's rather hard to induce this. Microsoft has tried hard to move people to IE7 by making it a Windows Update. However, the simple truth is that when developing web sites, you must support many different browsers, including those old, terrible versions that you wished had never been released.
There is a small problem with this: finding these older versions of IE is rather difficult. Oh they are out there somewhere, but it's not easy and obvious. I usually resort to downloading some version of IE 5.5 off of a sketchy website, and then proceed to do multiple AV scans of the file I've downloaded.
And what should come to the rescue...Linux, naturally! I recently found a project called IEs4Linux that installs IE 5, 5.5, and 6 off of Wine. It's meant to be used for web designers to test site compatibility in older IE versions. Personally, it will be useful for my own web testing -- and now I don't have to use Windows to test IE anymore.
There is a small problem with this: finding these older versions of IE is rather difficult. Oh they are out there somewhere, but it's not easy and obvious. I usually resort to downloading some version of IE 5.5 off of a sketchy website, and then proceed to do multiple AV scans of the file I've downloaded.
And what should come to the rescue...Linux, naturally! I recently found a project called IEs4Linux that installs IE 5, 5.5, and 6 off of Wine. It's meant to be used for web designers to test site compatibility in older IE versions. Personally, it will be useful for my own web testing -- and now I don't have to use Windows to test IE anymore.
Friday, April 6, 2007
Apple Rising
Paul Graham, that most prolific of writers for any tech geek, recently posted an interesting article titled "Microsoft is Dead". I think the title was for dramatic effect, as what he means is that Microsoft is dead in the same way IBM is dead, i.e., not innovative or interesting anymore. I believe he's right, for the most part.
One of the reasons he touted as a sure sign of Microsoft's death piqued my interest:
Without a doubt, Apple is rising. It's something I've discussed with friends, and spent time trying to decipher. At any tech conference I go to, there are tons of people with Macs. Whenever I visit the Google campus, there are macs all over the place.
What does this mean? Why are people moving to Apple, rather than installing Linux on their laptops with cheaper commodity hardware? There are a myriad of small reasons that I can come up with, but there is no denying the trend. Or fad. I'm not sure which it is.
One of the reasons he touted as a sure sign of Microsoft's death piqued my interest:
"Nearly all the people we fund at Y Combinator use Apple laptops. It was the same in the audience at startup school. All the computer people use Macs now. Windows is for grandmas, like Macs used to be in the 90s."
Without a doubt, Apple is rising. It's something I've discussed with friends, and spent time trying to decipher. At any tech conference I go to, there are tons of people with Macs. Whenever I visit the Google campus, there are macs all over the place.
What does this mean? Why are people moving to Apple, rather than installing Linux on their laptops with cheaper commodity hardware? There are a myriad of small reasons that I can come up with, but there is no denying the trend. Or fad. I'm not sure which it is.
A good quote to be reminded of
A recently came across a Peter Drucker quote that struck me as very true.
"Growth that increases productivity is healthy, growth without an increase in productivity is fat, and growth that decreases productivity is cancer...."I think it's often easy to forget that, in software at least, output is king. If what you're doing doesn't contribute to output or productivity, you may want to rethink what you're doing.
Thursday, April 5, 2007
AppVerifier
I've been introduced to a pretty interesting test tool that was released by Microsoft: AppVerifier. From the intro page of the tool:
Pretty nice. AppVerifier works by attaching to the program you want to test, and performing configurable tests whenever the program is accessed. There are five main areas you can configure for testing:
"AppVerifier encompasses areas that are specifically designed to test for commonly encountered application compatibility issues, and some very subtle kernel issues."
- Memory: This ensures that virtual memory API's are used correctly
- Thread Local Storage: This just ensures that the TLS API's are used correctly.
- Low Resource Simulation: This simulates a low resource environment
- Limited User Account Predictor: This lets you know whether a non-Admin user can properly use the program
- Misc: Dirty stacks and dangerous API's, according to the website
Wednesday, March 28, 2007
Wasting time at Microsoft
Ryan's Tech Blog (good name) recently had a nice ranting post about how Microsoft wasted 6 months of developer time re-creating a source control client for Codeplex.
It's hard to argue with his logic. He makes a strong argument about the need to work with extensible tools that can plug into each other. I think this exemplifies one of the basic differences between Windows and Unix tools that I've noticed: Windows tools are designed as "solutions", whereas Unix tools are designed to be extensible. It's almost unheard of to see Windows tools that are designed to be pluggable, but in the Unix world, the opposite is true.
"This problem is ingrained at Microsoft, which feels the need to brand everything, but it is in no way limited to them. A search on Sourceforge for “issue tracker” gives 585 results. Sifting through those to pick a winner is difficult."
It's hard to argue with his logic. He makes a strong argument about the need to work with extensible tools that can plug into each other. I think this exemplifies one of the basic differences between Windows and Unix tools that I've noticed: Windows tools are designed as "solutions", whereas Unix tools are designed to be extensible. It's almost unheard of to see Windows tools that are designed to be pluggable, but in the Unix world, the opposite is true.
Monday, March 26, 2007
Collaborative Filtering: An introduction
Recommendation algorithms are hot right now. NetFlix, Amazon, and StumbleUpon all have examples of working recommendation algorithms that are not only interesting, but useful. Collaborative filtering is the method of prediction that has been widely adopted across these sites.
The underlying assumption with collaborative filtering is that your preferences in the past will help predict your preferences in the future. Assuming that this is true, CF makes itself unique by finding users with similar preferences as you. Once this neighborhood of users is found, it can be reasonably assumed any items they like, you will also like, and hence, an intelligent recommendation can be made.
Using the NetFlix analogy, if you and another user named "Jack", who is completely unknown to you, have rated the same movies in the same way, it can be assumed that you have similar tastes. Therefore, if Jack rates a new movie very highly, the likelihood that you will also like that movie is very high, and it can be recommended.
There are issues with this algorithm, of course. First, you have to build up a reasonable list of preferences for the algorithm to make successful matches of other users. If you only rate 1 item ever, you can't expect great recommendations to come your way. Second, if your tastes change, the algorithm will be able to pick it up, but only with time. There must be time for the new change to affect the weight of your past selections. Last, this algorithm doesn't scale very well. For massive sites like NetFlix and Amazon, I'm sure there are some sort of caching or iterative methods involved so that the recommendations aren't recomputed everytime you login to the site. With millions of users and millions of products, it would simply take too long to load any sort of user-specific recommendation.
There is another method way to implement CF outside of the user-centric CF model described above. You can do an item-centric recommendation too. Rather than find a neighborhood of users with similar tastes, you can look at the specific items you've rated well and find a neighborhood of items with that share similar traits. If you're in a world where the number of items doesn't grow as fast as the number of users, this algorithm would scale much better.
The underlying assumption with collaborative filtering is that your preferences in the past will help predict your preferences in the future. Assuming that this is true, CF makes itself unique by finding users with similar preferences as you. Once this neighborhood of users is found, it can be reasonably assumed any items they like, you will also like, and hence, an intelligent recommendation can be made.
Using the NetFlix analogy, if you and another user named "Jack", who is completely unknown to you, have rated the same movies in the same way, it can be assumed that you have similar tastes. Therefore, if Jack rates a new movie very highly, the likelihood that you will also like that movie is very high, and it can be recommended.
There are issues with this algorithm, of course. First, you have to build up a reasonable list of preferences for the algorithm to make successful matches of other users. If you only rate 1 item ever, you can't expect great recommendations to come your way. Second, if your tastes change, the algorithm will be able to pick it up, but only with time. There must be time for the new change to affect the weight of your past selections. Last, this algorithm doesn't scale very well. For massive sites like NetFlix and Amazon, I'm sure there are some sort of caching or iterative methods involved so that the recommendations aren't recomputed everytime you login to the site. With millions of users and millions of products, it would simply take too long to load any sort of user-specific recommendation.
There is another method way to implement CF outside of the user-centric CF model described above. You can do an item-centric recommendation too. Rather than find a neighborhood of users with similar tastes, you can look at the specific items you've rated well and find a neighborhood of items with that share similar traits. If you're in a world where the number of items doesn't grow as fast as the number of users, this algorithm would scale much better.
Monday, March 19, 2007
Adobe Apollo Launches
Adobe Lab's project Apollo has officially launched today. It's an interesting project that lies in the realm of creating "Rich Internet Applications", a buzzword I've been seeing a lot lately.
Apollo is essentially a runtime, similar to Flash. What makes this new and different is that the Apollo technologies are focused on bringing web functionality back to the desktop. It tries to solve the online-offline data problem that people currently have with mail, contacts, news, etc. Gmail is a great application, but when I'm offline (on a place, perhaps), I can't write emails in Gmail to send later. If, however, there was a Desktop app that synced with Gmail when you were back online, that would be pretty sweet.
Apollo is a recognition that as wired as we are, and as popular as the web has become, some major functionality and synchronization issues were not addressed in the jump from the Desktop to the web. Some smooth transition apps that allow you to host your data online on central nodes, but then have that data available locally on distributed nodes, would be very nice to have.
Read more about it here.
Apollo is essentially a runtime, similar to Flash. What makes this new and different is that the Apollo technologies are focused on bringing web functionality back to the desktop. It tries to solve the online-offline data problem that people currently have with mail, contacts, news, etc. Gmail is a great application, but when I'm offline (on a place, perhaps), I can't write emails in Gmail to send later. If, however, there was a Desktop app that synced with Gmail when you were back online, that would be pretty sweet.
Apollo is a recognition that as wired as we are, and as popular as the web has become, some major functionality and synchronization issues were not addressed in the jump from the Desktop to the web. Some smooth transition apps that allow you to host your data online on central nodes, but then have that data available locally on distributed nodes, would be very nice to have.
Read more about it here.
Wednesday, March 14, 2007
Bash has a debugger
One of my biggest complaints with interpreted languages in general is that the debugging tools available are rather limited. When trying to find the root of a problem, having some sexy surgical tools available to cut into your program are nice. No, more than nice - sometimes they're necessary.
To my surprise, I discovered that Bash has a debugger! Bashdb, where have you been all my life? You can set breakpoints in your code, and step through the script line by line if desired. In terms of basic functionality, it's matches gdb fairly closely. Amazing!
I've used the PHP debugger XDebug before, but I found it to be a bit intrusive into the source code. You have to add PHP commands to the source to enable XDebug. I've never used Python or JS debuggers. Anyone know if good debuggers exist for those languages?
To my surprise, I discovered that Bash has a debugger! Bashdb, where have you been all my life? You can set breakpoints in your code, and step through the script line by line if desired. In terms of basic functionality, it's matches gdb fairly closely. Amazing!
I've used the PHP debugger XDebug before, but I found it to be a bit intrusive into the source code. You have to add PHP commands to the source to enable XDebug. I've never used Python or JS debuggers. Anyone know if good debuggers exist for those languages?
Wednesday, March 7, 2007
WiX: Windows Installer XML Toolset
While reading "The Build Master" this week, I was introduced to the tool that I find quite interesting: WiX. It's a MSFT produced and supported tool that has been open-sourced (gasp!). Open-source? Microsoft? Useful? My head hurts.
The WiX toolset allows you to create an MSI rather painlessly using an XML format that they've defined. It's actually quite intuitive and useful. The WISE and InstallShield guys have had the market on creating Windows Installers for the longest time. In addition, the horror stories I've heard from people who have the strength to use the WISE / InstallShield tools really caused me to do a double-take when I saw this tool.
In addition, there is no "setup" to speak of with this toolset. It's literally a set of binaries that you can execute from the command-line. There is no installation overhead, making these tools easy to store and version for build escrow purposes.
There is another open-source tool called Votive that ties in to Visual Studio and allows you to create and control the MSI build using WiX from within the GUI. As Borat would say, Niiice.
The WiX toolset allows you to create an MSI rather painlessly using an XML format that they've defined. It's actually quite intuitive and useful. The WISE and InstallShield guys have had the market on creating Windows Installers for the longest time. In addition, the horror stories I've heard from people who have the strength to use the WISE / InstallShield tools really caused me to do a double-take when I saw this tool.
In addition, there is no "setup" to speak of with this toolset. It's literally a set of binaries that you can execute from the command-line. There is no installation overhead, making these tools easy to store and version for build escrow purposes.
There is another open-source tool called Votive that ties in to Visual Studio and allows you to create and control the MSI build using WiX from within the GUI. As Borat would say, Niiice.
Sunday, March 4, 2007
The Comfort Zone
Steve Yegge had a really interesting blog post last week about the average programmer. Take a looksey:
Very true. He goes on to discuss how programmers are ranked relative to each other, since there is no quantitative method of ranking programmers, which in itself is completely subjective (and hence, his complaint). Programmers are ranked according to the context they are in.
His main complaint is about the comfort zone we all find ourselves in. He believes that since it is difficult to quantify the skill of a programmer, incentive is provided for pigeonholing yourself into an area you feel comfortable with.
I think this is true in most professions, not just programming, but I find this very true in testing as well. It is very easy to turn to the same tools and oracles to help with your problems and give advice.
Word. Hopefully I'll be that inspired and insightful after 16 or 17 years of working.
"But how do programmers compete? Generally, they just don't. Not in the way chess players or golfers compete, anyway. The reason? You can't compare programmers quantitatively, so you can't compute a score or a rank. Competitions and competitors have to be scored. Sure, you can set up scored programming competitions, but they're so tightly controlled that they don't resemble real-world software development anymore. Professional programmers basically just don't compete with each other.
Hence, you're probably not pushing yourself. Even if you're trying to improve your programming skills, you're probably just doing it in areas you're already comfortable in. And your improvements probably still aren't happening as fast as they would if you were competing to improve them."
Very true. He goes on to discuss how programmers are ranked relative to each other, since there is no quantitative method of ranking programmers, which in itself is completely subjective (and hence, his complaint). Programmers are ranked according to the context they are in.
His main complaint is about the comfort zone we all find ourselves in. He believes that since it is difficult to quantify the skill of a programmer, incentive is provided for pigeonholing yourself into an area you feel comfortable with.
I think this is true in most professions, not just programming, but I find this very true in testing as well. It is very easy to turn to the same tools and oracles to help with your problems and give advice.
"See, I always thought I was a perfectly competent programmer: as good as you can get, basically. I was building cool stuff, doing seemingly complicated things, and I felt I knew a tremendous amount of lore about the art of programming. I had won or placed in programming competitions, could program in Java for weeks on end without referring to the API docs, and pretty much felt on top of things.
Every few years, I would read some critical book, or have some weighty flash of insight, and realize that I'd been operating all this time in what could only be termed "clueless mode", and that I hadn't really known what I was doing after all. Amusingly, I was always relieved that now I could consider myself to be a good programmer, since I now knew whatever it was I'd been missing before.
Last year it finally dawned on me, after 16 or 17 years of this, that I just might possibly still be clueless about something important that I really ought to know, something that would make me a much better programmer."
Word. Hopefully I'll be that inspired and insightful after 16 or 17 years of working.
Monday, February 19, 2007
Test Personality Traits
I've begun to wonder what affect personality traits have in testing. The main question I pose to you is: are there specific personality traits that one should look for in good testers? This isn't to say that there aren't people who are excellent testers who have personality traits that aren't amiable to testing, but I'm looking for some generic pattern.
Take cynicism. Cynical people could be described as untrusting. I think it could be argued that this is a good trait to have in a tester. If by nature one does not trust the very inputs, outputs, data flows, and processes of a system system under test, it is possible that one could find more problems, assuming that there is proper motivation to do so.
On the opposite side of the spectrum, what about optimists? If you by nature are a very trusting person, would you unknowingly overlook problems due to your sunny outlook? Or is it possible to seperate your innate optimism from the technical job at hand?
You can probably already read my biases in the writing above, but I'm curious what the world at hand thinks. I'm not suggesting that hiring people should be based on personality traits, especially since I think it is impossible to get a good feel for someone's personality in 4-5 hours; but perhaps, just maybe, it should be taken into account.
Take cynicism. Cynical people could be described as untrusting. I think it could be argued that this is a good trait to have in a tester. If by nature one does not trust the very inputs, outputs, data flows, and processes of a system system under test, it is possible that one could find more problems, assuming that there is proper motivation to do so.
On the opposite side of the spectrum, what about optimists? If you by nature are a very trusting person, would you unknowingly overlook problems due to your sunny outlook? Or is it possible to seperate your innate optimism from the technical job at hand?
You can probably already read my biases in the writing above, but I'm curious what the world at hand thinks. I'm not suggesting that hiring people should be based on personality traits, especially since I think it is impossible to get a good feel for someone's personality in 4-5 hours; but perhaps, just maybe, it should be taken into account.
The Brotherhood of Building and Testing
In the past few weeks, I've been embedded in the world of build engineering and configuration management. It has been an eye-opening experience, believe me. There are many splinters and specializations in the software industry, and the area of build engineering is one that I've never really paid attention to until recently. From what I've seen thus far, there are far more connections between building and testing than I've previously known.
There is a web of trust in the process of building any project that one takes a bit for granted, and indeed one must take it for granted. It's impossible to know every detail that every responsibility entails, so you must take for granted that the program manager is doing his job, and the product manager's requirements have been adequately reviewed and vetted out. The one area where the web of trust stops is between developer and testers. The testers are present to be cynical, and in general not trust the output of the developers. This ensures more quality in the software so that the users are not the ones consuming poor products.
I've recently started to think that the web of trust should be breaking down in another place: building. I would say that most engineers take the output of the build engineer a bit for granted. You assume that the build is an instrumented build because the build engineer says so. You assume that the build has the correct fixes in it because the build engineer says so. Well what happens if the build engineer is wrong? Having the correct build is the first step in the testing process, and depending on where you are in the life cycle, the main consumer of the build will be the testers.
I don't think it's too far-fetched to say that a new specialization should open up: build test engineer. The process of building can be complicated enough, depending on the size of your project, and can be a full-time job by itself. Adding value on top of that could be another full-time job. The value I'm referring to is build testing, build metrics, and build consumption.
Just as testers are considered a trusted third-party that can tell you with certainty (i.e., data) the risks and issues found in a product, a build test engineer would be a trusted third-party who could tell you with certainty that the build has or contains the attributes it says it does.
There is a web of trust in the process of building any project that one takes a bit for granted, and indeed one must take it for granted. It's impossible to know every detail that every responsibility entails, so you must take for granted that the program manager is doing his job, and the product manager's requirements have been adequately reviewed and vetted out. The one area where the web of trust stops is between developer and testers. The testers are present to be cynical, and in general not trust the output of the developers. This ensures more quality in the software so that the users are not the ones consuming poor products.
I've recently started to think that the web of trust should be breaking down in another place: building. I would say that most engineers take the output of the build engineer a bit for granted. You assume that the build is an instrumented build because the build engineer says so. You assume that the build has the correct fixes in it because the build engineer says so. Well what happens if the build engineer is wrong? Having the correct build is the first step in the testing process, and depending on where you are in the life cycle, the main consumer of the build will be the testers.
I don't think it's too far-fetched to say that a new specialization should open up: build test engineer. The process of building can be complicated enough, depending on the size of your project, and can be a full-time job by itself. Adding value on top of that could be another full-time job. The value I'm referring to is build testing, build metrics, and build consumption.
Just as testers are considered a trusted third-party that can tell you with certainty (i.e., data) the risks and issues found in a product, a build test engineer would be a trusted third-party who could tell you with certainty that the build has or contains the attributes it says it does.
Thursday, February 15, 2007
Nerdiest. Comic. Ever.
This is a bit off topic, but xkcd had quite possibly the nerdiest comic ever tonight, and I thought I should share with you all. It involves Lisp, God, and Perl.
http://xkcd.com/c224.html
Amazing.
http://xkcd.com/c224.html
Amazing.
Wednesday, February 7, 2007
The Desire to Build
My girlfriend and I had an interesting conversation last week about how so many test engineers feel the need to create and recreate tools upon more tools, even if a tool already exists that does what they need to do. This is a situation that has its own acronym: YATT (yet another test tool).
It brings up an interesting question: does this desire come from the very core of what it means to be an engineer, to build and create; or is this possibly something more? Is it possible that this desire stems from deeper feelings of inadequacies at not being a "developer", at least in title? When you need a test tool to perform an action, and there are twenty tools out there already that do this, why should one create another tool?
It's true that some stigma remains around the role of the tester. When people ask what I do, I feel the need to stress that I'm an engineer, but in the testing field. There are times at work when I feel I should be creating something in order to be taken seriously by my fellow engineers. I feel this is true, at least in part. Doing your job, and doing it well, is an exceptional way of getting respect, but there is nothing like showing off something cool to get instant cred. It's like we're in 8th grade again.
Which brings up an interesting conclusion that we came to in our coversation. The testing field needs to a mature a little before such problems will be alleviated. I think that when testing is taken as a serious discipline across the industry, testers will stop trying to be developers and start trying to be testers.
It brings up an interesting question: does this desire come from the very core of what it means to be an engineer, to build and create; or is this possibly something more? Is it possible that this desire stems from deeper feelings of inadequacies at not being a "developer", at least in title? When you need a test tool to perform an action, and there are twenty tools out there already that do this, why should one create another tool?
It's true that some stigma remains around the role of the tester. When people ask what I do, I feel the need to stress that I'm an engineer, but in the testing field. There are times at work when I feel I should be creating something in order to be taken seriously by my fellow engineers. I feel this is true, at least in part. Doing your job, and doing it well, is an exceptional way of getting respect, but there is nothing like showing off something cool to get instant cred. It's like we're in 8th grade again.
Which brings up an interesting conclusion that we came to in our coversation. The testing field needs to a mature a little before such problems will be alleviated. I think that when testing is taken as a serious discipline across the industry, testers will stop trying to be developers and start trying to be testers.
Tuesday, January 23, 2007
Ethical Hacking
A common question I get from people is, "What do you like about QA?". Actually, it's also a common question I ask people during interviews, but I digress. It's a question I've thought about much during my time in the industry, and I have a few answers, but my favorite one is that I consider testing to be a form of ethical hacking.
Putting the negative connotations aside, hacking (in my opinion) is understanding a system so thoroughly that you can discover weaknesses to exploit in order to gain some sort of advantage. These advantages could be an escalation of privileges, using the system for ways not intended or devised, or simply disturbing the data in the system. Not to be too picky, but the actual term for this is "cracking". The idea of hacking extends well beyond computers, as many of you know I'm sure. Using the correct combination of coupons to get a good deal at Best Buy (as seen frequently on Techbargains) is using the system in a way that wasn't intended, and is a form of hacking.
Since hacking unfortunately already has negative connotations with it, I like to use the phrase "ethical hacking". As part of my job, I must understand a system well enough to exploit it. Truly there are an infinite number of combinations of actions that one can try to exploit a system. As members of a larger system, one must understand more than just the application under test. You must also have knowledge of the underlying operating system(s) it may be used on, any network protocols that may be in use, the pitfalls of the underlying architecture, the caveats of the language it was written in, etc., etc. It is the understanding of the external world housing the application that one must learn, and in that, there is so much to learn. Which keeps life interesting.
Putting the negative connotations aside, hacking (in my opinion) is understanding a system so thoroughly that you can discover weaknesses to exploit in order to gain some sort of advantage. These advantages could be an escalation of privileges, using the system for ways not intended or devised, or simply disturbing the data in the system. Not to be too picky, but the actual term for this is "cracking". The idea of hacking extends well beyond computers, as many of you know I'm sure. Using the correct combination of coupons to get a good deal at Best Buy (as seen frequently on Techbargains) is using the system in a way that wasn't intended, and is a form of hacking.
Since hacking unfortunately already has negative connotations with it, I like to use the phrase "ethical hacking". As part of my job, I must understand a system well enough to exploit it. Truly there are an infinite number of combinations of actions that one can try to exploit a system. As members of a larger system, one must understand more than just the application under test. You must also have knowledge of the underlying operating system(s) it may be used on, any network protocols that may be in use, the pitfalls of the underlying architecture, the caveats of the language it was written in, etc., etc. It is the understanding of the external world housing the application that one must learn, and in that, there is so much to learn. Which keeps life interesting.
Friday, January 19, 2007
Inducing Low Memory Conditions
Testing under stress and load conditions are important for every application. Having empirical knowledge of what the breaking point is for any computer hosting an application helps you to plan for such an occaision, as well as giving you an idea of what sort of load the machine can handle before it buckles. If you're aware that having 1000 simultaneous users on a web application you've built is the down-hill tipping point for failure, and you're seeing peak usage at 1200 users a day, you know that it's time to either get another machine or tweak the code to allow for more users on the machine.
Recently I did some searching for a program that would induce low memory conditions on my linux box. I wanted to do some stress testing on a web app while the machine was low in memory. Unfortunately I couldn't find a decent program, so I wrote one! Since I've found that "tutorial" sites are very popular (I visit them myself daily), I thought posting this might be good for the blog. And what's good for the blog, is good for Ryan.
Recently I did some searching for a program that would induce low memory conditions on my linux box. I wanted to do some stress testing on a web app while the machine was low in memory. Unfortunately I couldn't find a decent program, so I wrote one! Since I've found that "tutorial" sites are very popular (I visit them myself daily), I thought posting this might be good for the blog. And what's good for the blog, is good for Ryan.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define MEGABYTE 1024*1024
#define LOWMEMORY 30000
/*
* SUMMARY: Induce low memory conditions by allocating memory in 1 MB chunks and checking free memory until there is only LOWMEMORY KB of free memory left
* REQUIREMENTS: Must be run on a Linux machine, and the LOWMEMORY number should be in KB
* AUTHOR: Ryan Gerard, ryan dot gerard at gmail dot com
*/
int main(int argc, char *argv[]) {
void *myblock = NULL;
int count = 0;
int freeMem = 0;
//Allocate memory in 1 MB chunks as long as the free mem can be found and the free mem is greater than LOWMEMORY KB
while( (freeMem = getFreeMem()) != -1 && freeMem > LOWMEMORY)
{
myblock = (void *) malloc(MEGABYTE);
if (!myblock) break;
memset(myblock,1, MEGABYTE);
printf("Currently allocating %d MB\n",++count);
printf("Free mem is %d KB\n", freeMem);
}
//Print out amount of free memory left
printf("%d free memory left\n", getFreeMem());
exit(0);
}
/*
* Parse the output of 'vmstat' to determine the amount of free memory left on the system
*/
int getFreeMem()
{
FILE *readFile;
char string[256];
int cnt = 0;
int freeMem = -1;
//Output memory information to file tmpfile
system("vmstat -a 1 1 > tmpfile");
//Open the file for reading
if ( (readFile = fopen("tmpfile", "r")) != NULL )
{
//Get the third line of text from the file
while( fgets(string, 256, readFile) && cnt < 2 )
cnt++
cnt = 0;
char* token = strtok(string, " ");
while(token != NULL && cnt < 3)
{
token = strtok(NULL, " ");
cnt++;
}
//Convert to an integer to return
freeMem = atoi(token);
//Close the file
fclose(readFile);
}
return freeMem;
}
Monday, January 15, 2007
A Timeout for Amazon
I wanted to take a timeout from testing to discuss an area I find interesting: the web services platforms being built by Amazon. I am, of course, referring to S3, EC2, SQS, and Mechanical Turk. I was reminded of these services by Marc Hedlund's post on the O'Reilly Radar today.
The game is afoot in the race to develop the next best thing, and Amazon has released these rather interesting technologies that enable developers to build that next best thing cheaply. Jeff Bezos strikes again with services that leverage the idle parts of Amazon's computing infrastructure and provides it to the people at extremely reasonable prices. Here is a quick overview of the various technologies described above.
Mechanical Turk
This web service is in reality a masked human-computer interface. Instead of humans asking computers to perform actions, computers are asking humans to perform actions, and using the results in whatever computations are being performed. The entire developing area of funnelling work off that is easy for humans to do but hard for computers (visual pattern recognition, reading obscured text, etc) is interesting, and efficient, assuming you can assemble people to actually do the work.
S3: Simple Storage Service
This is truly a simple web service that allows you to store and retrieve any amount of data from the Amazon infrastructure, and you only pay for what you use.
This is a very simple idea, and easy to grasp the significance. Storage is cheap, so make is available anywhere through standardized methods. The first obvious idea around this technology, online backup, has been done by a few companies already. The cost, you ask? $0.15 per GB-month of storage used, and $0.20 per GB of data transferred.
E2: Elastic Computing Cloud
E2 is to computing what S3 is to storage. E2 provides a web service that allows you to increase your computing capacity on the fly.
The first obvious idea I can see with this is reducing the Digg /Slasdot / reddit effect on smaller sites. I hate clicking on a link from one of those sites to only find out that it maxed out its bandwidth limits two hours ago.
Virtualizing all computer resources like this is quite interesting - I'd be interested in reading a story by someone who completely runs a site from Amazon's infrastructure. Requests made to the domain would use the EC2 computing cloud to retrieve the page requested from the S3 storage service. Traffic would be monitored, and in the case of a statistical spike, more computer resources could be called within minutes to serve your new readers.
Again, the cost is very reasonable. $0.10 per instance-hour consumed, $0.20 per GB of data transferred, and any associated S3 costs.
SQS: Simple Queue Service
SQS is a giant distributed queue using Amazon's messaging infrastructure. This service-based asynchronous message technology is a little less obvious in terms of realistic functionality, but very geeky, and hence, very interesting. It reminds me of when Google released Google Sets. My first thought was, "What sane person would use this?", and my second thought was, "I wish I could think of a reason to use this".
The game is afoot in the race to develop the next best thing, and Amazon has released these rather interesting technologies that enable developers to build that next best thing cheaply. Jeff Bezos strikes again with services that leverage the idle parts of Amazon's computing infrastructure and provides it to the people at extremely reasonable prices. Here is a quick overview of the various technologies described above.
Mechanical Turk
"Developers use the Amazon Mechanical Turk web services API to submit tasks to the Amazon Mechanical Turk web site, approve completed tasks, and incorporate the answers into their software applications."
This web service is in reality a masked human-computer interface. Instead of humans asking computers to perform actions, computers are asking humans to perform actions, and using the results in whatever computations are being performed. The entire developing area of funnelling work off that is easy for humans to do but hard for computers (visual pattern recognition, reading obscured text, etc) is interesting, and efficient, assuming you can assemble people to actually do the work.
S3: Simple Storage Service
This is truly a simple web service that allows you to store and retrieve any amount of data from the Amazon infrastructure, and you only pay for what you use.
"It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites."
This is a very simple idea, and easy to grasp the significance. Storage is cheap, so make is available anywhere through standardized methods. The first obvious idea around this technology, online backup, has been done by a few companies already. The cost, you ask? $0.15 per GB-month of storage used, and $0.20 per GB of data transferred.
E2: Elastic Computing Cloud
E2 is to computing what S3 is to storage. E2 provides a web service that allows you to increase your computing capacity on the fly.
"Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change."
The first obvious idea I can see with this is reducing the Digg /Slasdot / reddit effect on smaller sites. I hate clicking on a link from one of those sites to only find out that it maxed out its bandwidth limits two hours ago.
Virtualizing all computer resources like this is quite interesting - I'd be interested in reading a story by someone who completely runs a site from Amazon's infrastructure. Requests made to the domain would use the EC2 computing cloud to retrieve the page requested from the S3 storage service. Traffic would be monitored, and in the case of a statistical spike, more computer resources could be called within minutes to serve your new readers.
Again, the cost is very reasonable. $0.10 per instance-hour consumed, $0.20 per GB of data transferred, and any associated S3 costs.
SQS: Simple Queue Service
"Amazon SQS offers a reliable, highly scalable hosted queue for storing messages as they travel between computers."
SQS is a giant distributed queue using Amazon's messaging infrastructure. This service-based asynchronous message technology is a little less obvious in terms of realistic functionality, but very geeky, and hence, very interesting. It reminds me of when Google released Google Sets. My first thought was, "What sane person would use this?", and my second thought was, "I wish I could think of a reason to use this".
Friday, January 12, 2007
Extreme Testing
I'm sure many to most of you are familiar with the principles of extreme programming. It can be quickly characterized by writing unit tests before coding and pair programming, but you can read more about it from this webpage.
I was thinking on the way home today about possibly modifying the rules of engagement into something called Extreme Testing. It would focus more on the relationship of the two people in the pair. In particular, one of the two would be a QA engineer. They would work closely together in the planning and designing phase - not much would change there. During the coding phase, the two would spend half their time together in a modified pair programming mode; most likely the main developer would do most of the coding, and the QA person would be there making recommendations and understanding what's going on. The other half of the time would be spent on their own: the developer doing his thing, and the QA person developing test plans, documenting the design, and creating process flows and timing analyses. Then in the end there is a big celebration with much rejoicing.
Why would one think of such a scheme? What is the possible value of this to you? A more effective method of knowledge transfer. Very loosely, here is how the software development process works today: the developer designs the component and hands you a design doc. That developer then builds his component, and hands you (the QA person) a finished product to test. Meanwhile, you must learn the intricacies of this component on your own, for the most part. You have a design document, and checked it code to sift through to understand the inner magic.
Being in QA, understanding the component better than the author is your quest. You must know it so well that you can break it with your left pinky. However, the process of transferring that understanding and knowledge is somewhat broken. Here is the current process: developer's head --> design document --> QA's head. Stuff is lost inbetween, I assure you. Developers stereotypically dislike documenting their component, making the knowledge transfer process more difficult. Seriously, I love a well-written design doc. It's like crack-cocaine for me. I would mainline a well-written design doc if I could only fit it on that little spoon.
So what is the solution? Well how about some extreme testing? Working side by side with the developer while he's working, and documenting the component and processes involved are a better way in my opinion of guaranteeing that the knowledge is tranferred more effectively.
I was thinking on the way home today about possibly modifying the rules of engagement into something called Extreme Testing. It would focus more on the relationship of the two people in the pair. In particular, one of the two would be a QA engineer. They would work closely together in the planning and designing phase - not much would change there. During the coding phase, the two would spend half their time together in a modified pair programming mode; most likely the main developer would do most of the coding, and the QA person would be there making recommendations and understanding what's going on. The other half of the time would be spent on their own: the developer doing his thing, and the QA person developing test plans, documenting the design, and creating process flows and timing analyses. Then in the end there is a big celebration with much rejoicing.
Why would one think of such a scheme? What is the possible value of this to you? A more effective method of knowledge transfer. Very loosely, here is how the software development process works today: the developer designs the component and hands you a design doc. That developer then builds his component, and hands you (the QA person) a finished product to test. Meanwhile, you must learn the intricacies of this component on your own, for the most part. You have a design document, and checked it code to sift through to understand the inner magic.
Being in QA, understanding the component better than the author is your quest. You must know it so well that you can break it with your left pinky. However, the process of transferring that understanding and knowledge is somewhat broken. Here is the current process: developer's head --> design document --> QA's head. Stuff is lost inbetween, I assure you. Developers stereotypically dislike documenting their component, making the knowledge transfer process more difficult. Seriously, I love a well-written design doc. It's like crack-cocaine for me. I would mainline a well-written design doc if I could only fit it on that little spoon.
So what is the solution? Well how about some extreme testing? Working side by side with the developer while he's working, and documenting the component and processes involved are a better way in my opinion of guaranteeing that the knowledge is tranferred more effectively.
Tuesday, January 9, 2007
Quality Problem Solving
All good engineers are at their roots good problems solvers. Despite the fact that my degree is in computer science, I feel that I should have received a bachelor's in "Technical Problem Solving". This may be obvious to some of those reading this blog, but this point was definitely not emphasized going through school. Until you're pretty deep into the program, you are led to believe that this education is in how to program, and nothing else. I'll have to write later about the issues I have with computer science education before I digress any further, as this post is about a book.
I've started a book that has turned out to be quite interesting. The book is called Quality Problem Solving by Gerald Smith. Now before you laugh at the fact that I found this interesting, hear me out. Being that I'm a member of the QA community, it is problems of quality that currently concern me. I find the book to be very realistic in its view of quality and problem solving in general, without any hokey aphorism's or a-b-c methods of solving problems. For instance, take this gem:
Gerald starts the book with a very sober view of the word "quality" and how it has been manipulated and had its meaning raided by corporate vandals, citing TQM and other quality fads frequently. He rightly states that these movements had good principles and ideas behind them, and if used in the right context were very useful. However many were applied to solve every problem known to man and beast, and hence, fell out of favor after failure in those contexts.
He continues on to lay down a philosophical yet clear foundation for solving problems, discussing problem identification, problem definition, diagnosis, and alternative generation. He discusses these topics in general terms, and provides good examples for each.
The last part of the book applies the problem solving foundations to specific problems related to quality. I can't evaluate that part of the book, as I have not yet read it, but it looks promising. The TOC lists the topics of conformance, efficiency, product design, process design, and unstructured performance problems.
Basically what I'm saying is this book is good, and I recommend it.
I've started a book that has turned out to be quite interesting. The book is called Quality Problem Solving by Gerald Smith. Now before you laugh at the fact that I found this interesting, hear me out. Being that I'm a member of the QA community, it is problems of quality that currently concern me. I find the book to be very realistic in its view of quality and problem solving in general, without any hokey aphorism's or a-b-c methods of solving problems. For instance, take this gem:
"The most common weakness in practical reasoning, as in problem solving, is incompleteness. Poor outcomes result not so much from the mistakes we make as from the possibilities we overlook."
Gerald starts the book with a very sober view of the word "quality" and how it has been manipulated and had its meaning raided by corporate vandals, citing TQM and other quality fads frequently. He rightly states that these movements had good principles and ideas behind them, and if used in the right context were very useful. However many were applied to solve every problem known to man and beast, and hence, fell out of favor after failure in those contexts.
He continues on to lay down a philosophical yet clear foundation for solving problems, discussing problem identification, problem definition, diagnosis, and alternative generation. He discusses these topics in general terms, and provides good examples for each.
The last part of the book applies the problem solving foundations to specific problems related to quality. I can't evaluate that part of the book, as I have not yet read it, but it looks promising. The TOC lists the topics of conformance, efficiency, product design, process design, and unstructured performance problems.
Basically what I'm saying is this book is good, and I recommend it.
Saturday, January 6, 2007
More learning materials
Continuing with the last past of highlighting good learning materials, I saw on Michael Howard's blog a post about online security sessions from the Microsoft TechEd IT Forum. It looks like pretty awesome material. There are videos profiling "real" hacking techniques live, Vista UAC internals, and Vista kernel changes.
For anyone in QA that will be working on Vista, these look like good resources.
For anyone in QA that will be working on Vista, these look like good resources.
Google London Test Automation Conference Video's
I realize that this is old news (September 2006), but I wanted to remind people that there are some fantastic videos available through Google Video. Specifically, the London Test Automation Conference video's being hosted. Some great stuff is covered if you have the time, including a video on Selenium (a programmatic cross-browser web GUI test tool), and distributed testing with SmartFrog.
These videos are highly recommended if you have the time and inclination.
These videos are highly recommended if you have the time and inclination.
Wednesday, January 3, 2007
Exploratory Eating
My girlfriend and I sometimes perform these food experiments. For one month, we do or eat something differently just to see what happens. For instance, we were vegetarians for one month. This month of January, we're cutting all sweets and desserts out of our diet.
I guess these are more like body experiments. The point is just to see what happens when we change our diet. You can call it "exploratory eating". Do we feel more energized? Healthier? More tired? Is there no change?
The point I'm trying to make is that the testing mindset can (and should) be applied outside of software. Applying your skills in testing on your finances, health, and mental well-being could be a interesting (and possibly rewarding) experience.
I guess these are more like body experiments. The point is just to see what happens when we change our diet. You can call it "exploratory eating". Do we feel more energized? Healthier? More tired? Is there no change?
The point I'm trying to make is that the testing mindset can (and should) be applied outside of software. Applying your skills in testing on your finances, health, and mental well-being could be a interesting (and possibly rewarding) experience.
Subscribe to:
Posts (Atom)