Requirements gathering techniques include the easy to send, but sometimes hard to develop, survey method to obtain data from a wide variety of people located anywhere. Surveys, however, are notorious for many faults such as ambiguity and a lack of response.
But surveys can produce a large volume of information for the gathering parties to peruse and collate, so developing good surveys is important for both the respondents who have to understand the questions and for the collators to get useful data.
Statistics Prove “IT”
It has been said you can prove anything with statistics if you ask the right question. Obviously, you need to ask the right questions to get the right data. Right?
Reality can be different. The interpretation of an ambiguous question, an open-ended question, a leading question, or a question made up of words most people do not understand, can be very detrimental to getting the answers you need.
For instance, if the requirements were related to virtualization of your datacenter equipment, you should avoid being too general or assume too much knowledge on the part of the respondent.
Ambiguous Question examples:
How does it help?
Who can use this?
Open Ended Question examples:
Why do you need this feature?
What services would help you?
Leading Question examples:
If this service were to fail, who would be affected?
Without web based access to this device, what can be done?
Who amongst your group has the necessary required certifications?
The “Right” Questions
So what is the right question to ask? You do not want to insult your respondent nor do you want to assume they understand the situation or concepts entirely.
How can you ask the right question? Ask an eight year old. Or rather, pretend the audience is eight years old. Assume they do NOT have a vast knowledge of the topic, a slanted knowledge of the world, or possibly a jaded view of the system. Use simple language that is less likely to be interpreted differently.
Provide respondents with some background information or assumptions at the beginning of the survey, or point them to relevant documents or websites they can use to clarify the concepts and issues.
Avoiding Acronyms and Industry Jargon
Often times, in the technical world, we have developed our own jargon or acronyms for the myriad technologies, services, applications, and such that we developed and/or have to deal with every day. For instance, SMB is one acronym that has vastly different meanings depending on your perspective. A recent book had this acronym within the title, and it was very confusing for the network specialist who later found out the book was meant for a general business type audience. What does it mean to you? In the book context, it was Small- to-Medium Business, to the technician, Server Message Block. To my kids on their instant messaging service, it meant So Much Better.
The standard axiom of KISS (or, Keep It Simple, Stupid), applies for questions – keep them short and simple but clearly to the point. In some cases, a preliminary paragraph stating the base assumptions will help to clarify the premise for the questions.
In Requirements Gathering
A common method of gathering requirements from many stakeholders, when interviewing or workshops are not prudent, is to send a general topic survey of questions in the early phase of discovery. These general questions should lead toward more specific questions as the discovery phase progresses. This is when specific questions become more critical. The questions must be very topical or delivery-oriented, maybe even very technical in nature, and yet they must ensure the respondent clearly understands both the premise and concepts.
For instance, a general trend today is toward using virtualization. Most people have no idea what this really means or even implies. To make your survey results more coherent, you may need to include references to websites, local or external, that explain the concepts. Have multiple people within your organization peruse these materials to assess their usefulness, and employ a wide cross-section of people to gauge if the content is technical in nature. You should not rely exclusively on technicians for feedback.
Survey Says …
The right questions on a survey may not always be enough. In some cases, the options to respond and/or the choice of responses can affect your survey results.
Fixed answers, such as in multiple choice, should give the reader plenty of leeway for their decision. Be as wordy as necessary for each choice, without too many overlapping or confusing variations of the same request.
Assumptions
Surveys have to assume some level of knowledge for the respondent, but that is not always achievable when dealing within different technical or managerial staffing levels.
For instance, the following question assumes a great deal of knowledge on the part of the respondent:
Q. How would your department benefit from using virtualization?
This is a rather open-ended question requiring the respondent to be very familiar with the topic and critical about the lack of virtualization use for a long time. To improve this question, provide some options that will make sense rather than have the respondent come up with all the answers:
Q. In an effort to reduce our data-center footprint, the company has been looking into the possibility of moving toward the use of specialized hardware that maintains the separate host services on a centralized, redundant virtual computing platform. Each current host that meets all the criteria (see website //xxx/criteria for details) for being virtualized will be transitioned to the virtual environment. How would your department benefit from using virtualization?
Redundancy [ ] Backups [ ] Ease of recovery [ ] Availability [ ]
Reduce licensing requirements [ ] Reduce number of physical boxes [ ]
Reduce management overhead [ ] Reduce resources [ ]
Expand service levels [ ] Ease of testing [ ] Ease of deployment [ ]
This may be too limiting, instead provide a range of responses
Q. [same intro] How would your department benefit from using virtualization?
(Scaled response with 1 being the least important and 5 being the most important)
This might be followed-up later with questions that specifically define some of the key attributes for switching to a virtualized platform or not.
Q. Select all that apply to your server:
Server provides: [ ] file [ ] print [ ] dns [ ] AD [ ] other _____
Comments: _________
Service must be: [ ] kept separate [ ] own IP [ ] behind firewall
Comments: _________
Setup requires: [ ] Win2K [ ] Win2K3 [ ] Win2K8 [ ] XP [ ] Linux
Comments: _________
Survey Choice Option Default Settings
Surveys often provide a scale for their responses. Some include as few as three options, while others may have 10 or more within the range. There is no best answer, although five to eight choices are probably more than adequate for most ranges.
Imagine a now defunct restaurant asked you to fill out their web-based survey as you left the restaurant. They failed, however, to put an easy-to-find link to the survey on their website. After searching the entire main page, and a few subsequent pages, a small, bottom menu link was eventually found. Within the survey web page, there was a simple explanation of their scale being from 1 to 5, for the 10 questions presented. Each question ended with the same simple drop down list. The default drop down value was preset to 1 for all 10 questions. This may have been planned or not, but it might produce a lot of low results.
Common survey choice ranges include:
“Excellent, Good, Fair, Poor, N/A”
“Strongly Agree, Agree, neutral, Disagree, Strongly Disagree”
“1=Very Bad to 5=Excellent”
“1 is worst, and 10 is best”
Clearly, each of these ranges requires questions that would elicit such a response from the expected target audience. However, do not expect that you will get the top-level choice. How often are you told you are doing an “excellent” job as apposed to a “good” job? Probably not very often. So why would you expect the general public to give a 5 or 10, an Excellent or Best response? Why would you then assume anything else is unfavorable if you have not clearly defined to the respondent what each choice represents?
Do Ranges Reflect Expectations?
The option “Good” could mean many things to many people such as: good, good enough, acceptable, expected norm, at least as good as you expect, etc. This is especially true when the “Excellent” response is the only one good enough. If anything other than “Excellent” is not good enough, ensure the respondent knows that they have to explain why they did not select “Excellent.”
In many hotels, you are given a document requesting that you call the local manager if there is anything that is not to your liking at any time. They want to provide you with a “10 out of 10” experience every visit. They conveniently provide a survey form for when you leave. What do you think happens if you do not fill out the survey for your stay? Perhaps an automatic “10” for that visit?
In some local grocery and department stores, they now have “Outstanding Service” forms you can fill out to honor anyone who has gone out of their way to provide exemplary service for you. No mention of where to enter complaints.
Can you remember your last few visits to a restaurant and a large department store? What was your experience? What was your expectation? What would you expect to be asked? Which range would you expect for the answers?
Surveys Everywhere
Surveys are now ubiquitous throughout the world and we have developed a love-hate relationship. We love to send them but hate to respond to them. We, and this is a collective “we,” tend to ignore most surveys unless there is something in it for us. There are companies on the Internet that offer rewards for filling out surveys from their clients.
Many years ago, a former boss had a great philosophy about surveys done by large groups of customers. He would sort them according to rank, from best to worst, and then ignore the first and last to get the average. He assumed that there was always one in the crowd who had a weird idea of what “good” meant, and he balanced this against those who just quickly gave a general “good” or “excellent.”
There are lots of ways to interpret survey results – many of them bad!
The professional sports teams have all sorts of methods, or “improvements,” to make their stats look good. There are rules for how stats are accepted and interpreted. They usually keep a running total, not just the single events. Keep this in mind when you are working on your results. Taken in small groupings, there can be wide swings in averages. Taken in gross, these averages of averages are more consistent.
In Summary
The usefulness of a survey is in the responses that are received. Be as helpful as possible to the respondents by ensuring they have sufficient information and knowledge to complete the survey. Strive to provide a reasonable and clearly stated range if used. Restrict the space for open-ended questions and keep the questions as simple and succinct as possible, so that anyone can understand each question. Adding an incentive for respondents to complete the survey quickly may also speed up the process. Surveys can take a long time to prepare, collect, and collate but with careful planning, a well-executed survey can simplify the process of gathering requirements.
David Egan is a Course Director and developer of content for more than 20 Global Knowledge courses. David has more than 20 years of teaching experience including more than five years of teaching ‘on-line’ virtual classes over the Internet using Centra, Interwise and iLinc virtual classroom services. David has also written technical books and delivered adult learning seminars on UNIX, Linux, Microsoft Operating Systems and Services, as well as Business Process Analysis and Project Management since 2005. David and his family live in a suburb of Vancouver, B.C. This article was originally published in Global Knowledge’s Management in Motion e-newsletter, Business Brief. Global Knowledge (www.globalknowledge.com) delivers comprehensive hands-on project management, business analysis, ITIL, and professional skills training.
Copyright © Global Knowledge Training LLC. All rights reserved.
