QA Software Testing Blog

A formal test plan is a document that provides and records important information about a test project. Read more about this…

A test plan records important information about a test project like for example:

1. Project assumptions
2. Project background information
3. Available resources
4. Project Schedule
5. Entry and exit criteria
6. Test milestones
7. Use cases and/or test cases

For a range of reasons — both good and bad — many software and web development projects don’t budget enough time for complete and comprehensive testing. A quality test team must be able to test a product or system quickly and constructively in order to provide some value to the project. This essay describes how to test a web site or application in the absence of a detailed test plan and facing short or unreasonable deadlines.

Identify High-Level Functions First
High-level functions are those functions that are most important to the central purpose(s) of the site or application. A test plan would typically provide a breakdown of an application’s functional groups as defined by the developers; for example, the functional groups of a commerce web site might be defined as shopping cart application, address book, registration/user information, order submission, search, and o­nline customer service chat. If this site’s purpose is to sell goods o­nline, then you have a quick-and-dirty prioritization of:

1. Shopping cart – credit card validation and security.
2. Registration/user information
3. Taking Orders
4. Search the site
5. Online customer service like chat, email etc

I’ve prioritized these functions according to their significance to a user’s ability to complete a transaction. I’ve ignored some of the lower-level functions for now, such as the modify shopping cart quantity and edit saved address functions because they are a little less important than the higher-level functions from a test point-of-view at the beginning of testing.
Your opinion of the prioritization may disagree with mine, but the point here is that time is critical and in the absence of defined priorities in a test plan, you must test something now. You will make mistakes, and you will find yourself making changes o­nce testing has started, but you need to determine your test direction as soon as possible.

Test Functions Before Display
Any web site should be tested for cross-browser and cross-platform compatibility — this is a primary rule of web site quality assurance. However, wait o­n the compatibility testing until after the site can be verified to just plain work. Test the site’s functionality using a browser/OS/platform that is expected to work correctly — use what the designers and coders use to review their work.

Concentrate o­n Ideal User Actions First
Ideal User Actions are those actions and steps most likely to be performed by users. For example, o­n a typical commerce site, a user is likely to

1. identify an item of interest
2. add that item to the shopping cart
3. buy it o­nline with a credit card
4. ship it to himself/herself

Now, this describes what the user would want to do, but many sites require a few more functions, so the user must go through some more steps, for example:

1. login to an existing registration account (if o­ne exists)
2. register as a user if no account exists
3. provide billing & bill-to address information
4. provide ship-to address information
5. provide shipping & shipping method information
6. provide payment information
7. agree or disagree to receiving site emails and newsletters

Most sites offer (or force) an even wider range of actions o­n the user:

1. change product quantity in the shopping cart
2. remove product from shopping cart
3. edit user information (or ship-to information or bill-to information)
4. save default information (like default shipping preferences or credit card information)

All of these actions and steps may be important to some users some of the time (and some developers and marketers all of the time), but the majority of users will not use every function every time. Focus o­n the ideal path and identify those factors most likely to be used in a majority of user interactions.

Concentrate o­n Intrinsic Factors First
Intrinsic factors are those factors or characteristics that are part of the system or product being tested. An intrinsic factor is an internal factor. So, for a typical commerce site, the HTML page code that the browser uses to display the shopping cart pages is intrinsic to the site: change the page code and the site itself is changed. The code logic called by a submit button is intrinsic to the site.
Extrinsic factors are external to the site or application. Your crappy computer with o­nly 8 megs of RAM is extrinsic to the site, so your home computer can crash without affecting the commerce site, and adding more memory to your computer doesn’t mean a whit to the commerce site or its functioning.
Given a severe shortage of test time, focus first o­n factors intrinsic to the site:

1. does the site work?
2. do the functions work? (again with the functionality, because it is so basic)
3. do the links work?
4. are the files present and accounted for?
5. are the graphics MIME types correct? (I used to think that this couldn’t be screwed up)

Once the intrinsic factors are squared away, then start o­n the extrinsic points:

1. cross-browser and cross-platform compatibility
2. clients with cookies disabled
3. clients with javascript disabled
4. monitor resolution
5. browser sizing
6. connection speed differences

The point here is that with myriad possible client configurations and user-defined environmental factors to think about, think first about those that relate to the product or application itself. When you run out of time, better to know that the system works rather than that all monitor resolutions safely render the main pages.

Boundary Test From Reasonable to Extreme
You can’t just verify that an application works correctly if all input and all actions have been correct. People do make mistakes, so you must test error handling and error states. The systematic testing of error handling is called boundary testing (actually, boundary testing describes much more, but this is enough for this discussion).
During your pedal-to-the-floor, no-test-plan testing project, boundary testing refers to the testing of forms and data inputs, starting from known good values, and progressing through reasonable but invalid inputs all the way to known extreme and invalid values.

Good Values
Enter in data formatted as the interface requires. Include all required fields. Use valid and current information (what “valid and current” means will depend o­n the test system, so some systems will have a set of data points that are valid for the context of that test system). Do not try to cause errors.

Expected Bad Values
Some invalid data entries are intrinsic to the interface and concept domain. For example, any credit card information form will expect expired credit card dates — and should trap for them. Every form that specifies some fields as required should trap for those fields being left blank. Every form that has drop-down menus that default to an instruction (”select o­ne”, etc.) should trap for that instruction. What about punctuation in name fields?

Reasonable and Predictable Mistakes
People will make some mistakes based o­n the design of the form, the implementation of the interface, or the interface’s interpretation of the relevant concept domain(s). For example, people will inadvertently enter in trailing or leading spaces into form fields. People might enter a first and middle name into a first name form field (”Mary Jane”).
Not a mistake, per se, but how does the form field handle case? Is the information case-sensitive? Or does the address form handle a PO address? Does the address form handle a business name?

Compatibility Test From Good to Bad
Once you get to cross-browser and cross-platform compatibility testing, follow the same philosophy of starting with the most important (as defined by prevalence among expected user base) or most common based o­n prior experience and working towards the less common and less important.
Do not make the assumption that because a site was designed for a previous version of a browser, OS, or platform it will also work o­n newer releases. Instead, make a list of the browsers and operating systems in order of popularity o­n the Internet in general, and then move those that are of special importance to your site (or your marketers and/or executives) to the top of the list.

The Drawbacks of This Testing Approach
Many projects are not mature and are not rational (at least from the point-of-view of the quality assurance team), and so the test team must scramble to test as effectively as possibly within a very short time frame. I’ve spelled out how to test quickly without a structured test plan, and this method is much better than chaos and somewhat better than letting the developers tell you what and how to test.
This approach has definite quality implications:

1. Incomplete functional coverage — this is no way to exercise all of the software’s functions comprehensively.
2. No risk management — this is no way to measure overall risk issues regarding code coverage and quality metrics. Effective quality assurance measures quality over time and starting from a known base of evaluation.
3. Too little emphasis o­n user tasks — because testers will focus o­n ideal paths instead of real paths. With no time to prepare, ideal paths are defined according to best guesses or developer feedback rather than by careful consideration of how users will understand the system or how users understand real-world analogues to the application tasks. With no time to prepare, testers will be using a very restricted set input data, rather than using real data (from user activity logs, from logical scenarios, from careful consideration of the concept domain).
4. Difficulty reproducing — because testers are making up the tests as they go along, reproducing the specific errors found can be difficult, but also reproducing the tests performed will be tough. This will cause problems when trying to measure quality over successive code cycles.
5. Project management may believe that this approach to testing is good enough – because you can do some good testing by following this process, management may assume that full and structured testing, along with careful test preparation and test results analysis, isn’t necessary. That misapprehension is a very bad sign for the continued quality of any product or web site.
6. Inefficient over the long term — quality assurance involves a range of tasks and foci. Effective quality assurance programs expand their base of documentation o­n the product and o­n the testing process over time, increasing the coverage and granularity of tests over time. Great testing requires good test setup and preparation, but success with the kind testplan-less approach described in this essay may reinforce bad project and test methodologies. A continued pattern of quick-and-dirty testing like this is a sign that the product or application is unsustainable in the long run.

This simple article discusses Defect Severity and Defect Priority…

This document defines the defect Severity scale for determining defect criticality and the associated defect Priority levels to be assigned to errors found in software. It is a scale which can be easily adapted to other automated test management tools.

ANSI/IEEE Std 729-1983 Glossary of Software Engineering Terminology defines Criticality as,
“A classification of a software error or fault based o­n an evaluation of the degree of impact that error or fault o­n the development or operation of a system (often used to determine whether or when a fault will be corrected).”

The severity framework for assigning defect criticality that has proven most useful in actual testing practice is a five level scale. The criticality associated with each level is based o­n the answers to several questions.

First, it must be determined if the defect resulted in a system failure. ANSI/IEEE Std 729-1983 defines a failure as,
“The termination of the ability of a functional unit to perform its required function.”

Second, the probability of failure recovery must be determined. ANSI/IEEE 729-1983 defines failure recovery as,
“The return of a system to a reliable operating state after failure.”

Third, it must be determined if the system can do this o­n its own or if remedial measures must be implemented in order to return the system to reliable operation.

Fourth, it must be determined if the system can operate reliably with the defect present if it is not manifested as a failure.

Fifth, it must be determined if the defect should or should not be repaired.
The following five level scale of defect criticality addresses the these questions.

The five Levels are:

1. Critical
2. Major
3. Average
4. Minor
5. Exception

1. Critical – The defect results in the failure of the complete software system, of a subsystem, or of a software unit (program or module) within the system.

2. Major – The defect results in the failure of the complete software system, of a subsystem, or of a software unit (program or module) within the system. There is no way to make the failed component(s), however, there are acceptable processing alternatives which will yield the desired result.

3. Average – The defect does not result in a failure, but causes the system to produce incorrect, incomplete, or inconsistent results, or the defect impairs the systems usability.

4. Minor – The defect does not cause a failure, does not impair usability, and the desired processing results are easily obtained by working around the defect.

5. Exception – The defect is the result of non-conformance to a standard, is related to the aesthetics of the system, or is a request for an enhancement. Defects at this level may be deferred or even ignored.
In addition to the defect severity level defined above, defect priority level can be used with severity categories to determine the immediacy of repair.

A five repair priority scale has also be used in common testing practice. The levels are:

1. Resolve Immediately
2. Give High Attention
3. Normal Queue
4. Low Priority
5. Defer

1. Resolve Immediately – Further development and/or testing cannot occur until the defect has been repaired. The system cannot be used until the repair has been effected.

2. Give High Attention – The defect must be resolved as soon as possible because it is impairing development/and or testing activities. System use will be severely affected until the defect is fixed.

3. Normal Queue – The defect should be resolved in the normal course of development activities. It can wait until a new build or version is created.

4. Low Priority – The defect is an irritant which should be repaired but which can be repaired after more serious defect have been fixed.

5. Defer – The defect repair can be put of indefinitely. It can be resolved in a future major system revision or not resolved at all.

A detailed look into Usability Testing.

What is Usability Testing?

Usability testing is the process of working with end-users directly and indirectly to asses how the user perceives a software package and how they interact with it. This process will uncover areas of difficulty for users as well as areas of strength. The goal of usability testing should be to limit and remove difficulties for users and to leverage areas of strength for maximum usability.

This testing should ideally involve direct user feedback, indirect feedback (observed behavior), and when possible computer supported feedback. Computer supported feedback is often (if not always) left out of this process. Computer supported feedback can be as simple as a timer o­n a dialog to monitor how long it takes users to use the dialog and counters to determine how often certain conditions occur (ie. error messages, help messages, etc). Often, this involves trivial modifications to existing software, but can result in tremendous return o­n investment.

Ultimately, usability testing should result in changes to the delivered product in line with the discoveries made regarding usability. These changes should be directly related to real-world usability by average users. As much as possible, documentation should be written supporting changes so that in the future, similar situations can be handled with ease.

When to Begin

This process can and should begin as early in the development process as possible. It need not involve a large number of users, in fact 3-10 is probably ideal depending o­n the scale of the product. These users should ideally be intended users for the product (ie. alpha/beta testers) and should represent a decent cross section of users targeted by the application.

The real question should be “when to end”. I believe that this is an incremental process with many beginnings and endings. Ultimately, the job is not done until the product has reached the end of its lifecycle (not just development cycle). Each incremental step should be relatively short with changes and supporting documentation made often (up until initial delivery of the product.) o­nce a product is initially delivered it can be difficult to make these kind of changes without affecting active users. o­nce the product is shipped, changes should be considered more carefully with special concern for how active users will be affected and how future users will benefit.

However, it is never too late to start. Even if you are nearing the end of the development cycle, usability testing can still yield enormous results. Sometimes even minor changes to the UI, help system, reports, etc can make the product more appealing to users.

How to Begin

Usability testing can be quite simple. There are 4 basic ingredients that are essential to success. These are

1. The usability testing person/team needs to include a software developer who is open-minded about changes and not offended by criticism. The goal of usability testing is not to criticize, but improve and learn. If any member of the team is not ready to receive criticism with an open-mind, the testing will almost certainly fail. This person needs to have a good working knowledge of the workflow process the application is designed to facilitate and needs to have good communication skills. Good note taking skills are also essential.
2. The users selected as test subjects need to be average users (not all power users, and not all entry-level users.) They should understand what the application is designed to do and should be able to communicate their needs reasonably well. Often, these users will not be able to communicate their needs in technical terms, but will be able to identify problems they are having.
3. The development person/team needs to be prepared to make changes o­n an incremental basis in relatively fast paced environment. The best situation is to make needed changes quickly so that those changes can be incorporated into the continuing testing process.
4. Patience. The usability testing and refinement process can take some time and will sometimes go in the wrong direction. By keeping changes small and incremental, it is usually easy to back-track and rework the problem without significant setbacks.

There are 3 methods of feedback that need to be incorporated into our testing. These are:

1. Direct user feedback. This type of feedback usually occurs by having the test users use the software alone and report back to the usability team/person. Ideally, reporting back should occur o­n a regular basis (daily or weekly at most).
2. Observed behavior. This type of feedback can occur in conjunction with direct user feedback. This occurs when the testing team/person observes how users use the software.
3. Computer supported feedback. This type of feedback can occur o­n an o­ngoing basis throughout the testing process. As mentioned this is usually quite simple involving timers and hit counters.

Each of these feedback methods should be used to achieve the ultimate goal.

How to best Leverage Feedback Methods
Direct Feedback

1. Provide users with notebooks, pre-printed forms, a simple program, web page, etc to record their problems as they are having them. They tend to forget the details if this is not done.
2. Regularly review the issues users report.
3. Meet with users o­n a scheduled (somewhat) basis to discuss their issues and make sure that you fully understand it before proceeding. Be prepared for this meeting by reviewing their issues prior to the meeting.
4. Keep a good communication dialog open with all users.
5. Prioritize direct-feedback issues highly. Users need to see results relatively quickly or they get discouraged with the process.

Observed Behavior

1. Use multiple opportunities to observe behavior. Whether you are training them, discussing problems found earlier, or just walking past their desks, these are opportunities to observe their behavior. (I doubt many of us have 1 way glass to watch users, so I won’t go into that here.)
2. Take notes (use same forms, software, web pages, etc as users use if possible. This will insure you do not forget what you observed.
3. Compare the notes you are taking against the notes users are taking. If users are not reporting nearly as many problems as you are finding, it is possible that they are not comfortable with the process yet.
4. Keep a good communication dialog open with all users.
5. Don’t interfere with their normal process. The goal is not to train them o­n the “right way”, but rather to have the software work “their way”.
6. Be prepared to misinterpret user behavior. Sometimes you might observe a user having an apparent problem, but in-fact you are misunderstanding what they are doing.
7. Prioritize observed behavior after direct-feedback. Users are not expecting these changes and they can cause confusion. Carefully review your observations and discuss them with your team and the users.

Computer Supported Feedback

1. It is hard to know where to use computer supported feedback early in the development and testing cycles. Therefore start simple and grow from there.
2. Be careful that the code that supports this does not interfere with the users workflow and by all means does not crash the software. (Been there, done that.)
3. If possible, log all computer-feedback issues into a simple database (Access worked well for me.)
4. When reviewing the log, be very careful to not overlook issues and not misinterpret the data. (Your method need not be statistically valid, just reasonable.)
5. When you see an issue in the data, try and support it through direct feedback and observed behavior methods. This data can be very helpful in knowing what to look for when working with users.
6. Consider leaving this capability in the final product carefully. o­nly do so, if the log can be disabled safely and it has been thoroughly tested against all issues (especially exceptional growth in log size.

Download Black Box Testing Course Video for Free!!!

Cem Kaner has posted, free, an evolving o­nline version (with video, course notes, examples, exercises, papers, tests, and more) of his Black Box Software Testing course.

From the course overview:”Black box testing is the craft of testing a program from the external view. We look at how the program operates in its context, getting to know needs and reactions of the users, hardware and software platforms, and programs that communicate with it.”This course is an introduction to black box testing. It is a superset of the Software Testing 1 introductory courses that Florida Tech requires in its undergraduate (CSE 3411) and graduate (SWE 5411) software engineering degree programs.

The full set of materials are equivalent to about a two-semester course.”…and from the learning objectives for the course:”Testing is a type of technical investigation. If you think of it as a routine task, you’ll do crummy testing. So, the essence of what I hopeyou’ll learn in the course is how to investigate, how to plan your investigation, how to report the results, and how to tell whether you’re doing a good job. Excellence in the course will require you to extend your critical thinking, writing, and teamworking skills. Finally, the work that you do in this course might help you land a job.

Many of the course tasks were designed to be realistic or impressive (to an employer) and to give you a chance to do professional-quality work that you can show off during a job interview.

http://www.testingeducation.org/BBST/index.html

Black box testing or functional testing is used to check that the outputs of a program, given certain inputs, conform to the functional specification of the program.

The term black box indicates that the internal implementation of the program being executed is not examined by the tester. For this reason black box testing is not normally carried out by the programmer.

Black Box Testing is testing without knowledge of the internal workings of the item being tested. For example, when black box testing is applied to software engineering, the tester would o­nly know the “legal” inputs and what the expected outputs should be, but not how the program actually arrives at those outputs. It is because of this that black box testing can be considered testing with respect to the specifications, no other knowledge of the program is necessary.
For this reason, the tester and the programmer can be independent of o­ne another, avoiding programmer bias toward his own work. For this testing, test groups are often used, “Test groups are sometimes called professional idiots…people who are good at designing incorrect data.”

The opposite of this would be glass box testing, where test data are derived from direct examination of the code to be tested. For glass box testing, the test cases cannot be determined until the code has actually been written. Both of these testing techniques have advantages and disadvantages, but when combined, they help to ensure thorough testing of the product.
Also, do to the nature of black box testing, the test planning can begin as soon as the specifications are written.

Advantages of Black Box Testing
There are many advantages of Black Box Testing…

1. More effective o­n larger units of code than glass box testing
2. The tester needs no knowledge of implementation, including specific programming languages
3. The tester and programmer are independent of each other
4. Tests are done from a user’s point of view
5. Will help to expose any ambiguities or inconsistencies in the specifications
6. Test cases can be designed as soon as the specifications are complete
7. The test is unbiased because the designer and the tester are independent of each other.
8. The tester does not need knowledge of any specific programming languages.
9. The test is done from the point of view of the user, not the designer.
10. Test cases can be designed as soon as the specifications are complete.

Disadvantages of Black Box Testing

1. Only a small number of possible inputs can actually be tested, to test every possible input stream would take nearly forever
2. Without clear and concise specifications, test cases are hard to design
3. There may be unnecessary repetition of test inputs if the tester is not informed of test cases the programmer has already tried
4. May leave many program paths untested
5. Cannot be directed toward specific segments of code which may be very complex (and therefore more error prone)
6. Most testing related research has been directed toward glass box testing
7. The test can be redundant if the software designer has already run a test case.
8. The test cases are difficult to design.
9. Testing every possible input stream is unrealistic because it would take a inordinate amount of time; therefore, many program paths will go untested.

Testing Strategies/Techniques

1. Black box testing should make use of randomly generated inputs (only a test range should be specified by the tester), to eliminate any guess work by the tester as to the methods of the function
2. Data outside of the specified input range should be tested to check the robustness of the program
3. Boundary cases should be tested (top and bottom of specified range) to make sure the highest and lowest allowable inputs produce proper output
4. The number zero should be tested when numerical data is to be input
5. Stress testing should be performed (try to overload the program with inputs to see where it reaches its maximum capacity), especially with real time systems
6. Crash testing should be performed to see what it takes to bring the system down
7. Test monitoring tools should be used whenever possible to track which tests have already been performed and the outputs of these tests to avoid repetition and to aid in the software maintenance
8. Other functional testing techniques include: transaction testing, syntax testing, domain testing, logic testing, and state testing.
9. Finite state machine models can be used as a guide to design functional tests
10. According to Beizer the following is a general order by which tests should be designed:
11. Clean tests against requirements.
12. Additional structural tests for branch coverage, as needed.
13. Additional tests for data-flow coverage as needed.
14. Domain tests not covered by the above.
15. Special techniques as appropriate–syntax, loop, state, etc.
16. Any dirty tests not covered by the above.

Glass box testing requires the intimate knowledge of program internals, while black box testing is based solely o­n the knowledge of the system requirements. Being primarily concerned with program internals, it is obvious in SE literature that the primary effort to develop a testing methodology has been devoted to glass box tests. However, since the importance of black box testing has gained general acknowledgement, also a certain number of useful black box testing techniques were developed.
It is important to understand that these methods are used during the test design phase, and their influence is hard to see in the tests o­nce they’re implemented.

Advantages

1. Forces test developer to reason carefully about implementation
2. Approximates the partitioning done by execution equivalence
3. Reveals errors in “hidden” code:
4. Beneficent side-effects
5. Optimizations (e.g. charTable that changes reps when size > 100)

Disadvantages

1. Expensive
2. Miss cases omitted in the code

Software testing is an extremely creative and intellectually challenging task. When testing follows the principles given below, the creative element of test design and execution rivals any of the preceding software development steps.

Following are some of the principles of Software Testing, which when followed would yeild good quality products.

Testing must be done by an independent party.

Testing should not be performed by the person or team that developed the software since they tend to defend the correctness of the program. The developer is driven by delivery and he will try to finish the testing as early as possible. The developer is gentle in testing his code and has a soft corner for his code. The independent tester is driven by quality and will try to break the code. Hence testing should be done by an independent person.

Assign best personnel to the task.

Because testing requires high creativity and responsibility o­nly the best personnel must be assigned to design, implement, and analyze test cases, test data and test results. Also note that from point 1, the testing is being done by an independent tester, the system is new to him and he has to understand the entire system to test it. It is possible to find bugs and issues with with software o­nly when you know the system thoroughly.

Testing should not be planned under the tacit assumption that no errors will be found.

Testing is the process of executing a program with the intent of finding errors. Good testing will find bugs… and testing is never complete, it is infinte. There should be no assumption that the software has no erros. Such an assumption would leave tremendous amount of holes in the system.

Test for invalid and unexpected input conditions as well as valid conditions.

The program should generate correct results when a valid input is given, this is called as positive testing; Also the software should give correct error messages when an invalid test is encountered, this is called as negative testing. We need to give a different range of inputs with varying sets of values. For example, if the input field is taking a positive integer, we should try with all sorts of integers, positive, negative, zero, large positive, large negative etc. Also we should try giving a character, string etc and the expect the correct error message.

The probability of the existence of more errors in a module or group of modules is directly proportional to the number of errors already found.

If you find a bug in a particular module, the developers just tend to resolve that issue and close the bug. But there is a more probability that there will be more issues in that area. You need to look around, why that particular bug is present and look for similar issues in that module, you are going to hit many issues.

Keep software static during test.

The program must not be modified during the implementation of the set of designed test cases. Modifying the program while the testing has started will leave a lot of loop holes in the testing; you would have finished testing module A and started testing o­n module B, but by that time, module A would have changed and the changes are not tested!!!

Document test cases and test results.

This is very important for testers, you need to document the test cases, which can be used for later testing, may be for the next release. The test results should be documented so that they can used for analyzing if required at a later stage.

Provide expected test results if possible.

A necessary part of test documentation is the specification of expected results, even if providing such results is impractical. Using the expected results you can verify the software which is under test and declare whether the software meets the expectations or not.

A few simple definitions of  Testing Terminology.

Error - The difference between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition.

Fault - An incorrect step, process, or data definition in a computer program.

Debug - To detect, locate, and correct faults in a computer program.

Failure - The inability of a system or component to perform its required functions within specified performance requirements. It is manifested as a fault.

Testing - The process of analyzing a software item to detect the differences between existing and required conditions (that is, bugs) and to evaluate the features of the software items.

Static analysis - The process of evaluating a system or component based o­n its form, structure, content, or documentation.

Dynamic analysis - The process of evaluating a system or component based o­n its behavior during execution.

Correctness - The degree to which a system or component is free from faults in its specification, design, and implementation. The degree to which software, documentation, or other items meet specified requirements. The degree to which software, documentation, or other items meet user needs and expectations, whether specified or not.

Verification - The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase. Formal proof of program correctness.

Validation - The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements.

A comprehensive list of QA Check List.

Following is the check list which can be adopted by the QA.

General
Pages fit within the resolution(800×600)

• Design works with liquid tables to fill the user’s window size.
• Separate print versions provided for long documents (liquid tables may negate this necessity). Accommodates A4 size paper.
• Site doesn’t use frames.
• Complex tables are minimized.
• Newer technologies are generally avoided for 1-2 years from release, or if used alternative traditional forms of content are easily available.

Home vs. Subsequent Pages & Sections

• Home page logo is larger and more centrally placed than o­n other pages.
• Home page includes navigation, summary of news/promotions, and a search feature.
• Home page answers: Where am I; What does this site do; How do I find what I want?
• Larger navigation space o­n home page, smaller o­n subsequent pages.
• Logo is present and consistently placed o­n all subsequent pages (towards upper left hand corner).
• “Home” link is present o­n all subsequent pages (but not home page).
• If subsites are present, each has a home page, and includes a link back to the global home page.

Navigation

• Navigation supports user scenarios gathered in the User Task Assessment phase (prior to design).
• Users can see all levels of navigation leading to any page.
• Breadcrumb navigation is present (for larger and some smaller sites).
• Site uses DHTML pop-up to show alternative destinations for that navigation level.
• Navigation can be easily learned.
• Navigation is consistently placed and changes in response to rollover or selection.
• Navigation is available when needed (especially when the user is finished doing something).
• Supplimental navigation is offered appropriately (links o­n each page, a site map/index, a search engine).
• Navigation uses visual hierarchies like movement, color, position, size, etc., to differentiate it from other page elements.
• Navigation uses precise, descriptive labels in the user’s language. Icon navigation is accompanied by text descriptors.
• Navigation answers: Where am I (relative to site structure); Where have I been (obvious visited links); Where can I go (embedded, structural, and associative links)?
• Redundant navigation is avoided.

Functional Items

• Terms like “previous/back” and “next” are replaced by more descriptive labels indicating the information to be found.
• Pull-down menus include a go button.
• Logins are brief.
• Forms are short and o­n o­ne page (or demonstrate step X of Y, and why collecting a larger amount of data is important and how the user will benefit).
• Documentation pages are searchable and have an abundance of examples. Instructions are task-oriented and step-by-step. A short conceptual model of the system is available, including a diagram that explains how the different parts work together. Terms or difficult concepts are linked to a glossary.

Linking General:

• Links are underlined.
• Size of large pages and multi-media files is indicated next to the link, with estimated dowload times.
• Important links are above the fold.
• Links to releated information appear at bottom of content or above/near the top.
• Linked titles make sense out of context.
• If site requires registration or subscription, provides special URLs for free linking. Indicates the pages are freely linkable, and includes and easy method to discover the URL.
• If site is running an ad, it links to a page with the relevant content, not the corporate home page.
• In content:
• Keeps linked phrases short to aid scanning (2-4 words).
• Links o­n meaningful words and phrases. Avoids phrases like, “click here.”
• Includs a brief description of what the user should expect o­n the linked page.
• In code:
• Uses relative links when linking between pages in a site. Uses absolute links to pages o­n unrelated sites.
• Uses link titles in the code for IE users (preferably less than 60 characters, no more than 80).

Search Capabilities

• A search feature appears o­n every page (exceptions include pop-up forms and the like).
• Search box is wide to allow for visible search parameters.
• Advanced Search, if included, is named just that (to scare off novices).
• Search system performs a spelling check and offers synonym expansion.
• Site avoids scoped searching. If included it indicates scope at top of both query and results pages, and additionally offers an automatic extended site search immediately with the same parameters.
• Results do not include a visible scoring system.
• Eliminates duplicate occurrences of the same results (e.g., foo.com/bar vs. foo.com/bar/ vs. foo.com/bar/index.html).

Page Design

• Content accounts for 50% to 80% of a page’s design (what’s left over after logos, navigation, non-content imagery, ads, white space, footers, etc.).
• Page elements are consistent, and important information is above the fold.
• Pages load in 10 seconds or less o­n users bandwidth.
• Pages degrade adequately o­n older browsers.
• Text is over plain background, and there is high contrast between the two.
• Link styles are minimal (generally o­ne each of link, visited, hover, and active states). Additional link styles are used o­nly if necessary.
• Specified the layout of any liquid areas (usually content) in terms of percentages.

Fonts and Graphics

• Graphics are properly optimized.
• Text in graphics is generally avoided.
• Preferred fonts are used: Verdana, Arial, Geneva, sans-serif.
• Fonts, when enlarged, don’t destroy layout.
• Images are reused rather than rotated.
• Page still works with graphics turned off.
• Graphics included are necessary to support the message.
• Fonts are large enough and scalable.
• Browser chrome is removed from screen shots.
• Animation and 3D graphics are generally avoided.

Content Design

• Uses bullets, lists, very short paragraphs, etc. to make content scannable.
• Articles are structured with scannable nested headings.
• Content is formatted in chunks targeted to user interest, not just broken into multiple pages.
• No moving text; most is left-justified; sans-serif for small text; no upper-case sentences/paragraphs; italics and bold are used sparingly.
• Dates follow the international format (year-month-day) or are written out (August 30, 2001).

Writing

• Writing is brief, concise, and well edited.
• Information has persistent value.
• Avoids vanity pages.
• Starts each page with the conclusion, and o­nly gradually added the detail supporting that conclusion.
• One idea per paragraph.
• Uses simple sentence structures and words.
• Gives users just the facts. Uses humor with caution.
• Uses objective language.

Folder Structure

• Folder names are all lower-case and follow the alpha-numeric rules found under “Naming Conventions” below.
• Segmented the site sections according to:

1. Root directory (the “images” folder usually goes at the top level within the root folder)
2. Sub-directories (usually o­ne for each area of the site, plus an images folder at the top level within the root directory)
3. Images are restricted to o­ne folder (”images”) at the top level within the root directory (for global images) and then if a great number of images are going to be used o­nly section-specifically, those are stored in local “images” folders

Naming Conventions

• Uses clients preferred naming method. If possible, uses longer descriptive names (like “content_design.htm” vs. “contdesi.htm”).
• Uses alphanumeric characters (a-z, 0-9) and – (dash) or _ (underscore)
• Doesn’t use spaces in file names.
• Avoids characters which require a shift key to create, or any punctuation other than a period.
• Uses o­nly lower-case letters.
• Ends filenames in .htm (not .html).

Multimedia

• Any files taking longer than 10 seconds to download include a size warning (>50kb o­n a 56kbps modem, >200kb o­n fast connections). Also includes the running time of video clips or animations, and indicate any non-standard formats.
• Includes a short summary (and a still clip) of the linked object.
• If appropriate to the content, includes links to helper applications, like Adobe Acrobat Reader if the file is a .pdf.

Page Titles

• Follows title strategy … Page Content Descriptor : Site Name, Site section (E.g.: Content Implementation Guidelines : CDG Solutions, Usability Process )
• Tries to use o­nly two to six words, and makes their meaning clear when taken out of context.
• The first word(s) are important information-carrying o­ne(s).
• Avoids making several page titles start with the same word.

Headlines

• Describes the article in terms that relate to the user.
• Uses plain language.
• Avoids enticing teasers that don’t describe.

CSS

• Uses CSS to format content appearance (as supported by browsers), rather than older HTML methods.
• Uses a browser detect and serve the visitor a CSS file that is appropriate for their browser/platform combination.
• Uses linked style sheets.

Documentation and Help Pages

• When using screen shots, browser chrome was cropped out.
• Hired a professional to write help sections (a technical writer).
• Documentation pages are searchable.
• Documentation section has an abundance of examples.
• Instructions are task-oriented and step-by-step.
• A short conceptual model of the system is provided, including a diagram that explains how the different parts work together.
• Terms or difficult concepts are linked to a glossary.

Content Management
Site has procedures in place to remove outdated information immediately (such as calendar events which have passed.

A look into the pro’s and con’s of Automated Testing and Manual Testing.

I met with my team’s automation experts a few weeks back to get their input o­n when to automate and when to manually test. The general rule of thumb has always been to use common sense. If you’re o­nly going to run the test o­ne or two times or the test is really expensive to automation, it is most likely a manual test. But then again, what good is saying “use common sense” when you need to come up with deterministic set of guidelines o­n how and when to automate?

Pros of Automation

1. If you have to run a set of tests repeatedly, automation is a huge win for you
2. It gives you the ability to run automation against code that frequently changes to catch regressions in a timely manner
3. It gives you the ability to run automation in mainstream scenarios to catch regressions in a timely manner (see What is a Nighlty)
4. Aids in testing a large test matrix (different languages o­n different OS platforms). Automated tests can be run at the same time o­n different machines, whereas the manual tests would have to be run sequentially.

Cons of Automation

1. It costs more to automate. Writing the test cases and writing or configuring the automate framework you’re using costs more initially than running the test manually.
2. Can’t automate visual references, for example, if you can’t tell the font color via code or the automation tool, it is a manual test.

Pros of Manual

1. If the test case o­nly runs twice a coding milestone, it most likely should be a manual test. Less cost than automating it.
2. It allows the tester to perform more ad-hoc (random testing). In my experiences, more bugs are found via ad-hoc than via automation. And, the more time a tester spends playing with the feature, the greater the odds of finding real user bugs.

Cons of Manual

1. Running tests manually can be very time consuming
2. Each time there is a new build, the tester must rerun all required tests – which after a while would become very mundane and tiresome.

Other deciding factors

1. What you automate depends o­n the tools you use. If the tools have any limitations, those tests are manual.
2. Is the return o­n investment worth automating? Is what you get out of automation worth the cost of setting up and supporting the test cases, the automation framework, and the system that runs the test cases?

Criteria for automating
There are two sets of questions to determine whether automation is right for your test case:

Is this test scenario automatable?

1. Yes, and it will cost a little
2. Yes, but it will cost a lot
3. No, it is no possible to automate

How important is this test scenario?

1. I must absolutely test this scenario whenever possible
2. I need to test this scenario regularly
3. I o­nly need to test this scenario o­nce in a while

If you answered #1 to both questions – definitely automate that test
If you answered #1 or #2 to both questions – you should automate that test
If you answered #2 to both questions – you need to consider if it is really worth the investment to automate

What happens if you can’t automate?
Let’s say that you have a test that you absolutely need to run whenever possible, but it isn’t possible to automate. Your options are

1. Reevaluate – do I really need to run this test this often?
2. What’s the cost of doing this test manually?
3. Look for new testing tools
4. Consider test hooks

A short discussion about Test Plan.

A software project test plan is a document that describes the objectives, scope, approach, and focus of a software testing effort. The process of preparing a test plan is a useful way to think through the efforts needed to validate the acceptability of a software product. The completed document will help people outside the test group understand the ‘why’ and ‘how’ of product validation. It should be thorough enough to be useful but not so thorough that no o­ne outside the test group will read it.

A test plan states what the items to be tested are, at what level they will be tested, what sequence they are to be tested in, how the test strategy will be applied to the testing of each item, and describes the test environment.

A test plan should ideally be organisation wide, being applicable to all of an organisations software developments.

The objective of each test plan is to provide a plan for verification, by testing the software, the software produced fulfils the functional or design statements of the appropriate software specification. In the case of acceptance testing and system testing, this generally means the Functional Specification.

The first consideration when preparing the Test Plan is who the intended audience is – i.e. the audience for a Unit Test Plan would be different, and thus the content would have to be adjusted accordingly.

You should begin the test plan as soon as possible. Generally it is desirable to begin the master test plan as the same time the Requirements documents and the Project Plan are being developed. Test planning can (and should) have an impact o­n the Project Plan. Even though plans that are written early will have to be changed during the course of the development and testing, but that is important, because it records the progress of the testing and helps planners become more proficient.

What to consider for the Test Plan:
1. Test Plan Identifier
2. References
3. Introduction
4. Test Items
5. Software Risk Issues
6. Features to be Tested
7. Features not to be Tested
8. Approach 9. Item Pass/Fail Criteria
10. Suspension Criteria and Resumption Requirements
11. Test Deliverables
12. Remaining Test Tasks
13. Environmental Needs
14. Staffing and Training Needs
15. Responsibilities
16. Schedule
17. Planning Risks and Contingencies
18. Approvals
19. Glossary

Standards for Software Test Plans
Several standards suggest what a test plan should contain, including the IEEE.

ANSI/IEEE 829-1983 IEEE Standard for Software Test Documentation -Description
http://standards.ieee.org/reading/ieee/std_public/description/se/829-1983_desc.html