Software Testing Basics for Programmers

Learn the fundamentals of software testing, including responsibilities, cost/benefit analysis, testing types, automation, and levels, tailored for programmers and testers, with a focus on Cobol on z/OS. No prior testing experience required.

The course has approximately 70% presentation and 30% hands-on exercises.

This course can be done using our z/OS labs, but results will be better if done using the client's z/OS system.

• Programmers/software engineers interested in learning or strengthening their testing skills, with a focus on Cobol on z/OS. Previous testing experience not required.
• Software testers interested in gaining a sense of the programmer's perspective on testing and how the two roles can reinforce each other's value.

Some experience working in any role directly (not only as an end user) with software applications in any language and on any platform.

• Understand the responsibilities of programmers and testers with respect to software quality
• Understand how to assess the cost/benefit of investing effort in testing
• Understand the differences in testing, checking, and test-driven development
• Understand basic concepts of effective software testing
• Understand basic concepts of test automation
• Understand different levels of testing such as unit, component, integration, and end-to-end

• Changing hats
  • Programmer's hat - ensure it can work, at least under ideal conditions
  • Tester's hat - discover how it can fail, including under suboptimal conditions
• Basic goals (simplified)
  • Programmers - verify the application functions as specified
  • Testers - discover information about the application that was not anticipated or planned
• The game
  • Programmers - make sure your testing team cannot find any issues
  • Testers - make sure no issues slip past you
• Benefit obtained vs. level of effort
  • Goal of testing
    • Goal is to avoid negative business impact
    • Goal is not to achieve absolute perfection
  • Dude's Law: Value = Why / How
  • Business risk of software errors
    • Risk is a function of the probability and cost of issues r = f(p*c)
    • High p + high c: Yes, test this. Example: Multiple funds transfer operations representing the same transaction, submitted maliciously, result in incorrect account balances and possibly direct financial losses.
    • High p + low c: Yes, test this. Example: Incorrect values for a field in an inbound data feed; our application can supply reasonable default values to prevent catastrophic failure, but if defaults are poorly chosen the cumulative effect of multiple occurrences may be costly.
    • Low p + high c: Yes, test this. Example: An incorrect update operation corrupts sensitive data in a system of record; the system is stable and there is little or no history of errors. (Think: Chernobyl, Bhopal)
    • Low p + low c: Test this if doing so is easy and cheap. Value is questionable. Example: Incorrect value in a date field that is not a key or index field, and that is not used in a significant calculation (such as the birth date of a person applying for insurance).
• Testing, checking, test automation, and test-driven development
  • What's the difference between testing and checking?
  • What do we mean by "manual" and "automated" testing?
  • How is test-driven development a software design technique vs. a testing technique?
  • How does test-driven development support automated functional/regression testing?
• Testing with purpose
  • Mission and testing strategy
  • Concept of test oracles
  • Whole-team and multi-team exploratory testing
• Common types of functional verification (manual and automated)
  • Verifying expected output under controlled conditions
  • Boundary analysis
  • Combinatorial testing
  • Make no assumptions; if it can happen, it will happen
• Working with existing code bases
  • Characterization tests
    • Characterization tests (general)
    • Characterization tests for batch COBOL applications
    • Characterization tests for CICS applications
  • Refactoring to isolate code units to enable automated functional checks
    • When is refactoring useful?
    • When is refactoring not useful?
    • How do you judge when to stop refactoring?
    • Simple refactorings to isolate small units of COBOL code
    • Tools (or the lack thereof) to support refactoring
• Automate repetitive, predictable tasks
  • Ensures consistent results by reducing the chance of human error
  • Frees time for "real" work ("thinking" work)
  • With respect to "testing," this means functional validation under controlled conditions with specific test data; running test case A today is identical to running it yesterday, so we can detect changes in the software's behavior - not possible using manual methods
  • Other repetitive, predictable tasks include e.g., static analysis, packaging, deployment, audit trail
• Is "regression testing" different from "regular" testing?
  • What is a regression?
  • Repeatable test cases that validate expected behavior under controlled conditions also detect regressions
• Levels of checking and testing
  • Rules of thumb
    • Many small-scale tests covering well-isolated units of code
    • Somewhat fewer slightly-larger-scale tests covering integration points and functionality that involves multiple programs
    • Far fewer large-scale, end-to-end functional checks of batch jobs and online transactions
    • Verify as much as possible at lower levels, where test cases are smaller, easier to understand, and cheaper/faster to execute
    • Do not expect any single "level" of testing to be sufficient
    • Use exploratory testing (not automated) to surface unknowns about the system under test
  • Some aspects of "testing in production"
    • Some conditions can't be replicated in a test environment prior to deployment; active production monitoring is crucial