Chaos Engineering and Verification
Difference with testing.
Experimentation creates new knowledge. Experiments propose a hypothesis, and as long as the hypothesis is not disproven, confidence grows in that hypothesis. If it is disproven, then we learn something new. This kicks off an inquiry to figure out why our hypothesis is wrong. In a complex system, the reason why something happens is often not obvious. Experimentation either builds confidence, or it teaches us new properties about our own system. It is an exploration of the unknown.
Verification-oriented
In observing a system we can distinguish two types of procedures.
- Verification: Verification of a complex system is a process of analyzing output at a system boundary. A homeowner can verify the quality of the water (output) coming from a sink (system boundary) by testing it for contaminants without knowing anything about how plumbing or municipal water service (system parts) functions.
- Validation: Validation of a complex system is a process of analyzing the parts of the system and building mental models that reflect the interaction of those parts. A homeowner can validate the quality of water by inspecting all of the pipes and infrastructure (system parts) involved in capturing, cleaning, and delivering water (mental model of functional parts) to a residential area and eventually to the house in question.Both of these practices are potentially useful, and both build confidence in the output of the system. As software engineers we often feel a compulsion to dive into code and validate that it reflects our mental model of how it should be working. Contrary to this predilection, Chaos Engineering strongly prefers verification over validation. Chaos Engineering cares whether something works, not how.
The Chaos Engineering Learning Loop - Note that you need to replace validation with verification