I will examine the practical implications, at the code level, of testing for “right-ish” and “wrong-ish” using small machine learning models. We can usefully bring in openness and learning into a space previously dominated by closed, idempotent logic. The examples we consider will reveal that this is a watershed change radically altering how test-automation code looks and works.

I have long been focused on other attributes of automated integration and end-to-end tests, beyond just passing and failing. These attributes include different meaningful test trends calculated over time. This talk will pick up that thread again because the move toward less deterministic tests I present here puts real pressures on test result management. But solving that problem also gives us an exceptionally relevant advantage in our current dilemma of accelerating A.I. generated pull requests: a stochastic approach that pinpoints the most likely acute failures may be the only way to rescue quality in the face of an A.I. generated code onslaught.