A false sense of security through testing
Published: – Leave a comment
When announcing version 2.0.0 of Block Control, I was very happy and satisfied with its new tests. How wrong I was …
This new version was the first one that contained end-to-end testing, meaning that there were actual tests within a browser to click through the plugin’s functionality. It also ran the tests through all supported PHP versions, which was explicitly made so that no errors in PHP versions arrive I don’t use personally on any of my systems.
The symptom
After releasing Block Control 2.0.0, I waited for the plugin release to get available, since you currently have to wait 6 hours after publishing it via WordPress.org. In theory, you could download it manually – which I should have done – but I didn’t. And so, on the next day, several reports stated that the update completely broke their site with a fatal error in PHP.
The problem
The actual problem was identified fast: after all running tests, the plugin files get send to WordPress.org. Some files and folders are excluded from it, usually some project files only required for development. One of those folders was the folder src. Within the plugin itself, that’s where the raw React code lays that interacts with the block editor. However, the third-party library to detect mobile devices also has a src folder and it contains the actual PHP code for the detection. Leaving it out meant that PHP files were missing – and thus resulting in a fatal error.
The actual error was that I didn’t limit the exclusion of only my src folder.
The solution
As an immediate workaround, I just excluded my own src folder and published it as version 2.0.1, which worked fine then. But that was only one part of it and just to minimize the damage. The following days, I worked on an actual solution: During deployment, I now create a ZIP file that has the exact same structure as the data sent to WordPress.org. This one gets then installed and activated within a WordPress Playground so that I can make sure that this version won’t result in a fatal error during regular use.
In the end, testing is only as good as its coverage. If you miss something and don’t test it, it may break. Don’t feel overly secure with your available testing methods. You’re most certainly missing something. 🥲
To all having updated to version 2.0.0 before the fixed version: I’m truly sorry!
Reposts