The honest Truth about Plugin Testing

After my post about content analysis with the WordPress SEO plugin, which covered the recent updates to that plugin, I got some comments that we should test better, especially because the commenter had issues with a certain slider plugin. I laughed when I read that comment, but then it dawned on me that most in the WordPress community might not have the insight that we here at Yoast have into the WordPress world. Most people are (of course) unaware of the enormous amount of other plugins used by our users of the WordPress SEO plugin.

We’ve been doing anonymous and opt-in tracking of WordPress installs for just over a year now. This allows us to check whether or not people update our plugins and much more. Also, it allows us to see which other plugins people use on their websites. Of course, we test compatibility of every plugin update with the most commonly used plugins. To test compatibility with every plugin, however, is impossible. Let me show you why by giving you some numbers.

The WordPress SEO plugin runs on at least 1 million and in fact probably more than 2 million sites. All of these sites use different plugins. The 691,797 sites we currently track in our tracking system use approximately 83,000 different plugins, in innumerable combinations. After the 10 most used plugins, none of them are run on more than 10% of those sites, yet most sites run more than 5 plugins. In fact, let me show you the distribution of number of plugins per site:

number of active plugins per site

This doesn’t even take into consideration the fact that all of those people run a different theme too, and that they each have different settings for each plugin and theme as well. All of these can (and actually do) interact and interfere with each other. Because of that, testing if a certain update works for all of our users and their plugins is – and I hate saying this - impossible.

What to do if you’re updating?

If you are installing an update of a plugin, any plugin, - especially when you have a webshop or are otherwise making money from your site – you should always test the new version first on a staging environment. Be aware of plugins with very little users and of plugins developed by people with limited WordPress experience. Of course, this plugin may just have the exact functionality you need, but these plugins will come with a higher risk of compatibility issues.

What you should do is figure out the most important things people should be able to do on your website, create specific test plans to reproduce them and then test them against every (plugin/theme/core) update before you put it live. We as plugin developers are not responsible if something breaks because of incompatibilities, you are. Of course we’ll try and prevent it, but… well, you’ve read the above. This means that installing another plugin for feature X does always come with a cost, even when you’re not paying for the plugin.

This is also why I don’t really get the free versus premium debate. Some people seem to hate paying for plugins, but they don’t realize the biggest investment is actually their time, not the money. Paying money so that a plugin developer can properly test it because he has the resources is actually a good thing.

A few WordPress hosts get this particular issue quite well, especially WP Engine (aff) seems to have taken their one-click staging to such a level that it makes this kind of testing really easy. I assume from having conversations with other hosting parties that they will soon follow.

A word on versioning

I thought it’d be wise to add a note to this post on our versioning system. We try to stick with Semantic Versioning, which means that upgrading from 1.4.17 to 1.4.18 for instance should be painless as the upgrade only contains bug fixes. We have in the past made the mistake of adding functionality in a minor release, we won’t do that again. Those minor releases are for bug fixes, new features will be marked by upgrading from 1.4 to 1.5, like the upcoming release.

When we go to 2.0, which we are planning for, we will drop some (though only a small part) of our old API, so at that point we increase the major version number.

Check out our WordPress SEO Premium plugin

This post first appeared on Yoast. Whoopity Doo!