A few weeks ago I wrote an article in which I complained about a few aspects of the PHP development process that I thought were inexcusable and harmful. The article was surprisingly well-received with only a handful of people responding by demeaning me; given the nature of the article, I’d say that is a big win! The generally positive response does not make up for the flaws in the article itself, however. You see, I started out writing the article with the serious intention of making not only an honest critique but also providing some practical feedback for how the process could be improved. I feel my critique was completely honest, but I failed spectacularly in my second goal, and any merit that the article had was overshadowed by its negative tone.
It is no secret that the PHP development process has never been a shining example of project organization or quality assurance. Until recently, some of the most important aspects of any project’s development cycle were either entirely lacking or were ill-defined. Worse, there was little in the form of systemic quality assurance. Fortunately, the core devs did not ignore these issues, and they’ve been pushing really hard to improve on these areas over the past few years: They’ve created a new release process, a new voting procedure, and have been building up the automated test coverage. They’re shaping the development process into a more open, clear, and modern process that the whole community can be proud of.
This type of transformation takes time, however, and they are not at the finish line yet. There are specific characteristics of the development team and process that are still very much a huge problem today. An acceptance of failing unit tests along with a lack of an automated release process that incorporates testing requirements resulted in the recent disaster that was the PHP 5.3.7 release. A lack of clear responsibilities continues to disrupt developer trust in the PHP project. Insufficient means of communication and insufficient use of those means frequently keeps the large number of non-contributing PHP developers in the dark. I’ll address each of these individually for clarity sake:
Automated tests are ineffective if failures are acceptable
So much progress has been made in recent years with the PHP test suite – the coverage for PHP 5.3 is at 70%. Obviously this isn’t ideal, but PHP is a massive project, so it is understandable that it takes some serious time to build the testing suite toward 100% coverage. The big problem with the automated PHP tests is not that there are not enough but rather that it is not a requirement that tests pass. To make matters worse, it does not appear that new code is required to have significant test coverage either. At the time of this writing, there are 128 failing unit tests in PHP 5.3 (the current stable branch), and there are 129 failing unit tests in PHP 5.4. There is actually a small decrease in test coverage between the two versions as well (70.2% in 5.3 vs 70.0% in 5.4); I’m not sure whether this is the result of existing tests being removed or new code being committed without corresponding tests, but it is definitely not a trend that should continue.
The consequences of accepting failing tests were never more apparent than with the stable release of PHP 5.3.7. The details of which have been outlined numerous times all over the place (including in my last post), but the gist is that a serious bug that broke a common security component was introduced into the PHP repository, the automated tests actually identified that the bug existed, but the release happened anyway. There has been no official explanation for how this happened, but it appears to be the result of an unfortunate human error. At the time there were close to 200 tests failing, so it seems reasonable to assume that a human may have missed that one of those failed tests was actually due to a severe issue.
There are a couple things that could be done in order to solve this problem. Defining a coverage goal would be an excellent starting point. Ideally this goal would be 100%, but any goal above 80% would likely be a huge step in the right direction. Once a sufficient coverage goal has been defined, all new features committed to the HEAD should be required to have corresponding tests that match or exceed the coverage goal. Unless existing tests are removed, this will ensure that the test coverage can only increase over time. In addition, all current automated tests should run successfully. There appears to be an effort to identify which tests should be expected to fail, but that only partly addresses the problem. If a test is no longer applicable for some reason, then it should be removed. If it is applicable, then a bug exists that needs to be addressed.
The coverage goal and a requirement that all new commits need to exceed that goal can be done within the existing development process. Those are the easy parts of the issue to address. The more difficult issue is resolving the errors from the existing tests. Theoretically, more contributors could help a lot here; some of the failing tests probably belong to components that no longer have active maintainers, so the currently active core developers may not be the best to address the issue. Realistically, the hardest contributors to find are those that are comfortable writing C, and this is also the most difficult type of contribution to make when you’re not already involved in the project. Resolving all of the failing tests will no doubt require both an increased effort by the existing core developers as well as an increase in the number of contributors.
Developer trust is shaken due to lack of communication
The developers managing the PHP project are only human, so mistakes are not only reasonable but also inevitable. The community (none more so than myself) needs to accept that fact and remember this when reacting to future mistakes. Nonetheless, incidents such as the failed release of 5.3.7 have shaken the trust that many community members place in the core developers to manage the project in a professional and stable way, and as a result there is a whole lot of mudslinging that happens when a mistake occurs.
Communication is the key, here. In my opinion, the worst single part of the release of PHP 5.3.7 was not the human error in deployment but rather a breakdown in sufficient communication. The two official means for people to follow important PHP announcements are the php.announce mailing list and the main news feed on the homepage of PHP.net. We are rapidly approaching 2012, but PHP has no developer blog much less an official social networking account (e.g. Twitter, Google+) to follow for technical updates.
The php.announce mailing list used to include security advisories in addition to releases, but this appeared to stop long ago. The announcement mailing list really should include all new releases, security advisories, and notices of unstable releases (e.g. 5.3.7). This should allow any one of the core developers to immediately draw attention to any stability-related/critical issues to anyone subscribed to the list. After 5.3.7 was released, it took four days for any official word to come down the chute that a problem existed. In that time, any number of people could have upgraded their production environments to the broken release, and it didn’t have to play out that way! At least two core developers jumped on and fixed the issue the next day after release. That’s pretty damn good turn-around. Unfortunately, it was still three days before an official notice was posted and a total of five days before a new release fixed the issue. In this case, a swift reaction from core developers was overshadowed by less-than-ideal communication, and that is truly a shame.
It would be nice to see an official development blog for PHP where any of the core developers could post articles about features that are being worked on and, more importantly, post-mortems for when things go to crap (again, 5.3.7). I know the core PHP developers didn’t just brush 5.3.7 away from their minds the moment it happened. I know this because since then the number of failing unit tests has plummeted. I know this because I’ve been personally assured by a few core developers that measures are being taken to make sure similar problems do not happen for future releases. I do not know this based on any form of official communication from those managing the PHP project, and that is unfortunate.
We, the community, know that the core developers are working. We know that they’re trying hard to make sure the core is stable. We know that they are introducing cool new features to help keep PHP competitive and modern. But we know all of this only because we’ve heard it through the grapevine. A developer blog would allow the core developers to speak, on their own terms, about failures that happen and what steps they’re taking to avoid them. This is frequently all that is required to turn a horribly unfortunate situation into a positive experience for the community as a whole. If ever you needed a better example of that, compare the reactions from people losing their accounts and credit card information to Sony versus people losing their accounts and credit card information to Valve. For the former, the near-universal response was outrage. For the latter, most people responded positively. The gaming community did not rip Valve a new one because Valve did an excellent job communicating with them. Despite overwhelming failure, the community trusts Valve due to little more than communication alone.
Also, the features the core developers are working on are pretty damn sweet, so it definitely would be awesome to hear more from them about those ongoing pieces of development. I don’t know if it would help on the “trust” front, but it would be awesome.
PHP itself is not broken; my title is sensational rubbish
I preserved the title from the previous article so that people can quickly make the connection between the two, but the original title really does not reflect my thoughts on the matter.
The PHP community is doing really awesome things. From extraordinary second generation frameworks down to one-click install applications that have unparalleled portability, the extent to which we lead the web is still amazing and humbling to me. The core PHP developers have long been a key component of that, and none of progress that modern PHP applications have made would be possible without their ongoing efforts. As a result of those efforts, PHP is a stable, secure, and beautifully-practical language that is both easy for novices to wrap their heads around and experts to build the most-used web applications the world has ever seen. PHP is not broken, but there are areas that can definitely be improved. Fixing unit tests and polishing methods of communication should seem minor in comparison to the monumental accomplishments of the past 15 years.