Today Microsoft released the stable 1.0 version of their WinCache accelerator. Although Linux is still the most frequently used platform at Ibuildings, we have a significant and growing number of PHP on Windows deployments, and since WinCache is designed to speed up PHP applications it is an interesting product to us. The best way to put an accelerator product to the test is to benchmark it. While it's important to realise that benchmarks have their drawbacks, they are very useful to make relative comparisons, and that is exactly what we will be doing in this article.

The benchmarks were carried out by Mark van der Velden, one of our PHP on Windows platform specialists.
I'm going to describe his setup in a minute, but for those unfamiliar with the concept of acceleration, I'd like to point to this Wikipedia article which is a good primer on how PHP accelerators work and why you should use one.

The Infrastructure

To conduct the benchmark, we set up 2 servers with the following specs:

• Dual Quad-Core 2.5Ghz Intel processors
• 8GB RAM
• Windows Server 2008 RC2
• IIS 7.5

To keep the results clean, we ensured that there were no other users using the server at the time of testing. Mark used one server as the web server, and the other as the benchmark server. The benchmark machine simulated load on the web server, and measured the performance of the web server in requests per second.

The Applications

It is irrelevant to benchmark a PHP stack using 'hello world' or similar scripts. Firstly because it doesn't reflect a real world situation, secondly because in the case of accelerators, it would skew the results significantly (the compilation step of a hello world script is so much more complex than actually rendering 'hello world' that we would see a ridiculous performance gain on such a script). To make the benchmark as close to a real world test as possible, we wanted to conduct the test using real world applications that represent typical usage of a PHP stack. We chose applications that use a database since this is the most common type of web application. While this means that the benchmark does not purely test the accelerator, it does give us meaningful results, because the same applications will be tested both with and without accelerators, and the database would be a constant factor between both tests.

To conduct our tests we chose installations of the following applications:

• WordPress; the defacto blogging platform. It is frequently used, has a decently sized codebase and is comparable to many real world applications.
• SilverStripe; a CMS rising in popularity. The reason for selecting SilverStripe is that much more than WordPress or other CMS systems, it is a well-designed, object oriented application; the type of application that we see more and more. Its heavy use of OO makes it a good candidate to test the performance of the accelerator on OO code.

In both cases, we set up a default install of both applications and added a little bit of content.

The Stack

We are benchmarking the performance of WinCache but for comparison we've also benchmarked it against a popular competing accelerator to see how it fares. Both accelerators were tested in their own PHP stack, but on the same server using the same IIS configuration. All stacks were configured to run PHP 5.3.

The stacks tested are:

• Microsoft Web Platform Installer (WebPI); this is Microsoft's PHP stack, which comes bundled with with the php.net community build for Windows. Tested both with and without WinCache enabled.
• Zend Server; this is our reference environment for comparison. It is Zend's PHP stack that's been out for a while. We've used the beta of Zend Server 5 as it's the most recently released version and we know that it has some improvements in its accelerator. This stack was tested both with and without Optimizer+ (its accelerator component) enabled.

The Test

The actual test consisted of simulating load on the applications running on the stacks. All tests were conducted for a couple of minutes at a time, and all tests were executed five times to even out any system anomalies and get good average results. Mark tried to keep the load of the machines just under 100% to avoid the risk of overloading the servers and causing the scripts to malfunction.

The first couple of tests were a disappointment; Mark was reporting numbers that suggested that the WebPI stack was twice as slow as the Zend Server environment, even with WinCache enabled, it would be slower than Zend Server without its accelerator. The odd thing however was that the CPU usage on the WebPI stack would refuse to go beyond 50%, so we knew it should be able to perform better, but no matter how hard we hit it, it wouldn't let us consume more than 50% cpu. We consulted Microsoft and quickly discovered why we were seeing disappointing results. As it happens, a default install of the WebPI configures IIS to allow only 4 concurrent FastCGI instances, whereas a default install of Zend Server configures 10. For an environment without accelerator 4 instances might be sufficient, but the accelerator makes the scripts so much faster that the machine can run more than double the amount of FastCGI instances. Ruslan Yakushev (our IIS contact at Microsoft who is involved with the WinCache project) actually has an article on exactly this problem; when you install WebPI with WinCache, be sure to take this into account and reconfigure your 'maxInstances'.

After solving the initial hurdle by giving both stacks 10 maxInstances we had comparable setups and were able to get good results. (Note: according to Ruslan's post we could have used a higher value, but for our level of concurrency we were getting good results with 10.)

The Numbers

Below are the results from Mark's benchmark tests. We are interested in the amount of requests per second each environment can handle, as this is a good indication of an application's performance.

Benchmark Results Breakdown

Again, I'd like to stress that it's not useful to look at the absolute numbers. If you run tests on your own servers on different applications, you will get different numbers. The key information comes when you look at the difference between the tests with and without accelerators. To make this easier, I've added a 'gain' number, which indicates how much faster a stack has become in comparison with its no-acceleration counterpart (which has a factor of 1).

The Analysis

Based on the above numbers we can observe the following:

• The out of the box performance of Zend Server and the Web Platform Installer are very much comparable. Zend Server is slightly faster.
• In all cases we see an increase in performance after enabling the accelerator by a factor of 2. Regardless of the outcome of the benchmark, it is always useful to run an accelerator.
• In absolute numbers, with SilverStripe the Zend Server stack is faster than the WebPI stack, but given that it's non-accelerated result was already faster, the actual acceleration of WinCache is higher (2.0 vs. Zend's 1.9).
• With WordPress WebPI is faster, with SilverStripe Zend Server is faster. The probable cause of this is that Zend's stack is more optimized to run OO code, whereas WebPI is more optimized to run procedural code. This however is not proven by the benchmark so it's an assumption. In both cases, WordPress is accelerated more than SilverStripe is.
• WinCache is a very young project, which makes it all the more interesting that its results are so close (even slightly better) than Zend Server's accelerator, which has been around for a couple of years. While Zend's accelerator is proprietary, Microsoft's accelerator is open source, available through PECL (some might expect this to be the other way round but this is how it is). Being open to improvement from anybody, we might see WinCache's results improve significantly over time.

Variations

During the tests, Mark experimented with a couple of additional configurations and we found some interesting outcomes from this:

• PHP 5.2; Some of the tests were run using PHP 5.2 too but the results were similar enough not to list them in the benchmark results. One interesting thing to not from the php 5.2 tests was that without accelerators, out of the box php 5.3 performed significantly faster than php 5.2, but the use of an accelerator compensated this: with accelerators turned on, both php 5.3 and 5.2 seemed to have equal performance.
• Tuning the accelerators; The benchmarks were performed against the out of the box configuration (with the exception of the changed maxInstances configuration), but Mark tried to tune both of the configurations by playing with their settings. In both cases, we were able to gain some performance, but also in both cases we noticed that it's very easy to misconfigure the settings and degrade performance. We recommend to test thoroughly if you want to play with the settings of an accelerator. While you may think you're increasing the performance, you might just as easily be degrading it.
• Tainted environments; If you run the benchmarks on your own environment, make sure to do clean tests on freshly installed stacks. We initially tried to install both stacks and disable one or the other. This however gives tainted results, because the stacks change environment variables that influence each other. Microsoft recommended us to uninstall Zend Server when running tests against WebPI and vice versa, and we concur with this advice.

Final thoughts

Our benchmark illustrates that WinCache, despite the fact that it's very young, is a must-have add-on if you run PHP on Windows using Microsoft's Web Platform Installer. Since it's free (both in the beer and speech senses), there is no reason not to use it, and considering it not only makes your applications run faster but also helps save energy, using an accelerator is the right thing to do.

I'd like to thank Microsoft for providing us with the final build of WinCache shortly before the release date so we could prepare this article for release right after the final version shipped. This happened at our request and Microsoft made no effort to dictate how the benchmarks should be run or to influence the outcome. They were aware that we would publish our findings regardless of the results, yet they still were very helpful providing us with the tools and helping us solve installation issues. I'd also like to thank Microsoft's Open Source Technology Center for providing us with dedicated Windows machines and the necessary licenses.

Finally I'd like to thank Mark van der Velden who has spent a significant amount of time designing, preparing and running the benchmarks.

Suggestion for further reading: Cal Evans has recently written a whitepaper about PHP on Windows that might be of interest to those of you using or considering deploying PHP on Windows.