Microbenchmarking PHP: performant code, now with 20% less superstitious handwaving

Last week, Matt Farina tossed me a question about the best approach to introspecting code in PHP, particularly in relation to whether or not the situation was a good candidate for using PHP's Reflection API. The original (now outdated) patch he gave me as an example had the following block of code in it:

I've used Reflection happily in the past. I've even advocated for it in situations where I later realized it was the totally wrong tool for the job. But more importantly, I'd accepted as 'common knowledge' that Reflection was slow. Dog-slow, even. But Matt's question was specific enough that it got me wondering just how big the gap ACTUALLY was between the code he'd shown me, and the Reflection-based equivalent. The results surprised me. To the point where I ended up writing a PHP microbenching framework, and digging in quite a bit deeper.

My hope is that these findings can help us make more educated judgments about things - like Reflection, or even OO in general - that are sometimes unfairly getting the boot for being performance dogs. But let's start with just the essential question Matt originally posed, and I'll break out the whole framework a later.

FYI, my final and definitive round of benchmarks were performed on a P4 3.4GHz with HyperThreading riding the 32-bit RAM cap (~3.4GB), running 5.2.11-pl1-gentoo, with Suhosin and APC. With Linux kernels, I strongly prefer single core machines for microbenching; I'm told that time calls on 2.6-line kernels get scheduled badly, and introduce a lot of jiggle into the results.

Is Reflection Really That Slow?

NO! In this case, a direct comparison between reflection methods and their procedural counterparts reveals them to be neck in neck. Where Reflection incurs additional cost is the initial object creation. Here's the exact code that was benchmarked, and the time for each step:

The comparison between these two functions isn't 100% exact, as ReflectionClass::getInterfaceNames() generate an indexed array of interfaces, whereas class_implements() generates an associative array where both keys and values are the interface names. That may account for the small disparity.

While it wasn't part of Matt's original question, curiosity prompted me to test method_exists() against ReflectionClass::hasMethod(), as it's the only other really direct comparison that can be made. The results were very similar:

These direct comparisons are interesting, but simply not the best answer to Matt's specific question. Although the procedural logic can be mirrored with Reflection, Reflection provides a single step to achieve the exact same answer as took several procedurally:

This logic achieves the same goal more directly, and so is more appropriate for comparison. It's also a nice example of how the Reflection system makes up for some of its initial object instanciation costs by providing a more robust set of tools. Now, the above numbers don't exactly sing great praises for Reflection, but given all the finger-wagging I'd heard, I was expecting Reflection to do quite a bit worse. As it is, Reflection is generally on par with its procedural equivalents; the big difference is in object instanciation. It's hard to say much more about these results, though, without a better basis for comparison. So let's do that.

More Useful Results

Benchmarking results are only as good as the context they're situated in. So, when I cast around in search of a baseline for comparison, I was delighted to find a suitable candidate in something we do an awful lot: call userspace functions! That is:

Because foo() has an empty function body, the time we're concerned with here is _only_ the cost of making the call to the userspace function. Note that adding parameters to foo()'s signature has a negligible effect on call time. So let's recast those earlier results as numbers of userspace function calls:

1. Checking interfaces
   • class_implements(): 3.6 function calls
   • ReflectionClass::getInterfaceNames(): 3.7 function calls
2. Checking methods
   • method_exists(): 2.0 function calls
   • ReflectionClass::hasMethod(): 2.7 function calls
3. Logic from Matt's original patch
   • Approach from original patch: 4.5 function calls
   • Approach using reflection: 8.7 function calls (3.6 if ReflectionClass object instanciation time is ignored)

These numbers should provide a good, practical basis for comparison; let 'em percolate.

Let's sum up: as an introspection tool, Reflection is roughly as fast as its procedural equivalents. The internal implementations seem to be just as efficient, as the primary cost seems to have more to do with the overhead of method calls and object creation. Though creating a ReflectionClass object is fairly cheap as object instanciation goes, the cost is still non-negligible.

My interpretation of these results: Given that Reflection offers more tools for robust introspection and is considerably more self-documenting than the procedural/associative arrays approach (see slide 8 of http://www.slideshare.net/tobias382/new-spl-features-in-php-53), I personally will be defaulting to using Reflection in the future. And, if using the additional introspective capabilities of a system like Reflection early on Drupal's critical path (bootstrap, routing, etc.) means we can make a more modular, selectively-loaded system, then their use is absolutely justified. At the end of the day, Reflection should be an acceptable choice even for the performance-conscious.

...With an important caveat: The thing to avoid is the runaway creation of huge numbers of objects. Many reflection methods (ReflectionClass::getInterfaces(), for example) create a whole mess of new objects. This IS expensive, although my benchmarks indicate each additional object instanciation is roughly 1/3 to 1/2 the cost of instanciating ReflectionClass directly. So be sensible about when those methods are used.

My Little Framework

To do all this benchmarking, I wrote a small framework that does four crucial things:

1. Allows the function to be benchmarked to be specified externally
2. Runs two loops for each benchmarking run - an inner loop containing the actual function to be benchmarked, which is iterated a configurable number of times, and an outer loop that creates an sample set (of configurable size) with each entry being the result of the inner loop
3. Processes results, calculating standard deviation & coefficient of variance; additional mean result values are also calculated by factoring out both a configurable time offset, as well as the time offset incurred by processing overhead for the framework itself (the internal offset is calculated on the fly)
4. Repeats a benchmarking run if the result set's coefficient of variance > a configurable target value

Since I had the framework already together, I ran some more tests in addition to the ones above, mostly focusing on object instanciation costs. The results are in this Google Doc. In addition to the results from the Reflection Comparisons tab (which are from the first part of the blog post), there's also data on the costs for most other Reflection types with a wide range of arguments under Reflection Instanciation. The Object Instanciation tab, there is data on the instanciation time for a small variety of classes; the range of times they require is quite interesting.

Some oddities

Though I put forward static calls as a baseline before, if you look at the framework, you'll notice that it uses a dynamic call. Interestingly, dynamic function calls work almost exactly as fast:

I glossed over this earlier because, within the confines of the framework, these two have almost exactly the same execution time (variations are totally within error ranges), whether or not an opcode cache is active. This strikes me as strange, as there's no way dynamic function calls can be known at compile-time...not that that's the only relevant consideration. But I don't know the internals of PHP, let alone APC, well enough to grok how that all works. So for the purposes of these benchmarks, I assumed the two to be interchangeable for the purposes of results-gathering. However, because I don't trust those results to be accurate without confirmation from someone with greater expertise, I'd rather people not make that assumption when writing real code.

Also, there is one case where Reflection differs notably from its procedural counterparts: object instanciation. While the other methods were generally on par, the cost of $refl->newInstance() vs. new $class() consistently differed by approx 0.21s@100k, or around 3 function calls (see the results for _do_refl_instanciate() vs. _do_proc_instanciate() under the Reflection Comparisons data). I suspect this is a result of the difference between a method call vs. a language construct, as the difference is similar to that of the difference between a static function call and call_user_func().