First, here’s the tl;dr summary … Benchmarking is for losers, Profiling rulez!
I’ve noticed a couple blog entries in the Planet Perl Iron Man feed discussing which way of stripping whitespace from both ends of a string is fastest.
Both of these entries discuss examples of benchmarking. Programmers love benchmarks. After all, it’s a great chance to whip out one’s performance-penis and compare sizes, trying to come up with the fastest algorithm.
Unfortunately, this is pointless posturing. Who cares that one version of a strip-whitespace operation is three times faster than another? The important question is whether the speed matters.
Until you answer that question, all the benchmarking in the world won’t help you, and that brings us to profiling.
Profiling is a lot harder than benchmarking, which may be why people talk about it less often. Profiling doesn’t compare multiple versions of the same operation, it tells us where the slowest parts of our code base are.
In order to make profiling useful, we need to write code that simulates typical end user use of the code we’re profiling. Then we run that code under a profiler, and we know what’s worth optimizing.
Once we know that, then we can start speeding up our code. At this point, benchmarking might be handy. If, for example, on some crazy bizarro world, our program spent a lot of its runtime trimming whitespace from strings, we could benchmark different approaches, and use the fastest.
Of course, in the real world, this will never be the slowest thing your program is doing. In most cases, the slowest parts of the program are usually the parts with IO, such as reading files, talking to a DBMS, or making network calls. If this isn’t the case, we may be operating on a lot of data in memory with some sort of non-trivial algorithm, and that’s the slowest part.
Either way, without profiling, benchmarking is just a pointless distraction.
Of course, I’d be remiss if I didn’t point out that Perl has an absolutely fantastic profiler available these days, Devel::NYTProf. It actually works (no segfaults!), and produces fantastically useful reports, so use it.