I have always worked on software that needed to squeeze the last bit of practical performance out of a piece of hardware. Anything less, and I’m bored at work.
For most of the 2000s, this has been financial software, which is a brilliant example of how Moore’s Law solves nothing. See, every year, your average $2000 computer is going to be faster and more capable than last. But with the stock market, it just means that if I can execute 100 trades in the time it took me to execute 90 last year, I’ll probably find a way to make 10 more trades, because if I don’t, somebody will steal profits from me because they’re certainly going to execute 10 more trades. Or I’ll run my models for 48 months instead of 36, so as to have a more accurate programmed model. Remember, the finance markets are like a bad experiment in social darwinism, to a fault. How else can you explain why all of those stupid “me too” startups were funded in the dotcom bust?
This has implications for those of us who have to deal with this stream of data.
Programming tools
Take, for example, profiling. Simple profilers have been around since 1979. But you can’t use one for this kind of software because any instrumenting profiler is going to slow down the software enough that any instrumentation results aren’t going to be as accurate anymore for finding slow code because it won’t be able to keep up with a realistic feed. On the other hand, a modern sampling profiler like OProfile or VTune isn’t going to have these problems.
But there aren’t any good sampling profilers for OpenVMS, so you need to manually instrument code if you can’t run a full profiler.
The same goes for memory debuggers… they wrote their own malloc tracker because nothing else worked.
All of these things sucked up a lot of silent time. Remember, if you put two mythical identical engineers, one armed with OProfile, one without, both are going to look like productive little engineers while they solve a performance problem, it’s just that the one using OProfile will probably be done faster.
This is actually when I realized that I needed to write this article — I was debugging a performance problem at my current job and I realized just how handy it was to toss OProfile at it.
Even things like the compiler were productivity-killers. It hasn’t been until fairly recently that the majority of commonly available C++ compilers all implemented a decent amount of the standard including the library and templates. But once that situation got mostly fixed, there’s a lot of completely brilliant C++ libraries out there, and brilliant articles about how to program C++. Except that any library needs to be ported and the technique may not work. Remember, there are some incredible bits of code in Boost that simply won’t compile on an old C++ compiler.
Thing is, when you are on a minority platform, even if they had used the STL, you know that it wouldn’t have been the same sort of quality that the one that comes with a mainstream platform would have.
Kids these days… sheesh…
Now, you also have to consider the main recruiting push from the company — mostly kids out of CS programs from good universities. Overall, this is a good thing because it provides a constant infusion of fresh blood and new ideas. In the eighties, most of those kids had fond memories of coding on various DEC hardware. In 2000, using OpenVMS was foreign and legacy and they would all be unhappy because things didn’t work like they did in Windows or Unix. So they probably lost at least some folks simply because, even though VMS fans still think that Unix is a piece of crap, nobody cares. People wanted something modern at work, like Linux or Windows.
Similarly, people needed to take some time to get used to the quirks of the almost-but-not-quite STL vector and string classes and the quirks of the compiler. So, every time you got a new person, even if they lived and breathed C and C++, you needed to re-train them.
All of these things suck up time and most of these are hidden costs. It didn’t help that there was some hardware that was fairly questionable. But before we get to that, I’d like to talk about the other side of the coin. In my next article..