For my first 15 years as an analyst of information management solutions, in every vendor briefing, whenever I could find absolutely nothing else to criticize, I would look the presenter in the eye and ask, “So, what do you do about reorg?”
It was always amazing to see how few of the techie marketing people knew what reorg does. For the first five years, almost invariably, all I got was a blank stare—a teachable moment. For the next five years, increasingly, I also got a “Well, we have this tool … but does it really matter?” Only in the last few years has it been the case that most, if not all, vendors have reorg that operates, somewhat appropriately, automatically, without need for user intervention, often without the user’s knowledge. Even today, I bet, if I went into any large vendor’s briefing or any large enterprise’s IT shop, the two reactions most people would have would be: “What is it, anyway? Does it really matter?” Oh, yes, it matters.
Fundamentally, reorg (database reorganization) is about the fact that every year, in every data management system, the way we use the data we store changes fundamentally. We store proportionally more of one data type, proportionately less of another. We add new data types. We change the database design to handle new kinds of queries or new user demands. We add other data stores to the same disk, then don’t coordinate the new data-store storage with the old. Inevitably, slowly or quickly, a database’s data-store storage becomes more and more suboptimal and so does performance. It just keeps getting worse—until you do reorg.
Reorg fixes things using an “optimization snapshot.” It ripples through the entire data store and optimizes storage as if this database was fresh out of the box and the vendor had just finished tuning it for your initial workloads. Is your indexing suboptimal for the new mix of data? Are there large physical gaps between adjacent fields in a row stored on disk, and is that row likely to be used in a mission-critical query so the disk head is constantly ping-ponging from one end of the cylinder to the other? Is the cache window no longer optimal, so the database is constantly doing page swaps from disk instead of operations in main memory? Reorg handles it all.
When I first started as an analyst, it was interesting, although not too surprising, to discover that Small-to-Medium Business (SMB) database users, such as Progress Software users, badly needed automated reorg, as the non-techie head of the local SMB office had to do all the backups, etc. by himself or herself. What really opened my eyes, however, was when Tandem announced that its NonStop database had easily outperformed IBM’s FastPath at an IBM customer site. An IBM rep came in to explain: It was unfair! The customer had forgotten reorg for roughly a year and a half, so Tandem simply ran NonStop on the reorged data. On the same optimized data store, FastPath would have won! Um … how many IBM customers didn’t do reorg because you didn’t make sure they understood its importance, or you didn’t provide automated tools? Well … I’d better check that out.
During the ’90s, things greatly improved in mainframe databases and elsewhere. Not only was reorg typically scheduled by the database automatically in most information management solutions, but as the demand grew for databases that were available 24x7, online reorg arrived, which did its task (with performance overhead) while the database was running. And so, these days, in most cases, the mainframe databases out there, whether it be Oracle, DB2, or a CA database, run reorg at least once a year, and “database performance sclerosis,” which inexplicably slows response time to a crawl and then an eternity, is pretty much a thing of the past.
But these are vendor tools, and vendors don’t care too much about reorg, because they think you don’t. So, frequently, the reorg you have isn’t set right and the vendor hasn’t reminded you to reset it. Reorg happens too rarely, so half the time your application performance is 20 to 40 percent slower than it could be. It happens too frequently, so doubling of response times at odd times is visible enough to be annoying to your users. It’s scheduled for the wrong day/time of day, so, in times of peak load, your users have additional response time aggravations. It’s online when you should move it offline, or vice versa.
So, IT, do you know what your reorg is doing today? Yes? Happy IT mainframe administrator, take a bow. No? Why don’t you find out, and then ensure you insert a reorg review/fix in your process every year? No one may ever thank you for it, but the response-time benefits for most critical applications should be substantial.
And while you’re at it, please join my ode to reorg. O foster-child of performance and fast time, thou shalt remain, in midst of other data than ours, a friend to IT. Enough Keats. Back to work.