Aug 20 ’14

Top 10 Ways to Waste Money on CPU

by Craig Hodgins in Enterprise Executive

Sometimes a little reverse psychology is necessary to effect a positive outcome. In this case, we show you the best ways to tune your applications by showing you how not to tune them. Consider it a worst practices list. If you don’t care about wasting money on poorly tuned applications, or you’re running your z/OS system for charity, you needn’t bother reading any further.  

Worst Practice #1: Fix Problems in Production Only

The first (but not the only) way to waste money on CPU is to fix a performance problem only when the application goes into production. Who cares about pre-production performance when you have a deadline to meet and a user to satisfy? It will sort itself out eventually. The end users, many of whom will be paying customers (think mobile apps), will endure slow response times and even lengthy outages. Your public relations staff will be extremely busy in the days that follow, explaining the situation to the press and responding to unpleasant comments on social media.

Accenture did a study some years ago where they found that it costs up to eight times more to fix a performance defect in production than it does in design. This is because it’s much easier and cheaper for a programmer to change code or infrastructure in the design, unit test or even system test phases. The associated costs increase with each development phase up to and including the production roll out. Production changes are more costly because they require change controls, code or infrastructure changes and more testing, all while your paying customers are waiting for the application to be available again.

Worst Practice #2: Don’t Use Tools

The Pareto Principle, also known as the 80-20 rule, is alive and well in performance. For example, 80 percent of your application CPU will be consumed by only 20 percent of the application code. Chances are, your DB2 performance issue is an SQL statement buried deep within the application. It probably isn’t hard to fix; it’s only hard to find. You might want to get some help, but then again, tools are for amateurs, right? You read dumps and love hex output. It may take you two or three days to find that needle in the haystack (once you’ve found the correct haystack), but it was productive time well spent. Just keep on truckin’ by the seat of your pants and let your competition fool around with those fandangled performance tools.

Worst Practice #3: Don’t Worry About Buffers 

In the old days (circa 1964), buffers were a huge part of performance tuning. There wasn’t a lot of memory to go around, so programmers had to allocate buffers sparingly. Neil Armstrong got to the moon using a computer with 64KB of memory! But today, we have so much memory and everything runs so fast, we no longer need to worry about such things as buffers. Except when we don’t use enough of them.

If you haven’t allocated enough buffers for a busy data set, for example, your I/O subsystem will gladly thrash, resulting in many more EXCP calls than are really needed in order to satisfy one logical request from the application. This is particularly true for VSAM indexing, where the default buffering is rarely sufficient for excellent performance. Each EXCP makes the response time longer, and eats up a little more CPU along the way. There are tools available that will indicate when buffering is inadequate, but your team can’t afford to spend time learning how to use them because they’re so busy fighting fires!

Worst Practice #4: Fight Fires All the Time

You thrive on the adrenalin rush of getting involved in a new performance problem. Hours of investigation and dozens of coffees later, you’ve resolved another problem. Meanwhile, your customers are frustrated and you have wasted CPU.

Take a lesson from the fire department. They don’t want to extinguish fires—they want to prevent fires. Hosing down a house that’s on fire may eventually save the foundation, but the cost is huge compared to avoiding the fire in the first place. Fire detectors are good at this, and likewise, so is performance software. Preventing fires is much cheaper than extinguishing them. This worst practice is often the result of other worst practices, such as not tracking performance and not having a methodology.

Worst Practice #5: Don’t Use a Performance Methodology

Airline pilots use a detailed checklist on every flight. Police officers have a use-of-force specification so they will be consistent in dealing with dangerous situations. Doctors have procedures they follow to determine the root cause of an illness. In fact, there are plenty of capability maturity models that define how to get from poor to best in many areas of life. Although these methodologies have been developed over time using real-world experience, we don’t need them in IT.

Reinventing the wheel is fun and everyone should do it! Or maybe you subscribe to the principle of “not invented here” equals “no good.” Don’t waste time trying to learn from other peoples’ experiences, just keep wasting CPU. There will be another upgrade in a few months. (By the way, methodologies are different from best practices in that a methodology defines the process of how and when to use the best practices.)

Worst Practice #6: Don’t Track Performance

The first question asked when something goes wrong in IT is “what changed?” Specifically, with performance, it might be “so, how fast should it be?” If we don’t know exactly how well module ABC executes when things are good, how will we know when things are bad, and just how bad.

You should be keeping performance statistics on all your major application components (transactions, batch programs, database calls) to use as reference points and develop trends. But who has the time? After all, you have fires to fight!

Worst Practice #7: Have a War-Room Mentality

Go ahead, spend the whole weekend on a conference call with 50 of your colleagues, each of you pointing fingers and getting defensive. You could rely on objective software tools that identify the root cause of the performance issue, but why do that when you could depend on Joe the developer’s gut feel and 30 years of experience? After all, he writes the most complex code you’ve ever seen!

“It’s not my fault” may make you feel good, but what does the war-room mentality cost the company’s bottom line? And did you lose any customers as a result of the extended war-room discussions?

Worst Practice #8: Don’t Tune SQL, Just DB2

DB2 is a huge part of most production systems in IT organizations today. Much time is spent tuning the DB2 subsystem, with all of its buffer pools and DSNZ parms. This is all well and good, but the rubber hits the road when an SQL statement is executed. A poorly written SQL statement can take down a well-tuned DB2 subsystem every time. The DB2 optimizer will try to choose the most efficient path to retrieve the required data, but in many cases, your cowboy coder has made the SQL statement so complicated (showing off his technical skills) that the optimizer just gives up and provides everything contained in the tablespace. Not efficient, but it gets the job done. Tuning SQL may be the biggest opportunity you have to tune your application, but just let the subsystem handle it and it will be alright.

Worst Practice #9: Reinvent the Wheel Yet Again

The mainframe has been around now for 50 years. Much progress has been made in reliability, scalability, security and other important aspects of software development. However, as long as human beings are the ones writing applications, we have to realize the same mistakes will be made over and over again. Who doesn’t start writing a new module by copying Bill’s old code as a template (with all of Bill’s performance defects intact)? Some organizations have developed, after years of experience in the trenches, a best practices guide on how to develop applications with performance in mind. These best practices help new, less experienced programmers avoid making the same mistakes over and over. But your team doesn’t have time to create or teach best practices because they have a deadline to meet! Oops, we’re back to worst practice #1. Oh my. When is that new hardware upgrade going to happen?     

Worst Practice #10: Do Nothing
It’s easy to not care about application performance. But it’s not cheap. And your competitors do care.

Now you have 10 worst practices for getting the most you can from your applications. So, go and have some more coffee before your company has to end the free coffee program to help pay for the excess CPU costs.