It was the worst ever. Month-end was here and they had laid siege to us. Egad! Now I know how they felt at Dunkirk. We fell back to the machine room. They charged, lobbing jobs at us at horrific rates. “Who is it?” cried management. “It’s the ad hoc users,” we replied, “and a cohort of developers on our flank!” Management turned and ran. “OK, men,” I rallied. “It’s up to us now. We’re going to take back the data center!”
This article examines how a squad of performance management specialists organized and streamlined batch operations using automation and other vendor products.
“You guys hold this position,” I shouted over the crushing din. “I’m going out on reconnaissance.” I dashed out of the machine room, where we had holed up, and made my way to our office area. A phone went off near me; instinctively, I hit the deck. I crawled over to it and checked the caller ID. Sacred bovine! It was Bob Zimway, one of our ad hoc power users. I gritted my teeth and lifted the receiver off the hook as if I were disarming a booby trap. “Hello,” I croaked into the phone. He instantly commenced his verbal assault. I did my best to take the call, cramming my words in between his undeleted expletives. “Yes, Mr. Zimway, I know who you are,” I grated. He went off again like a phosphorus grenade. “We’re doing the best we can,” I rasped. He was having none of it. “A group of actuaries launched a surprise attack early this morning and submitted a whole raft of long-running jobs.” I was shouting by now. “There was nothing we could do!” I heard him fill his lungs deeply with a witty rejoinder, so I cut him off. “We’ll review the situation and do what we can” I said, and hung up the phone as hard as I dared.
I quickly checked z/OS’ System Display and Search Facility (SDSF), and TMON, and then dodged my way back to the machine room to report. I signaled a huddle and my fellow squad members rounded on my position. There was Sgt. Vincent, management liaison and group leader; Jurgenson, our electronic surveillance specialist, qualified on all systems monitors; Meyer, counterintelligence expert, he worked undercover in development back in ’88—he knew the deepest recesses of their minds; and my buddy, Brewer; he defected from open systems a few years ago. Fortunately, he got out in time, before any real damage was done. I was on point that day.
“It’s bad,” I spat out, gasping for air. “The queues are backed up with jobs from all departments. It looks like most of the initiators are tied up with long-running jobs the actuaries submitted this morning. If any more jobs like that initiate . . . throughput will go to zero.” We gave each other hard looks as I caught my breath. “There’s more,” I said in a warning tone. My team studied me intently. “They got the tape drives . . . all of them! They never had a chance.”
These were seasoned veterans, but this was hard to take. “Are there allocation waits?” Sgt. Vincent asked, dourly. I couldn’t look him in the eye. “Yes, Sarge,” I replied quietly to the raised floor. No one spoke for a few minutes. You could cut the tension with a hacksaw. When young Brewer suddenly broke the silence, his words tore the air. “We can’t let them do this!” he barked. “No, we can’t,” agreed Vincent. Jurgenson and Meyer perked up. “It’s time to prepare for battle!” Vincent ordered with savage determination. Suddenly, the room became bright with hope. “I have some ideas I want to work on,” I said. “I’ll check TMON and see what I can do to get some more cycles back,” Jurgenson said. “And I’ll make a few phone calls,” Meyer added. With shoulders back and chins up, we strode out of the machine room and returned to our burlap bunkers, filled with a sense of purpose. We had a mission.
Sgt. Vincent promptly ordered our squad out on patrol and we discovered a number of problems. Several opportunistic actuaries had submitted large numbers of jobs at dawn, monopolizing the initiators. Meyer’s intelligence report indicated they entered the office under the cover of darkness. It was hard not to admire such a dedicated adversary. Many of these jobs were long-running. Once a long-running job started, it rendered the initiator ineffective for processing other jobs. One by one, initiators fell to invading forces and their occupation began in earnest. Jobs piled up behind them like refugees at a border crossing. We were hard-pressed to even react to this dastardly tactic. Our only option was an end-run around Workload Manager (WLM), manually starting jobs with the JES2 $SJ command. This is an effective countermeasure, but we do not want to use it too often. We run WLM-controlled initiators, allowing WLM to optimize initiator management.
Initial media reports indicated that the actuaries had taken control of all tape resources, but these reports were generalized speculations, foisted upon us by embedded news correspondents. Brewer infiltrated the tape library to serve as forward observer and discovered that developers were also conducting extensive data recovery operations. Their jobs captured and controlled numerous tape units. Brewer also noted that the local citizens were conscripted to fetch countless tapes one at a time, dooming them to hopeless drudgery. He could see no way to free them without a full, frontal assault that was certain to result in massive job cancellations.