This is my account of one of Cori's early implementation issues. The company at that time was located over in the Murray Business Center and I carried the phone for technical support and hosting. In fact, I was technical support and hosting and depended on the engineering group for second tier support. In those days I'd get a complaint call from a client, diagnose, and dash out to nab an engineer to assist me in working through the issue. Usually I found folks to assist based on who had what expertise. Sometimes it was a matter of who was just unlucky enough to be around when I came looking.
From: Terry Ishida
Sent: Friday, July 09, 1999 6:02 AM
To: Brent Weide; Kevin Goodman; engineers
Cc: Ted Spooner
Subject: RE: Basking in the light at the end of another tunnel.
I really want to congratulate everyone involved on the Crestar scramble. As I'm sure everyone knows, Crestar is our largest customer, and with the addition of OFX, they've gotten even larger. It is so important that we keep all of our clients happy not only because it is one of the basic tenets of our company, but because it really differentiates us from S1 or Edify. Thank you to all of the engineers who responded quickly and decisively to Brent's alert. And thanks, Brent and Kevin, for all of your hard work.
Sent: Friday, July 09, 1999 10:32 AM
Subject: Basking in the light at the end of another tunnel.
With the rollout to OFX Legacy users in additon to it's growing html/ofx user population, Crestar's Production Internet Banking/Bill Pay system came under very heavy load early this week. In fact, the systems 90,000 users began experiencing timeouts and long delays in accessing the systems.
To make a long story short...All is well in the world as of late last night.
Here's what we did to fix our world's problems...
*Upgrade bill pay components to latest build (ccexecutive, etc.) This reduces bill pay processing time from around 3.5 hrs to just over .25 hrs.
*Repair unwieldy stored procedure for ofx add payee/payment (this affects most installations of Voyager, if you have one, contact Paul M.)
*Increase bandwidth of frame relay to Crestar/CheckFree. (56k to 256k)
In it's hayday, this problem manifested itself by producing delays in Voyager's ability to send data to the client mainframe and receive requested data back again. This delayed users currently on the system and caused delays in subsequent users waiting their turn to run a transaction. Tbrstatus displayed up to 700+ user connections, IIS became flooded and unresponsive, Internet Protocol packets were dropped causing retries and compounding the traffic problem through the frame relay.
Our client, Crestar Bank worked with us on this issue by assisting in debugging, providing user information, and agreeing to accelerate a major frame relay upgrade. Jeff Madison, Kevin Goodman, and the three Pauls (Ainsworth, Murphy, Gomes) did an outstanding job of diagnosing the Voyager issues, and quickly resolving the problems. CheckFree provided very good support in the frame issues.
As with all dark moments, a few good things did come out of this. For one, I learned not to trust AT&T for valid bandwidth utilization, and most importantly, I picked up a new language skill! Most people do not know this but Jeff Madison begins speaking in perfectly unintelligible phrases after only about 20 hours of work!
I now know the sound that Jeff's head makes as it bounces off of one of the racks in the lab. He had fallen asleep in a chair and his head lolled off of the chair at somewhere around three in the morning.
I also learned how much it costs to get my truck out of impound...We began work at about 10:30pm on Wednesday night. As on most late night shifts, I parked my truck under the overhead next to Ted's spot. Well, the next day, we worked through the day and I forgot to move to another parking spot. The wannabe cop that patrols the lot took one look at my beater truck in the middle of the line of shiny BMW's (standing out like a turd in the fog), and stomped upstairs to notify Lynnette that it was gonna get towed. I found about it as Amanda dashed into my office, shook me and shouted something like "your car.....up on two wheels.........tow truck........gone........now!!" and ran out.
It was too late by the time I ran out so Kevin gave me a ride over to the tow place where I tried to pick out a newer model but they wouldn't give and insisted in giving me my old truck back instead (but not for free!).
Thanks again to everyone who helped out.