Wow its been a long week. I have been very busy at work. Had a little bit of drama on Thursday and Friday. I was told to reboot all of our SAN attached servers on Thursday night to include the Brocade fibre channel switches. Well it was about 7 in the evening and I had been racking and configuring servers all day, and I just wasn’t thinking, so I rebooted ALL of the servers at the same time. Bad move. We have 4 mail nodes that operate in 2 clusters using Microsoft Cluster Services. Well for those that are not familiar with the way cluster services work, you have to have external storage for use by the cluster, this is where the SAN comes in. Only one node in the cluster can have control of the shared storage at any one time, or you risk corruption. Well stoopid me, I rebooted all of the cluster nodes at the same time, thus they were all coming active at the same time. So of course the mail nodes that serve the entire LA area, including Las Vegas, San Diego, Phoenix and Honolulu attached to the SAN and tried to take control of the discs. It was at this moment that I realized that I had screwed up royally, and shut down one of the nodes in each of the clusters, hoping and praying that I didn’t screw something up. I wasn’t so lucky. After I rebooted again, the mail nodes could no longer identify the Quorum disk (disk that is used to manage the cluster) ad well as the Information Store (drive where all the Jet databases are stored) and the Transaction Logs (similar to SQL trans logs). So here it was 7:30 PST and nobody back in DC or anywhere else that would have any idea on how to recover everything without restoring everything from tape. So I went into work at 5 AM and made the calls back to DC, and those guys back there were already working on it. So to wrap this up, it took about 6 hours plus many calls to Microsoft to finally get this thing resolved. It turns out that instead of corrupting the discs, it simply lost the signature for the discs, so they appeared as RAW partitions in the Disk Management snap-in. Once we rewrote the signatures everything came back up and running. So really the fix was only about 5 minutes. But still about 2200 people were pissed off most of the day. I’m probably gonna get yelled at next week for sure.
OK that was pretty damn geeky. Other then that not much else is going on. Oh one more thing, if you are in need of a Gmail account let me know, 1GB of google indexed mail rules.
Kelly got REALLY burned yesterday while at her shoot for Girls Behaving Badly, so we are probably just going to take it easy this weekend. I spent most of last night watching the rest of American Choppers and Motorcycle Mania 3 that I TiVoed on Fathers Day. Maybe we will go and get some wings and beer for dinner tonight. No wings and no beer make Casey something, something…..
It looks like the ITech conference is going to be held in Orlando, FL this year instead of Scottsdale, AZ like originally planned. Which is OK with me, since I would like to go back to Disney World, and check it out again. Kelly and I messed up going to the Magic Kingdom on our honeymoon. So maybe we can go back to Epcot or something. There is still speculation that i will be helping teach a class, but no official word yet.