I know I bitch about work a lot but seriously, the ass-clown that designed this system had shit for brains. After implementing what was in the documentation I sat down and looked it over really well and made a few changes. In the Veritas install in particular I was able to go from a 30% success rate and a 20 hour backup window to 96% completion rate and a total backup time of a little over 11 hours. When I told this to the organization leads yesterday I swear they shit their pants, really. They asked why wasn’t it designed like this initially, I just shrugged my shoulders and told them I didn’t know. And then the came back and said “Well it must have been for a good reason”. WTF? The management was totally scared/ amazed of the Engineer that was in my place before I, they thought he was a genius, nobody questioned his motives and designs. I have had to reengineer half of this whole design, I simplified it, made it more robust and fault tolerant, and I get little to no gratitude for the effort. I implemented MirrorView on the pair of Clariion CX500s that we have at our sites and set up a full LUN replication at a block level ensuring that if one site was to be removed from the face of the earth all data would remain intact. The entire system uses about 8-12Mbps at load and next to nothing at idle time on their 100Mbps MAN that they are using between the facilities, and they moan and complain that perhaps we should have implemented the pervious solution. We are using a synchronous replication that ensures ZERO data loss and a minimal RTO, in the previous implementation we were taking snaps of LUNs and then doing a full LUN replication on a 4-8 hour schedule. Meaning that every few hours we were copying an ENTIRE LUN across the network instead of just replicating the changed blocks. Again WTF?
Anyway I finished up most of the hard work and we are approaching our Pilot date (Tuesday). I’m really hoping to have another contract lined up in the month or so following our Pilot and then Main Rollout, perhaps I can design the next system from scratch and not have to deal with other peoples asinine mistakes.
OK my rant is done