15-Feb-2004 (Sun) Wherein we mostly dodge the power-failure bullet.

Friday night I was meeting some folks for dinner, and we met at the club before opening. "Why don't I give you the 'lights on' tour of the club", I said, thus jinxing everything: for at around 9:30pm, the power went out on the whole block. The end of the tour was the 'lights out' tour. My friend Brad, who runs his own business, asked why I wasn't freaking out. I said, "because there are other people here to freak out for me, if freaking is warranted. They'll handle it." It's tough learning how to delegate, and he's trying to figure out how to do that better. So a little while later, Alexis had the yellow pages out and was looking for generator rentals... "See?" I said.

Fortunately the power came back on shortly after 10pm, because it seems that we didn't have a prayer of actually getting this place up and running on a generator, and being closed on a friday would have cost us a ton of money... Alexis got a hold of the electrician who installed our new system to ask him what we should order and how to hook it up, and his answer was something along the lines of, "hmmm, well, that's pretty complicated..." Apparently the fact that (as described in the July 2000 "Ask Dr. Science" interlude) we have "delta" three-phase power instead of "star" three-phase power, and that we actually use the mostly-useless 208v line (most people just ignore that one, and only use the two 120v lines) makes it tricky to hook a generator in...

So, that was a bullet dodged. I guess we're going to have our electrician come back out one of these days and like, put in some terminals that say "pull switch and attach generator here" or something.

    "Push to test."
    "Release to detonate."

We also learned that our five UPS boxes can only keep the eleven PCs in the back office running without wall power for something like twenty minutes. That's like, not so good. So we missed the beginning of the webcast, since the power only came back on after 10pm, and the machines didn't run the 10pm "start the webcast" cron job until I did it by hand...

So later that night, I was showing Brad one of the screen savers that runs on the kiosks that scrolls text around in an arty way; the source of that text is set up to be the most-recent posts to LiveJournal, the blogging web site that Brad created. I started this screen saver and the first thing it printed was "open your wrists and the pain ends forever"! It continued on in that 10th-grade-goth-poetry way for a while, and he accused me of setting that up ahead of time. I hadn't, it was real-time. "Oh my god my users are so dumb!" he exclaimed.

Just then, some guy came up to us and said, "yo, yo, you gotta go to pimps and hoes dot com! Check it out!"

"You think your users are bad?" I asked.

24 Responses:

  1. That last bit about the "pimps & hoes" dot com dude was genius. And completely unscripted? Wow.

  2. feren says:

    I'm sure there's some sort of ordinance for your neighborhood that will require a hearing and a permit if you were to try and have a generator permanently installed. Frankly I'd be surprised if you could rent one temporarily without inciting somebody's wrath.

    When my employer went to have a generator installed for the new data center it was quite an ordeal. First we (MIS) argued with the company facilities managers about the need for a generator. Once they conceeded that shutting the entire business down was a bad thing we then had to argue with them about what kind of generator we needed (natural gas versus diesel... why this was a debate is anybody's guess) and then how big it should be. If you get a diesel generator that's providing too much power and you don't make use of it all you can fall victim to "wet stacking." Then we had to argue with the building managers who ran the office complex we rent space in, because they didn't want it installed on the roof and due to city zoning laws for office high-rises they couldn't lose any of the parking space around the data center, which eliminated the possibility of us annexing a space or two for a slab to be poured on. Once we got that worked out to everyone's satisfaction we applied for the permit to get the contractors on-site for wiring and installation -- said permit was promptly denied by the city council. It seems the building that houses the data center is right behind a condominum complex for semi-rich yuppie folk and they objected to the presence of a genny because "it would disturb the peace and lower their property values were it to be run on a regular basis." The fact that we were hoping the generator would in fact not have to run on a regular basis apparently escaped them.

    • travisd says:

      Actually, if you're not running it weekly or at least monthly it likely won't work when you need it.

      Our 800KW and prior 400KW both sounded like a Semi-tractor with the accellerator floored within about 3 seconds of kicking on. And we run it weekly for about 30 minutes at a crack to "exercise" it.

      A colleague up in NJ has been dealing with generator zoning issues. The most workable solution seems to be get a "rental" and leave it on a properly road-licensed trailer in a parking spot outside the building.

      Of course, if you have the room you could go completely inside with it. Our NYC datacenter has two 800KW diesels in the basement of a high-rise.

      • feren says:

        [Actually, if you're not running it weekly or at least monthly it likely won't work when you need it.]

        This happened to my father a few years ago. The city installed a generator for his firestation to power the dispatch center's radios and keep the apparatus floor online. They had a power failure in the summer and they stayed dark. After fifteen minutes of absolute silence my dad went out to check the system and discovered that it had never been run since the initial load test. The batteries had gone dead -- it couldn't start. Not terribly useful under those circumstances, especially since something that big can't be jump-started by your average pickup truck.

        Ultimately what my employer did was cut a deal: we got patched into the big genny that was already installed on the property by another company for their datacenter, which was conveniently located underneath our own. So, in essence, we're leasing time on their unit from them. It's way over-sized (apparently they didn't care any about wetstacking, and told the power company to jam the biggest system they could into the room), so it's more than adequate to run our NOC and their own. The owners run a monthly test to keep it lubricated and the batteries charged while we check our transfer switch and the genny on a quarterly basis. It's quite a thing to behold, that V-12 Caterpillar. As you said, when it first starts up it sounds like the world is going to end.

    • jwz says:

      We wouldn't get a generator permanently installed: we don't need it often enough for that to be worth it. I was just talking about having our electrician install some obvious patch points for a rental.

      (No wait, what we should really do is get a two ton flywheel installed on the roof, like above.net has...)

      • billemon says:

        I was just imagining someone saying it would "not be appropriate" to have a generator, ... or a flywheel on the roof for that matter ... :)

      • kfringe says:

        Get the flywheel installed inside the club. You may have to knock down walls, but it has advantages.

        1. It's really cool to look at.
        2. You could rig some kind of floating stage over it. Stage diving would then be an "extreme sport."
        3. Annoying promoter disposal.
        4. Less dancing == more drinking.
        5. It would be really cool to look at.
        6. People could finally say "Jamie's lost it entirely," and be right.
  3. giantlaser says:

    Man, am I glad we don't have to deal with zoning laws in Baghdad. :)

    Here, every rich person and business has a backup power scheme. Some have a shared generator between several houses - my office/house gives free power to the three nearest houses. Some have small ones on the roof for critical systems. Some have enormous engines that run diesel and emit enormous clouds of black smoke. Some have mufflers. Some sound like Harley-Davidsons sized for 4-meter giants.

    I haven't seen a house yet that doesn't have a giant mad-scientist-like switch for moving between city power and local. Ours has an automatic kick-in next to it that starts the generator when the voltage starts to drop, but the UPS units usually beep once before it fully starts.

    My university physics classes on electricity didn't do shit to prepare me for the realities of Iraqi electrical systems. I can tell you what the average voltage and cycle rate is for city power and our generator, but I have no idea what size or capacity the generator actually is. Variable-voltage PC power supplies become your best friend.

  4. bifrosty2k says:

    Seeing as how I've had to do some of this research, I can probably hook you up with a generator guy. It took way longer than I'd wanted to get the whole generator thing sorted out but its all done now.

    Renting a portable generator costs about $1000 every 8-12 hours, especially for the size your're looking at. Its probably not so big of a deal for you if you have to do it every 9-12 months; but if you do it more often, you can buy a used generator for about 10-15k if you know where to go.

    If you can arrange to park it somewhere nearby, you're golden.
    The nice thing about portables, is you don't need a permit :)

  5. tjcrowley says:

    This is where you need someone like <lj user=netik> and <lj user=baconmonkey> to jerry rig the sound system with extension cords from around the corner where they have power like they did up the street at Studio Z.

    Oh wait, you have <lj user=baconmonkey>. Those two are like Scotty and Geordi Laforge working together.

  6. harryh says:

    we actually use the mostly-useless 208v line

    For what exactly? Curious...

    • jwz says:

      Part of the problem is that I don't remember exactly. It's either

      1. The 208v line goes to a transformer that then runs the lights, or
      2. The fans and compressors on the roof are wired three-phase.

      I don't think those can simultaneously be true, but it's one or the other.

      • luminalflux says:

        There are two 120V and one 208V lines? Isn't three-phase "over there" symmetric (as in 3 220V lines, here in Europe), or is this just another of the DNA's oddities?

        • jwz says:

          Let Google be your guide: Delta vs. Wye Power: "Wye connected power has two different voltages available. The Phase to Phase voltage is the main system voltage (typically 208 VAC or 480 VAC in the United States). The Phase to Neutral voltage is also available, and is typically used for small single phase loads (120 VAC or 277 VAC)."

          • luminalflux says:

            But what makes one of those wires 208 V and the others 120? Or am I missing something?

            • forthdude says:

              The voltage between any two of the phases, say phase A and B, is 208 V. The voltage from neutral to any phase is 120 V. The phases are seperated by 120 degrees of phase shift, hence the term "phases".

      • txdave says:

        208V is commonly used in commercial buildings for lighting. Something about florescents ballasts being more efficient. Because the low volt (120) panel is fed through a transformer off a breaker in the high volt panel, you should be able to simply power the HV panel with a transfer switch and a 208V 3phase generator (which it looks like rentalservice.com have)

  7. sfllaw says:

    You might want to look into anacron, for your webcasting cron job. That might prevent you from having to kick things off manually.

    • mjgardner says:

      I'll second anacron -- it's great on my PowerBook for those maintenance jobs that OS X likes to run at night when both it and I are usually sleeping. I open it up, it notices the jobs haven't run and lets 'em fly.

  8. asan102 says:

    Disadvantages: Current flywheels have low specific energy. There are safety concerns associated with flywheels due to their high speed rotor and the possibility of it breaking loose and releasing all of it's energy in an uncontrolled manner. Flywheels are a less mature technology than chemical batteries, and the current cost is too high to make them competitive in the market.

    Wow, If I knew there was such an extreme form of energy backup, I would have one installed already.

  9. kingfox says:

    So, any chance of that hilarious screen saver being made available for the masses?