29-Apr-2006 (Sat)
Wherein we take two steps forward and six steps back.

We seem to have solved the "missing right channel" problem. It was, in fact, a software problem. We were running Fedora 4, and when we installed the latest patches on March 31, that's when the right channel vanished. We tried downgrading to the version of the kernel and ALSA as of three months ago, and that didn't fix it. But, Jonathan took all the sound cards home and tried them in his machine, and they all worked fine there. He was running Fedora 5. So we upgraded to that, and the problem went away.

That's right: upgrading to the latest FC4: breaks the world. Giving up on FC4 and going to FC5: un-breaks it. Nicely done, guys.

(By the way, this means it was almost three weeks before anyone mailed me to let me know that the archives were screwed up. Ten of the archives got screwed like this before I even knew to start looking into it...)

Last night was the first unattended archive after we thought we fixed it, so this morning I gave it a listen, and -- hooray! -- it's stereo again. Then I skipped forward a bit, and was almost deafened by a squealing test tone. It seems that some time around 1:08 AM, the webcast abruptly turns to a high pitched shriek and stays that way for three hours.

I'm thinking of the immortal words of Socrates who said, "OMG WTF?"


For years I've had it drummed into my head that you always have to keep your systems patched, if you aren't running the latest security fixes, the script kiddies will eat you alive, running a six month old OS is like leaving your front door wide open, blah blah blah. Well you know what? Fuck that noise. I'm done upgrading anything ever. The next time I get this shit into a state that seems even remotely stable, I'm never touching it again. If we get hacked, oh well. I have backups. It has got to be less work to recover from than constantly dealing with this kind of nonsense.

37 Responses:

  1. otterley says:

    First rule of administration: "If it ain't broke, don't fix it."

    • codenazi says:

      hmm... would it be considered "sane" to do the following?

      1. Keep workhorse system stable (read: no patches)
      2. Put some fancy [hardware|software|magick-fairy-dust] firewall in front of it

      So you can have the KISS benefits of not "fixing" a "not-broken" system, while still having some security from the outside world?

      I suppose your shoutcast/icecast/whatever would be transparent through the firewall, but that would only be one thing to keep patched on the workhorse system, not everything?

      Blarg. Soooo many people (F/OSS in particular) don't seem to understand the idea of regression testing.

      • dagbrown says:

        I would go with the "stuff it behind a firewall and leave it the hell alone" option, myself.

        To make a completely worthless recommendation: Yamaha makes some pretty decent (as in "not toy") firewall/router boxes, but haha, good luck finding them outside of Japan.

        I asked another geek I know and he said "Get a Linksys and use OpenWrt", which suggestion I also include for its comedy value.

        Actual useful ideas? You got me.

  2. allartburns says:

    Aren't you the person who used to lecture me about free software being worth every penny you pay for it?

    Seriously, tho, why not just buy something that runs OSX?

    • jwz says:

      I'm seriously considering it, but I'd still have to run a lot of the same old open sores like LAME and Icecast...

    • rodgerd says:

      Jamie has never had any problems whatsoever with his MacOS X boxes.

      There will always be pain. It's just what kind of pain you want.

  3. jered says:

    There is hope for the future. My understanding is that <lj user="xiphmont"/> is working for RedHat on fixing all the audio brokenness. Monty is perhaps the most obsessive person I know, from writing audio compression routines, to raising bettas, to restoring bassoons (a prerequisite for learning how to play, of course), and so forth. I have high hopes.

    Of course, I just use a Mac for anything I care about now.

  4. chromal says:

    *sigh* FC4 is doin' much to damage the 'name' of linux distros. It's never been not-flaky. I knew it would be trouble from day one, when it refused to install on standard PC hardware.

    Full aware that this may come off sounding like "recompile the kernel!"... I don't know if this is a Fedora team problem, a GCC 4.0 problem, or what, but I tell you, you'd be much better off running CentOS if you find bullshit patch problems and things bein' fundamentally broken an irritating trait in your OSes. FC4 has been nothing but a "let's let the users do all FC's alpha testing" release.

    • georgedorn says:

      And before that, Red Hat wasn't so hot, either.

      • chromal says:

        heh, don't get me started about RH < 7.2's RPM problems. :} > 7.2-9 seemed mostly tolerable.

        • yakko says:

          It was, in fact, RHS 7 that prompted me to never run RedHat on any of my machines ever again. I don't run it at work, even, preferring to use Solaris or another Linux instead.

          That said, FC3 and FC4 seemed to work OK on the machines I tried it on at work. Then again, I only used it for mundane tasks, like DHCP, an internal web server and a glorified X terminal.

    • codenazi says:

      O.O

      *boggle*

      FC4 is using GCC 4.0?!

      Is that even considered remotely stable?

      I knew I bailed from the RH line of things a long time ago to debian and/or gentoo, but wow... even my "I totally don't care about it bleeding edge" gentoo box is still running GCC 3.4.6 with good success.

      I'm suddenly very glad I have never looked at the FC series. Maybe they are sane if you pay them big $$$ for the support contract? ...or maybe "a fool and his money are soon parted"?

      ouch!

      From the supposedly "stable corporate" linux disto, too. The more I think about it, the more I like the "debian-stable" theory of things. Who the fsck cares if it's recent software for most things... stable is more important...

      • chromal says:

        Well, I think part of the idea behind the FC project was to provide a community-based testbench OS for the development/testing of ideas and code to eventually be rolled into RHEL. As such, it's free-- I don't think there is such a thing as a FC support contract, nor can it be purchased (I might be wrong?). FC is not really a replacement for the OS track that ended with RH9; it's something a bit different. I became concerned when OS patches to FC3 (I think that there were over 500 bugfixes/errata by the time FC4 was released) started to *change* things; I remember one morning a patch to mount changed it so that it would no longer mount UDP-only NFS shares, breaking things across the enterprise. Well, I guess it served us right for thinking we could transition from RH9 to FC3.

        But, yes, that was the overall impression that I got; GCC 4.0.0 broke code that compiled fine in GCC 3.x.

        Really, CentOS seems to be where it's at if you like RPM-based distros. Free, but not a broken-code playground, so it's enterprise-ready (or just no-nonsense ready).

        • codenazi says:

          ahh... you have dug up bad memories of why I was blocking FC from my mind. One of those distos that should have been a nitch thing, yet everybody seems to run because it's "RedHat" or something.

          I wonder if they actually got their entire distro to compile with GCC 4.0.0; I remember downloading it when it came out, due to the potential of "cool new features", and finding that things failed to compile more often than not. I would be very surprised if they got that to work. I'm more expecting they used 3.x to compile the distro, and just ship with 4.x, just to fsck with everybody's head.

          waiting at least another year before looking at GCC 4.x for sanity reasons...

        • Fedora is positioned as an open R&D environment, and Red Hat doesn't offer support contracts for it.

          FC is a bit more leading edge than RHL, while RHEL is a bit more stable than RHL was.

          And yes, FC4 uses GCC 4.0.2.

  5. For years I've had it drummed into my head that you always have to keep your systems patched, if you aren't running the latest security fixes, the script kiddies will eat you alive, running a six month old OS is like leaving your front door wide open, blah blah blah. Well you know what? Fuck that noise. I'm done upgrading anything ever. The next time I get this shit into a state that seems even remotely stable, I'm never touching it again. If we get hacked, oh well. I have backups. It has got to be less work to recover from than constantly dealing with this kind of nonsense.

    Isn't this sort of one of those bad idea "Bring 'em on!" statements?

  6. edouardp says:

    Isn't that upgrade thing the reason SGI (hey - *still* in business) went to a dual upgrade path? You could choose either the "Maintainance" upgrades , or the "Feature" upgrades. The theory going that some people actually cared about the stability of their systems, and wanted to only apply the critical upgrades, and not have to get all the other, unwanted, possibly-broken, junk?

    But of course, what did they know - after all they produced proprietary (spit!) software. All hail RMS! All hail 15-year-olds writing device drivers!

    • codenazi says:

      which would be the Debian theory on things, if I remember right.

      "stable" branch only gets security updates (no feature changes, security patches back-ported, in theory)

      "unstable" having everything else

      I think it pisses off a lot of people (read: trend-following developers, users, etc) that some of their software is a 2-3 year old version... but it works well if that's what you care about.

    • rapier1 says:

      In the way back when I worked at UPenn and one of the sysadmins regularly got chewed out because she kept applying the latest irix patches and breaking everything because... the first rev of the irix patches *always* broke everything. That was the first step towards realising that I desperately wanted to not be a sysadmin. Which, fortunately, I no longer am.

  7. fantasygoat says:

    Imagine the glorious world we would live in if nerds actually tested shit before moving on to the next shiny object.

  8. ronbar says:

    The security guys at work are ALWAYS bugging me to install the Solaris "recommended" patches on everything every week. They're a humongous set of patches to individual packages which can be backed out individually or en-masse. I refuse because a) I don't know what they're upgrading and why, b) it ain't broke, don't fix it, and c) good hygiene demands a reboot after installing them.

    They say if we get hacked I'll lose my job. I say I keep up with the CERT advisories; that's the best you can do. Every other patch is useless and a potential headache unless it's specifically addressing something that isn't working for us.

    In any case I have the bare minimum installed on my machines. OpenSSH, OpenSSL, a few useful things like top and lsof and nano, and the software each server is supposed to be running. No GNOME, no X, no CDE, no emacs, no Netscape/Mozilla, no XEmacs. (Nothing personal on the last two.) I don't even put video cards in them-- if I do the security idiots will inevitably log in to the console after forgetting how that thar SSH works again, lock the console instead of logging off (never mind the "security policy" on that), and three weeks later X messily shits itself, sometimes taking everything else down with it.

    All that other crap can be installed on my workstation. If it breaks it's not keeping anybody else from working.

  9. malokai says:

    Use centos (www.centos.org), they recompile the RHAS stuff. Works rather well for me, especially on servers.

    Only problem I ever had was that their httpd-devel packages seem to be continously fucked in magical ways that never happens on my RHAS boxes.. Other than, don't upgrade software that isn't broken? Granted slinging beer is a time consuming process but if you have a SA, why not make him read the package changelogs?

    • mark242 says:

      You're new here, aren't you?

      • malokai says:

        No. I just like pretending jwz has gotten bitter enough to make other people do this kind of work & to make those decisions.

    • kamaraga says:

      why not make him read the package changelogs?

      ...and while we're fantasizing, I'd like to have enough spare time to read the changelogs for all 134 updated packages that the Fedora repositories shipped this week, and for these to contain useful information like, "alsa-driver-1.0.5fu-1: Broke stereo implementation and added loud screeching noises", "openoffice.org-calc-2.0.1.1.1.1-2: Liberally sprinkled code with additional memory leaks rather than fixing the six-month-old bug everyone keeps yelling about where the whole program hangs solid if you try to move cells with drag-and-drop", and "sendmail-8.13.6-7: Still haven't gotten around to including the patches against remote root exploits released months ago". Oh, and I'm still waiting for that pony.

      • malokai says:

        Hence the centos suggestion, which doesn't update quite as often.. Mind you I have a mixed RHAS/centos environment so I get the nice alerts from RHN whenever there's a problem, so I get a better idea of what's going on. That and I don't trust problematic packages from vendors when the original provider tends to do a better job (postgresql, openoffice, eclipse..).

        Despite my attempts to use it, I still say FC is trash, purely for the reasons you just listed.

        Hell, I haven't even updated any packages(recently) and now my X server doesn't init the display properly on startup or something and I have to switch to a virtual terminal and back to the X display in other for anything to show up. The whole internet experience over the last 15 years has made me want to retire out into a shack in the woods.

        • kamaraga says:

          I mentioned ALSA, OpenOffice and sendmail specifically because these programs have been broken as long as I've used them. When authors ship defective code, distributors are forced to make difficult decisions. Fedora ships early and often to allow authors to fix their own bugs, RedHat tries to ship the least broken versions of packages, while OpenBSD tries to rewrite broken code, etc. Meanwhile, proprietary vendors ask users to bet the farm on their promises to care. All these options are wrought with peril and individuals are stuck picking the least-awful alternative.

          Jamie's decisions to go with Apple was a good one, and the kindest thing he's likely to say about CentOS would be, "Same shit, different label". I acknowledge the complaints made about Fedora in this thread, but their significance depends on the user. When ALSA broke, I took a minute to switch to ARTS. When the bundled httpd package didn't meet my needs, I rolled my own. When the new NFS package hung a test box, I found a workaround and deployed new mount maps before it affected users. When a sendmail patch came out, I applied it myself because I knew better than to wait for the vendor. It never even occurred to me that I was supposed to feel angry because this was "business as usual" on every OS that I've administered professionally in 14 years. Obviously "It Should Just Work(TM)", but the last OS that lived up to that claim was Netware 3.x, and that was mostly because it didn't do much.

          The whole internet experience over the last 15 years has made me want to retire out into a shack in the woods.

          You should run for the hills now before you're overwhelmed with bitterness at the upcoming crapitude of dreadfully misguided attempts to use CSS, AJAX, Flash 'Blaze', and Web 2.0.

          • malokai says:

            The "I patch because they said to" mentality is what's broken with it, especially with something as bleeding edge as fedora. Sendmail itself seems to have chronic problems which is why I replaced it a long time ago with qmail, then postfix.

            And wtf is flash "blaze". Goddamn it, something else I'm gonna have to shoot down when the developers try to pad their resumes with on the company dime I suspect.

            I got used to everything that you described wrt to patches not done by the vendor shortly after realizing that the commercial unix industry had the same problems. Especially sun.. "Here, this sendmail has MX support compiled in." "No it doesn't." "Oops, this one does." "Nada." "Errr, this one!" "Look, just give me a fucking compiler already."

            You get what you(don't) pay for, and when you pay for it you don't get what you wanted.

            I wish I could find that post by the glibc guys about what gentoo did to glibc for a while (disabled some compile time security checks to stop broken & insecure code from being able to run or something along those viens).

            The centos suggestion was merely for the sake of his admin, it's a much slower moving target for people that are busy. I tried to keep track of my fedora install once and gave up on that shortly there after - I've got real work to do.

            • kamaraga says:

              Ooops, sorry for the late reply.

              Especially sun.. "Here, this sendmail has MX support compiled in." "No it doesn't." "Oops, this one does." "Nada." "Errr, this one!" "Look, just give me a fucking compiler already."

              As long as vendors consistently ship broken code, admins are stuck applying a flurry of half-baked patches in the hope that this latest one actually works. RedHat, Fedora and Sun are 10X better than most other vendors in these matters -- be very glad if you've never had the displeasure of working with IBM, Sequent or Oracle's patches.

              And wtf is flash "blaze". Goddamn it, something else I'm gonna have to shoot down when the developers try to pad their resumes with on the company dime I suspect.

              "Blaze" is the upcoming version of Flash that you can code against using a general purpose programming language and is almost as powerful as Java, and the demos show them using it to quickly code an MP3 player and email reader. Like CSS and AJAX, Blaze is a very good thing but is also likely to be misused in remarkably annoying ways.

  10. kamaraga says:

    Character: JWZ
    Alignment: Lawful Evil
    Class: Bardic Sage

    Modifiers:
    +6 Take fantastic photos under otherwise impossible conditions
    +5 Software Distortion Field induces fatal bugs in otherwise flawless binaries at 50 paces
    +4 Manage nightclub under conditions that would reduce mere mortals to tears
    +4 Induce rage at bugs most people have resigned themselves to accept or switched to avoid
    +3 Attract attention of lame city ordinances, licenses and fees
    -3 Saving throw against inept general contractors
    -6 Saving throw against bathroom vandals

    Mortal enemies:
    - Dreaded Brain Wyrms of David McCusker
    - Misguided Database Practitioners
    - Cascade of Attention-Deficit Teenagers

  11. dasht_brk says:

    Why not do something about it? MacOS sucks. Fedora sucks. There's nothing that doesn't suck in this space. Let's take over. You've already admitted that yr not *that* busy and, hey, I'm not saying you should spend lots of money here... just some brain cycles. Ok, a *little* money. Not much though. You can pay for the web site once things get that far and buy a few pizza-lunches meanwhile. Should take about 20 of us to flip this bird on it's ass.

    -t

  12. bifrosty2k says:

    I am always glad you are around to remind me that my choice to abandon Linux was the right one.

  13. makomk says:

    Hmmm - there was an interesting alsalib bug a few months back that could cause a missing right channel on recordings. (More specifically, it mangled the mixer settings such that the right channel came from Mic In, but everything looked OK in alsamixer and similar programs.)