7-Mar-2020 (Sat) Wherein I need someone to replace our webcast

Our webcast cameras are so infuriating and flaky. The thing that I have learned, after all these years, is that you must never, ever try to do an HDMI or SDI run longer than ten feet. Anyone who tells you that it's possible to do otherwise is a liar or a fool.

So I've got 16 SDI cameras and associated switching gear, and it's all useless. I think the only way to make this work is for the cameras to be IP-based. Such cameras are probably not too expensive these days, but "not too expensive" multiplied by "sixteen" still means "there is no way I can justify spending that much money on this".

Hey, you! The rich techie who reads this blog and rarely goes out! How about you donate a new system to us? I need 16 IP cameras plus whatever hardware/software combo is necessary to switch between them under scriptable control. (And probably 16 more Cat5 home-runs, which is like... half a mile of cable.) Eventually the video has to make its way into OBS.

I know a lot of you watch this dumb thing, from wherever you are out in the world. So which one of you is gonna step up?

24 Responses:

  1. Jieves says:

    So, if you decide on a system what if we looked to chip in 1 camera each until there were 16?

    • jwz says:

      This is not really the kind of thing you can do a little at a time. It's sort of an all or nothing deal.

  2. I'll be happy to help on the software/streaming side of things.

    On the network side, is there a specific reason you're considering 16 individual home runs, rather than having a few PoE switches in strategic places with an uplink run for each of those back to the closet where the OBS box is? Even at 4k, we're talking ~45mbit per camera, 1080p is closer to 8mbit, so aggregating multiple shouldn't be a problem for gigabit, and could make the whole project a lot easier.

    Looking at the 3d model of the club and having a rough idea of where the cameras are, it seems like you could probably do a switch in Above DNA up near the door somewhere, and maybe another up on the right side of the main club or in the DJ booth up top.

    For cameras - probably best to do a trial run with a couple different models of camera to see which one handles your lighting situation the best. I'd be willing to loan you a couple cameras I've got sitting around to test with, they fit the "cheap" requirement but I'm not sure if they'd give you anything but hot garbage with all the flashing lights and fog.

    Once you've narrowed down which camera would be the minimum acceptable, maybe a Kickstarter would be one way to go for funding this? Or perhaps a level of Patreon donor that would give access to high-res streams while the general public gets low-res?

    • jwz says:

      In the distant past, I tried to minimize cable runs by having several hubs at strategic locations and it was Bad and Wrong. And people who claim to know things seemed to have the consensus of "home runs or GTFO".

      Though if you do the math it "should" be ok to just dump these cameras on our regular network, if "should" means "not" as it often does then that means these cameras make our POS or our wifi fail and that's just more crap I don't want to have to fix, so putting them on their own physical layer seems prudent.

      Anyway, the wiring is the least complicated and probably least expensive aspect of this shitshow.

      As to testing cameras first: yes, that would be wise, but it's also difficult, given the flag-day, all-or-nothing aspects of this stupid project. Until the whole damned thing is deployed you can't really know which component is the weak link. I've been stomped to death in this rodeo before.

    • jwz says:

      Oh yeah, and "higher rez streams" can't happen because Sonic still doesn't have fiber in SOMA. There's no way for us to get a better uplink than DSL without paying thousands more per month.

      (Before anyone suggests putting a dish on the roof, we tried that years ago and it was unreliable bullshit.)

      • Yeah I wasn't gonna poke that bear, but since you brought it up... :D

        IIRC you're still on the old-school 6mbit ADSL offering rather than the newer "Fusion" bonded ADSL2+/VDSL offerings? Sonic is really bad about telling existing customers when they can get faster speeds for less than their current costs - I made the switch at home a few years back and got 4x the speed for half the price.

        I assume C***ast is out on moral grounds? As much as I hate the company and the monopoly, they really are the only decent option for good speeds in most places.

        • jwz says:

          You brought it up!

          We have two Fusion bonded services (4 lines total). One modem is for webcast and POS, the other is for public wifi. We are getting about 13 down 1.2 up on each. At home (a mile away) I get 26 down, but that's DSL for you.

          • Oh yeah, I guess I did mention 4k and 1080p for factoring the internal network bandwidth requirements, but I figured it would still be downsampled to hell and back for the upload. My bad :D

            That corner of SOMA is definitely a dead zone according to http://www.broadbandmap.ca.gov/ - most of your neighboring blocks list 5-6 offerings available, but then your block drops down to just Sonic.net and Comcast. Odd that AT&T doesn't list it as a service area, given...well, you know the whole deal. Maybe there really is just something messed up with the lines there, and AT&T is like "this far but no further" while Sonic was like "well...we'll see what we can do"

            Comcast claims to be offering 1gbit service for $89/mo (* for the first 12 months) in that area though. The 200mbit service ($49.99*) would probably be more than enough though, even for multiple HD streams.

            Maybe drop one of the busted fusion lines, keep the business/POS stuff on the other (I've got both at home too, and Sonic is absolutely more reliable than Comcast), and use the Comcast one for the stream plus the guest network?

            • Zach says:

              I'd be curious if Webpass (er, Google Fiber Webpass as they now call themselves, sigh) is an option. They aren't on the state broadband map, but their own site claims to already serve multiple buildings on DNA Lounge's block (their office is a few blocks away on 8th). It would be a ton more bandwidth than what you're probably paying a lot for from Sonic, but I'd be worried Webpass's installation for business service could be too pricey.

      • pagrus says:

        Monkeybrains' network is a lot more redundant and stable now than it was a few years ago, it might be worth giving them a call. Plus their office is like a block away from you.

        Full disclosure: I work for them, just upgraded my house and get 440Mbps

        • jwz says:

          "Network" being stable has nothing to do with the fact that getting your internet via a dish on the roof is great as long as you don't want internet when there is wind. Or rain. Or a light fog. Or a pigeon glances at it funny.

          No. Just no. No no no.

          • pagrus says:

            OK well I certainly know better than to try to convince you, but the offer stands if you would like to chat.

  3. andris kasparovics says:

    If you can't run SDI cameras longer than 10' you're doing it wrong. You are attempting to fix a situation by making it more complicated and painful going IP cameras with all new infrastructure. Most of the work I do is all switched SDI multicam with runs that regularly go 100' or more. If the existing gear you have is not total crap then you have a chance of making this work with what you already have. Shoot me a list of what you've currently got installed and I can see if there is a way to make your webcast life suck less.

    • jwz says:

      The SDI cameras do long runs (20' to 200') into a Blackmagic Design Smart Videohub 20x20 video switcher in the sound booth. The SDI output from that switcher does a long run (150' or so) into a Roland VC-1DL frame sync in the back office, to smooth out the glitches that occur every time the switcher switches. (Without that, downstream HDMI devices freak out every time the input changes, since the source cameras do not have external sync.) Then that SDI output does a short run into a Blackmagic UltraStudio MiniRecorder which converts SDI to Thunderbolt. That goes into a Mac Mini, and we run OBS to encode the video and upload it to our video hosting service (currently JanusXR).

      Some of these runs have SDI repeaters on them. They don't seem to help.

      Camera models vary, but I think they are mostly Everfocus EDH6210.

      The failure mode is that, often, when one camera is selected, we just see a still frame of the last frame of the previous camera that was working. (I think this behavior is from the Frame Sync.) Sometimes, a single camera is always dead. Sometimes, it's intermittent and will freeze or stutter even without having been switched.

      We have no known method to fix it once a camera goes wonky. Sometimes, some combination of repeatedly power cycling everything and jigging cables does it. Sometimes not. Every connector has been replaced countless times. The switcher is inside a closed box that does not get touched or jiggled.

      All of this gear is from 2013, so it's kind of old, but I can attest that it is neither more nor less flaky today than it was on the day we installed it.

      • cbw says:

        My strong suspicion is you've got an SDI clock sync issue. It's been a few years since I worked in the mystical, magical world of broadcast, but SDI revolves around everything being synced to a reference sync source.

        I couldn't find the model of camera you referenced, but looking at Everfocus's site, it looks like they aren't broadcast cameras, and probably don't take a reference sync signal (basically in a broadcast environment, you've got a master clock like this providing that reference signal, and you fan it out through the studio, feeding it into every video source... the cameras all have a reference input port on them. when you're bringing in external signals, you'd run them through a frame synchronizer - something functionally similar to that Roland box you've got).

        The Blackmagic Videohub is a router not a switcher, and I am assuming it's expecting its inputs to already be synced, and just passing through Weird Things when it makes a source change not on a frame boundary.

        Something like the ATEM Television Studio HD would do the trick: The switcher will convert inputs to the switcher’s standard, and combined with input re-sync, this means you can simply plug in and it just works! (they can't say it just works if it's not true!).

        You'll get clean switches out of this, and could dump the Roland frame sync as well since you're outputting a clean signal (and bonus! can do transitions, effects, and graphics, oooh!). The BlackMagic Web Presenter may be a cleaner solution to getting the bits into the Mac, but if the MiniRecorder is working as expected, then probably no reason to fuss with it.

        "Buy more broadcast shit" may not be the answer you were looking for, but it's likely cheaper than buying a bunch of network cameras and running all new cable, and whatever associated nightmare comes with trying to get a bunch of IP cameras to act like broadcast cameras.

        • cbw says:

          Also, FWIW they make a 12x12 Smart Videohub CleanSwitch model now which has a frame sync on every input.

        • jwz says:

          You are correct, the cameras have no external clock and the switcher is unsynchronized. That's why the Roland VC-1DL frame sync is sitting right in front of the Blackmagic UltraStudio MiniRecorder.

          • cbw says:

            Have you tried just running a single camera through without making a switch on the router for an extended period of time to see if it's stable?

            I'm wondering if the Roland just sucks/has a bug/etc. and eventually freaks out. A frame sync is designed to run off a reference sync source as well, so you're already tempting fate by letting it float.

            • jwz says:

              No, it's so intermittent that I'd have to like... do that for days, and watch it the whole time. I don't have any automated way to detect it.

    • lcrs says:

      Agree that SDI is normally bomb-proof - every broadcast studio in the world is full of long SDI runs with people standing on the cables all the time, spilling drinks on connectors etc. Maybe you have some horrifying grounding situation? HDMI, sure, is trash.

      • jwz says:

        Well if someone wants to come in and fix it, that would be fantastic. But "works for me" is not a particularly helpful take. Having fought this crap for years I'm ready to table-flip it and burn it all to the ground. Because at this point I wish we had never "upgraded" from NTSC.

        • lcrs says:

          Fair - but "works for every broadcaster in the world" is surely a hint that SDI is unlikely to be at fault no? My first port of call would be a sycning switcher - that videohub is absolutely not designed to switch inputs live and provide a coherent output. There's a newer model which states "If you cannot guarantee perfectly timed source signals but you still need to clean switch then Smart Videohub CleanSwitch 12x12 is the special model of Videohub that features resynchronization on all inputs so you get perfect clean feeds every time". Sadly only 12 inputs but I really suspect that would help you out and remove the need for the band-aid frame sync box.

          • jwz says:

            Ok but "every broadcaster in the world" presumably spent 100x as much on their build-out than I did, as I am not a TV studio, I am an unfrozen caveman.

Comments are closed because this post is 3 years old.