The dangers of an all Ethernet setup

[pucho812]

Was at a studio today and everything is Dante and Ethernet connections. I/o, servers, crestron, dadman, the works.
So they had a power outage and their servers won’t come back up. So they are effectively dead in the water. Can’t even look at pro tools on screen as their computers are far away and there is a kvm server controlling all the kvm switches and such.

There is something to be said about running copper and analog lines everywhere. Not that analog equipment did not break but at least we could work around it when it did.
Nothing like a crashed DNS server to prevent you from working. I won’t even address their ups which all need battery replacement.

 

[Brian Roth]

I recall 25-ish (?) years ago when Alesis released the HD24 multitrack. A long-time friend and client ran a 24 track MCI analog studio and decided to buy a HD24 and begin phasing out his JH-24 2".

I warned him he needed to be "religious" about making constant backups. "Dave, have you ever seen a roll of 2" tape just vanish into thin air before your eyes?" He said something like "Yeah, yeah...OK"

A year or two later he lost several album projects at once and had no backup.

Bri

 

[pucho812]

I recall 25-ish (?) years ago when Alesis released the HD24 multitrack. A long-time friend and client ran a 24 track MCI analog studio and decided to buy a HD24 and begin phasing out his JH-24 2".

I warned him he needed to be "religious" about making constant backups. "Dave, have you ever seen a roll of 2" tape just vanish into thin air before your eyes?" He said something like "Yeah, yeah...OK"

A year or two later he lost several album projects at once and had no backup.

Bri

Well if this were a broadcast facility there would be redundancy. There is, sort of. They have duel power circuits on devices that have duel power supplies but they do not do any good if they are both fed from the same UPS. It also doesn't do any good when the UPS has had flashing battery warnings and no one has done anything. The latest is debugging the network switches and how they all go together so that things can be routed back online.

 

[Brian Roth]

Younger folks will laugh at me, but.....if the Mains dropped out during an overdub on an analog multitrack all you lost was that portion of the take. Not the entire album...lol.

And UPS units are notoriously flaky, even if the "bad battery" indicator isn't flashing. You really need a UPS backup for the UPS backup. The less expensive APC units are very flaky.

Which reminds me....I need a small reliable UPS to keep my cable modem/router alive for a few hours while my main machine (a laptop connected to a largish external monitor and keyboard/mouse) can run from it's own internal battery. I can tolerate the small laptop screen for a few hours during a weather outage. I also have two battery powered radios (AAA or 9V cells). I keep those batteries outside of the radios in a ziplock bag sitting nearby.

Bri

 

[pucho812]

Younger folks will laugh at me, but.....if the Mains dropped out during an overdub on an analog multitrack all you lost was that portion of the take. Not the entire album...lol.

And UPS units are notoriously flaky, even if the "bad battery" indicator isn't flashing. You really need a UPS backup for the UPS backup. The less expensive APC units are very flaky.

Which reminds me....I need a small reliable UPS to keep my cable modem/router alive for a few hours while my main machine (a laptop connected to a largish external monitor and keyboard/mouse) can run from it's own internal battery. I can tolerate the small laptop screen for a few hours during a weather outage. I also have two battery powered radios (AAA or 9V cells). I keep those batteries outside of the radios in a ziplock bag sitting nearby.

Bri

well I didn't design this setup and configuration. Sure it's fancy switching with a crestron and things but when it all takes a dump due to a power outage it takes ages to get back and working. As it's been inherited, debugging networks and connections has been primary goal. Now I have a goal in front of that, as the core switch which handles all that traffic has a compatibility issue with the other switches. Some brands do not like to talk to other brands without major hoops. Annoying stuff. I rather just solder all day working on gear or be in the studio recording bands. This stuff although necessary is not fun for me at all.

 

[Spiritworks]

This reminds me of the new Harrison 32C desk, which comes standard with built-in Dante and conversion....
I would never buy it simply for that reason, even if I could afford it.

 

[pucho812]

This reminds me of the new Harrison 32C desk, which comes standard with built-in Dante and conversion....
I would never buy it simply for that reason, even if I could afford it.

It has analog too though. I heard the desk prototype build. it was nice. I had a few qualms like no center detent on eq boost and cut, no center detent on pan pots. The worst offender was hearing the digital noise in the audio path when the monitor pot was at full volume. I assume those were addressed before they did production. But outside of that, the desk sounded like a 3232C if that's your bag, cool.

 

[fazer]

Bleeding edge of audio. Never turn your back on digital.

 

[pucho812]

Never in my years of audio would I have ever thought a dhcp server would be my wheel house but it was. Most of it returned once I reestablished the dhcp servers working correctly.

 

[pucho812]

So today’s adventure was the avid S6….
Again all Ethernet setup… while lotta $$$$ for a mouse. I get the tactile but it’s not really necessary

 

[totoxraymond]

Well, in my opinion, a good controller is a must when you work in post. Speeds up the workflow a lot.

But the S6 in not really a good controller in my opinion and WAY overpriced.
Faders are awesome, but the central screen is awkward, and you basically have to spend half a day setting it up to really get the benefits. Which in my world doesn't make any sense (i mainly mix tv shows and cartoons, so i'll spend between 1 hour and 3 days on a program).

 

[pucho812]

Well, in my opinion, a good controller is a must when you work in post. Speeds up the workflow a lot.

But the S6 in not really a good controller in my opinion and WAY overpriced.
Faders are awesome, but the central screen is awkward, and you basically have to spend half a day setting it up to really get the benefits. Which in my world doesn't make any sense (i mainly mix tv shows and cartoons, so i'll spend between 1 hour and 3 days on a program).

Exactly.

 

[pucho812]

Well, in my opinion, a good controller is a must when you work in post. Speeds up the workflow a lot.

But the S6 in not really a good controller in my opinion and WAY overpriced.
Faders are awesome, but the central screen is awkward, and you basically have to spend half a day setting it up to really get the benefits. Which in my world doesn't make any sense (i mainly mix tv shows and cartoons, so i'll spend between 1 hour and 3 days on a program).

also my OP is pretty about connectivity. Since the power outage at the place there has been a mad dash to get it all back and running.
Leanred a lot of landmines were left by the previous team. The company learned a lot of what needs to happen which they had an excuse for it all. Your main work server for projects with a 120TB of data did not have a mirror backup. Took 4 days to rebuild it and get it working again. took a whole day in itself to work out the DHCP servers which fixed several things but not everything. Now I have some stragglers that I have to attend to.

 

[totoxraymond]

120TB of data did not have a mirror backup

That's a huge mistake...
Although 120TB doesn't look large to me. Video servers have gone wild this last decade. The company were i started as an assistant has about 40 editing rooms and 5 mixing rooms.
I don't know what they have as storage capacity these days, but i know the machine room had dozens of bays full of Digital Betacam when i started. Now they've all been replaced by Avid Nexis.

Also, I'm curious why a company like this had to rely on a DHCP server? This is clearly above my game in terms of network management, but I would think every machine should be on manual IP settings and DHCP could be used for clients macbooks and phones. (And i probably would make this a whole different network physically disconnected from the company's servers)

 

[pucho812]

That's a huge mistake...
Although 120TB doesn't look large to me. Video servers have gone wild this last decade. The company were i started as an assistant has about 40 editing rooms and 5 mixing rooms.
I don't know what they have as storage capacity these days, but i know the machine room had dozens of bays full of Digital Betacam when i started. Now they've all been replaced by Avid Nexis.

Also, I'm curious why a company like this had to rely on a DHCP server? This is clearly above my game in terms of network management, but I would think every machine should be on manual IP settings and DHCP could be used for clients macbooks and phones. (And i probably would make this a whole different network physically disconnected from the company's servers)

I couldn’t tell you why, I walked into the middle of this when I started.
DHCP does make things easier; you don’t have to give everyone their own static ip address. But since I started I have moved specific things to their own static address. Easier for me to manage that way.

 

[JohnRoberts]

It is classic systems engineering to avoid single point failures that can collapse entire systems. Back in the early days the space program used redundant computer systems to survive one computer going down.

It is impossible to anticipate exactly what will bite us next time.... reliability engineering is a rigorous discipline.

[edit] Over the years I've killed a lot of brain cells thinking about reliability engineering. During my 15 years over a significant fraction of Peavey SKUs and when you have that many out in the world getting abused by less than professional customers the designs get stress tested. Routinely weak/insufficient components reveal themselves in field failures. Using massively duplicative parallel designs actually increase the complexity and opportunities for failure. If you can identify one weak link the best fix is to beef up that one weak link.

OK here's two Peavey anecdotes.

1- I had an amplifier module in a powered mixer start failing. The weak link was an anti-saturation diode in the power amp modules pre driver stage. Mainly in the in the 230V markets this diode was failing from over-voltage. In the early days of 230V normalization there were a number of countries pushing higher than 230V, plus the older Peavey SKUs had 220/240V selector switches so customers figured out that they could run their amps in 220V position and cheat out a few more watts of power.

2-when we considered extending our warranty to 5 years (or something like that) for marketing reasons, we did a deep dive into warranty claims to evaluate the cost of extending the warranty. The good news is that most field failures were infant and once the SKUs survived the first 6 months they usually kept working for years. However, in the course of this investigation I discovered one SKU. A modest power amp, that had a horrible service record, but nobody complained because it was such a low volume seller and the service techs knew how to fix it. As soon as I/we determined there was a problem I turned it over to the head of analog engineering. He identified the weak link and upgraded to a more robust part costing almost nada.

3- OK one more story. When I was product manager over all power amps I identified one power amp that was a thermal hand grenade waiting to melt down. It was an example of value engineering that cut a little too much (IMO). I added 4 more TO3 output devices, two to each side, and turned it into a BSH (brick something something). This engineering change cost me less than $10 in parts and as product manager I had the levers to work to sell it profitably by reintroducing it with a new model name/number (you're welcome). [/edit]

JR