This show has been an adventure already.  I needed to set up a new networking/node inventory for my newer-to-me team.  I’ve done this before, so I wasn’t worried.  But then some mysterious issues kept popping up.  I ended up chasing my tail for a quite a while.

One source of challenges was the newer Zyxel gen of switches (3rd gen) requires a fussy-to-me interface to open up the switches for configuration.  It would appear to save the manual IP configuration, but when I powered them back up after a while, they would be back in default DHCP mode.  But the far sneakier one is two of five brand-new nodes we bought were bad.  Not bad like- they don’t turn on or won’t ping.  No, they were bad like dropping universes after two or three minutes.  So the issue looks exactly like a multicast problem, which is deeply concerning to me.  Even more confusing for troubleshooting- both nodes were on the same set.  If it’s helpful, here’s a bit of the list I built to isolate the problem:

• Start with basics- check all IP addresses, IGMP snooping status, subnet and Gateway.
• Remove/bypass switches between one set and the other
• Trade cables
• Transfer the console over to the misbehaving set and see if the behavior changes
• Use a different console (my IonXE had just been upgraded to Windows 10, so it was technically a variable)
• Retest between switches using Multicaster 2 (greatest tool for lighting networking ever)
• Remove the Querier switch and test with Broadcast
• Shift the trouble nodes over to the good set- does the problem follow the hardware?

Eventually, I called in a friend (the simply amazing Chris Balo) to double check everything I did since the problem remained.  We both came to the conclusion that the nodes were the culprits.  Extra confusing since they were brand new and from a company I have always had great experiences with.  They worked with me to RMA and get replacements and now everything is cool.  But since it took a lot of my time and energy, I always look for the lesson or technique to help me not get into a similar mess again.

I think my takeaway when setting up new nodes and switches needs to be a “prove it” approach.  Moving forward, I hope to be able to set up all network devices first thing during prep with 100’ network cables between and “burn it in” for the week.  Doing prep isn’t exactly the same as show conditions, but it’s close enough to test for many issues.  I can just keep an eye out for dropped universes and test each port one at a time when I have hard line fixtures to test. (Interesting side note- we tested using Artnet when sACN was being fussy and it didn’t work in a whole other way. Turns out the two misbehaving nodes were also not following the “art net offset” setting as well. )

It’s a simple idea that takes advantage of the prep time as networking test time as well.  In retrospect, it’s an obvious idea.  But we don’t tend to come up with good ideas until we have things go wrong.

What networking system tips and tricks have helped you?  Share in the comments.  It takes a village to set up a good network!

Photo by Scott Rodgerson on Unsplash

***

Mark LaPierre is a programmer in film, television and theater based out of Albuquerque.  He grew up in live entertainment and has been a designer/programmer for musicals, concert dance, live music, circus and corporate.  Mark is a proud member of IATSE, an ETC Eos trainer and an enthusiastic trainer of many other platforms and subjects.  He offers remote console coaching in 30 minute to 4 hour times as well as a full range of training in person.  If you enjoy his content, please consider commenting on his posts on the website to appease the Algorithm.