That MBB switch on the signals is not hot either. You are shorting two outputs together at each transition. If you look inside a passive unbalanced switching box, all grounds are bussed together, L and R, usually RCA's bussed in a cluster. Only the signals themselves are switched, usually with interlocking DPDT pushbutton switches. Rotary is better with BBM.
I second the "ground as pop" theory. To start with, all equipment should have 3rd pin power cords. A lot of unbal stuff does not. A simple source switch should not require any additional components if it is wired correctly with the correct switch or relay. When I install my relay based monitor switchers, the popper is usually an unbalanced amp with a 2 prong cord. Connect the chassis to power 3 rd pin and no pop. My switchers, balanced, buss all L shields together and all R shields together.
Mike