A small mixer, gains at parity, is probably the best solution I agree.
If you want to knock this up however, here is how I would do it. The tie-down resistors should minimise switch pops by grounding DC at each switch position. A transformer balanced preamp input with no phantom applied should work fine with this, but I agree that pulling the monitors down might be prudent, at least until you verify that it is safe to leave them up.

(XLR pins to the left, switch contacts on the right, duplicate per channel.)