Any thoughts on the switch debouncer mentioned earlier? Forget the diode, a 5V regulator would probably work better and switch common would be routed to the same ground as the fader.
That device appears to be a switch debounce device to debounce a switch that comes before it. And it's a logic level device rather than an analogue switch ? And it has 20ms delay.
Debouncing is not your problem here. The relay looks like your best and no compromise solution as I see it.
But if you can I'd still look at increasing the supply voltage on the 4053.