This schemo does not make much sense., the full diagram of each amp goes attached here and you are welcome to comment on it.
The potentiometer swings from an in-phase signal to an out-of-phase signal.
I wonder how you came to this gain structure.
What overall gain are you targetting? Including the elusive headphone amp's gain?
What acoustic level? what capsule sensitivity?
Anyway, your global gain switching doesn't work; you need a 3PDT switch.