Yea,
I agree with Butta- maybe lose the output pot altogether? As soon as you apply any attenuation it's effectively placing a resistor in series with the transformer primary, and the subsequent loading effect will definitely affect the output.
Try placing the output level pot at max, and listen to the sound of the pre. Then back it down a bit, and make up the gain, and see what happens. If you have had the output pot at say 60%, and you've relied on the preamp to boost the gain from the mic, it's possible that the stretched-gain of the pre has caused bandwidth-limiting, and combined with the series-load of the pot on the output stage, has caused a "darker" sound.
An output transformer in a circuit like this must be driven from a low impedance source.
Mark