Efinque,
happy to know you decided to play safe and forget the main/line experimentation for now...
Could you make a sketch of what you have done?
Because from what i understand you used an riaa preamp to drive a pot which drive a transformer.
You'll have sound but this is far from ideal... not enough overall gain to drive line output, not enough current too. At least you'll need a buffer stage before the transformer but even with that i doubt this would work as expected...
Usually an riaa for MM cartdrige (moving magnet, the one used by most dj) will give you something in the 40/45db gain range.
A more or less standard MM dj cartdrige will output something around 5mv (0.005 volt).
With only an RIAA max output will be around -4 to 0 dbu max output (and so without headroom: 0.5 to 0.77 volt!).
I've also been thinking of giving up on the upper "gain" knobs since they don't do much else than act as an additional resistor in the circuit..
Have you ever studied a mixer schematic?
Because... to have gain in a circuit (beside the use of a step up transformer but forget that for a moment) you need an active stage (something with a transistor, an opamp or a tube) to provide the needed gain.
The pot you want to discard is usually placed
after the riaa module and
before or around* an opamp to give something around 20db gain before going to (eq or in your case a) fader.
*: before or after will depend of the kind of topology used for the gain stage for opamp.
Take a look at the attached schematic for example:
for the 4th chanel (the upper most on the schematic with phono4 input) you have tl072 ic113 1/2 which is the Riaa preamp from which the output is send to tl072 ic114 1/2 which is the gain stage which will drive the eq part which in turn drive the fader (well in this case it don't drive a fader but a vca but let's assume this is a fader for easier understanding of the principle).
The pot r116 1/2 is a dual 50k pot which determine the gain of this stage. This is the pot you want to discard and as you can see this is not passive and it needs gain...