1 and 9 dB readings are delta (ie difference). So the RMS voltage depends on the reference level. You can monitor your output signal level in the DAW and take a note of your reference bypassed level in dBFS. Then your goal is to set the threshold so that the output level is 9 dBFS less (adjust the threshold and ratio only, don't touch your signal gen!). For example, if you get -10 dBFS reading in bypass, you should get approximately -19 dBFS when compressing.
If you insist on AC volts RMS readings, I suggest setting your output level to a convenient reference (for example, 0 dBV), and work from there. You should be able to calculate V RMS for -9dBV easily.