The LF distortion is caused by the ripple on the CV, but for not the reasons you think. If the bridge, and associated components are not matched extremely well, low frequency CV feedthrough components will appear at the output when the bridge is modulated with the LF audio and the CV, creating the distortion. This is prevalent in tube processors and some VCAs as well, but the mechanism is different. (called limiter "thump" or pedestal low frequency components.)
Since there doesn't appear to be any way to trim the circuit, to achieve lowest LF distortion, components will need to be "selected" for minimum feed through. This includes not only the bridge, but buffering/amplifying circuitry that drives the transformer.
Delay/release CV circuits have been implemented before with these circuits, notably, CBS Volumax, Moseley TFL-280, in addition to close matching of component parts.
Good Luck.
jdiamantis