> i am ignorantly assuming that a highend phonograph or tape player will more accurately reproduce complex waveforms.
At 7.5 or 15ips, tape will have a pretty steep low-pass filter at the top of the frequency response. You have to FIGHT to get the top octave to go on and off of tape: use micro-fine oxide, big recording boost, ultra-thin playback gap. You can only peak it up so far, then you have to give up, and it falls off pretty fast.
And put any shape of 20KHz wave onto 7.5 or 15ips tape, it will come back all rounded and sine-y.
Yes, pros like 30ips. Partly because it isn't a big fight to get 20KHz on and off, but also because the S/N ratio improves from OK to good.
Do note that there is NO musical instrument that plays 20KHz. The highest fundamentals are 4KHz, and rarely played. (Think what the top note of most pianos sounds like!) A 4KHz fundamental will have 8KHz, 12KHz, 16KHz, and 20KHz harmonics. 24, 28, and 32KHz, too though maybe only your dog will notice. However all real instruments have roll-offs too, and in fact a piano makes very little output above 6KHz or 10KHz. That's part of why the top note is so "tink-y": it does not have a lot of overtones, the darn string is too stiff, the soundboard too damped to make or pass 5KHz and up. Piccolo will blow 4KHz and several harmonics, but a 20KHz wave mostly gets sideways in the pipe and self-cancels. Anything above 15KHz is random blow-noise, not tones.
> in using the roll off, i started to find the music more pleasing.
> i am concluding that digital high end is irritating to some extent, relative to analog.
I'm not sure the observation supports the conclusion.
Many people do hate digital sound. Many possible reasons. The low-pass filter is VERY steep, even steeper than a 7.5ips tape cranked-up to 20KHz bandpass, and has some response ripple and significant internal ringing. People who can hear 23KHz may miss the little bit lost behind a 20KHz filter. Digital "noise" is very low, but very annoying, much more than tape hiss. Digital boxes are designed by digital engineers who don't know squat about that last inch in the analog domain, and are likely to use 4558 chips soaked in digital garbage. And digital tools (like ProTools) allow much more stupid messing with the sound, layered processing we could never afford to do with analog boxes. Rounding errors can accumulate real quick, making a muck of your sound. I learned the hard way to convert to 24-bit before doing any extended whacking on 16-bit audio.
> what was lacking in my studio mixes compared to the car radio(FM).
That is unfair. The "sound" of pop-radio is a VERY big and cut-throat business. There are guys who get big bucks for setting up processing to make a radio station sound louder/better than all others in town, so you will stop and linger on that station (and increase listenership and the value of ad-time). The really spectacular advances(?) in automatic compression and EQ came with radio limiters of the 1980s and 1990s. There is more happening in a good car radio and car speakers. My Honda's speakers are clearly un-flat yet very pleasing.
Also: the mixing engineer should let someone else finish the job. Mastering is really a special art and skill. While the same person can either mix or master, s/he should not do both on the same project. Same reason books and newspapers have writers and also editors. The person who assembles all the little pieces tends to lose sight of the whole pie, and tends to read/see what s/he intended instead of what s/he actually typed or recorded. Let someone else with good ears have a whack at your stereo master and see what difference it makes.
FWIW: US FM is generally limited to 15KHz, and always sharply notched at 19KHz. However I really don't think the difference between 15KHz and 20KHz or even 25KHz is what you are hearing. And I suspect that some FM sound-directors cut above 10KHz, especially loud 10KHz, because many listeners won't hear it, few car speakers handle it well, and dropping the top octave may allow 0.5dB more in the rest of the audio band without overmodulation. (The FM pre/de-emphasis curve predates modern top-heavy music, and a "flat" FM channel can easily over-modulate the highs before the midrange is maxed-out. Same problem on 78s: the NAB-78 EQ was too much for late-1930s musical balance.)