> 100 dB SPL in a pair of headphones is pretty darn loud.
100dB SPL *average* is petty loud.
But peaks can go higher.
The goal for high-fidelity living-room systems was sometimes 105dB SPL peak. If you take 85dB SPL as a reasonable average, and tack on 20dB for peaks (or 90dB SPL for "enthusiast" levels and 15dB peaks) you get in this zone.
100-106dB SPL per mW is pretty high. Some 'phones are much lower.
It is seriously likely that you will want a 50mW amp for critical listening.
The V*I situation is more complicated if you consider the broad range of impedances, 4r to 2K.
Situation is more consolidated now that "everything" has to play OK from a 3V device. 1V RMS in nominal 32 Ohms is 31mW. Much closer to '50mW" than your specific "1mW" case. For critical work we might want to aim for 2Vrms, but we no longer have to cater for 600r phones and many-Volt peaks.
1V in 32r is 31mA RMS. 44mA peak.
16 Ohm phones exist. 2Vrms in 16r is 250mW. Some of these are derived from hearing-aid mechanisms and would be VERY LOUD at 2V. 0.9Vrms might be ample. 55mA RMS. 80mA peak.
Note that 44mA-80mA are above the limit-level for general-purpose opamps. There are higher-current chips, or you can (correctly) lash-up a team.
Note that assuming 2V is enuff for modern 'phones means 5.6V peak-to-peak, or say a 8V or +/-4V power rails. 8V total supply is barely enough to get some general-purpose opamps fully-alive.
Note that happy rails for a low-Z 'phone driver are much lower than we customarily carry in "line" stages. The average and peak voltages are lower. We don't normally monitor every point inside a console, and may have hot levels in an intermediate stage; headphone levels are monitored by ear-pain. So it may be awkward to power both from the same supply. OTOH separate supplies are awkward. It is often done to power the 'phone amp from the line-amp rails, and let the excess voltage heat the chips of a build-out resistor.
Note that none of the general-purpose chip opamps are at all happy driving 32 Ohms, or even the 128 Ohms each part of a 4-pack would carry. The output load spoils the internal gain and reduces NFB. Since mis-loading is part of selling chips to sloppy designers, the chips usually don't show any surprise non-linearity, they just get mushy.
> What's with overbuilt headphone amplifiers?
Given the poor match with "simple" designs, and the low cost of OVER-design, it is popular to throw +/-15V supplies and 8-Amp transistors in with a 40VA PT. The Silicon-cost may still be less than the classy knobs. The heavy iron gives "heft". It is "better built" than a Mercedes, while the total price may be similar to one loan-payment on the Benz, and the punks won't steal your hubcaps.
As said, we want milliWatts not Watts. A 50mW 'phone amp is "minimum" like your would want 50 Watts in your speakers. So on the face of it, a 'phone amp costs 1,000 times less than a speaker amp. Not really, but many over-kill ideas which are preposterous on speakers are trivial at 'phone power levels.
OTOH, as Brian says, in days of higher-Z phones and many phones in the studio, the standard headphone amp was a speaker amp and a box of resistors or pots.
> a power supply for a .... a headphone amp
Power supply current is not load current. For a totem-pole Class B stage, at FULL power, the PS current is about 1/6 of the peak load current, or 1/4 of the RMS load current. Taking 80mA peak, current from the supply is about 13mA.
If you do NOT do full-power sine tests, only un-clipped speech/music, and use reservoir caps which can carry speech/music peaks, your average current will be 10dB or 15dB lower. Round to 12dB, so 1/4 the current, not even 4mA. For extreme linearity you would like output device emitter impedance less than low-load, 16 Ohms, which for BJT points to the 2mA idle zone. And your driver system is liable to be a couple mA all the time. So even if "class B", your max supply current may not be much above your idle current.
OTOH if you load in 16 Ohms and power with +/-5V and sine-test it at clipping, your power supply current approaches 100mA. 1VA of DC load, so with rectifier and regulator losses you need 3VA of PT core size.
All above for one ear. Double for two ears. Multiply again for friends and cohorts. (Though often it makes-sense to give each her/his own amp and knob.)
That's for semi-class-B designs. If you do a CCS-loaded SE amp, on the above numbers you need a standing current >80mA (I have advocated 150mA to be sure the amp never clips in hi-level monitoring). If you go resistor-loaded and assume peak swing is half the supply, you need 160mA or 300mA standing current!! (This is over-done, you can optimize significantly, though it will still be a Hot Box.)
