Every power supply, except perhaps batteries, brings in unwanted signal, ripple, HF noise, rectifier transitions, load dependent feedback that needs to shunted to ground.
Every capacitor has an impedance vs. frequency curve, with a big dip at its resonance point. Below the resonance the impedance is capacitive, above inductive. The closer to resonance the better the bypassing.
The caps internal inductance determines the resonance frequency. Paralleling several cap values and types has been done for a very long time also in audio amps to shunt unwanted signal to ground. A plate loaded gain stage will include roughly half of the power supply noise in the output signal. Constant current loaded gain stages are more resistive to supply noise, but will couple noise capacitively into the load, so a smaller number is better.
The resulting bypassing effort can of course be checked with a scope.
Signal coupling caps MAY benefit from paralleling, but this is not often done.
Much is written on benefits of various coupling caps, some prefer one type or another. Dielectric absorption is s factor, so film caps with better numbers for this would be a first choice.
Low ESR of caps are related to dielectric properties. ESL more to the construction technique. Film and foil may have higher inductance than metallized dielectric.
An extreme example of a capacitors are those used in radar circuits where thousands of amps is discharged every second, and metal foils brought out in a very low inductance fashion, thus caps also have a dv/dt limit as well.
Selecting bypass caps for low noise performance requirements is a balance between cost, size, longevity, temperature, and noise involved.
"Noise" not necessarily gaussian white noise but any unwanted signal.
The "best" cap may be a 100uF 600V NPO/C0G until you find out the cost of it.
Polypropylene caps are reasonable, and available, Teflon, polystyrene are considered better. Mylar are OK but not premium signal coupling caps, but good for bypassing.
Some swear by the use of paper-in-oil caps yet these have not so stellar dielectric absorption.
Tantalum caps have a bad rep since long ago, but some may like them. Besides expensive they also can fail spectacularly, and is a conflict material.
Niobium and organic electrolytes I have not looked into, perhaps OK for low voltage.
Electrolytics are indispensable but have a finite lifespan, which have to be considered in servicing equipment. Temp- and ripple current ratings have to be checked against the use case. Their ripple filtering capability may be more important than their energy storage value, and selected for lowest ESR values.
This is a pretty big subject, and it is hard to avoid using capacitors.
In the early days of electronics when caps were unobtanium transformer coupling was used. The Western Electric 47 mike used no coupling cap but ran plate current thru the output transformer which of course also provided galvanic isolation and voltage stepping.
Transformers were also used in RF circuits to keep kilovolts away from antennas