> replace the elctrolytic with a large value ceramic?
Until quite recently, a 10uFd ceramic was absurd.
Remember the power pins ARE inputs. They should be low-low-gain. But in several popular topologies the PSRR drops to nothing at a MHz. Junk on the rail is junk on the output. If another circuit is kicking the rail, that can sneak around and cause outright oscillation or worse, undetected instability and fuzzy transients.
You need a SMALL CLOSE cap to take the MHz spikes. When working at the edge you want one AT each chip. In many cases one between adjacent chips works fine.
You need a LARGE cap to absorb KHz wobble, though this does not have to be close and may be an electrolytic per handful of board.
Electrolytics have got much better and many smaller slower audio systems do OK with a few good electrolytics within a few inches. There's some super-fast "audio" chips, DSL drivers, which won't be stable without screws and nails and glue on their power pins.
As JD says, there's such a thing as "too good". The circuit between the cap's legs is finite length and thus has inductance. Inductance and capacitance is a tuned circuit, happy to ring. At VHF it does not take much series resistance to damp the ring, but ceramics have really low series resistance.
What are you trying to do? Can you save space? Is space essential? Or are you trying to build for eternity?