It would help if you drop back to basic electronics. Unless I missed something in the last 60 years impedance matching is critical in only two situations: 1) power transfer must be maximized, 2) cable length is such that reflections can occur. For audio case 1 is almost never a concern, although it might be in specialized situations. Headphone powering is not one of those situations. For case 2 reflections start to be a concern if the cable approaches 1/10 wavelength of the highest frequency. At 20KHz the wavelength in a vacuum (I know... but this is for simplification) is 15000 meters. One tenth of that is 1500 meters, or almost 5000 feet. Are you using the furthest "B" box at about 5000 feet from box "A"? If "NO" then case 2 doesn't apply and forget about impedance matching.
That said, impedance is important. You want a low enough source impedance that you can provide a good signal down the bus for the maximum number of connected loads. You want a high enough load impedance that connecting or disconnecting any one (or number of, you pick) load(s) has negligible effect on the signal level of the bus. For example, let's say the source output impedance was 10 Ohms, the load input impedance was 100K Ohms, and you had a 1V RMS signal from the source with no load. Connect anywhere from 1 to 100 loads and the input never drops below 0.990V, a drop of 0.086dB. That is so little that no one will notice switching on (or off) 100 loads!
As to whether you need a buffer, it all depends on the impedances you are dealing with, and how many loads you expect to have. Design the system right and the correct answer is NO, or maybe YES. It's all in the tradeoffs. You need to understand the basics well before you can intelligently manage the tradeoffs.