What kind of connectors did they use? Do they roughly have the size of a XLR connector with 2 pins close together for the balanced signal and a pin a bit further away connected to ground? Also there is probably a threaded ring to lock the connector.
As far as I remember those connectors were produced in two sizes; a XLR like size and a bigger type.
I used to have a box of those old connectors, but because I did not have any use for them, I threw them away some time ago...
Isn't is possible to fit XLR's? Or is it the idea to keep everything as original as possible?
For the documentation you might contact AEG-Telefunken in Germany. I am sure that they will still have the documentation of those tape machines. Or try the technical department of the Dutch Broadcasting organisation, NOB in Hilversum, they also might still have some documentation in their archives because they used the M5's in the past.