So if we want to use two or four core screened cable unbalanced which is the correct end to connect the screen , instrument or amp end of the cable ?
For balanced connections you want the source end of the screen connected. I think that would apply to unbalanced as well, the principle is that the screen is connected to the reference voltage (aka "ground") in the source, so the common mode voltage of the source and the potential of the screen would move together, effectively cancelling the common mode part of the capacitance.
Electric guitars are getting difficult to record with all the RF mush floating around , for the the extra freedom from noise it might be worth a try , ok we take a hit on cable capacitance compared to typical low noise single core guitar cables .
Star quad is primarily effective at low frequencies, so depending on how loose you are with your use of the term "RF" it may not help at all. For interference in the high MHz range like wireless devices, connecting the shield at both ends with a low inductance connection, and filtering the signal lines are the most effective remedies. Star quad is most effective for things like transformer induced noise, or power line noise from lighting dimmers, motor controllers, etc.
And of course it usually doesn't matter what kind of cable you have if you are using single coil pickups, the noise induced in the pickups is usually going to dwarf anything induced in the cable.