Considering most painted gear is done with satin or semi-gloss paint, you don't want to buff it too much or you'll end up with a high gloss on that spot. Try some 2000 grit sandpaper (wet) with a small rubber block. This may be enough on it's own as it leaves a kind of satiny surface.
good luck
Freddy