Start with transformer types: mic input, line input, single ended output, P-P output (gapped/ungapped). The circuit itself determines which of these you'll need.
Vast oversimplification follows, but it's intended to give you a rough idea of the considerations in selecting a good transformer for the job at hand.
Output transformer selection: (let's assume for this summary a line-level, pro audio scenario where you know you may need to drive a 600 ohm load). First, is the output single ended or push-pull? Is the output stage cap coupled? Then you'd choose an ungapped output transformer made for the specific stage topology (SE or P-P). Is it DC coupled? Then you'd choose a gapped output transformer that can handle DC made for the specific stage topology. The winding ratio choice is largely determined by the output impedance of the stage. It gets complex here, but to oversimplify, for lowest distortion you'd choose a transformer with a nominal primary impedance of 2x - 3x or more than the output impedance of the stage (higher=lower distortion; tradeoff is gain reduction). Core materials are largely a matter of choice. Higher steel content (up to 100%) = higher distortion, but generally more euphonic when pushed. Higher nickel content (up to about 80%) = wider bandwidth, lower distortion, but more abrupt/harsh distortion when pushed. Of course, level handling ability must be taken into account for the particular unit you’re dealing with.
Input transformers: Is it a line-in stage or mic-in stage? Match transformer type to stage type. Line-in requires significantly more level handling ability than mic-in, so line-in transformers will reflect this vs mic-in types. Higher max level = more headroom (but you might select a lower headroom model if you want to hear the transformer working - see the 1176 for example). Winding ratio ideally involves some math to match "optimal source impedance" of the input stage with the transformer secondary for lowest noise. JFET and tube input stages are higher impedance, so you'll generally use at least 1:10 ratio or higher with these. BJT/IC (non-FET types) inputs are lower impedance, so it's common to see anywhere from 1:1 up to 1:10, with 1:2 - 1:5 being the most common range. Core material selection is less of a factor with mic inputs, since the majority are typically high nickel content for maximum core excitation at lower levels. Line-in transformers can vary though, with 50/50 (steel/nickel) up to about 80% nickel being common.
And of course, with all of the above, you want to select transformers that are designed for the bandwidth you need. Ham radio transformers, for example, may look the part, but usually won’t do the job because they are limited to speech input frequencies (~200 - 3500 Hz).
Again, a total oversimplification, but an attempt to give you an idea of the considerations that go into a fairly rigorous selection process. Often, however, we just use experience to get us into the ballpark and try things out.