I think it will work correctly. You just sum all of the volume control outputs and either use volume commands or mutes sent to the volume control chips to implement 'switching'. There won't be so many channels as to cause problems from the fact that all of the chips are essentially active at the same time. Plus, if there's some silly reason why one would need to mix sources together, this would be easy to do by altering the software, but impossible if the sources were actually switched.
Further, it's usually a good idea to do some sort of a volume ramp up or down before doing a mute or a switch anyway, just to avoid abrupt changes or clicks. I think the chips have a gain setting as well as a mute, and while it's good to ramp down to a low gain to mute, the mute itself should also probably be asserted, just to avoid crosstalk issues. IIRC, most of these chips have separate volume and mute features, so this should be simple enough.