You can, but it might decrease the air throughput. I've had really good results from Panasonic fans. Some of them will accept both 4" and 6" ducts. If I needed to do it again, I'd buy one with a built-in moisture sensor. Rather than a timer, one of these would turn itself on once the moisture got within certain limits, and shut itself off later. The one I put into my mother's house, unless you carefully listen in a quiet time of the day, you may not hear it running.
In my place, I have an in-line fan as part of a tubular skylight, and it, too, partly because the motor is remote, is very quiet. You do not have to live with a noisy bathroom fan...best to start off with a quiet, quality design than trying to patch what you have.