The nomenclature can be somewhat confusing...a valve discharging water is more the indication of a pressure relieve valve than a pressure reduction valve. They do different things.
A pressure relief valve just literally opens up to relieve excess pressure above it's setting (some are adjustable, some are not).
A pressure reduction valve should never leak unless it has a fault. It reduces the pressure from the supply, to some lower pressure you configure on its outlet. When those fail, they generally do not leak, but allow their outlet pressure to rise, or reduce the flow.
Sometimes, people will put a pressure relief valve after a pressure reduction valve, to help protect things, if the PRV fails. Normally, the relief valve would never open up. If you have that situation (and it sounds like you do), it is an indication that the first device - a pressure reduction valve - has failed. The pressure relief valve may just be doing what it is supposed to do. It can also be an indication of a failed bladder or expansion tank. Older expansion tanks DID need to be drained periodically, as the water moving through them would absorb the air used as the 'cushion'. You really do not want to use one of those on a heating system, as adding air means the opportunity for things to rust. With a bladder type expansion tank, there should be no air in the system that touches the water, and once any that is there dissolved in the fill water is used up (by rusting things!), it becomes essentially quite inert.