People often like to use a 3x4 reducing closet bend, so if you excavate the 4" hole in the floor you may find that as the drain turns horizontal it reduces to 3".
It's fine to leave the half-bath rough-in as is if it's not in the way, but the closet flange should be sealed off with an internal plug made for the purpose, not dirt. What is missing from your diagram is likely that the blue WC fixture drains connects to the horizontal blue drain between the vertical blue vent above the floor (which was intended to receive the lav and wet vent the WC) and the downstream blue wye you show below the floor.
If you do want to make it go away, you'd remove the closet flange and the unpictured wye it goes to; then your new 3" drain could connect where that wye was (or if you don't connect there, you'd remove the pictured blue wye as well). You'd also change the blue/blue/grey vent tee above to just an elbow, removing the blue vent.
Cheers, Wayne