Retrofit is difficult but in new construction use tape to close off all the holes in the box and some spray foam or caulk to seal the wire penetration. Use some siliconized latex caulk to seal between the edge of the box and the drywall.
They sell inexpensive foam pieces designed to fit over the switch or receptacle. For multi-gang ones, you can overlap them. They'll help with the breeze.
If you are really motivated, you might consider opening up the wall to see where all that air is coming from. You may have a big leak or the insulators could have "missed" a stud bay.