With the thermostat off (i.e., not calling for cooling), double-check the voltage at the contactor's coil. If it has 24vac across it, double check it at the thermostat base. It should be open. without the thermostat in the circuit, if there's 24vac on the yellow to common, there's a short somewhere.
Essentially, the thermostat is just a smart switch...when cooling is called for, it connects the R to Y to supply voltage to the ac contactor (the return line is wired from the ac unit to the source of the 24vac). That turns the ac unit on. It may or may not immediately turn on the circulation fan.
IOW, you're looking for where the coil is getting 24vac when the thermostat is not calling for cooling. It could be a defective thermostat, a connection between the R and Y somewhere else, or a defective contactor, but that doesn't seem to be the case. But, if you miswired the contactor, it's possible it could be getting the coil energized, but it would be a weird set of circumstances.