First off, what kind of pipe are you using to your spout? The instructions are pretty specific about that...you CANNOT use PEX or CPVC to the tub spout as that will result in a restriction, and that can force some water up to the showerhead. That piping must be copper.
Then, consider that as you change to all cold, you're using one supply. When warm, you're tapping off of two pipes, so the available volume will be greater. People often equate pressure when they really are being affected by volume. The smaller the ID of the pipe, the more friction there is, so that could be an issue, too, but normally, the pressure is not the issue. When the water is not flowing, regardless of the pipe size, the pressure will be the same, but you can get lots more volume out of a fire hose than a soda straw! The faster you try to get water through a small diameter pipe, the more friction there is, and the lower the outlet pressure (dynamic pressure) there will be. Volume is a function of not only the ID of the pipe, but the number of fittings, changes of direction, the velocity, and distance it has to travel through it. Smaller pipe, it tries to flow faster, resulting in more friction, and less volume AND pressure.