It's fine to compress batting to ensure a tight fit. When compressed it gives a higher R/inch and is more air-retardent than at full loft:
An R13 batt designed for 3.5" deep cavites has exactly the same amount of material as R19s designed for 5.5" deep cavities.
When compressed into 5.5" the R19 actually performs at R18, which is R3.27/inch.
Compressing it into a 3.5" space it performs at R13, which is R3.71/inch.
Compressing it further into a 2.5" cavity delivers R10, which is R4/inch.
Basically compressing the fiberglass is generally GOOD for performance, voids are always BAD. Never be reluctant to pack it in there as long as it's still springy to finger-push. It takes quite a bit of compression to where it goes beyond the optimal density.
Using 2-3" of foam is NOT better than filling the joist bays with cellulose, and it's probably going to be more expensive to boot. Cellulose at open blow densities (as in attics) costs about 3 cents per square foot per R, and at mid-density (2.2-2.8lbs per cubic foot) it's about 5 cents R-foot. Dense packed to 3.5lbs it's more, but actual cost varies depending on the complexity of the project. Ceiling joist bays are dead-easy, walls with lots of complicated window & door framing blockages to work around are harder.
Closed cell foam at about a buck a board-foot is 15-17 cents per square foot per R. So for the same cost of 3" of closed cell foam ($3 per square foot, ~R19) you can get ($3.00/$0.05=) R60 of cellulose. Clearly the most you're really going to get into a 2x12 joist bay is about R40, but it's more than 2x the performance at 2/3 the cost (a 33% savings, give or take.)
And cellulose...
* is more fire retardent using less toxic fire retardent chemicals
*doesn't have any polymeric outgassing issues, should the installer get the mix or temperature slightly off
*does not use a high global warming potential blowing agent (HFC245fa at ~1000x CO2)
*is protective of structural wood by buffering/sharing the wintertime moisture burden, without damage or loss of performance
*reduces the thermal bridging of the joists by being a deeper fill at any R, lengthing the thermal path through the ~R1.2/inch wood (which means the thermal bridge goes from R3.6 in a 3" closed cell foam solution to R13.5 in 11.25" cellulose joist-fill solution.) Even at equal R-values at center cavity, the cellulose-fill outperforms the closed cell foam partial fill by a double-digit percentage due to the thermal bridging factor.
It's basically cleaner-greener-cheaper and more forgiving stuff, and when you have a whole joist bay of space to fill, the rationale for using the more expensive higher R/inch goods just isn't there. In locations where you need the higher R/inch due to lack of space orwhere you need it for vapor retardency control (such as at band joists or foundation walls) closed cell foam can be the best option, but that's not what you have here.
If you are still enamored of polyurethane foam solutions, a full 2x 12 cavity fill of open cell foam would deliver R38-R40 and cost about the same as 3" of closed cell (maybe 10-15% more). And since it's blown with water instead of HFC245fa it doesn't have the greenhouse gas issue (though it still has the other chemical issues). It would have to be blown in two 5.5" lifts to not be a fire hazard as it cures or have shrinkage/bonding issues, but done right it is even more air-tight than closed cell foam. When applied over a garage or in a cantilever overhang you have maximal drying capacity on the "cold" side of the issue, so you don't really need the lower vapor retardency of closed cell foam to protect the ceiling drywall or the bottom side sheathing of the cantilever. But open cell foam does not have the protective hygric buffering characteristics of cellulose any moisture that gets in via diffusion ends up as adsorb in the cold-side wood until it warms up enough to release it.
The microscopic hollow fiber characteristic of cellulose allows it to adsorb and redistribute quite a bit of moisture before losing function, and it is protective even at relatively modest density (even more protective if dense-packed.) For cheap stuff it's pretty good.
An R13 batt designed for 3.5" deep cavites has exactly the same amount of material as R19s designed for 5.5" deep cavities.
When compressed into 5.5" the R19 actually performs at R18, which is R3.27/inch.
Compressing it into a 3.5" space it performs at R13, which is R3.71/inch.
Compressing it further into a 2.5" cavity delivers R10, which is R4/inch.
Basically compressing the fiberglass is generally GOOD for performance, voids are always BAD. Never be reluctant to pack it in there as long as it's still springy to finger-push. It takes quite a bit of compression to where it goes beyond the optimal density.
Using 2-3" of foam is NOT better than filling the joist bays with cellulose, and it's probably going to be more expensive to boot. Cellulose at open blow densities (as in attics) costs about 3 cents per square foot per R, and at mid-density (2.2-2.8lbs per cubic foot) it's about 5 cents R-foot. Dense packed to 3.5lbs it's more, but actual cost varies depending on the complexity of the project. Ceiling joist bays are dead-easy, walls with lots of complicated window & door framing blockages to work around are harder.
Closed cell foam at about a buck a board-foot is 15-17 cents per square foot per R. So for the same cost of 3" of closed cell foam ($3 per square foot, ~R19) you can get ($3.00/$0.05=) R60 of cellulose. Clearly the most you're really going to get into a 2x12 joist bay is about R40, but it's more than 2x the performance at 2/3 the cost (a 33% savings, give or take.)
And cellulose...
* is more fire retardent using less toxic fire retardent chemicals
*doesn't have any polymeric outgassing issues, should the installer get the mix or temperature slightly off
*does not use a high global warming potential blowing agent (HFC245fa at ~1000x CO2)
*is protective of structural wood by buffering/sharing the wintertime moisture burden, without damage or loss of performance
*reduces the thermal bridging of the joists by being a deeper fill at any R, lengthing the thermal path through the ~R1.2/inch wood (which means the thermal bridge goes from R3.6 in a 3" closed cell foam solution to R13.5 in 11.25" cellulose joist-fill solution.) Even at equal R-values at center cavity, the cellulose-fill outperforms the closed cell foam partial fill by a double-digit percentage due to the thermal bridging factor.
It's basically cleaner-greener-cheaper and more forgiving stuff, and when you have a whole joist bay of space to fill, the rationale for using the more expensive higher R/inch goods just isn't there. In locations where you need the higher R/inch due to lack of space orwhere you need it for vapor retardency control (such as at band joists or foundation walls) closed cell foam can be the best option, but that's not what you have here.
If you are still enamored of polyurethane foam solutions, a full 2x 12 cavity fill of open cell foam would deliver R38-R40 and cost about the same as 3" of closed cell (maybe 10-15% more). And since it's blown with water instead of HFC245fa it doesn't have the greenhouse gas issue (though it still has the other chemical issues). It would have to be blown in two 5.5" lifts to not be a fire hazard as it cures or have shrinkage/bonding issues, but done right it is even more air-tight than closed cell foam. When applied over a garage or in a cantilever overhang you have maximal drying capacity on the "cold" side of the issue, so you don't really need the lower vapor retardency of closed cell foam to protect the ceiling drywall or the bottom side sheathing of the cantilever. But open cell foam does not have the protective hygric buffering characteristics of cellulose any moisture that gets in via diffusion ends up as adsorb in the cold-side wood until it warms up enough to release it.
The microscopic hollow fiber characteristic of cellulose allows it to adsorb and redistribute quite a bit of moisture before losing function, and it is protective even at relatively modest density (even more protective if dense-packed.) For cheap stuff it's pretty good.