Thanks for the reply Dana. This is a walkout basement. The 12 inch cantilever is over the walkout. It used to just have fiberglass insulation just balled up and shoved in the spaces between the floor joists. I removed all the fiberglass and was wondering if I just need to seal any gaps in the cantilever spaces with caulk since there is foamboard insulation on the outside under the vinyl siding. The walls of the basement are insulated with 2 inch XPS foamboard. There is no insulation up in the floor joists between the basement and the first floor above.
You need to seal any gaps in the subfloor, and seal whatever air-dam there is between the mini-cavity and the joist bay as it transitions into the basement to both the joists and the subfloor, making it air-tight between that mini-cavity and the rest of the house.
Then insulate that foot-wide section of floor with the insulation in
full contact with the subfloor. Dense-packed cellulose (3lbs/cubic foot min) or dense-packed fiberglass (1.8lbs min) would be ideal, but short of that, rock wool batts or "cathedral ceiling" batts carefully trimmed & fitted will work, as long as you caulk or can-foam every corner & seam where wood meets wood (including plywood/OSB.)
In your climate (US zone 5) you don't need or want a vapor barrier of any type in the cantilever, but if you DO have a vapor barrier it needs to be at the subfloor or at the air-barrier in the joist bay where it crosses the supporting wall.
This ^diagram^ shows polystyrene (XPS or EPS) being used as both the air dam, and an interior-side vapor retarder under the floor in a cantilevered overhang. Air sealing that air dam and air sealing the joists to the subflooring is critical to getting the performance out of the insulation if you are using fiber insulation. Also critical is filling the bay completely with a perfectly snug fit at the subfloor-joist and subfloor-air dam transistion. To achieve that with batts requires careful trimming, then tucking/compressing the edges in to fill any void/channels in those corner transitions, then tugging out lightly so that it's still a compression fit for the board you put on the bottom.
If the joists are 2x10s compressing a low-density unfaced R38s (probably the cheapest option) will fit snugly enough, and deliver pretty good R32-ish performance:
Alternatively, stacking an R23 +R15 rock wool batt (Roxul is sold by both of the big box store chains now) would deliver a true R38 center-cavity performance.
If they are 2 x 12s it's probably worth cutting & cobbling 1- 2" EPS or polyiso in there, gluing and sealing it to the sub-floor for a more compressive-than-usual fit on the R39s. Only use XPS if you have some scrap remainders lying around. The blowing agents used for XPS aren't very environmentally friendly (for either you or the planet), and cutting up new stock to scrap the rest would be a waste.
Alternately, a double-layer of R23 rock wool , would give you R46 performance in a 2x12 cavity, but at a higher cost.