18,080,000 BTU/1456 HDD= 12,418 BTU per degree-day.
12,418/24 hour= 517 BTU/degree-hour.
65F- 3F= 62 heating degrees, x 517 BTU/hr degree-hour = an implied load of ~32,000 BTU/hr.
Looks right to me! (Same with the base 60F numbers.)
The ASHRAE multipler is really a maximum when looking at cast-iron- you won't freeze if the boiler only puts out 40,000 BTU/hr. At 40K you're fully covered for 40,000 / 517= 77 heating degrees, or an outdoor temp of about 65F -77F = -12F before it begins to lose ground.
Given that the existing boiler is nearly 4x oversized and ancient it's not really operating at it's original nameplate efficiency, so your actual BTU/degree-hour constants are less than that. Odds are pretty good that the heat load with a right-sized boiler operating at the full rated combustion efficiency, with lower standby & distribution losses would come in between 25-30K.
So if going for another cast-iron beast, keep it under 50K-out (if you can), but you will need to either side-vent it or install an narrowing flue liner to go with cast-iron that small- the current flue is almost certainly going to be WAY too fat to not have condensation or back drafting issues. It will also need to have some protection against condensing entering water temperatures when all zones are calling for heat. 50,000 / 150' is 330 BTU/hr per running foot, which will balance at about 140F average water temp (AWT), which means the return water is likely to be under 130F, edging into the range where corrosive condensation can take place inside the boiler. At 40,000/150' you'd be at 265 BTU/ft-hr, and an AWT of 130F, with even higher condensation risk.
But that's maybe a good thing. Any small 50-80K condensing boiler with a min-fire input in the 12,000 BTU/hr or lower range a good fit, given the amount of baseboard on your main first-floor zone, and even longer second floor zone. At 12,000 BTU/hr-in at 95% efficiency it would deliver 11,400BTUhr-out , divided by 55' is 207 BTU/hr, which will balance at an AWT of 120F, ( 125F out, 115F-return water), which is about where most condensing boilers actually deliver 95% efficiency. You can probably even dial it in to where the minimum firing rate is 120F -in or a bit lower, but the output of fin-tube baseboard becomes fairly non-linear at temps lower than that, making it hard to adjust "outdoor reset" curves that work in those temperature ranges.
An outdoor reset control lowers the boiler's setpoint temperature in response to higher outdoor temperatures, raises the boiler temp with lower outdoor temperatures. At ~500 BTU/hr per degree hour and a minimum firing rate output of 10,000 BTU/hr it means the outdoor temperature at which it balances at min-fire is 20F below the balance point, or about 40-45F. Above that temp the boiler will cycle on/off no matter what temperature it's set to, but boiler can run nearly continuously whenever it's below that outdoor temp, as long as the outdoor reset temperature is set so that it doesn't satisfy the thermostat quickly. The lower you can set the temperature curve, the higher the combustion efficiency will be and the steadier the indoor temperatures will be. But with fin-tube the hard limit at the warm-outdoors end of the curve should be 115-120F, out.
With 150' of baseboard and a design load of 30K @ +3F that's still 200 BTU/hr per foot of baseboard, so you may not need to even set up much of a curve, since at 200BTU/hr per foot it's still balancing at 120F AWT, well into the condensing zone. At 300 BTU/ft , 45,000 BTU/hr out of the fin-tube it would still only need ~135F -140F water and at the edge of the condensing zone, but it would have to be cooler than -20F outside to really need that. But te actual temp it needs depends on the zone load/length ratios, not the whole-house ratio- this is just a first rough-cut. But even at the final cut it looks like you should be able to average much better than 90% annual efficiency out of it under outdoor-reset control. You can probably do just fine with a condensing water heater set up as a combi-unit, given the load/ft ratios you have.
With a condensing boiler you can close up the flue to block the infiltration, and side-vent the boiler with plastic vent pipe, or in some cases, utilize the flue as a vent pipe chase for both the combustion air & exhaust venting. If other appliances (say, a water heater) shares that flue there are other issues to consider. The existing flue is ridiculously oversized for the BTU output of a water heater, and will have wintertime exhaust condensation & backdrafting problems (look up "orphaned water heater"). With a condensing boiler (any size) an indirect fired water heater operated as a priority zone makes sense. But a condensing water heater with a ~75K BTU burner or large could serve both the space heating and hot water loads and still have faster recovery times than a typical 50 gallon atmospheric drafted water heater.
