I appreciate the comments, but I'm still trying to sort them out as to what I can do, vs should do. The venting issue is one reason I chose not to go with a higher efficiency unit. I can't direct vent out because this is in the basement. I don't quite get why a second pipe into this large flue would still get backflow from hot rising exhaust.
Replacing the 11 year old 50 gal HW heater with an exchanger has already occurred to me. I actually have one in a tenants apt that's part of a completely overdone heat system with a cast iron boiler for the 450 sg/ft unit. I could pull the boiler & heat exchanger tank out and replace them with a Bradford White water heater that also does the baseboard, as is working fine in 2 other identical units. I doubt my cost would be much higher than just getting a new one for myself and I would stop having cleaning issues with that damn boiler (not a DIY btw). My problem with this idea is that it seems like using that indirect tank system is horribly inefficient in summer, the room with the utilities is always too warm from passive losses even after I insulated all the zone piping to the tank.
But doing the above tank is probably much cheaper than running a new lined SS flue pipe the 45 ft up to the roofline, as I understand would be required to be truly to code.
An 85% efficient gas boiler's exhuast just ISN'T all that hot compared to an 80% efficient gas boiler. At 80% you'll have a net-stack temp differential of ~400F+, for an exchust temp right out of the boiler of 450-500F, whereas at 85% it'll be under 300F, with much lower buoyancy. With a high-efficiency condensing gas boilers the stack temps are under 200F, and plastic venting can often be used (but never a clay-lined flue.)
If the flue is SO oversized that it won't pressurize and force backflow to the water heater when the forced draft boiler is running it's way oversized for the load and needs to be narrowed to have sufficient draft to avoid flue condensation/rot issues. You would need a HUGE natural stack-effect draft to overcome this issue (maybe you could add 100 feet to the chimney height for better draft?
)
Side venting is almost always an option, even from basements, but it may require moving the installation to where it can meet clearances from doors & operable windows and avoid a long lateral vent run inside the basement. (Mine had to move from the south wall where the chimney is to the north wall 40' away to get a code-legal side vent, but it made it.)
An indirect with a 90k boiler behind it runs better than 40% efficiency in the summer. See:
http://www.nora-oilheat.org/site20/uploads/FullReportBrookhavenEfficiencyTest.pdf
The standby loss can be reduced dramatically by insulating the boiler loop to the tank to at least R4-R6 Better yet, insulate all of the near tank potable plumbing (including temperature/pressure valve outflow) to R4 at the same time. The smaller, less-massive the boiler, the lower the cycle loss- the CGi 4 will do better than any of the non-purging non-condensing boilers tested in that document. Also the lower you can keep the setpoint on the indirect, the lower the overall boiler/indirect standby will be. There may be lag issues at high flow from a lower-temp indirect on an initial cold start where the boiler needs a minute or three to come up to temp. This can be tweaked up for performance if need be, but storing 30+ gallons of water at 180F all summer is silly with a 90K boiler behind it, which is enough boiler to run a 2.5gpm shower 24/7. Start at 125-130F, and work your way up in 5F increments if the early morning shower has an intolerable lag on boiler warmup.
If the room is too hot in summer from standby losses, you must be keeping the boiler hot rather than cold-starting it whenever the indirect calls for heat, which would be as-inefficient as an internal-coil, or you have a mini-coil type indirect with sub-10gallon capacity. If it's over 30gallons and it won't cycle very often, so the boiler cools completely. The last thing you want to do is to use an internal "tankless" coil in the boiler, which requires the boiler to remain hot all summer. Running the indirect as a zone, the boiler should only fire when the tank is calling for heat.
A 50 gallon atmopsheric-drafted beast struggle to make 60% efficiency all year long unless you're a high-volume user (more than 60gallons per day, every day.) See:
http://www.aceee.org/conf/08whforum/presentations/1a_davis.pdf This too can be much improved with near-tank insulation on the plumbing.
Not measured in an efficiency test, the heat loss from the builiding due to infiltration induced by the open draft hood of the water heater adds up to 1-5% of the total heating bill (depending on a whole lot of factors.)