The preservation of space is a good thing. Right now, LED's are not standardized. Current Cans that maintain Edison sockets are great as you can put an LED bulb in them just the same, but when the bulb fails or some other situation comes up, you still have a 100 year old standard interface. California forces them to use the CFL plug, but the can's are the same. It's a manufacturing reality, no point to re-invent it all just for one state.
Non-edision base LED options (bulbs and fixtures) are something to stay away from right now as the only option you have when it breaks is to replace the whole fixture. There are efforts to standardize the form factors/interface of LED arrays that get used in these new compact solutions, but don't expect it to be complete soon. As people have LED fixture failures after a few years, they will learn that a 20 year expected life means nothing if you have a failure long before and have to replace a couple hundred dollar fixture, or 20 years later and all the fixtures in the house need to be replaced as the LEDs hit the end of their life. This will get ugly. Edison base fixtures, and old school 6 and 4" can lights maintain future proofing that proprietary new units don't. In time this will get fixed, and the world can shift over to what will be smaller. Probably won't be too fast though. There is really no reason to sell cans that aren't IC and Air Sealed, yet they do. Even though going to only IC and air sealed would prevent people from installing the wrong units, reduce shops carrying so many can's and make IC and Air sealed can's cheaper (think about how 4" cans cost more than 6" can's due to being less used). They used the special connector for California to prevent people from using incandescent bulbs, do the same with the IC and sealing. Using an IC and sealed unit where it doesn't need it wouldn't have a negative.
Also in the future that extra space in the can we don't use now might have a use we haven't thought of yet. Place to shove wireless routers, or pop out lights, deployable disco balls, etc. Doesn't hurt having it there.
I know where you are going though, I'd love to fill my basement ceiling with cans, but there is just no space. LED setup that would fit up in a 4" J box (not the ones that mount to a J-box, but into a J-box to keep things flush) would be great. Just not there yet.