Here in China removable batteries are a very common thing and most people live with it without any problem.
Some people even have removable lead acid batteries...
So if in your case it's lithium batteries, then it's not really heavy. Perfectly fine to live with it unless maybe if you're extremely lazy to carry it for a few meters.
The only important thing would be to charge it in a relatively safe spot, think in advance about what could happen in the very unlikely case that the battery overcharges and catches fire. The chance of it happening is extremely low, but there is a chance, so find a spot far away from flamable stuff, invest in a good fire extinguisher (though it isnt' always enough), or think about some place where you could easily kick the battery outside. Don't get paranoid, but have some common sense.
I never seen much people having issues with connectors, so I'd say it's not a really common thing, unless maybe if they were poorly engineered in the first place. Your battery is much more likely to die because a few cells gets old than because of the connector. Connectors are easy enough to replace anyways, no need to go for the proprietary stuff, if it fries just get any connector you want (and that matches the continuous current discharge it will have to face).