As with most electrical appliances these days it is allways best to unplug when not in use,or unattended.
It is the use of electronics and the absence of a switch to switch off the device. If left plugged in the electronics is still energised and draws power,although small( 2-5w) still capable of creating a fault leading to a fire, causing a lot of damage.
As regards wearing of the contacts in the plugs,you should pull/push plugs straight not rock from side to side which tends to open out the connections,a little silicone grease on the pins help too.
The wall mounted charger box also draws some current even when not in use,since this also contains electronic circuitry that can fail.If its contained in a metal box and has adequate fuse protection it should be ok.
Or install a suitable switch to the charger off,( disconnect altogether) however if you leave it connected to car you are still at a small risk!