I've lived in the South 34 years out of 52, and I tend to say "Ma'm" to just about every adult woman I meet in a non-social setting. It's just something that was considered good manners when I was growing up.

Yes, it's a bit archaic, but it's possible to be TOO informal, isn't it?