I have worn a plain hat for most of my adult life. It identifies me with other plain Christians. I never use it as an instrument of belligerence or insubordination; if other men remove (or should remove) their headgear, I remove mine.
I never wear my hat in prayer. Since meeting for worship is in its entirety an act of prayer for me, I never wear my hat in meeting.
Incidentally, I have saved myself from many cracks on the head (on the farm) by wearing my hat; it often gives a second or two warning when I will be at risk.