Well...I can see how cold numb fingers would cause you to drop your hat...makes prefect sense.

In fact a prefect is Uniform hat of a French prefect. Perhaps this hat may work for you (note he is wearing gloves as well).