My horses hair is being rubbed off by his halter?

I have a nylon halter that has started to rub off the hair behind my horse's ears, He only wears his halter when I have him out of his stall or he is out in the paddock, but it is still enough to rub the hair off. Is there a different type of halter I can get that will not do that? I heard you can get a leather one and get fleece covers for them. Will fleece covers be ok in Florida heat? Thank you :)