Blower motor is acting weird

So a few weeks ago my blower motor started working intermittently. I had all 4 speeds but it would just quit for half the day then start working again. A few days ago it stopped working all together. If I put the blower on the 3rd setting I can feel a very small amount of air coming out, but on position 4 there is nothing. I pulled the blower motor resistor and it looked ok to me, no black or scorched coils. I pulled the blower motor itself and left it plugged in. I turned it on and cycled through all the speeds and it worked just fine! I installed it back in the housing and its not working again.

Does anyone know what the heck is up with that?!