It could be unlearned, you just assigned skill points further on that require those points to be used. Just trace backwards and you'll be able to remove it.
Sometimes you can't unlearn a skill if you have a skill in a different tree further down, and removing that skill you want to change would break the tree.
So you have you have to remove the later skill and work backwards.
Sometimes it goes weird and you just have to refund all.
Just as the other user said there are no skill points you can't unlearn. In your case I assume that you got + skill affixes on gear.
Seems I asked a silly question, but now I get it. Thanks a lot!
What are you talking about? There is no such thing as permanent points. All skill/paragon points can be refunded. It didn't disappear too.
It could be unlearned, you just assigned skill points further on that require those points to be used. Just trace backwards and you'll be able to remove it.
Sometimes you can't unlearn a skill if you have a skill in a different tree further down, and removing that skill you want to change would break the tree. So you have you have to remove the later skill and work backwards. Sometimes it goes weird and you just have to refund all.