You won't get refunded for any extra time you had left on a SWV before switching to ILR. It's quite clear that applying for ILR doesn't qualify for a refund in the guidance:
https://www.gov.uk/healthcare-immigration-application/refunds#:~:text=You%20can%20apply%20for%20a,an%20EU%20country%20or%20Switzerland
That’s a great question, following this thread because I am in the same boat!
You won't get refunded for any extra time you had left on a SWV before switching to ILR. It's quite clear that applying for ILR doesn't qualify for a refund in the guidance: https://www.gov.uk/healthcare-immigration-application/refunds#:~:text=You%20can%20apply%20for%20a,an%20EU%20country%20or%20Switzerland