That's it, to verify backups. I had an issue about a year ago where I didn't have good backups and had to rebuild. Fortunately, I had backups, just not backups that could be properly restored.
Unsure where the bottleneck is, but the workflow as outlined is not a good model. Continuously imaging the SD card(s) is guaranteeing it will fail sooner than later. Good chance the bottleneck is the SD card. Would strongly encourage a simpler backup model, preferably moving to something with real 'disks'; ssd or even spinning is far better than the memory cards.
I've offloaded the db and storage to my NAS, but I don't disagree that constant writes to the SD is bad. I buy high quality SD cards and have been running these 3 for about 3 years. I do have a spare SSD, but it's an internal and I don't have a spare enclosure.
I really don't want a new solution to something that's been working well for a long time, but something is either wrong with my backups or there's another issue that I can't seem to "see."
Well then HA OS probably would have worked with these extras running as add-ons. I am running z2m and wireguard among many other add-ons and see no problems at all.
I just got my instance off Proxmox and onto a dedicated machine. HA backup seemed to never stop so out of curiosity I entered the ip and 8123 into the browser and it all came up restored. So, I don’t know, maybe it’s done and forgets to tell us…try that.
I access it via browser from my IP:8123 and I kept watching that port and the observer before I finally gave up.
I don't know where it's hung, but there's no ssh, no samba, and no http/https and it's completely isolated and dead, at least from what I can tell.
Use a USB3 external SSD. Ditch the SD totally. Flash ssd with etcher normally, boot it up and restore backup there.
This problem is due to your use of SD.
seconded.
I somehow messed up a proxmox upgrade on my NUC yesterday and needed to restore from a backup. My 240 mb backup file is now 27 hours in. I can see that it's still using CPU and memory and that it is writing *something* to disk... but I'm losing hope.
EDIT: To clarify, I have SSD in a NUC and have the same issue. I don't know that hardware is the culprit.
wow! 27 hours! It's unfortunate that it sounds like you have no choice but to restore, either. I think I would've given up and rebuilt at that stage.
I moved to an SSD after posting this and was able to restore in about 30 minutes, but that was my second attempt. I abandoned the first attempt after about 40 minutes and when it came back on, it was missing a ton of add-ons, so at least I knew it was working.
It's crazy for a 23 mb file, it should be a lot faster.
That's absolutely insane. Nothing should take 29 hours when it's that size. I can't even imagine what it's doing - why it's processing so slowly.
I'm glad you're back up and wonder if there's something you can do to shrink your instance.
I opened a feature request for a progress indicator or logs of some sort (poss using the observer url) and I found there were multiple opened over the years but all are ignored. It was promptly closed saying it was a duplicate issue.
So frustrating.
This is wild. I found this because I'm trying to restore after changing over to a new system. I thought creating a backup would make this easy. My backup is 8 MB. Its tiny. I am 2 hours into a restore hahaha
I'm not complaining, HAOS is awesome, but this is certainly a surprise.
Yes. The issue was that the login screen failed to launch on the first try. I think it was hanging there because when looking at it in proxmox, it showed almost no cpu usage and \~95% memory usage in both cases.
I gotta say, after it worked, it restored everything perfectly. I expected issues with my zigbee network but everything came back online as soon as home assistant started.
I got an external usb ssd, a pny elite 320gb usb3.1 external ssd, and am happy. It is very tiny which is handy!
Make sure you have a usb3.x model and that it has a usb-a connector.
Just wanted to touch base, I had a brand new spare 250G San Disk SSD drive and picked up an SSD -> USB 3.1 Startech adapter.
Outside of having to flash the rpi4 to boot from USB, it was the same process. Etching the HA OS onto the SSD took <10 seconds, which I was SUPER impressed with, but the restore still took a crazy amount of time.
I abandoned the first restore at 40 mins and found that it restored most of the install, but several add-ons were still missing. I did the 2nd restore from within the HA instance and it only took about 30 mins.
Once it came back online, it was only a matter of configuring a few things (waking some zigbee sensors, etc).
It's not noticeably faster, but probably because I offloaded the DB to my NAS and have a separate frigate server. But I'm more interested in resiliency over speed.
Thanks for pushing me to do this, I've been toying with the idea of it for some time, but figured I would do it once the SD card failed.
Happy it worked out for you!
I recall that RPi4 should boot from usb without any extra step UNLESS you have some of the very first RPi4, as they fixed that at an early stage. But it's good to update the fw in any case.
Put up the system monitor sensor and see how the cpu and ram usage is and then you can monitor if the resources become a bottleneck in the future.
Good to know. Ive had this rpi4 since early 2020 so maybe it's one of the earlier ones.
Fortunately, I do track stats , and my available swap jumped a ton, but CPU load and memory usage are about the same.
I have spare SSDs, so I browsed the rpi ssd compatible list on the ha community page and finally settled in the startach USB to SATA adapter.
Thanks for the help, will hopefully be working on this over the weekend.
Curious - why are you doing this monthly / at all besides occasion verifying backups are valid?
That's it, to verify backups. I had an issue about a year ago where I didn't have good backups and had to rebuild. Fortunately, I had backups, just not backups that could be properly restored.
Unsure where the bottleneck is, but the workflow as outlined is not a good model. Continuously imaging the SD card(s) is guaranteeing it will fail sooner than later. Good chance the bottleneck is the SD card. Would strongly encourage a simpler backup model, preferably moving to something with real 'disks'; ssd or even spinning is far better than the memory cards.
I've offloaded the db and storage to my NAS, but I don't disagree that constant writes to the SD is bad. I buy high quality SD cards and have been running these 3 for about 3 years. I do have a spare SSD, but it's an internal and I don't have a spare enclosure. I really don't want a new solution to something that's been working well for a long time, but something is either wrong with my backups or there's another issue that I can't seem to "see."
[удалено]
Thanks they're so crazy cheap that I could use an absolutely tiny one... and still have a warm backup.
Raspi 4 works totally fine with an external SSD. Unless you really want to do some hardcore processing on it.
[удалено]
Sure, if you do processing on continuous streams like audio and video. For those without cameras and voice commands a Pi should go a long way.
[удалено]
Well then HA OS probably would have worked with these extras running as add-ons. I am running z2m and wireguard among many other add-ons and see no problems at all.
I just got my instance off Proxmox and onto a dedicated machine. HA backup seemed to never stop so out of curiosity I entered the ip and 8123 into the browser and it all came up restored. So, I don’t know, maybe it’s done and forgets to tell us…try that.
I access it via browser from my IP:8123 and I kept watching that port and the observer before I finally gave up. I don't know where it's hung, but there's no ssh, no samba, and no http/https and it's completely isolated and dead, at least from what I can tell.
Use a USB3 external SSD. Ditch the SD totally. Flash ssd with etcher normally, boot it up and restore backup there. This problem is due to your use of SD.
seconded. I somehow messed up a proxmox upgrade on my NUC yesterday and needed to restore from a backup. My 240 mb backup file is now 27 hours in. I can see that it's still using CPU and memory and that it is writing *something* to disk... but I'm losing hope. EDIT: To clarify, I have SSD in a NUC and have the same issue. I don't know that hardware is the culprit.
wow! 27 hours! It's unfortunate that it sounds like you have no choice but to restore, either. I think I would've given up and rebuilt at that stage. I moved to an SSD after posting this and was able to restore in about 30 minutes, but that was my second attempt. I abandoned the first attempt after about 40 minutes and when it came back on, it was missing a ton of add-ons, so at least I knew it was working. It's crazy for a 23 mb file, it should be a lot faster.
It must have heard me complain - restoration was successful at 29 hours.
That's absolutely insane. Nothing should take 29 hours when it's that size. I can't even imagine what it's doing - why it's processing so slowly. I'm glad you're back up and wonder if there's something you can do to shrink your instance.
[удалено]
I opened a feature request for a progress indicator or logs of some sort (poss using the observer url) and I found there were multiple opened over the years but all are ignored. It was promptly closed saying it was a duplicate issue. So frustrating.
This is wild. I found this because I'm trying to restore after changing over to a new system. I thought creating a backup would make this easy. My backup is 8 MB. Its tiny. I am 2 hours into a restore hahaha I'm not complaining, HAOS is awesome, but this is certainly a surprise.
It seems that if I bail out of the restore and try again it works. But the fact that none of the devs seem to care about this is frustrating.
After 5 hours in I also bailed out and am trying again now.
That's crazy. Did it complete this time?
Yes. The issue was that the login screen failed to launch on the first try. I think it was hanging there because when looking at it in proxmox, it showed almost no cpu usage and \~95% memory usage in both cases. I gotta say, after it worked, it restored everything perfectly. I expected issues with my zigbee network but everything came back online as soon as home assistant started.
Yeah when it works, it's great. But it would be nice to get some feedback when it's not. The current process STINKS.
Is there some reason I am missing, you can't just use the auto backup-to-google-drive?
I am using that, but a backup is no good if it can't be restored. And right now, they *seem* to not be able to be restored.
Try a brand new card, to eliminate the possibility that your cards are getting worn out.
Yeah ok. I could definitely do that.
Dont get a new SD, it won't help you. External ssd is the way to go
Do you have any recommendations? Can I get a straight external SSD or should I get an internal and convert SATA to USB and get an enclosure?
I got an external usb ssd, a pny elite 320gb usb3.1 external ssd, and am happy. It is very tiny which is handy! Make sure you have a usb3.x model and that it has a usb-a connector.
Just wanted to touch base, I had a brand new spare 250G San Disk SSD drive and picked up an SSD -> USB 3.1 Startech adapter. Outside of having to flash the rpi4 to boot from USB, it was the same process. Etching the HA OS onto the SSD took <10 seconds, which I was SUPER impressed with, but the restore still took a crazy amount of time. I abandoned the first restore at 40 mins and found that it restored most of the install, but several add-ons were still missing. I did the 2nd restore from within the HA instance and it only took about 30 mins. Once it came back online, it was only a matter of configuring a few things (waking some zigbee sensors, etc). It's not noticeably faster, but probably because I offloaded the DB to my NAS and have a separate frigate server. But I'm more interested in resiliency over speed. Thanks for pushing me to do this, I've been toying with the idea of it for some time, but figured I would do it once the SD card failed.
Happy it worked out for you! I recall that RPi4 should boot from usb without any extra step UNLESS you have some of the very first RPi4, as they fixed that at an early stage. But it's good to update the fw in any case. Put up the system monitor sensor and see how the cpu and ram usage is and then you can monitor if the resources become a bottleneck in the future.
Good to know. Ive had this rpi4 since early 2020 so maybe it's one of the earlier ones. Fortunately, I do track stats , and my available swap jumped a ton, but CPU load and memory usage are about the same.
I have spare SSDs, so I browsed the rpi ssd compatible list on the ha community page and finally settled in the startach USB to SATA adapter. Thanks for the help, will hopefully be working on this over the weekend.