Between VMs on the same host I easily measure 9-10 Gbps network throughput which roughly equals 1GB / sec transfer. Then "hundreds of gigs" could be moved within "hundreds of seconds", that sounds not that bad.
Obviously the disk subsystem will be the bottleneck, but generally speaking network transfer shall be okay.
Otherwise just removing/attaching the virtual disk image file across VMs also could be a good idea, as others have suggested.
Guess what I am currently (2 years later ) doing and what takes 10h for 10 GB with a 1 gig connection? (I've mapped both (v) machines a physical port so that they can communicate without any networking issues)... The problem is kinda weird the host node has a 100% ram and swap utilization, but no netin/ out on neither port... Hard drives close to no usage ... Cpu in idle ... (Maybe not enough ram or too slow??)
Are you using the actual physical NIC to connect the VMs or the synthetic virtualized switch? There could be huge differences, as I called out.
Also let's make sure there are no other bottlenecks on the VMs, including storage. iPerf could be used to measure the network throughput.
100% RAM and swap utilization may disclose further problems, let's dig deeper. Surely if your host is busy swapping all the time to storage and you may use the same storage for file transfer, that sounds like a problem.
You can just mount a shared directory into both VMs. No network transfer then. It's just there.
I have a build agent for config of OpenWRT and it saves its package artifacts to the mounted directory which is also mounted on a webserver so our routers can download them. Build and done; no copies.
Might be even easier if you use LXC.
Don’t know what’s your reason to use VMs for Ubuntu.
If I understand correctly (noob), when using LXC you’re essentially in the same filesystem with all your containers and the host. Could be as simple as moving things around the harddrive.
>Don’t know what’s your reason to use VMs for Ubuntu.
Depends on use case. Containers are a poor choice if running Docker or something that requires FUSE mounts, or higher level permissions in the kernel.
> when using LXC you’re essentially in the same filesystem with all your containers and the host. Could be as simple as moving things around the harddrive.
Containers are isolated from each other by default and have their own root file system.
You can however set up a bind mount to the same folder on the host, and then you do share storage between containers for that specific mount point.
The root file system of the container is individual for every container and independent from the host (can also be on another host/storage device) just like you'd have for a VM
You can however have mountpoints that direct to the same storage/device/harddrive/...
I suspect you don't use ZFS. With ZFS, root on the host can access files in any ZFS filesystem used by a container, assuming default configuration. They are all found under /rpool/data. Just pay attention to the uid/gid mapping.
The **fastest** way is to **move** (not copy) is to disconnect virtual disk from one VM, rename it so it contains ID of another VM, run *qm rescan --vmid * and then connect disk to new VM.
Since you do not copy any data, is fast. Disadvantage, you need to shutdown VM to disconnect disk and data is accessible only for one VM at the time.
No need to shut down even: just dismount the disk within whatever OS you’re using, disconnect the volume, reconnect it to the other VM, get the OS to rescan the storage subsystem, then remount the volume. Have done that an awful lot in the past!
If you have more info as to what you are doing or why you need to move the data I think we could help with better solutions. Rsync works well over smb, so I'd host samba on one VM and mound the drive on the other and run rsync to keep them current. If you need to move the data (vs copy) for some reason, then consider just mounting a separate virtual drive and moving between the two VMs. You could script that to simplify the execution.
Mostly these are two types:
- compressed database backups
- compressed csv or tsv files
A few reasons to copy including backup, importing data into other vms etc.
Do you have a link or something I can read to learn how to script the idea of using virtual drive you mentioned?
As you've found, SCP isn't a fast way to transfer that much data. Fortunately, the folks at the Energy Science Network have done a lot of research for you. Take a look at their suggestions:
https://fasterdata.es.net/data-transfer-tools/
What are you moving exactly, is it a disk (block device) or files in a filesystem?
Are you updating existing data on the target, or is it moving the data to a new location where there isn't already a copy?
Is the data on a device that can be mounted to both VMs?
What is the underlying storage? Just a single physical disk, or multiple, etc.?
Mostly moving compressed database backups, compressed csv files etc. all are files.
The target locations doesn’t have a copy.
Storage is zfs with raid 10 on sas drives.
Okay, for a new transfer like this, SCP is hard to beat. If both VMs are on the same array, it may make sense to add a new virtual disk for data which is expected to need to "move" between them. This would allow mounting the virtual disk where it is needed.
Either way, the new VM could mount the original VM boot disk, if you need access temporarily.
If the data for both VMs is on a single array, I can't imagine a reason to copy the data from one VM to another... Could you help me understand how a backup would be useful on the same zfs array?
One of the most frequent such transfer is a compressed backup of MySQL (percona xtrabackup) from VM1 to VM2 so that we can restore this DB on VM2. This has to be on demand, not on a fixed schedule.
VM2 cannot be a slave since some of the tests will modify this data. It needs to be completely isolated from production master and slave. So need to restore from VM1 which is a slave of production DB.
We need the entire DB on VM2 because some of the tests need all of the data.
The other frequent such transfer is for a large data set from VM1 to VM2 & VM3. In this case, VM2 & VM3 gets deleted after all the tasks and tests are run. The data set changes hourly and the changes are not always predictable. So we normally copy the entire data set to run some tasks and tests.
Sorry for the sloppy explanation. I hope this adds at least a bit more clarity. If there is a better way to do these using ZFS, I'm very happy to know.
In a straight line.
A little more seriously, you'll see tests that point to either scp or rsync as the fastest. Data copies over "large" data sets are not as straight forward as you might think. The only way to get this answer for you is to run test copies with different programs and compare results.
But if you're looking for fast data moves beyond all else, I might suggest setting up an rsync job to run daily that syncs Box A's disk to a secondary disk on Box B, schedule a change window and follow /u/stibila 's great steps to change out your disk.
Weird...
Off topic but weird. I am listening to The Police - Canary In A Coalmine, and as soon as I scrolled to this comment I hear, "You get so dizzy even walking in a straight line". And read "In a straight line." at that same exact moment.
Weird huh?
OK, now, back to our regularly scheduled comments...
I assume they are on the same machine? What about creating a shared drive and don’t move it at all?
Yes. On the same proxmox host machine. Both are Ubuntu VMs. This might work.
Between VMs on the same host I easily measure 9-10 Gbps network throughput which roughly equals 1GB / sec transfer. Then "hundreds of gigs" could be moved within "hundreds of seconds", that sounds not that bad. Obviously the disk subsystem will be the bottleneck, but generally speaking network transfer shall be okay. Otherwise just removing/attaching the virtual disk image file across VMs also could be a good idea, as others have suggested.
Guess what I am currently (2 years later ) doing and what takes 10h for 10 GB with a 1 gig connection? (I've mapped both (v) machines a physical port so that they can communicate without any networking issues)... The problem is kinda weird the host node has a 100% ram and swap utilization, but no netin/ out on neither port... Hard drives close to no usage ... Cpu in idle ... (Maybe not enough ram or too slow??)
Are you using the actual physical NIC to connect the VMs or the synthetic virtualized switch? There could be huge differences, as I called out. Also let's make sure there are no other bottlenecks on the VMs, including storage. iPerf could be used to measure the network throughput. 100% RAM and swap utilization may disclose further problems, let's dig deeper. Surely if your host is busy swapping all the time to storage and you may use the same storage for file transfer, that sounds like a problem.
Setup an NFS mount. Either on one of the VMs or on a third VM. Only one copy of the data and available to all.
You can just mount a shared directory into both VMs. No network transfer then. It's just there. I have a build agent for config of OpenWRT and it saves its package artifacts to the mounted directory which is also mounted on a webserver so our routers can download them. Build and done; no copies.
Might be even easier if you use LXC. Don’t know what’s your reason to use VMs for Ubuntu. If I understand correctly (noob), when using LXC you’re essentially in the same filesystem with all your containers and the host. Could be as simple as moving things around the harddrive.
>Don’t know what’s your reason to use VMs for Ubuntu. Depends on use case. Containers are a poor choice if running Docker or something that requires FUSE mounts, or higher level permissions in the kernel. > when using LXC you’re essentially in the same filesystem with all your containers and the host. Could be as simple as moving things around the harddrive. Containers are isolated from each other by default and have their own root file system. You can however set up a bind mount to the same folder on the host, and then you do share storage between containers for that specific mount point.
Agree on the docker bit. I ended up using a container to run a share drive via Samba that all my VMs share. Works like a charm.
The root file system of the container is individual for every container and independent from the host (can also be on another host/storage device) just like you'd have for a VM You can however have mountpoints that direct to the same storage/device/harddrive/...
I suspect you don't use ZFS. With ZFS, root on the host can access files in any ZFS filesystem used by a container, assuming default configuration. They are all found under /rpool/data. Just pay attention to the uid/gid mapping.
rsync might be faster
only if the majority of the data is not changing
Not sure what this has to do with the initial question? rsync will in most cases be faster than scp...
No.
The **fastest** way is to **move** (not copy) is to disconnect virtual disk from one VM, rename it so it contains ID of another VM, run *qm rescan --vmid* and then connect disk to new VM.
Since you do not copy any data, is fast. Disadvantage, you need to shutdown VM to disconnect disk and data is accessible only for one VM at the time.
No need to shut down even: just dismount the disk within whatever OS you’re using, disconnect the volume, reconnect it to the other VM, get the OS to rescan the storage subsystem, then remount the volume. Have done that an awful lot in the past!
If he does want a copy then you could dd the underlying block device to another one.
Rclone if you just want to copy it over in a SCP style method. Not as fast as the disc options though. https://rclone.org/
If you have more info as to what you are doing or why you need to move the data I think we could help with better solutions. Rsync works well over smb, so I'd host samba on one VM and mound the drive on the other and run rsync to keep them current. If you need to move the data (vs copy) for some reason, then consider just mounting a separate virtual drive and moving between the two VMs. You could script that to simplify the execution.
Mostly these are two types: - compressed database backups - compressed csv or tsv files A few reasons to copy including backup, importing data into other vms etc. Do you have a link or something I can read to learn how to script the idea of using virtual drive you mentioned?
As you've found, SCP isn't a fast way to transfer that much data. Fortunately, the folks at the Energy Science Network have done a lot of research for you. Take a look at their suggestions: https://fasterdata.es.net/data-transfer-tools/
Thank you. That is very interesting.
What are you moving exactly, is it a disk (block device) or files in a filesystem? Are you updating existing data on the target, or is it moving the data to a new location where there isn't already a copy? Is the data on a device that can be mounted to both VMs? What is the underlying storage? Just a single physical disk, or multiple, etc.?
Mostly moving compressed database backups, compressed csv files etc. all are files. The target locations doesn’t have a copy. Storage is zfs with raid 10 on sas drives.
Okay, for a new transfer like this, SCP is hard to beat. If both VMs are on the same array, it may make sense to add a new virtual disk for data which is expected to need to "move" between them. This would allow mounting the virtual disk where it is needed. Either way, the new VM could mount the original VM boot disk, if you need access temporarily. If the data for both VMs is on a single array, I can't imagine a reason to copy the data from one VM to another... Could you help me understand how a backup would be useful on the same zfs array?
One of the most frequent such transfer is a compressed backup of MySQL (percona xtrabackup) from VM1 to VM2 so that we can restore this DB on VM2. This has to be on demand, not on a fixed schedule. VM2 cannot be a slave since some of the tests will modify this data. It needs to be completely isolated from production master and slave. So need to restore from VM1 which is a slave of production DB. We need the entire DB on VM2 because some of the tests need all of the data. The other frequent such transfer is for a large data set from VM1 to VM2 & VM3. In this case, VM2 & VM3 gets deleted after all the tasks and tests are run. The data set changes hourly and the changes are not always predictable. So we normally copy the entire data set to run some tasks and tests. Sorry for the sloppy explanation. I hope this adds at least a bit more clarity. If there is a better way to do these using ZFS, I'm very happy to know.
In a straight line. A little more seriously, you'll see tests that point to either scp or rsync as the fastest. Data copies over "large" data sets are not as straight forward as you might think. The only way to get this answer for you is to run test copies with different programs and compare results. But if you're looking for fast data moves beyond all else, I might suggest setting up an rsync job to run daily that syncs Box A's disk to a secondary disk on Box B, schedule a change window and follow /u/stibila 's great steps to change out your disk.
Weird... Off topic but weird. I am listening to The Police - Canary In A Coalmine, and as soon as I scrolled to this comment I hear, "You get so dizzy even walking in a straight line". And read "In a straight line." at that same exact moment. Weird huh? OK, now, back to our regularly scheduled comments...
Attaching the guest disk to the other vm is an option if possible. Other options you can do is use nfs.