Migrate VM From One Node to Another Gradually Print

671088640+0 records in
671088640+0 records out
343597383680 bytes (344 GB) copied, 37518.1 seconds, 9.2 MB/s
671088640+0 records in
671088640+0 records out
343597383680 bytes (344 GB) copied, 37518.6 s, 9.2 MB/s

- Ensure no errors occur. If any, fix it and restart the mirroring process.

5. Once done, ensure successful imaging while preparing for next step, by mounting the destination LV. In the destination node:-
# mount /dev/vg_server06/vm911_img /home/xen/vm911/mnt
# df -h

.....
/dev/mapper/vg_server06-vm911_img315G 218G 81G 73% /home/xen/vm911/mnt

# cd /home/xen/vm911/mnt
- Look around the mount point to ensure roughly the same files as the source VM.

6. If imaging confirmed successful, remove the snapshot VM in the source node:-

# lvremove VolGroup00/vm578_snap
Do you really want to remove active logical volume vm578_snap? [y/n]: y
Logical volume "vm578_snap" successfully removed

Now you should have 2 identical VMs, in both source and destination nodes, with the following situation:-
- The source VM is still running and operational with no downtime, with constant data changes as normal
- The destination VM is shut down, not operational, and holds a snapshot of the source VM data.

#!/bin/sh

# Change this according to node type and VM config locations
node_type=xen # xen/kvm
vm_home=/home/$node_type

rsync=/usr/bin/rsync
ssh=/usr/bin/ssh
ssh_port=22222
rsync_args="-ave '$ssh -p $ssh_port' --exclude='backup/' --exclude='sys/' --exclude='proc/' --delete --force"

if [[ -z "$1" || -z "$2" ]]; then
echo "Usage: $0 <Live Source VM IP Address (internal IP recommended)> <Dormant Destination VM ID>"
echo
exit
fi

src_vm_host=$1
dst_vm_id=$2
dst_vm_data=$vm_home/$dst_vm_id/mnt/

if [ ! -d "$dst_vm_data/home" ]; then
echo "ERROR: Destination VM data ( $dst_vm_data ) does NOT exist. Make sure it is typed and mounted properly."
echo
exit
fi

echo "-------------------------------------------------------------------"
echo "Begin $src_vm_host VM Sync to $dst_vm_data: "`date`
echo "-------------------------------------------------------------------"
eval $rsync $rsync_args $src_vm_host:/ $dst_vm_data
echo "-------------------------------------------------------------------"
echo "End $src_vm_host VM Sync to $dst_vm_data: "`date`
echo "-------------------------------------------------------------------"
echo

----- end script -----

3. In the destination node, in a screen session, test run this script to ensure correct operations ( it should sync the currently running source VM in source node, to the dormant source VM in destination node ).
# screen

# /usr/local/scripts/sync-vm.sh
Usage: /usr/local/scripts/sync-vm.sh <Live Source VM IP Address (internal IP recommended)> <Dormant Destination VM ID>

# /usr/local/scripts/sync-vm.sh vm2 vm911

-------------------------------------------------------------------
Begin vm2 VM Sync to /home/xen/vm911/mnt/: Ahd Sep 13 11:58:26 MYT 2015
-------------------------------------------------------------------
...... (some file transfers)

-------------------------------------------------------------------
End vm2 VM Sync to /home/xen/vm911/mnt/: Ahd Sep 13 12:58:26 MYT 2015
-------------------------------------------------------------------

4. If the script completes successfully, schedule the script to run a few times during off hours in /etc/crontab :-
# vim /etc/crontab
0 19,0,5 * * * root /usr/local/scripts/sync-vm.sh vm2 vm911 >> /var/log/sync-vm2.log 2>&1
:wq
- In this example, sync-vm.sh is scheduled to run daily at 7pm, 12am, and 5am (off hours for the VM). Adjust according to the relative off hours of the customer.

5. From time to time, you can monitor the sync operations by :-
# tail -f /var/log/sync-vm2.log


IV. PREPARATION FOR SWITCHOVER TO DESTINATION VM

1. Ensure the destination node is able to SSH into the source node smoothly with RSA key pairs. 
- SSH to destination node, copy contents of /root/.ssh/id_rsa.pub to source node's /root/.ssh/authorized_keys ( if .ssh does not exist, create using ssh-keygen -t rsa ).
- Test by SSH from destination node to source node as root. In this example, destination node is "server06", and source node is "server03". From server06:-
# ssh -v -l root -p 22222 server03
( You should be logged in to server03 without having to key in any passwords ).

2. Prepare pre-switchover sync script as below, save as /usr/local/scripts/sync-vm-offline.sh in destination node. This script is similar to the online sync script above, but the difference is both source and destination VMs are shutdown during the sync.

----- begin script -----

#!/bin/sh

# Change this according to node type and VM config locations
node_type=xen # xen/kvm
vm_home=/home/$node_type

rsync=/usr/bin/rsync
ssh=/usr/bin/ssh
ssh_port=22222
rsync_args="-ave '$ssh -p $ssh_port' --exclude='backup/' --exclude='sys/' --exclude='proc/' --delete --force"

if [[ -z "$1" || -z "$2" ]]; then
echo "Usage: $0 <Source node IP Address (internal IP recommended)> <Dormant Source Destination VM ID> <Dormant Destination VM ID>"
echo
exit
fi

src_node_host=$1
src_vm_id=$2
dst_vm_id=$3
src_vm_data=$vm_home/$src_vm_id/mnt/
dst_vm_data=$vm_home/$dst_vm_id/mnt/

if $ssh -p $ssh_port $src_node_host test -d $src_vm_data/home; then
echo "OK: Source VM data ( $src_vm_data ) exists and mounted."
echo
else
echo "ERROR: Source VM data ( $src_vm_data ) does NOT exist. Make sure it is typed and mounted properly."
echo
exit
fi

if [ ! -d "$dst_vm_data/home" ]; then
echo "ERROR: Destination VM data ( $dst_vm_data ) does NOT exist. Make sure it is typed and mounted properly."
echo
exit
fi

----- end script ----- 

3. In destination node, test run the script to ensure runnable and no errors:-

# /usr/local/scripts/sync-vm-offline.sh
Usage: /usr/local/scripts/sync-vm-offline.sh <Source node IP Address (internal IP recommended)> <Dormant Source Destination VM ID> <Dormant Destination VM ID>

4. Repeat the test run, this time with the actual arguments (Since source VM not prepared yet, it should throw error, but it should verify script ready to be run):-

# /usr/local/scripts/sync-vm-offline.sh server03.nocser.net vm578 vm911

ERROR: Source VM data ( /home/xen/vm578/mnt/ ) does NOT exist. Make sure it is typed and mounted properly.

- If you get the above error, you are OK and ready to switchover.


V. SWITCHOVER TO DESTINATION VM

1. When you are ready for switchover, schedule a 1-hour downtime for the VM with the customer, and remove all scheduled synchronization from crontab:-
- Remove the sync-vm.sh line corresponding to the VM that will be switched over:-
# vim /etc/crontab
#0 19,0,5 * * * root /usr/local/scripts/sync-vm.sh vm2 vm911 >> /var/log/sync-vm2.log 2>&1 <---- REMOVE THIS LINE
:wq

2. At the start of scheduled downtime, shutdown the source VM in SolusVM. Make sure both source and destination VMs are shutdown.

3. In source node, recall from your notes, the LV allocated to the source VM (Source LV).
- In this example, the allocated LV is /dev/VolGroup00/vm578_img .

4. After double checking that source VM is shutdown and not running, mount the Source LV:-

# mount /dev/VolGroup00/vm578_img /home/xen/vm578/mnt

# df -h

.....
/dev/mapper/VolGroup00-vm578_img 315G 218G 81G 73% /home/xen/vm578/mnt

# cd /home/xen/vm578/mnt
- Look around the mount point to ensure roughly the same files as the previously online source VM or the current dormant destination VM.

5. Now you are ready for the final sync before the switchover. SSH to destination node and run the sync-vm-offline.sh script you initially prepared, as follows:-

# /usr/local/scripts/sync-vm-offline.sh server03.nocser.net vm578 vm911

- This time, the script will run and copy the latest changes from source VM to destination VM.

-------------------------------------------------------------------
Begin server03.nocser.net:/home/xen/vm578/mnt/ VM Sync to /home/xen/vm911/mnt/: Isn Sep 14 10:48:26 MYT 2015
-------------------------------------------------------------------
..... some file transfers
-------------------------------------------------------------------
End server03.nocser.net:/home/xen/vm578/mnt/ VM Sync to /home/xen/vm911/mnt/: Isn Sep 14 12:48:26 MYT 2015
-------------------------------------------------------------------

6. Once done, unmount both source and destination VMs:-
- In source node:-
# umount /home/xen/vm578/mnt
# df -h 
- Ensure no more vm578 mounts
- In destination node:-
# umount /home/xen/vm911/mnt
# df -h
- Ensure no more vm911 mounts

7. In SolusVM, open source VM and take note of current VM main IP and assigned IPs. Then remove them. Replace with a single temporary main IP.

8. Then, still in SolusVM, open destination VM and remove all current IP allocations, and replace with the exact same IP allocations as the source VM previously. Boot the destination VM. It should boot the VM on the new node. Verify boot process OK, all allocated IP addresses operational, and all services running OK.

9. Leave source VM dormant until confident enough to remove (after VM running stable for some time in new node).

DONE.

-----

References:-
http://www.serveradminz.com/blog/?p=862
https://github.com/janeczku/vps-musings/wiki/Backup-or-clone-KVM-guest
https://www.linode.com/docs/migrate-to-linode/disk-images/copying-a-disk-image-over-ssh/