dd / nc netcat disk clone over SSH

ssh root@sourcehost 'dd if=/dev/sda'| dd of=/dev/target status=progress

Source: https://www.maketecheasier.com/ssh-pipes-linux/

Combined with the rsync tip, to copy 32 GB:

dd if=/dev/sda status=progress bs=4M count=8192 | ssh -T -c arcfour -o Compression=no -x user@targethost 'dd of=/folder/on/target/file.img'

Got 42,5 MB/s with this from SSD to HDD over 1 Gbit.

Verify the file

dd if=/dev/sda status=progress bs=4M count=8192|md5sum

md5sum /folder/on/target/file.img

Netcat - higher speed, unencrypted

On the source:
cat /path/to/source | pv | nc -l -p 5555

On the target:
nc <source-host> 5555 | pv > /path/to/target

Speeds were ~10-90 MB/s with this method.

Regarding VirtualBox VM:s

To increase speed between a host and a virtual machine switch network adapter on the host from Intel PRO 1000/MT Desktop to PCnet-FAST III, also replace NAT connection with a bridged connection if possible.

This increased speed by around +10 MB/s.

Synchronize only changes on block devices

If you already have an outdated copy of the block device you want to copy, then you can save by just updating it with the changes.

However, rsync does not work with block devices, but there are alternatives that do.

rsync alternative - bscp

cd /opt
git clone https://github.com/dop251/diskrsync
ln -s  /opt/bscp/bscp /usr/bin/bscp

bscp <source> <host>:<path> <block size> <hash, defaults to sha1>
bscp /dev/sdXY host:/path/to/device-or-file

bscp does NOT need to be installed on the remote side.

It is NOT possible to specify a user with user@ before the hostname, it tries to log in with the username of the current local user.

Specifying a directory that does not exist results in cryptic errors.

What hashes that are available is not sure.

Speed test 6 GB file over a wireless network: 07:15.

rsync alternative - diskrsync

Installation - note, the binary MUST be installed on both the client AND the host:
apt install golang
cd /opt
mkdir diskrsync
cd diskrsync
export GOPATH=$(pwd)
go get github.com/dop251/diskrsync/diskrsync
ln -s /opt/diskrsync/bin/diskrsync /usr/local/bin/diskrsync

diskrsync --no-compress --verbose <source> <host>:<target>
diskrsync --no-compress --verbose /dev/sdXY host:/path/to-device-or-file

diskrsync binary must exist on both client and server the client it runs it on the server.
Login must be done using a private/public SSH key.
By default it stores files compressed in a spgz format, to avoid this use --no-compress.

Speed test 6 GB file over a wireless network: 10:17.

This is a personal note. Last updated: 2021-01-07 19:04:22.


Don't forget to pay my friend a visit too. Joakim