Reparing a broken download with Rsync


Reparing a broken download with Rsync

2015/12/08

Tags: Linux SSH

If you can download a file with size in GBs in a single session, you should feel a great sense of accomplishment, specially in Nepal where internet connection follows Heisenberg’s uncertainty principle. Yesterday, I was downloading a ~1.2 GB file using a download manager on my phone.

Using inference from this post’s title, you can guess what happened to the download. Ye, the download was corrupted, and that download manager uses segmented requests so that I just couldn’t use wget -c to resume that download.

So, my choice boiled down to two options - restart the download or try a monkey brain to monkey patch it (Python, no pun intended). I felt that the second option resonated with my brain waves.

Story aside, you can use rsync to do a differential binary transfer i.e. only transfer those blocks which are different. On my server I downloaded that file at around 5MBps, so it didn’t took much time. When the file was at the server, I issued the following rsync magic:

$ rsync -Pa --checksum --inplace --no-whole-file -e ssh android-studio-bundle-141.2343393-windows.exe my-server:/home/aatish/

sending incremental file list

android-studio-bundle-141.2343393-windows.exe

  1,192,115,784 100%  178.41MB/s    0:00:06 (xfr#1, to-chk=0/1)

And voila, the download was fixed in a matter of seconds. I spent my saved time thanking the one who made this universe.

If you are wondering whether I read the man pages and figured out all those arguments and flags: No, I didn’t. I knew rsync could do that. So, I merely searched the web and got a wonderful article written by Chris. You should read that article at https://blog.christophersmart.com/2014/01/15/force-rsync-to-use-delta-transfer-to-fix-corrupt-remote-file/. It’s more technical, btw.

Happy Downloading…

References

Force rsync to use delta transfer to fix corrupt remote file