Backup Your Data on Slow Network

There might have been moment when you have wished to take your data backup on cloud. Well, there may be a couple of restriction, like you dont have any storage device, or you are not allowed to use your storage device in your systems (mostly enterprise systems) and you are left with no other options to use cloud storage, like Google drive or OneDrive.

So, if you have very good network connectivity, that’s fine. But what if you are on mobile networks. Copying large files can be one of the challenging task in that situation. Moreover, many times, due to network latencies and ‘n’ nos. of reasons, the upload fails; sometime on 50% or 99%.

What will you do?

Well. we have a solution for that, which is kind of based on algorithmic principal “Divide n Conquer”.

The idea is, to split your files into very smaller parts, say of 50mb or 10mb or if your network is very slow, can go with even 1mb files, and then upload. Simple! Isn’t it ?

Steps to achieve:

1. Compress your files to any popular format, say “Zip”. (easily extractable on both Mac and Windows system).
2. Split the files into smaller chunks of smaller sizes. (Using split utility in linux systems/Mac)
3. Upload it. (Latency and network issues effects will be drastically reduced due to small size)
4. Download it on destination system. (Or just keep it on cloud, as you wish ).
5. Merge the downloaded files in one output files. (Zip file)
6. Go ahead, and extract it. Bingo, you got all your content safely transferred .

Splitting you file

$ ls
image.jpg
$ split -b 1m image.jpg myfolder/prefix

split is a terminal command which splits your files in to small chunks with a given prefix (optional). So, if you have a file image.jpg, or movie.mp4 (say, 400mb), and you want split into smaller size of 10mb, can use this command:

split -b 10m movie.mp4 moviefolder/prefix1

it will create spliced files in your ‘moviefolder’.

Combining you files

$ cat moviefolder/prefix1* > output.zip
$ ls
output.zip

cat command is also a terminal utility to combine all spliced files with given prefix into one file. and once you got the output zip, you can simply extract it .

Why to Compress? can’t we directly use our files in native format, like, mp4 or jpeg?

Compressing a files has lot of advantages.

1. File size is reduced a bit.
2. Your metadata is preserved. That’s an important thing to consider. Metadata, like file creation date, last modified data etc play significant role sometimes. (for example, while use maps application, location metadata can play a role to locate the place where the picture was clicked. Additionally , an image can have info about, the device through which the image was clicked, which is stored as metadata in image.)
3. If you are compressing a folder containing multiple files (of may be different type), you don’t have to worry about uploading all files, just single file.

This method is not just useful for slow network, but also for the situation when you have much data and you don’t want to keep it on local storage device for privacy/safety purpose.

I tried this method to take back up of my data, while I was on Mobile Network, (not so fast, ~1–2Mbps) and had to backup GBs of data. Many times I tried to upload it as whole file of say (500MB or so), it used to fail in between. So, all my efforts 🙈 of uploading till 50mb/100mb was lost. So thought why not look for some way, this efforts could be utilised, and then landed on stackoverflow and see, we got a beautiful solution.

Thanks for reading.