They really don’t make good hardware anymore, specially hard drive, they seems to be not too reliable nowadays.
Here is my story, I have a gigantic external hard drive that is great to store lots of data. I use it for all my data files so that I’m not filling up my normal C drive, and thinking ahead, that it will be easier to migrate to a new machine in the future. Good strategy? yes until the external hard drive becomes unstable. The drive seems to have slowed down, and randomly giving CRC error when trying to read files (for those that do not know this 3 letter word, it means Cyclic Redundancy Check. It is kind of like a checksum created to ensure that the integrity of data can be confirmed).
Panic time? No worries. I should be able to save all the files by copying them to a new hard drive, right?
I quickly ordered another gigantic external hard drive (now double the size), and starting to copy all the files from the old drive to the new one. I’m a loyal window users, so I rely on the windows’ File Explorer for the copying. Now is the time that I discover some annoying limitation (I’m using Windows 10). Although it has the nice graphical interface showing you the file being copied, it is lacking a few very important and simple features that is absolutely needed when you are copying large number of files.
Panic time now? Yep! Time to scream.
So as a techi, I look around for a few things that can help me to get the job done. So I would like to share some of my findings with you, and hope that this may help if you are in a similar situation.
Let’s start with the basic, are there any command line interface in Windows that will do the job?
The answer is yes, copy and xcopy command. xcopy is the command that maybe suitable for this task. There are a number of options you can select, and research it a bit and find the following will work:
xcopy /I/E/Y/F/D/C/H <source folder> <target folder> 1><log file name> 2>&1
I know, you may want to put on your eyeglasses now. But trust me this will traversed through all the sub-directories, copy all the files, ignore the ones with error, and you will see the list of files (with full path) in the log file. The /D without a date will only override file if it is newer, so if you have to do this a few times, you don’t have to rewrite all the same files.
You can do a xcopy /? to find out all the available option. Click below to see the screenshot.
As a matter of fact, I had to run this command a few times. After some research with the list of files that has the CRC error, it turns out that if you retry a few times, it actually works, so that CRC error can sometime be intermittent.
Anyway, this was an interesting exercise, but I’m a bit disappointed on the amount of manual effort needed.
Let’s modernize a bit, are there any file copy utility software out there that will do this?
Preferably open source and free? So I started to search in the internet to find if there is any file copy utility software out there that will make my life easier. And yes, there are many, some good and some bad.
I will share with you the best one I found so far. It is called Free File Sync. It has a simple interface where you can select the source and target directory, do a compare first to see the differences, and then do a synchronize if you want to merge the files, there are a number of very useful options for both compare and synchronize task. Just to mention a few
- For Compare, you can compare file time and size, or you can also compare actual file content.
- For Synchronize, you can do two way or mirror synchronization and also option to give instruction for each file.
I’m quite impressed with this simple UI and it works amazing fast as well. There are many other features to explore, including all can be run in batch mode, so you can pretty much build a simple customized backup script for yourself. If you are interested, here is the link to their project page in Sourceforge.
Do you know how your hard drives are performing?
So we continue to purchase all the latest gadgets, with upgrades that are suppose to boast performances. Do you know how your new or old hard drive are performing? which one is faster? by how much?
So I found another interesting open source tool that can help with that. It is called Crystal Disk Mark, an hard drive benchmark software. It strikes me that it is really simple to use, the UI is simple and test result easy to understand. You select the drive you want to test, and the test you want to do, simplest thing is to do all the read and write test. It will come back to you with the benchmark result.
The pre-configured test are:
- Seq Q32T1 – Sequential 128KiB Read/Write with multi queues and threads
- 4K Q32T1 – Random 4KiB Read/Write with multi queues and threads
- Seq: Sequential 1MiB Read/Write with single thread
- 4K – Random 4KiB Read/Write with single queue and thread
You can adjust the test configuration base on your liking, but the default one is pretty good already. Bottom line is the higher the number, the better performance. I certainly find that my drive that is having intermittent CRC errors are performing really bad.
If you are interested, here is a link to the Crystal Disk Mark home page.
Time to invest on a backup strategy?
Now, we all have much of our memories and other important documents exist in electronic form only. For example for photos, I do have lots of physical films and paper photos in boxes. But I also have digital medias probably for the last 20+ years on my hard drive. They are a real treasures but how do I make sure that they are safe? Storing them in relatively new and reliable hard drives is a good start, but it is probably a good idea to have a backup copies, just in case. So how can I automate that so that I feel that I have a reliable setup?
I just ordered another 8 TB hard drive which I will use for backup purpose. While waiting for the new hard drive to arrive, I did some research to find a suitable backup software. I first review
some bundle backup software directly from the hard drive manufacturer, and they are not too impressive. I finally located one that looks really good, it is called EaseUS Todo Backup. There are few versions of the software, and yes, there is a free one, which amazingly give you most of the features needed for backup.
The software has a very modern, simple ,and intuitive user interface. In addition to backup features, the software provide features to do system transfer, disk cloning, create emergency disk, wipe data, … just mention a few.
So I quickly downloaded and installed the free version, and my first test project is to create a file backup for the all my documents in the C drive. The interface allows me to select the specific folders I’m interested, and the ability to do a one off backup, or create a backup plan.
I have about 60 GB of data (about 75,000 files) and the initial full backup took about about 30 minutes to complete. I then tested with daily incremental and differential backup plan, both of them takes less than 20 minutes to complete.
Wait, what is a differential backups? Here is a simple explanation of the 3 backup modes:
- Full backup – back up up all the files.
- Incremental backup – back up only the data that has changed since the last backup, be it a full or incremental backup.
- Differential backup – back up only the files that changed since the last full backup up.
Incremental backup runs faster but will have a more complicated and maybe longer recover process. To fully restore to date, you will need to recover the last full backup + each of the individual incremental backups.
Differential backup consume a bit more time in the backup process, but will simplify the recover process, as you will only need to recover the last full backup + the last differential backup.
You can workout your backup strategy according to your preference. I set up a daily differential backup to run at a specific time of the day. Option are available for weekly and monthly as well. An interesting premium feature is available for event driven backup, option to start at System Startup, System shutdown, User log on, User log off, and upon USB device plugged in. In order to use this feature, you will need to upgrade to a paid version.
If you are interested to explore more, here is the link to the EaseUS Todo Backup Free page.