Amanda backup on a Backblaze Storage Pod

Yup, sometimes something is better than nothing. We have all been there before in IT. You have no backup server. You have no budget to buy a “real” backup server. You have been told a million times hard drives are no substitute for a real backup solution. Tape is king. You are being foolish. This goes against all best practices, it may possibly violate some IEEE standard, the ACM is looking at you like Clint Eastwood, your docrotal advisor is sitting on some beach somewhere while scantily clad men fan her while she sips her tropical-fruity drink you can not even pronounce or ever afford and she is shaking her head at you. Yet, you still go on, like some poor beaten up dog that has to protect his turf, even if it means certain death.

Or something like that. You need backup. You need it on the cheap. You have a LOT of crap to backup. Like, 54TB of crap. And you want incrementals, and possibly more than one backup of those “important” things HR keeps nagging you about that if you loose, you will also loose your job. Yeah, those files. Then, in the midst of it all, you recall that a few years ago when the company had money, they tried those backblaze units out and found they just were not fast enough for a second tier storage against the parallel distributed file system you setup. Hrm, could they be used for backup? Surely, they are not reliable? But wait, are they not as reliable as the sum of the parts? We have taken all of those computer architecture classes, we know enough to design large multipliers, multi-level caches, ALUs, buses, every low transistor count SRAM cell in the universe, every adder from 14T to 4T, surely we can mathematically figure out if the backblaze is good enough? Screw it, its good enough. Lets take one of them and upgrade it and use it.

Amanda Install and Setup:

So, you are back from that upgrade already and want to install Amanda on the backblaze? Great, lets get to it. First, we need to grab Amanda for Debian, or whatever flavour you put on your backblaze. I recommend the current stable release, which at the time of this writing is 3.3.5.  For this article I will bold and italicize the user that is doing the operation. For example, download the amanda server and install it on the Debian box as root with:

 

dpkg -i amanda-backup-server_3.3.5-1Debian70_amd64.deb

 

Before we go into amanda, it is a good idea to make sure your email is working. If your mail is working from this box to another (e.g. you have changed /etc/alias to point root to youraccount@yourdomain.com or you have it going to your local SMTP server, great) or if you have it sending to your free email account somewhere and it works, fine. If it does not work, and you have installed Debian stock from the netinstall, reconfigure exim4 with:

 

dpkg-reconfigure exim4-config

 

Ok, now onto Amanda. There is a good document online called “Amanda in 15 Minutes” in pdf format. It is what most of this document is based on. I am just being Debian specific and specific to the backblaze pod. Also, I find that they do not talk at all about the drive space and dumping to harddisk. Let us just go forth and configure this. I am going to assume you have a few linux servers and perhaps one Windows client that you want to backup. I will call the linux server LinuxServer and the Windows client puke.

First we will set a password for the amanda user and also change the device we are using from tape to harddisk. Recall in the previous blog we mounted the 130TB array on /mnt/backup.  I will throw everything in /mnt/backup/amanda. As the root user:

passwd amandabackup

 

vi (you don’t use that operating system emacs, do you?) the file /etc/amanda/amanda-client.conf and change the line that reads: tapedev “tape:/dev/YOUR-TAPE-DEVICE-HERE” to be:

 

tapedev “file://mnt/backup/amanda/vtape/DailySet1”

 

Note that just like everything in Linux and UNIX, it is case sensitive. Before we install a client, let us go ahead and finish up some directory creation on the server. Still as root:

 

mkdir -p /mnt/backup/amanda/vtape/DailySet1

chown amandabackup:disk /mnt/backup/amanda/vtape/DailySet1

chmod -R 750 /mnt/backup/vtape/DailySet1

 

We also need to setup the vtape configuration. We have to do this as the amandabackup user:

 

su – amandabackup

amserverconfig DailySet1 –template harddisk –tapedev /mnt/backup/amanda/vtape/DailySet1 –mailto root@localhost –dumpcycle 1week –runspercycle 6 –tapecycle 12 –runtapes 2

 

Note here that I am specifying one week, 12 tapes, a runcycle of 6 days and 2 runtapes. You can change to match your needs. Before we go to a client, the amanda server config will generate some RSA SSH keys for us, but for some reason this does not work for me when I tried to use it on any host. I have done plenty of ssh keys in the past, so I ended up just going them by hand. Still as the amandabackup user:

 

ssh-keygen -t rsa

 

Now, open your favourite editor and copy and paste the /var/lib/amanda/.ssh/id_rsa.pub file into it for later use.

Ok, so let us go to a client, like the LinuxServer. Assume it is RedHat/CentOS, so you would download the appropriate rpm (3.3.5) from the download site, and then as root on the client:

 

rpm -ivh amanda-backup_client-3.3.5-1.rhel6.x86_64.rpm

passwd amandabackup

 

Again, change the tapedev line in /etc/amanda/amanda-client.conf to read:

 

tapedev “file://mnt/backup/amanda/vtape/DailySet1”

 

Still on the LinuxServer client? Great, now su to the amandabackup user (su – amandabckup) and edit /var/lib/amanda/.ssh/authorized_keys and paste the contents of the servers amanda id_rsa.pub file that we copied earlier into authorized_keys. Now amandabackup can login to LinuxServer via ssh without a password.  Now save and exit that file. One more thing, still as the amandabackup user, we need to edit the .amandahosts file and add a line:

 

cd

vi .amandahosts

 

At the end of that file, put your amanda servers hostname and amandabackup and amdump. For example, lets just say your amanda servers hostname is backblaze2.yourdomain.com, then add the following to the end of .amandahosts:

backblaze2.yourdomain.com amandabackup amdump

 

save that file and logout of the amandabackup users shell. There is one more item to cover, and that is not backing up some files. Sometimes we might not want to backup certain files on the server. I find that the easiest way to do this is with the exclude files in amanda. Lets say on LinuxServer you do not wish to backup files in the /dev directory. Edit a file in / called .amanda.excludes and add the following:

 

/dev

 

Save that file, then logout as root, we are done on the client.

Installing the amanda client on puke is pretty easy. Again, download the client from Amanda for 32Bit or 64Bit. Unpack it and run the setup file. The only things you need are the amandabackup password and the server name and IP address of the amandaserver (which in this case is the backblaze2 pod). I will admit, I have seen an error with the ZWC client in that if you go back later on and wish to install another backup server to the client (or change its IP address), it will fail and you will need to reinstall the ZWC client. For this reason, I suggest here that if you know you will have more than one Amanda server, you need to go ahead and list them all here now. Done, great!

We are going to add the Linux client first, as after adding the first client, Amanda will copy a number of files into the /etc/amanda directory, and we will edit a few of them later. So, lets add the LinuxServer.  Assume for this that LinuxServers host name is actually LinuxServer.yourdomain.com and that you have DNS on your domain setup correctly. If not, use the actual IP address for this. For this we are going to be on the amanda server, as the amandabackup  user.

 

amaddclient –config DailySet1 –client LinuxServer.yourdomain.com –diskdev / –dumptype comp-user-tar

 

Note that we called the disk device /. This is because we want to backup the entire machine, except for those things we excluded in the excludes file. Note also that the dumptype is comp-user-tar, which is with compression and it does it on the client, meaning the client will take a performance hit during this operation (not just in disk usage). You can look through the /etc/amanda/templates.d/dumptypes file and see if you would like to use one without compression, or one that does compression on the server (although I don’t recommend it).

Now, before we can add the puke clients to the server we need to add the dumptype for puke and those types of machines. As the amandabackup user (or root), vi /etc/amanda/template.d/dumptypes and at the end of that file add the following:

 

define dumptype zwc-normal {
global
program “DUMP”
}
define dumptype zwc-compress {
global
compress client fast
program “DUMP”
}

 

Important note: while we are editing this file, under each type of dump that you want to use (as an example lets go with comp-user-tar) we need to add a line that tells amanda to look for the excludes file. Going with our example of comp-user-tar it would end up looking like the following:

 

define dumptype comp-user-tar {
user-tar
compress client fast
exclude list optional “.amanda.excludes”
}

 

Now save and exit the dumptypes file. As the amandabackup user add puke, again assuming that pukes hostname is in DNS and is called puke.yourdomain.com, and that the directory you want to backup is C:\Users\awesomeuser

 

amaddclient –config DailySet1 puke.yourdomain.com –diskdev “C:/Users/awesomeuser” –dumptype zwc-compress

 

No, there is no typo, that is literally C:/ and not C:\. You need to include the quotes when you type the command. It should return that it failed, but that is just as it can not ssh into the windows client. All is good. Lets just make sure amanda thinks things are still OK.

 

amcheck DailySet1

 

This command should return no errors. It will have some warnings since you have never run it, but no errors. We have one final thing to do here.  Recall we setup the tapes early on as the amandabackup user with the amserverconfig command? Great, well as it turns out the defaults are too low for this to be really useful when backing up large clients. So we need to modify a few numbers in the configuration. You can do this as the amandabackup user:

 

vi /etc/amanda/DailySet1/advanced.conf

 

There is a line that reads: netusage and shows a number which represents the maximum net bandwidth usage in KB/s. This was far too low for our setup. You can calculate what you need, or you can go overboard, but it should also be more than the total number of the bandwidth of all interfaces (I am assuming if you are reading this brief introduction to amanda you are not making additional interfaces) and parallel backups. For the time being, I stuffed a number equal to 2Gb/s in that line. You will also need to change the line in the local interface:

 

define interface local {
comment “a local disk”
use <somethingmorethan 8000> kbps
}

 

When we used the stock 8000 number, we would only get 8000. We upped both the netusage and use to nearly 2Gb/s. Save and exit that file. The final file we need to edit is in the same directory

 

vi /etc/amanda/DailySet1/amanda.conf

 

In that file you will see near the end a definition of the tapetype “HARDDISK”. You will need to modify/add the length attribute. Since we are using 12 tapes and wanted some leeway, we divided 125TB by 12 which gave us roughly 10922700 mbytes, so the tapetype should end up looking like this:

 

define tapetype HARDDISK {
comment “Virtual Tapes”
length 10922700 mbytes
}

 

NOTE: If you do not have 125TB of storage, do not put the length line in above. Figure out how much space you can give to each virtual tape and put that number in the length attribute, in megabytes.

Save and exit that file. We are nearly done. As the amandabackup user check the setup one more time with:

 

amcheck DailySet1

 

It should return no errors. Like the manual says, lets run it one time by hand and see what happens:

 

amdump DailySet1 &

 

You can check the progress with:

 

amstatus DailySet1

 

After some time (depending on how much you have this could be hours, or a 1/2 a day) it should come back and email root with the output (time, what was backed up, compression, etc).  If you added an alias this should do to your personal email. If all of that is OK, just add a cron job for it to run M-Sat, still as the amandabackup user:

 

crontab -e

 

0 22 * * 1-6 /usr/sbin/amdump DailySet1

 

Save and exit that (if you are setup with vi as the default editor, you know what to do). Note, in the above crontab line, we are telling it to start at 0min, 10pm, every day of the month, every month of the year, on days Mon-Sat (1-6) of the week.

 And finally… Recovery:

To recover a client, let us first assume we want to recover some files from LinuxServer. As root on LinuxServer

 

amrecover DailySet1

listdisk

 

From there, it will give you a list of what it has backed up on the server. Lets just say that you want to restore /etc for some reason. The above command should have returned the list of disk for host LinuxServer as /. You would perform the following actions:

 

setdisk /

ls

add etc/*

extract

 

It will ask you if you are sure you want to restore, as it will overwrite the original files. You decide. After it finishes restoring you can type “exit” to leave the amrecovery console. The files should be restored in /etc. Note that you can restore individual files. For example, lets just say you wanted to restore /etc/resolv.conf You would do:

 

amrecover DailySet1

listdisk

setdisk /

add etc/resolv.conf

extract

 

And there you have it, it will ask you if you are sure, and if you want to load the tape, and it will then restore resolv.conf to /etc.

To restore files on the windows server, it is a bit more tricky. Since the puke windows client does not have a full fledged client, you will need to go to the Amanda server and restore the files to the Amanda server, and then copy them to the puke windows client. Ready, ok, login to your Amanda server as root and perform the following action (given that the root disk has enough space for this):

 

mkdir /restore

chown amandabackup:disk /restore/

su – amandabackup

amadmin DailySet1 find puke.yourdomain.com

cd /restore

amfetchdump DailySet1 puke “C:/Users/awesomeuser” <YYYYMMDDHHMMSS>

 

Ok, a few notes, the YYYYMMDDHHMMSS information is the year, month, day, hour, min and second which was given to you by the amadmin DailySet1 find puke.yourdomain.com command. Also, when it is completed, you will have a file like puke.yourdomain.com.C__Users__awesomeuser.date.0 which is a zip file. You will need to use unzip on the server or on the windows machine to unzip it, but those are the files form the last backup.

And there you have it. A small Amanda backup solution on top of 130TB of storage.

This entry was posted in Servers, Software. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *