ccurley at trib dot com
Revision History | ||
---|---|---|
Revision 0.02 | 2002-01-27 | Revised by: c^2 |
Added Mondo to resources | ||
Revision 0.01 | 2001-10-25 | Revised by: c^2 |
Initial version for LDP release |
Copyright Š 2001, 2002 Charles Curley and distributed under the terms of the GNU Free Documentation License (GFDL) license, stated below. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".
If you have any questions, please contact <linux-howto at metalab.unc.edu>.
No liability for the contents of this documents can be accepted by the author, the Linux Documentation Project or anyone else. Use the concepts, examples and other content at your own risk. There may be errors and inaccuracies that may be damaging to your system. Proceed with caution, and although errors are unlikely, the author(s) take no responsibility for them.
All copyrights are held by their by their respective owners, unless specifically noted otherwise. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark.
Naming of particular products or brands should not be seen as endorsements.
You are strongly recommended to take a backup of your system before major installation and backups at regular intervals.
In addition, you are strongly recommended to use a sacrificial experimental computer when mucking with the material, espcially the scripts, in this HOWTO.
You can find this document at its home page or at the Linux Documentation Project homepage in many formats. Please comment to <ccurley at trib dot com>
bzip2 compressed chunky (lots of small pages. Faster reading.) HTML.
bzip2 compressed smooth (one monster page -- no chunks. Easier to search.) HTML.
Use the source, Luke. It will be available here until I get this checked into the LDP CVS.
This document is derived from two articles originally published in Linux Journal. My thanks to Linux Journal for reverting the rights to those articles, thereby helping make this HOWTO possible.
Thanks to Joy Y Goodreau for excellent HOWTO editing.
The target computer for this HOWTO is a Pentium computer with a Red Hat 7.1 Linux server or workstation installation on one IDE hard drive. The target computer does not have vast amounts of data because the computer was set up as a "sacrificial" test bed. That is, I did not want to test this process with a production computer and production data. Also, I did a fresh installation before I started the testing so that I could always re-install if I needed to revert to a known configuration.
![]() | NOTE |
---|---|
The sample commands will show, in most cases, what I had to type to recover the target system. You may have to use similar commands, but with different parameters. It is up to you to be sure you duplicate your setup, and not the test computer's setup. |
The basic procedure is set out in W. Curtis Preston, Unix Backup & Recovery, O'Reilly & Associates, 1999, which I have favorably reviewed in Linux Journal. However, the book is a bit thin on specific, real-time questions. For example, exactly which files do you back up? What metadata do you need to preserve, and how?
Before beginning the process set forth in this HOWTO you will need to back up your system with a typical backup tool such as Amanda, BRU™, tar, Arkeia or cpio. The question, then, is how to get from toasted hardware to the point where you can run the restoration tool that will restore your data.
Users of Red Hat Package Manager (RPM) based Linux distributions should also save RPM metadata as part of their normal backups. Something like:
bash# rpm -Va > /etc/rpmVa.txt |
in your backup script will give you a basis for comparison after a bare metal restoration.
To get to this point, you need to have:
Your hardware up and running again, with replacement components as needed. The BIOS should be correctly configured, including time and date, and hard drive parameters. At the moment, there is no provision for using a different hard drive.
A parallel port Iomega® ZIP® drive or equivalent. You will need at least 30 MB of space.
Your backup media.
A minimal Linux system that will allow you to run the restoration software.
To get there, you need at least two stages of backup, and possibly three. Exactly what you back up and in which stage you back it up is determined by your restoration process. For example, if you are restoring a tape server, you may not need networking during the restoration process. So only back up networking in your regular backups.
You will restore in stages as well. In stage one, we build partitions, file systems, etc. and restore a minimal file system from the ZIP® disk. The goal of stage one is to be able to boot to a running computer with a network connection, tape drives, restoration software, or whatever we need for stage two.
The second stage, if it is necessary, consists of restoring backup software and any relevant databases. For example, suppose you use Arkeia and you are building a bare metal recovery ZIP® disk for your backup server. Arkeia keeps a huge database on the server's hard drives. You can recover the database from the tapes, if you want. Instead, why not tar and gzip the whole arkeia directory (at /usr/knox), and save that to another computer over nfs or ssh? Stage one, as we have defined it below, does not include X, so you will have some experimenting to do if you wish to back up X as well as your backup program. Some restore programs require X.
Of course, if you are using some other backup program, you may have some work to do to. You will have to find out the directories and files it needs to run. If you use tar, gzip, cpio, mt or dd for your backup and recovery tools, they will be saved to and restored from our ZIP® disk as part of the stage one process describe below.
The last stage is a total restoration from tape or other media. After you have done that last stage, you should be able to boot to a fully restored and operational system.
Hand build a new partition table and go from there. That is one reason why restore.metadata does not call the hard drive rebuilding script. (Another being that I don't know what hard drives you have.) Make sure you remove the call to fdisk from the rebuilding script.
The scripts shown here only handle ext2fs, FAT12 and FAT16. Until some eager volunteer supplies code for doing so in these scripts, you will need other tools for backing up and restoring file systems we haven't covered. Partition Image looks like a useful candidate here.
![]() | WARNING |
---|---|
Do your normal backups on their regular schedule. This HOWTO is useless if you don't do that. |
Build yourself a rescue disk. I use tomsrtbt. It is well documented and packs a lot of useful tools onto one floppy diskette. There is an active list for it, and the few questions I've had were quickly and accurately answered. I like that in a product my shop may depend on one day.
Next, figure out how to do the operating system backup you will need so that you can restore your normal backup. I followed Preston's advice and used an Iomega® parallel port ZIP® drive. The drives get approximately 90 MB of useful storage to a disk. I need about 85 MB to back up my desktop, so a 100MB ZIP® drive may be pushing your luck.
Installing the ZIP® drive is covered in the ZIP® Drive HOWTO, available at the Linux Documentation Project and at its home page, http://www.njtcom.com/dansie/zip-drive.html.
The script make.fdisk scans a hard drive for partition information, and saves it in two files. One is an executable script, called make.dev.x. The other, dev.x (where "x" is the name of the device file, e.g. hda), is the commands necessary for fdisk to build the partitions. You specify which hard drive you want to build scripts for (and thus the file names) by naming the associated device file as the argument to make.fdisk. For example, on a typical IDE system,
bash# make.fdisk /dev/hda |
spits out the script make.dev.hda and the input file for fdisk, dev.hda.
In addition, if make.fdisk encounters a FAT partition other than FAT32, it preserves the partition's boot sector in a file named dev.xy, where x is the drive's device name (e.g. sdc, hda) and y is the partition number. The boot sector is the first sector, 512 bytes, of the partition. This sector is restored at the same time the partitions are rebuilt, in the script make.dev.hda
Fortunately, the price of hard drives is plummeting almost as fast as the public's trust in politicians after an election. So it is good that the output files are text, and allow hand editing. Right now, that's the only way to rebuild on a larger replacement drive. (See the To Do list.)
Other metadata are preserved in the script save.metadata. The script saves the partition information in the file fdisk.hda in the root of the ZIP® disk. It is a good idea to print this file and your /etc/fstab so that you have hard copy should you ever have to restore the partition data manually. You can save a tree by toggling between two virtual consoles, running fdisk in one and catting /etc/fstab or /fdisk.hda as needed. However, doing so is error prone.
You will also want to preserve files relevant to your restoration method. For example, if you use nfs to save your data, you will need to preserve hosts.allow, hosts.deny, exports, etc. Also, if you are using any network-backed restoration process, such as Amanda or Quick Restore, you will need to preserve networking files like HOSTNAME, hosts, etc. and the relevant software tree.
The simplest way to handle these and similar questions is to preserve the entire etc directory.
There is no way a 100 MB ZIP® drive is going to hold a server installation of a modern distribution of Linux. We have to be much more selective than simply preserving the whole kazoo. What files do we need?
The boot directory.
The /etc directory and subdirectories.
Directories needed at boot time.
Device files in /dev.
To determine the directories needed at boot, we look at the boot initialization file /etc/rc.sysinit. It sets its own path like so:
PATH=/bin:/sbin:/usr/bin:/usr/sbin export PATH |
Trial and error indicated that we needed some other directories as well, such as /dev. In Linux, you can't do much without device files.
In reading the script save.metadata, note that we aren't necessarily saving files that are called with absolute paths.
We may require several iterations of back up, test the bare metal restore, re-install from CD and try again, before we have a working backup script. While I worked on this HOWTO, I made five such iterations before I had a successful restoration. That is one reason why it is essential to use scripts whenever possible. Test thoroughly!
Before booting tomsrtbt, make sure your ZIP® drive is placed on a parallel port, either /dev/lp0 or /dev/lp1. The start-up software will load the parallel port ZIP® drive driver for you.
The next step is to set the video mode. I usually like to see as much on the screen as I can. When the option to select a video mode comes, I use mode 6, 80 columns by 60 lines. Your hardware may or may not be able to handle high resolutions like that, so experiment with it.
Once tomsrtbt has booted and you have a console, mount the ZIP® drive. It is probably a good idea to mount it read only:
# mount /dev/sda1 /mnt -o ro |
Check to be sure it is there:
# ls -l /mnt |
Then change to the directory where the scripts are on the ZIP® drive.
# cd /mnt/root.bin |
Now run the script that will restore the partition information, e.g.:
# ./make.dev.hda |
This script will:
Clean out the first 1024 bytes of the hard drive, killing off any existing partition table and master boot record (MBR).
Recreate the partitions from the information gathered when you ran make.fdisk.
Make ext2 file system partitions and Linux swap partitions as appropriate.
Make some types of FAT partitions.
Make mount points and mount the ext2 partitions for you.
![]() | NOTE |
---|---|
If you have other operating systems to restore, now is a good time to do so. First, reboot to tomsrtbt to finish restoring Linux. You will have to remount the partitions you just built. Make a new, separate, script to mount the partitions from the tail end of the make.dev.x script. |
Once you have created all your directories and mounted partitions to them, you can run the script restore.metadata. This will restore the contents of the ZIP® drive to the hard drive.
You should see a directory of the ZIP® disk's root directory, then a list of the archive files as they are restored. Tar on tomsrtbt will tell you that tar's block size is 20, and that's fine. You can ignore it. Be sure that lilo prints out its results:
Added linux * |
That will be followed by the output from a "df -m" command.
If you normally boot directly to X, you could have some problems. To be safe, change your boot run level temporarily. In /target/etc/inittab, find the line that looks like this:
id:5:initdefault: |
and change it to this:
id:3:initdefault: |
Now, you can gracefully reboot. Remove the tomsrtbt floppy from your floppy drive if you haven't already done so, and give the computer the three fingered salute, or its equivalent:
shutdown -r now |
The computer will shut down and reboot.
As the computer reboots, go back to the BIOS and verify that the clock is more or less correct.
![]() | NOTE |
---|---|
As an alternative, you can boot to single user mode (at the lilo prompt, enter linux single), but you will have to configure your network manually and fire up sshd or whatever daemons you need to restore your system. How you do those things is very system specific. |
/etc/rc.d/init.d/arkeia start |
![]() | NOTE |
---|---|
When you restore, read the documentation for your restoration programs carefully. For example, tar does not normally restore certain characteristics of files, like suid bits. File permissions are set by the user's umask. To restore your files exactly as you saved them, use tar's p option. Similarly, make sure your restoration software will restore everything exactly as you saved it. |
bash# restore.all |
tar: usr/sbin/rpcinfo: Could not create file: File exists tar: usr/sbin/zdump: Could not create file: File exists tar: usr/sbin/zic: Could not create file: File exists tar: usr/sbin/ab: Could not create file: File exists |
bash# rpm -Va |
id:5:initdefault: |
You should now be ready for rock and roll -- and some aspirin and a couch.
This distribution is the one I use on my test computer. I have had no problems with it.
This version seems to require libcrack (in /usr/lib) and its attendant files in order to authenticate users. So in save.metadata, add to the line that saves /usr/lib the following: /usr/lib/*crack* and enable that line.
I have listed below notes about backing up particular applications.
Squid is a http proxy and cache. As such it keeps a lot of temporary data on the hard drive. There is no point in backing that up. Insert "--exclude /var/spool/squid" into the appropriate tar command in your second stage backup script. Then, get squid to rebuild its directory structure for you. Tack onto the tail end of the second stage restore script a command for squid to initialize itself. Here is how I did it over ssh in restore.tester:
ssh $target "mkdir /var/spool/squid ; chown squid:squid /var/spool/squid;\ /usr/sbin/squid -z;touch /var/spool/squid/.OPB_NOBACKUP" |
The last command creates a file of length 0 called .OPB_NOBACKUP. This is for the benefit of Arkeia, and tells Arkeia not to back up below this directory
Arkeia is a backup and restore program that runs on a wide variety of platforms. You can use Arkeia as part of a bare metal restoration scheme, but there are two caveats.
The first is probably the most problematic, as absent any more elegant solution you have to hand select the directories to restore in the navigator at restoration time. The reason is that, apparently, Arkeia has no mechanism for not restoring files already present on the disk, nothing anlogous to tar's -p option. If you simply allow a full restore, the restore will crash as Arkeia over-writes a library which is in use at restore time, e.g. lib/libc-2.1.1.so. Hand selection of directories to restore is at best dicy, so I recommend against it.
The second caveat is that you have to back up the Arkeia data dictionary and/or programs. To do that, modify the save.metatdata script by adding Arkeia to the list of directories to save:
# arkeia specific: tar cf - usr/knox | gzip -c > $zip/arkeia.tar.gz |
You must back up the data dictionary this way because Arkeia does not back up the data dictionary. This is one of my complaints about Arkeia, and I solve it on my own computer by saving the data dictionary to tape with The TOLIS Group's BRU™.
The data dictionary will be restored in the script restore.metadata automatically.
You should also have several tomsrtbt floppies and possibly some ZIP® drives in your off-site storage as well. Also, have copies of the tomsrtbt distribution on several of your computers so that they back each other up.
You should probably have copies of this HOWTO, with your site-specific annotations on it, with your backups and in your off-site backup storage.
A partition editor to adjust partition boundaries for a different hard drive, or the same one with different geometry, or to adjust partition sizes within the same hard drive. A GUI would probably be a good idea here. On the other tentacle, the FSF's parted looks like it will fill part of the bill. It does re-size existing partitions, but with restrictions.
make.fdisk currently spits out one script. Separate out the mount commands to another script, so you can run make.dev.hda, then reboot to do some other mischief, like build a partition for some exotic OS I've never heard of, or run parted, then reboot to tomsrtbt, mount all the Linux partitions, and continue.
Since tomsrtbt supports bzip2, convert the scripts to use bzip2, and see if there is a noticeable reduction in the first stage data saved.
make.fdisk currently only recognizes some FAT partitions, not all. Add code to make.fdisk to recognize others and make appropriate instructions to rebuild them in the output files.
For FAT12 or FAT16 partitions we do not format, write zeros into the partition so that Mess-DOS 6.x does not get confused. See the notes on fdisk for an explanation of the problem.
Make a script for putting ext2 file systems on ZIP® disks.
Translations into other (human) languages.
Find out how loadlin or similar programs affect this process.
Changes for GRUB
Change the scripts to use a CD-ROM. A CD-ROM that would boot to tomsrtbt, with the first stage restore data on the rest of it, would be just the ticket.
See the notes in the beginning of each script for a summary of what it does.
This script, run at backup time, creates a script similar to make.dev.hda, below, for you to run at restore time. It also produces data files similar to dev.hda, below. The name of the script and data file produced depends on the device given this script as a a parameter. That script, run at restore time, builds the partitions on the hard drive. make.fdisk is called from save.metadata, below.
#! /usr/bin/perl # A perl script to create a script and input file for fdisk to # re-create the partitions on the hard disk, and format the Linux and # Linux swap partitions. The first parameter is the fully qualified # path of the device of the hard disk, e.g. /dev/hda. The two # resulting files are the script make.dev.x and the data file dev.x # (where x is the hard drive described, e.g. hda, sdc). make.dev.x is # run at restore time to rebuild hard drive x, prior to running # restore.metadata. dev.x is the input file for fdisk. # Time-stamp: <2001-12-13 15:41:24 ccurley make.fdisk> # Copyright 2001 through the last date of modification Charles Curley # except for the subroutine cut2fmt. # cut2fmt Copyright (c) 1998 Tom Christiansen, Nathan Torkington and # O'Reilly & Associates, Inc. Permission is granted to use this code # freely EXCEPT for book publication. You may use this code for book # publication only with the explicit permission of O'Reilly & # Associates, Inc. # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # In addition, as a special exception, Tom Christiansen, Nathan # Torkington and O'Reilly & Associates, Inc. give permission to use # the code of this program with the subroutine cut2fmt (or with # modified versions of the subroutine cut2fmt that use the same # license as the subroutine cut2fmt), and distribute linked # combinations including the two. You must obey the GNU General # Public License in all respects for all of the code used other than # the subroutine cut2fmt. If you modify this file, you may extend # this exception to your version of the file, but you are not # obligated to do so. If you do not wish to do so, delete this # exception statement and the subroutine cut2fmt from your version. # You can also contact the Free Software Foundation at http://www.fsf.org/ # Changes: # 2001 11 25: Changed the way mke2fs gets its bad block # list. badblocks does not guess at the block size, so you have to get # it (from dumpe2fs) and feed it to badblocks. It is simpler to just # have mke2fs call badblocks, but you do loose the ability to have a # writing test easily. -- C^2 # 2001 11 25: Changed the regex that extracts partition labels from # the mount command. This change does not affect the results at all, # it just makes it possible to use Emacs' perl mode to indent # correctly. I just escaped the left bracket in the regex. -- C^2 # Discussion: # fdisk will spit out a file of the form below if you run it as "fdisk # -l". # root@tester ~/bin $ fdisk -l /dev/hda # Disk /dev/hda: 64 heads, 63 sectors, 1023 cylinders # Units = cylinders of 4032 * 512 bytes # Device Boot Start End Blocks Id System # /dev/hda1 1 9 18112+ 83 Linux # /dev/hda2 10 1023 2044224 5 Extended # /dev/hda5 10 368 723712+ 83 Linux # /dev/hda6 369 727 723712+ 83 Linux # /dev/hda7 728 858 264064+ 83 Linux # /dev/hda8 859 989 264064+ 83 Linux # /dev/hda9 990 1022 66496+ 82 Linux swap # What fdisk does not do is provide output suitable for later # importing into fdisk, a la sfdisk. This script parses the output # from fdisk and creates an input file for fdisk. Use the input file # like so: # fdisk /dev/hdx < dev.hdx # For the bare metal restore package, this script also builds a script # that will execute the above command so you can run it from your zip # disk. Because the bare metal restore scripts all are in /root/bin, # the data file and script created by this script are also placed # there. The same script also creates appropriate Linux file systems, # either ext2fs, or Linux swap. There is limited support for FAT12 and # FAT16. There is no support right now (hint, hint) for FAT32. For # anything else, you're on your own. # Note for FAT32: According to the MS KB, there are more than one # reserved sectors for FAT32, usually 32, but it can vary. Do a search # in M$'s KB for "boot sector" or BPB for the gory details. For more # info than you really need on how boot sectors are used, see # http://support.microsoft.com/support/kb/articles/Q140/4/18.asp # You can also edit dev.x to change the sizes of partitions. Don't # forget, if you change the size of a FAT partition across the 32MB # boundary, you need to change the type as well! Run "fdisk /dev/hda" # or some such, then the l command to see the available partition # types. Then go ahead and edit dev.x appropriately. Also, when moving # partition boundarys with hand edits, make sure you move both logical # and extended partition boundaries appropriately. # Bad block checking right now is a quick read of the partition. A # writing check is also possible but more difficult. You have to run # badblocks as a separate command, and pass the bad block list to # mke2fs in a file (in /tmp, which is a ram disk). You also have to # know how large the blocks are, which you learn by running # dumpe2fs. It gets messy and I haven't done it yet. You probably # don't need it for a new hard drive, but if you have had a hard drive # crash on you and you are reusing it (while you are waiting for its # replacement to come in, I presume), then I highly recommend it. Let # me know how you do it. # For more information contact the author, Charles Curley, at # http://w3.trib.com/~ccurley/. # cut2fmt figures out the format string for the unpack function we use # to slice and dice the output from fdisk. From Christiansen and # Torkington, Perl Cookbook 5. sub cut2fmt { my (@positions) = @_; my $template = ''; my $lastpos = 1; foreach $place (@positions) { $template .= "A" . ($place - $lastpos) . " "; $lastpos = $place; } $template .= "A*"; return $template; } # Provide a default device. if ( $ARGV[0] == "" ) { $device="/dev/hda"; } else { $device=$ARGV[0]; } # print "Device is $device.\n"; # prepare format string. $fmt = cut2fmt (11, 19, 24, 34, 45, 49); # print "Format string is $fmt.\n\n"; # define fields in the array @_. $dev = 0; $bootable = 1; $firstcyl = 2; $lastcyl = 3; $parttype = 5; $partstring = 6; $target = "\/target"; $outputfilename = $device; $outputfilename =~ s/\//./g; $outputfilename = substr ($outputfilename, 1, 100); $outputfilepath = "\/root\/bin\/"; # Make a hash of the labels. $mpid = open (MOUNT, "mount -l |") or die "Couldn't fork: $!\n"; while (<MOUNT>) { if ($_ =~ /^$device/i) { # is this a line with a partition in it? # print $_; # print it just for grins split; if ($_[6] ne "") { # only process if there actually is a label $_[6] =~ s/[\[\]]//g; # strike [ and ]. $labels{$_[0]} = $_[6]; # print "The label of file device $_[0] is $labels{$_[0]}.\n"; } # We only mount if it's ext2fs and read and write. if ($_[4] eq "ext2" and $_[5] eq "(rw)" ) { $mountpoints{$_[2]} = $_[0]; # print "$_[2] is the mountpoint for $mountpoints{$_[2]}.\n"; } } } close (MOUNT); $fpid = open (FDISK, "fdisk -l $device |") or die "Couldn't fork: $!\n"; open (OUTPUT, "> $outputfilepath${outputfilename}") or die "Couldn't open output file $outputfilepath${outputfilename}.\n"; while (<FDISK>) { if ($_ =~ /^$device/i) { # is this a line with a partition in it? # print $_; # print it just for grins chop; # kill trailing \r @_ = unpack ($fmt, $_); # now strip white spaces from cylinder numbers @_[$firstcyl] =~ s/[ \t]+//; @_[$lastcyl] =~ s/[ \t]+//; @_[$parttype] =~ s/[ \t]+//; $partnumber = substr(@_[$dev], 8, 10); # get partition number for this line # just for grins # print " $partnumber, @_[$firstcyl], @_[$lastcyl], @_[$partstring]\n"; # Here we start creating the input to recreate the partition # this line represents. print OUTPUT "n\n"; if ($partnumber < 5) { # primary Linux partition if (@_[$parttype] == 83) { print OUTPUT "p\n$partnumber\n@_[$firstcyl]\n"; if (@_[$firstcyl] ne @_[$lastcyl]) { # in case it's all on one cylinder print OUTPUT "@_[$lastcyl]\n"; } if ($labels{@_[$dev]}) { # do we have a label? # no bad block check # $format .= "mke2fs @_[$dev] -L $labels{@_[$dev]}\n"; $format .= "echo\necho formatting and checking @_[$dev]\n"; $format .= "mke2fs -c -L $labels{@_[$dev]} @_[$dev]\n\n"; } else { # $format .= "mke2fs @_[$dev]\n"; # no bad block check $format .= "echo\necho formatting and checking @_[$dev]\n"; $format .= "mke2fs -c @_[$dev]\n\n"; } # extended partition } elsif (@_[$parttype] == 5) { print OUTPUT "e\n$partnumber\n@_[$firstcyl]\n"; if (@_[$firstcyl] ne @_[$lastcyl]) { print OUTPUT "@_[$lastcyl]\n"; } # primary Linux swap partition } elsif (@_[$parttype] == 82) { print OUTPUT "p\n$partnumber\n@_[$firstcyl]\n"; if (@_[$firstcyl] ne @_[$lastcyl]) { print OUTPUT "@_[$lastcyl]\n"; } print OUTPUT "t\n$partnumber\n82\n"; $format .= "mkswap -c @_[$dev]\n\n"; # primary mess-dos partition. We don't handle FAT32, # which requires a command line switch for mkdosfs. } elsif ( @_[$parttype] == 1 || @_[$parttype] == 4 || @_[$parttype] == 6 ) { print ("dd if=@_[$dev] of=$outputfilepath$outputfilename$partnumber"); print (" bs=512 count=1\n"); system ("dd if=@_[$dev] of=$outputfilepath$outputfilename$partnumber bs=512 count=1"); print OUTPUT "p\n$partnumber\n@_[$firstcyl]\n"; if (@_[$firstcyl] ne @_[$lastcyl]) { # in case it's all on one cylinder print OUTPUT "@_[$lastcyl]\n"; } print OUTPUT "t\n$partnumber\n@_[$parttype]\n"; # $format .= "mkdosfs @_[$dev]\n"; # no bad block check $format .= "echo\necho formatting and checking @_[$dev]\n"; $format .= "mkdosfs -c @_[$dev]\n"; $format .= "# restore FAT boot sector.\n"; $format .= "dd if=$outputfilename$partnumber of=@_[$dev] bs=512 count=1\n\n"; } else { # anything else partition print OUTPUT "p\n@_[$firstcyl]\n"; if (@_[$firstcyl] ne @_[$lastcyl]) { print OUTPUT "@_[$lastcyl]\n"; } print OUTPUT "t\n$partnumber\n@_[$parttype]\n"; } } else { # logical Linux partition if (@_[$parttype] == 83) { print OUTPUT "l\n@_[$firstcyl]\n"; if (@_[$firstcyl] ne @_[$lastcyl]) { print OUTPUT "@_[$lastcyl]\n"; } if ($labels{@_[$dev]}) { # do we have a label? # no bad block check # $format .= "mke2fs @_[$dev] -L $labels{@_[$dev]}\n"; $format .= "echo\necho formatting and checking @_[$dev]\n"; $format .= "mke2fs -c -L $labels{@_[$dev]} @_[$dev]\n\n"; } else { # $format .= "mke2fs @_[$dev]\n"; # no bad block check $format .= "echo\necho formatting and checking @_[$dev]\n"; $format .= "mke2fs -c @_[$dev]\n\n"; } # logical Linux swap partition } elsif (@_[$parttype] == 82 ) { print OUTPUT "l\n@_[$firstcyl]\n"; if (@_[$firstcyl] ne @_[$lastcyl]) { print OUTPUT "@_[$lastcyl]\n"; } print OUTPUT "t\n$partnumber\n82\n"; $format .= "mkswap -c @_[$dev]\n\n"; # primary mess-dos partition. We don't handle FAT32, # which requires a command line switch for mkdosfs. } elsif ( @_[$parttype] == 1 || @_[$parttype] == 4 || @_[$parttype] == 6 ) { print ("dd if=@_[$dev] of=$outputfilepath$outputfilename$partnumber"); print (" bs=512 count=1\n"); system ("dd if=@_[$dev] of=$outputfilepath$outputfilename$partnumber bs=512 count=1"); print OUTPUT "p\n$partnumber\n@_[$firstcyl]\n"; if (@_[$firstcyl] ne @_[$lastcyl]) { # in case it's all on one cylinder print OUTPUT "@_[$lastcyl]\n"; } print OUTPUT "t\n$partnumber\n@_[$parttype]\n"; # $format .= "mkdosfs @_[$dev]\n"; # no bad block check $format .= "echo\necho formatting and checking @_[$dev]\n"; $format .= "mkdosfs -c @_[$dev]\n"; $format .= "# restore FAT boot sector.\n"; $format .= "dd if=$outputfilename$partnumber of=@_[$dev] bs=512 count=1\n\n"; } else { # anything else partition print OUTPUT "l\n@_[$firstcyl]\n"; if (@_[$firstcyl] ne @_[$lastcyl]) { print OUTPUT "@_[$lastcyl]\n"; } print OUTPUT "t\n$partnumber\n@_[$parttype]\n"; } } # handle bootable partitions if (@_[$bootable] =~ /\*/) { print OUTPUT "a\n$partnumber\n"; } } } print OUTPUT "v\nw\n"; close (OUTPUT); close (FDISK); open (OUTPUT, "> ${outputfilepath}make.$outputfilename") or die "Couldn't open output file ${outputfilepath}make.$outputfilename.\n"; print OUTPUT <<FINIS; #! /bin/sh # A script to restore the partition data of a hard drive and format # the partitions. Created at bare metal backup time by the Perl script # make.fdisk. # Copyright 2001 through the last date of modification Charles Curley. # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # You can also contact the Free Software Foundation at http://www.fsf.org/ # For more information contact the author, Charles Curley, at # http://w3.trib.com/~ccurley/. FINIS # Clean the old partition table out. print OUTPUT "dd if=/dev/zero of=$device bs=512 count=2\n\nsync\n\n"; print OUTPUT "fdisk $device \< $outputfilename\n\nsync\n\n"; print OUTPUT $format; # Now build the mount points on the root and other partitions. # We have a hash of mount points and devices in %mountpoints. However, # we have to process them such that directories are built on the # appropriate target partition. E.g. where /usr/local is on its own # partition, we have to mount /usr before we build /usr/local. We can # ensure this by sorting them. Shorter mount point paths will be built # first. We can't sort a hash directly, so we use an array. # We build commands to create the appropriate mount points and then # mount the partitions to the mount points. This is in preparation for # untarring the contents of the ZIP disk, done in restore.metadata. foreach $point ( sort keys %mountpoints) { print OUTPUT "\n# $point is the mountpoint for $mountpoints{$point}.\n"; print OUTPUT "mkdir $target$point\n"; print OUTPUT "mount $mountpoints{$point} $target$point\n"; } close (OUTPUT); # These scripts are dangerous & should only be visible to root. chmod 0700, "${outputfilepath}make.$outputfilename"; chmod 0600, "${outputfilepath}$outputfilename"; |
This script is a sample of the sort produced by make.fdisk, above. It uses data files like dev.hda, below. It builds partitions and puts file systems on some of them. This is the first script run at restore time.
If you are brave enough to edit dev.hda (q.v.), say, to add a new partition, you may need to edit this script as well.
#! /bin/sh # A script to restore the partition data of a hard drive and format # the partitions. Created at bare metal backup time by the Perl script # make.fdisk. # Copyright 2001 through the last date of modification Charles Curley. # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # You can also contact the Free Software Foundation at http://www.fsf.org/ # For more information contact the author, Charles Curley, at # http://w3.trib.com/~ccurley/. dd if=/dev/zero of=/dev/hda bs=512 count=2 sync fdisk /dev/hda < dev.hda sync echo echo formatting and checking /dev/hda1 mkdosfs -c /dev/hda1 # restore FAT boot sector. dd if=dev.hda1 of=/dev/hda1 bs=512 count=1 echo echo formatting and checking /dev/hda2 badblocks -c 128 -o /tmp/dev.hda2 /dev/hda2 mke2fs -L /boot -l /tmp/dev.hda2 /dev/hda2 echo echo formatting and checking /dev/hda5 badblocks -c 128 -o /tmp/dev.hda5 /dev/hda5 mke2fs -L / -l /tmp/dev.hda5 /dev/hda5 mkswap -c /dev/hda6 # / is the mountpoint for /dev/hda5. mkdir /target/ mount /dev/hda5 /target/ # /boot is the mountpoint for /dev/hda2. mkdir /target/boot mount /dev/hda2 /target/boot |
This data file is used at restore time. It is fed to fdisk by the script make.dev.hda. It is produced at backup time by make.fdisk. Those familiar with fdisk will recognize that each line is an fdisk command or value, such as a cylinder number. Thus, it is possible to change the partition sizes and add new partitions by editing this file. That's why the penultimate command is v, to verify the partition table before it is written.
n p 1 1 29 t 1 6 a 1 n p 2 30 44 n e 3 45 1023 n l 45 944 n l 945 1023 t 6 82 v w |
This is the first script to run as part of the backup process. It calls make.fdisk, above. If you have a SCSI hard drive or multiple hard drives to back up, edit the call to make.fdisk appropriately.
#! /bin/sh # A script to save certain meta-data off to the boot partition. Useful for # restoration. # Time-stamp: <2001-11-19 09:18:52 ccurley save.metadata> # Copyright 2000 through the last date of modification, Charles Curley. # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # You can also contact the Free Software Foundation at http://www.fsf.org/ # For more information contact the author, Charles Curley, at # http://w3.trib.com/~ccurley/. export zip="/mnt/zip"; # export save="/mnt/save"; echo "saving hard drive info" make.fdisk /dev/hda # back up RPM metadata echo "Verifying RPMs." rpm -Va > /etc/rpmVa.txt echo "Finished verifying RPMs; now mounting the ZIP drive." umount $zip modprobe ppa mount /dev/sda1 # clean it all out rm -r $zip/* echo "Building the ZIP drive backups." mkdir $zip/lost+found fdisk -l /dev/hda > $zip/fdisk.hda ls -al /mnt > $zip/ls.mnt.txt ls -al / > $zip/ls.root.txt mkdir $zip/etc; cp -p /etc/* $zip/etc cd / # These appear to be required so we can restore later on. tar cf - boot | gzip -c > $zip/boot.tar.gz tar cf - root | gzip -c > $zip/root.tar.gz tar cf - etc | gzip -c > $zip/etc.tar.gz tar cf - lib | gzip -c > $zip/lib.tar.gz tar cf - usr/sbin | gzip -c > $zip/usr.sbin.tar.gz tar cf - usr/bin | gzip -c > $zip/usr.bin.tar.gz tar cf - sbin | gzip -c > $zip/sbin.tar.gz tar cf - bin | gzip -c > $zip/bin.tar.gz tar cf - dev | gzip -c > $zip/dev.tar.gz # Now optional saves. # arkeia specific: tar cf - usr/knox | gzip -c > $zip/arkeia.tar.gz # save these so we can use ssh for restore. *crack* for RH 7.0 login # authentication. tar cf - usr/lib/*crack* usr/lib/libz* usr/lib/libssl* usr/lib/libcrypto*\ | gzip -c > $zip/usr.lib.tar.gz # save the scripts we used to create the zip disk and the ones we will # use to restore it. mkdir $zip/root.bin cp -p /root/bin/* $zip/root.bin rm $zip/root.bin/*~ $zip/root.bin/#*# echo "Test our results." find $zip -iname "*.gz" | xargs gzip -t # Not a normal part of the process: we duplicate the ZIP disk onto an # NFS mount elsewhere. # echo "Backing the ZIP drive to the NFS mount." # umount $save # mount $save # rm -r $save/zip # mkdir $save/zip # cp -pr $zip $save df -m eject $zip |
This script restores metadata from the ZIP® disk as a first stage restore.
#! /bin/sh # A script to restore the meta-data from the ZIP disk. This runs under # tomsrtbt only after partitions have been rebuilt, file systems made, # and mounted. It also assumes the ZIP disk has already been # mounted. Mounting the ZIP disk read only is probably a good idea. # Time-stamp: <2001-11-19 09:21:51 ccurley restore.metadata> # Copyright 2000 through the last date of modification Charles Curley. # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # You can also contact the Free Software Foundation at http://www.fsf.org/ # For more information contact the author, Charles Curley, at # http://w3.trib.com/~ccurley/. zip="/mnt"; target="/target"; # warm fuzzies for the user. ls -lt $zip cd $target # restore the archived metadata files. for archive in $( ls $zip/*.gz ); do echo $archive ls -al $archive gzip -dc $archive | tar -xf - done mkdir $target/proc chmod a-w $target/proc # build a mount point directory so we'll have it on reboot. mkdir $target/mnt # Build the mount points for our second stage restoration and other # things. mkdir $target/mnt/save mkdir $target/mnt/zip mkdir $target/mnt/cdrom mkdir $target/mnt/floppy mkdir $target/mnt/imports # Restore the scripts we used to create the ZIP disk and the ones we will # use to restore it. These should be the latest & greatest in case we had # to do any editing during 1st stage restore. cp -p $zip/root.bin/* $target/root/bin # Now install the boot sector. chroot $target /sbin/lilo -C /etc/lilo.conf df -m |
These scripts run on the computer being backed up or restored.
#! /bin/sh # Back up the entire system to another computer's drive. To make this # work, we need a convenient chunk of disk space on the remote computer we # can nfs mount as /mnt/save. # Time-stamp: <2001-11-19 09:19:35 ccurley back.up.all> # Copyright 2000 through the last date of modification Charles Curley. # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # You can also contact the Free Software Foundation at http://www.fsf.org/ # For more information contact the author, Charles Curley, at # http://w3.trib.com/~ccurley/. save="/mnt/save" # Make sure it's there umount $save mount $save cd / rm $save/tester.tar.old.gz mv $save/tester.tar.gz $save/tester.tar.old.gz # save everything except /mnt, /proc, and nfs mounted directories. time tar cf - / --exclude /mnt --exclude /proc --exclude $save\ | gzip -c > $save/tester.tar.gz |
This script does exactly what back.up.all does, but it uses SSH instead of nfs.
#! /bin/sh # Back up the entire system to another computer's drive. To make this # work, we need a convenient chunk of disk space on the remote # computer. This version uses ssh to do its transfer, and compresses # using bz2. This means this script has to know more about the other # computer, which does not make for good modularization. # Time-stamp: <2001-11-19 09:19:48 ccurley back.up.all.ssh> # Copyright 2000 through the last date of modification Charles Curley. # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # You can also contact the Free Software Foundation at http://www.fsf.org/ # For more information contact the author, Charles Curley, at # http://w3.trib.com/~ccurley/. save="/backs/tester" backup_server="charlesc" # rotate the old backups. Do it all in one line to minimze authentication overhead. ssh $backup_server "rm $save/tester.tar.old.bz2; mv $save/tester.tar.bz2 \ $save/tester.tar.old.bz2" # save everything except /mnt, /proc, and squid directories. time tar cf - / --exclude /mnt --exclude /proc --exclude /var/spool/squid\ | ssh $backup_server "bzip2 -9 > $save/tester.tar.bz2" |
This is the restore script to use if you backed up using back.up.all.
#! /bin/sh # A script to restore all of the data from an nfs mount. This is our final # stage restore. # Time-stamp: <2001-11-19 09:21:22 ccurley restore.all> # Copyright 2000 through the last date of modification Charles Curley. # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # You can also contact the Free Software Foundation at http://www.fsf.org/ # For more information contact the author, Charles Curley, at # http://w3.trib.com/~ccurley/. export save="/mnt/save" mount $save cd / gunzip -dc $save/tester.tar.gz | tar -xpkf - rm /var/run/*.pid lilo |
This is the restoration script to use if you used back.up.all.ssh to back up.
#! /bin/sh # A script to restore all of the data using ssh and bunzip2. This is # our final stage restore. # Copyright 2000 through the last date of modification Charles Curley. # Time-stamp: <2001-11-19 09:21:33 ccurley restore.all.ssh> # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # You can also contact the Free Software Foundation at http://www.fsf.org/ # For more information contact the author, Charles Curley, at # http://w3.trib.com/~ccurley/. save="/backs/tester/" backup_server="charlesc" cd / ssh $backup_server "cat $save/tester.tar.bz2" | bunzip2 | tar -xpkf - rm /var/run/*.pid lilo |
I use these scripts routinely.
#! /bin/sh # Back up another computer's drive to this system. To make this work, we # need a convenient chunk of disk space on this computer. This version # uses ssh to do its transfer, and compresses using bz2. This version was # developed so that the system to be backed up won't be authenticated to # log onto the backup computer. This script is intended to be used on a # firewall. You don't want the firewall to be authenticated to the backup # system in case the firewall is cracked. # Time-stamp: <2001-11-19 09:20:06 ccurley get.tester> # Copyright 2000 through the last date of modification Charles Curley. # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # You can also contact the Free Software Foundation at http://www.fsf.org/ # For more information contact the author, Charles Curley, at # http://w3.trib.com/~ccurley/. # The host name of the computer to be backed up. target=tester echo Backing up $target echo Aging the ZIP disk backups. rm -r $target.oldzip mv $target.zip $target.oldzip ssh $target "modprobe ppa ; mount -r /mnt/zip" echo Copying the ZIP disk. # -r for recursive copy, -p to preserve times and permissions, -q for # -quiet: no progress meter. scp -qpr $target:/mnt/zip $target.zip echo Testing the results. find . -iname "*.gz" | xargs gunzip -t ssh $target "eject $zip" echo Aging the archives rm $target.tar.old.bz2 mv $target.tar.bz2 $target.tar.old.bz2 echo Backing up $target to the backup server. ssh $target tar -cf - / --exclude /mnt --exclude /proc --exclude /var/spool/squid\ | bzip2 -9 | cat > $target.tar.bz2 echo Testing the results. find . -iname "*.bz2" | xargs bunzip2 -t |
#! /bin/sh # A script to restore all of the data to tester via ssh. This is our final # stage restore. # Time-stamp: <2001-11-19 09:22:01 ccurley restore.tester> # Copyright 2000 through the last date of modification Charles Curley. # This program is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by the # Free Software Foundation; either version 2 of the License, or (at your # option) any later version. # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # You can also contact the Free Software Foundation at http://www.fsf.org/ # For more information contact the author, Charles Curley, at # http://w3.trib.com/~ccurley/. # The host name of the computer to be restored. target=tester bunzip2 -dc $target.tar.bz2 | ssh $target "cd / ; tar -xpkf - " ssh $target lilo |
W. Curtis Preston's excellent Unix Backup & Recovery. This is the book that got me started on this bare metal recovery stuff. I highly recommend it; read my review.
A list of small Linux disties.
tomsrtbt, "The most Linux on 1 floppy disk." Tom also has links to other small disties.
The Linux Documentation Project. See particularly the "LILO, Linux Crash Rescue HOW-TO"
The Free Software Foundation's parted for editing (enlarging, shrinking, moving) partitions.
Partition Image for backing up partitions.
Hugo Rabson's Mondo "... creates one or more bootable Rescue CD's (or tape+floppies) containing some or all of your filesystem. In the event of catastrophic data loss, you will be able to restore from bare metal."
Copyright (C) 2000 Free Software Foundation, Inc. 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.
Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.
Copyright (c) YEAR YOUR NAME. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with the Invariant Sections being LIST THEIR TITLES, with the Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST. A copy of the license is included in the section entitled "GNU Free Documentation License".
If you have no Invariant Sections, write "with no Invariant Sections" instead of saying which ones are invariant. If you have no Front-Cover Texts, write "no Front-Cover Texts" instead of "Front-Cover Texts being LIST"; likewise for Back-Cover Texts.
If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.