to scdbackup homepage


scdbackup
- Examples -




to top of page


A sample configuration with both a CD and a DVD writer


This is an installation for a normal user. Provided this user can already run programs cdrecord and growisofs, or cdrskin, or xorriso, there will be no urgent need for superuser action.
Nevertheless the superuser should later follow these steps to create a private installation for system backups and eventual bundled user backups.

The user's actions are shown in bold kbd typeface.

Step 1 of the installation procedure (see README) is to unpack the archive in an arbitrary directory (here: $HOME/scdbackup) and to go to the newly created directory scdbackup-0.9.2 :
$ mkdir $HOME/scdbackup
$ cd $HOME/scdbackup
$ tar xzf ..../scdbackup-0.9.2.tar.gz
$ cd scdbackup-0.9.2/inst
I read README paragraph "Planning" to prepare myself for the coming questions:
I found out the device file address of my DVD writer and briefly tested it with xorriso. I decided not to need a split directory for overly large files and no buffer file for avoiding failed burns with legacy hardware.

Now i start the configuration script for CD (step 2a). If you are solely interested in DVD you may jump to Step 2b , DVD configuration.
$ ./CONFIGURE_CD
 
scdbackup for CD installation script 0.9.2
 
Evaluating availability of ISO 9660 formatter programs:
...
Evaluating availability of CD burn programs:
...
========================================================================
Note: Empty input leaves the current setting unchanged
 
Note: Choose a ISO 9660 formatter program. See list in () brackets.
You may add "-J" and "-joliet-long" after the program name.
 
What ISO 9660 formatter program to use ? ( xorriso genisoimage mkisofs )
current setting  : ""
Your input please:
xorriso
 
Note: Choose a CD burn program. Available local programs are:
    cdrskin cdrecord xorriso wodim
...
"-" disables use of any burn program.
 
What CD burn program to use ? ( cdrskin cdrecord xorriso wodim )
current setting  : ""
Your input please:
xorriso
The chosen CD burn program will then show an overview of the burner devices.
----------------------------------------------------- devices
/dev/sr0 : 'TSSTcorp' 'CDDVDW SH-S203B'
/dev/sr1 : 'PHILIPS ' 'SPD3300L'
/dev/hda : 'HL-DT-ST' 'DVD-ROM GDR8162B'
/dev/hdc : 'LITE-ON ' 'LTR-48125S'
----------------------------------------------------- end of device list
 
Note: Adresses of detected drives are shown in the () list.
Enter "-" to skip this input and disable burning for now.
 
Address of CD recorder ? ( /dev/sr0 /dev/sr1 /dev/hda /dev/hdc )
current setting  : ""
Your input please:
/dev/sr0
 
Evaluating CD burner at address /dev/sr0 . This may last some seconds.
Burner evaluation done
 
Speed ? (e.g. 2)
current setting  : ""
Your input please:
48
 
Note: erasing CDs is usable for rewritable CD-RW media only
 
Automatically erase CD before writing ? (yj1/n0)
current setting  : ""
Your input please:
y
 
Note: Files which do not fit on the media usually are excluded from
the backup. But a directory outside the backup area
may be used to buffer pieces of such files which then become
part of the backup. The directory has to be empty now or has
already to contain a file scdbackup_split_file_directory_mark.
Its filesystem has to be able to carry a full media content
(650MB). Take care not to use the same directory as with DVD !
If you enter "-" then large files are excluded.
Your local xorriso is suitable for "-xorriso_cut_out"
which needs no extra disk space.
 
Large file split directory ? ( "-" = exclude large files)
current setting  : ""
Your input please:
-xorriso_cut_out
 
Note: Recording CDs without buffering at higher speed may fail
if the system is slow or its load by other processes is high.
Compressed afio (system) backups may produce arbitrarily sparse
output which strongly increases the risk of such a failure.
Therefore compression usually is disabled without buffer.
On the other hand, the buffer is as large as the CD (~ 650MB)
and must be outside the backup area or explicitely excluded.
Take care not to use the same file as with DVD !
Special name "-" disables buffering. "-afio_compress" does the
same but enables afio compression (needs burnproof hardware).
 
Buffer file name ? ( or "-" or "-afio_compress" )
current setting  : ""
Your input please:
-afio_compress
 
Note: if you enter 0 then the directory size is determined
by a very slow method which previews the compressed
afio sizes.
 
Estimated capacity for a compressed afio CD ? (eg. 1024m)
current setting  : ""
Your input please:
1200m
 
Exclusion list of directories for system backup ?
current setting  : "/cdrom /cdrw /floppy /home /mnt /opt /proc /tmp"
Your input please:
Normal users just press Enter here. This input is only of interest for the superuser.
Please note that -if ever- a single line is expected here, despite your browser right now may show more than one.
unchanged: "/cdrom /cdrw /floppy /home /mnt /opt /proc /tmp"
The interogation ends now and the parameters get written to their files.
In future runs of CONFIGURE_CD you may confirm your settings by empty input lines and only have to enter those values which you want to change. The content of those files may be temporarily overriden by shell variables.
See README Appendix CONFIGURATION for a definition of the files' content (if it isn't obvious by name and sequence :)

writing file scdbackup_use_genisoimage_value
done
writing file scdbackup_use_cdrskin_value
done
...
Now applied : permission scheme 'owner_only'
 
Note: You may change the permission scheme by ./SET_PERMISSIONS
 
scdbackup for CD 0.9.2 : First stage of installation done.
--------------------------------------------------------
The commands of your scdbackup installation are now available in :
/home/thomas/scdbackup/scdbackup-0.9.2/cmd
 
You may want to add this address to your PATH variable
or ask your superuser to run ./SUPERUSER_CONFIGURE in directory
/home/thomas/scdbackup/scdbackup-0.9.2/inst
--------------------------------------------------------
If you got no DVD writer or if you want to test CD in advance, then you may skip now to Step 3

Now for DVD configuration (step 2b) :
$ ./CONFIGURE_DVD
 
scdbackup for DVD installation script 0.9.2
 
Evaluating availability of ISO 9660 formatter programs:
...
Evaluating availability of DVD burn programs:
...
========================================================================
Note: Empty input leaves the current setting unchanged
 
Note: Choose a ISO 9660 formatter program. See list in () brackets.
You may add "-J" and "-joliet-long" after the program name.
 
What ISO 9660 formatter program to use ? ( xorriso genisoimage mkisofs )
current setting  : ""
Your input please:
xorriso
 
Note: Choose a DVD burn program. Available local programs are:
    growisofs cdrskin xorriso cdrecord wodim
...
 
What DVD burn program to use ? ( growisofs cdrskin xorriso cdrecord wodim )
current setting  : "xorriso"
Your input please:
xorriso
 
----------------------------------------------------- devices
...
Address of DVD recorder ? ( /dev/sr0 /dev/sr1 /dev/hda /dev/hdc )
current setting  : ""
Your input please:
/dev/sr0
 
Note: Speed is an upper limit. The drive will eventually curb it.
By default the burn program will do fast DVD-RAM and BD
recording (option stream_recording=on).
Option stream_recording=off causes full slow checkreading.
 
Speed ? (e.g. any , 1)
current setting  : ""
Your input please:
16
 
Note: Erasing DVDs is needed for rewritable DVD-RW media only
 
Automatically erase DVD before writing ? (yj1/n0)
current setting  : ""
Your input please:
y
 
Note: Files of size 2GB or larger are likely to cause trouble when
read from an ISO filesystem. A directory outside the backup area
may be used to buffer pieces of such files which then become
part of the backup. The directory has to be empty now or has
already to contain a file scdbackup_split_file_directory_mark.
Its filesystem has to be able to carry a full media content
(4.7GB). Take care not to use the same directory as with CD !
If you enter "-" then large files are excluded. If you enter
"-allow_large_files" then files just have to fit on the media.
Your local xorriso is suitable for "-xorriso_cut_out"
which needs no extra disk space.
 
Large file split directory ? ( "-" = exclude large files)
current setting  : ""
Your input please:
-xorriso_cut_out
 
Note: Compressed afio (system) backups may produce arbitrarily sparse
output which is quite demanding to the writer hardware.
Therefore compression usually is disabled without buffer.
On the other hand, the buffer is as large as the DVD (~ 4.7 GB)
and must be outside the backup area or explicitely excluded.
Take care not to use the same file as with CD !
Special name "-" disables buffering. "-afio_compress" does the
same but enables afio compression (works fine with my DVD+RW).
 
Buffer file name ? ( or "-" or "-afio_compress" )
current setting  : ""
Your input please:
-afio_compress
 
Note: if you enter 0 then the directory size is determined
by a very slow method which previews the compressed
afio sizes.
 
Estimated capacity for a compressed afio DVD ? (eg. 8000m)
current setting  : ""
Your input please:
8000m
 
Exclusion list of directories for system backup ?
current setting  : "/cdrom /cdrw /floppy /home /mnt /opt /proc /tmp"
Your input please:
Normal users just press Enter here. This input is only of interest for the superuser.
Please note that -if ever- a single line is expected here, despite your browser right now may show more than one.

Now the parameters get written to their files.
In future runs of CONFIGURE_DVD you may confirm your settings by empty input lines and only have to enter those values which you want to change. The content of those files may be temporarily overriden by shell variables.
See README Appendix CONFIGURATION for a definition of the files' content (if it isn't obvious by name and sequence :)

writing file scdbackup_use_genisoimage_value
done
...
scdbackup for DVD 0.9.2 : First stage of installation done.
--------------------------------------------------------
The commands of your scdbackup installation are now available in :
   /home/thomas/scdbackup/scdbackup-0.9.2/cmd
 
You may want to add this address to your PATH variable
or ask your superuser to run ./SUPERUSER_CONFIGURE in directory
/home/thomas/scdbackup/scdbackup-0.9.2/inst
--------------------------------------------------------
A first test :
$ $HOME/scdbackup/scdbackup-0.9.2/cmd/scdbackup_home -version
scdbackup 0.9.2 2009.12.13.072542
The user could now extend the variable PATH in order to enable the shell to find the commands of scdbackup :
$ PATH="$PATH":$HOME/scdbackup/scdbackup-0.9.2/cmd
This will only help with that single shell session. So the user now has to find a place in the shell's startup scripts where to make this enhancement permanently.
If there was already scdbackup-0.8 or older installed, then the superuser will have to remove the old public commands.

As an alternative for both (PATH and command removal), the superuser may install the commands in a public command directory where they can be found without such precautions. In case of an upgrade installation this has to be the same directory as before, of course.
So, i performStep 3 of the installation procedure.
$ su
Password:
# ./SUPERUSER_CONFIGURE
 
scdbackup command installation script 0.9.2
 
This script may install following commands owned by the superuser
in a public command directory of your choice :
scdbackup access: a+rx,u+w,go-w
scdbackup_home access: a+rx,u+w,go-w
scdbackup_afio access: a+rx,u+w,go-w
scdbackup_verify access: a+rx,u+w,go-w
scdbackup_blank access: a+rx,u+w,go-w
scdbackup_askme access: a+rx,u+w,go-w
scdbackup_clear access: a+rx,u+w,go-w
scdbackup_sys access: u+rwx,go+r,go-wx
For use with the DVD writer it installs accordant commands: sdvdbackup*
like: sdvdbackup , sdvdbackup_home , sdvdbackup_verify , ...
 
Public command directory ? (e.g. /usr/bin , "-" = do not install)
current setting: "/usr/local/bin"
/usr/local/bin
confirmed
... the installed commands and their permissions get shown ...
The superuser should have an own scdbackup installation. To get one
follow steps 1 and 2 of README "Installation".
But instead of step 1b (./FETCH_CONFIGURATION) do :
./SUPERUSER_FETCH_CONF /home/thomas/scdbackup/scdbackup-0.9.2
Do not forget step 2c : ./ADD_USER
 
Since i want to run some backups as superuser, i will follow this advice.
I read README paragraph "Superuser Planning" to decide what to put into the system backup and what therfore has to be excluded.
For the sake of security and because i cannot guarantee for all the software which is run by myself as a desktop user, i now will make a designated installation for the superuser. It is a bit more inconvenient than to simply perform ./ADD_USER now. But it will help to keep up the system security in the same quality as it was before scdbackup's installation.
 
# mkdir $HOME/scdbackup
# cd $HOME/scdbackup
# tar xzf ..../scdbackup-0.9.2.tar.gz
# cd scdbackup-0.9.2/inst
# chown root ../../scdbackup-0.9.2/*/* ../../scdbackup-0.9.2/* ../../scdbackup-0.9.2
# ./SUPERUSER_FETCH_CONF /home/thomas/scdbackup/scdbackup-0.9.2
Fetching files ...
scdbackup_use_genisoimage_value :
xorriso
...
done
# ./CONFIGURE_CD
No fear, you will mainly have to press the Enter key and confirm the offered settings.
Only with the questions for "Large file split directory" and "Buffer file name" you will have to invent new directory and file addresses, where you used some with the desktop user's installation. Do not use the same addresses as other users. This would be a security hole.
Make sure the superuser's disk buffer objects are properly protected by restrictive access permissions. Best is to have no such buffers.

This time you also will have to make an appropriate input to question "Exclusion list of directories for system backup ?". The README paragraph "Superuser Planning" gives the necessary information about what addresses belong into this list (everything that's volatile or owned by normal users).
...
# ./CONFIGURE_DVD
...
# ./ADD_USER
publishing directory address : /root/scdbackup/scdbackup-0.9.2 in /root/.scdbackup_dir
done
# exit
And again a test :
$ scdbackup_home -version
scdbackup 0.9.2 2009.12.13.072542
If this yields a result different from the first test then the user has to find the obtrusive old command's location by help of which scdbackup_home. The superuser then will have to take appropriate action.
I advise against running old versions of scdbackup for a prolonged time.

Since i do not use the cdrecord_prog which comes with scdbackup, there is no need to run ./SUPERUSER_CONFIGURE again. The public commands will stay valid until a user's ./CONFIGURE_* run with a future scdbackup version might complain about them being outdated.

Note that other users may now install scdbackup on their own - provided they got read access to scdbackup-0.9.2.tar.gz . Nevertheless that will not give them any more permissons than they would have had using the system installed DVD burn programs directly.


to top of page | to scdbackup homepage


Backup run with scdbackup_home


The installation of this very antique example was made in /home/test . Usage of scdbackup_home has not changed since then.

The user's actions (1 command, 5 Enter keys) are shown in bold typeface

$ scdbackup_home
The disk is rattling for about five minutes now.
My box was a K6/350 and below my HOME directory were about 3 GB.
From time to time one of these information lines appears:

... splitting .
... splitting ./p
... splitting ./p/sorten
########### 1 (9 items 638.8m bytes)
########### 2 (16 items 634.3m bytes)
########### 3 (10 items 638.6m bytes)
########### 4 (335 items 579.3m bytes)
... splitting ./.netscape
########### 5 (4 items 411.1m bytes)
########## part 1 of 5 (9 items 638.8m bytes)
insert CD and press Enter
The distribution plan for five CDs is made and the result script took over control now.
This is the time to insert a CD-R and press the Enter key.

The following messages come from mkisofs and cdrecord:
Total extents scheduled to be written = 326878
1.53% done, estimate finish Sat Nov 27 22:37:54 1999
...
99.43% done, estimate finish Sat Nov 27 22:28:10 1999
...
326878 extents written (638 Mb)
Cdrecord release 1.6 Copyright (C) 1995-1998 Jörg Schilling
...
/home/test/scdbackup-0.5/cdrecord_prog: fifo was 0 times empty and 20071 times full, min fill was 96%.
Fixating time: 62.121s
The first CD is done, gets ejected and scdbackup asks for the next one:
########## part 2 of 5 (16 items 634.3m bytes)
insert CD and press Enter
This is again the time to insert a CD-R and press the Enter key.
...
########## part 3 of 5 (10 items 638.6m bytes)
insert CD and press Enter
Press Enter
...
########## part 4 of 5 (335 items 579.3m bytes)
insert CD and press Enter
Press Enter
...
########## part 5 of 5 (4 items 411.1m bytes)
insert CD and press Enter
Press Enter
$
It is done. Five CDs. I watched two movies in TV meanwhile and changed CDs every half hour. (Speed factor 4)

I label my CD-RWs at first use with a unique id number. After each use i write a leaflet with this id number, the date , the format ("ISO" or "afio"), the content ("system", "music collection", whatever). Very important is the volume number and the total number of volumes ("1 of 5", "2 of 5" ...). With incremental backups i additionaly record level number and date of the previous level.
Leaflet and CD are stored together (most time). As soon as a CD-RW gets blanked i destroy the old leaflet to avoid any confusion.

to scdbackup homepage

to top of page


Resuming an interrupted backup run


After a backup run has been interrupted, it can be resumed without distributing the data over the CDs again. This guarantees that the new CDs match the old ones. Care should be taken that not too many new files have been created meanwhile. Otherwise particular CDs might overflow.
If you expect an interupted backup then you should decrease the size of the pieces in order to get more buffer space for subsequent size increases.
Use -max_size 600m to reserve 50 MB on each CD (with 650 MB capacity).

If you want to interrupt a lengthy backup session then kill the script when it asks for the next CD (e.g. by pressing CTRL+C rather than Enter). The interrupted backup can be resumed until you start a new backup.

Let us assume that the run was interrupted before or during the forth piece:
$ scdbackup_home -resume
...
########## part 4 of 5 (335 items 579.3m bytes)
insert CD and press Enter
Press Enter
...
########## part 5 of 5 (4 items 411.1m bytes)
insert CD and press Enter
Press Enter
...
$
It is done again. Two missing CDs have been created.

If a certain volume shall be done it can be addressed by an optional volume number like in :
$ scdbackup_home -resume 2

to scdbackup homepage

to top of page


Verifying a result CD


Some CD recorders and some CD media are not as reliable as they should be. It may be necessary to verify the readabilty and correctness of the backup CDs.
Be aware that this check does not ensure the completeness of your backup. Any faults of scdbackup or yourself will not be detected. Only the success of your CD recorder's efforts, the individual CD and your CD reading device will be tested.

So you should look at the total backup size wether it is plausible. After verifying the single CDs you should also use ASKME to locate some files and check wether they really are on their CD.

The most convenient way to verify a CD is to put it into the tray of /dev/cdrom and to execute :
$ scdbackup_verify -auto
scdbackup_verify 0.8.8 en
Loaded 994 checksum records from /home/thomas/scdbackup/scdbackup-0.8.8/logs/backup_log
/ 103.5m bytes
 
Found matching record :
1_1 A40904.234957 109447168 695bc386687f9594e2885f87e9983c68
 
Note: Checksum tag found by scanning confirms match.
 
Note: read 340464 surplus bytes
 
OK. Matching Record found . Passender Record gefunden .
 
104.4m bytes
So this CD is part 1 of 1, written 2004 September 4 23:49:57, with 109447168 bytes.
The note about the scanned checksum tag means that not only the matching list record was found and verified but also an identical checksum tag on the backup media. It would have been sufficient to verify the media if there had been no matching record in the list file.
The note about surplus bytes is typical for CD reading devices. To avoid problems, scdbackup appends a generous amount of padding data when writing. An amount of more than 400000 with a CD would be quite suspicious, though.

In case of failed verification there would be no "OK" but :
/ 103.5m bytes
 
-search_md5 : NO MATCHING RECORD FOUND. KEIN PASSENDER RECORD GEFUNDEN :
- - -
 
104.4m bytes
In such a case you should hurry to burn a new CD #1 by scdbackup -resume 1 . Do this as long as the data from the disk still fit on the CD.

If you want to verify at a device other than /dev/cdrom, give that address as first argument :
$ scdbackup_verify /dev/cdrecorder -auto

With a previously used rewritable DVD (and command sdvdbackup_verify ) one may get a lot of surplus data which are the remainder of previous media contents. This may be annoying. So one may use option -auto_end which stops reading after the first match. Check well what record has been found and better rerun with -auto if size or timestamp are not as expected.

You may also compare the CD against the most recent checksum record in the list. This will not complain about surplus bytes and eliminates the very small risk to incidentially match a faulty CD with an older record.
$ scdbackup_verify
 
Comparing with record :
1_1 A30110.222002 260765696 c30d13d53977cecb87017896d72616ae
 
OK. Record and data input match. Daten passen zum Record.
 
248.7m bytes

In case of failure there would be no "OK" but for example :
SIZE MISMATCH. FALSCHE GROESSE :
record=260796416 stdin=255420416
or
MD5 CHECKSUM MISMATCH. FALSCHE MD5 PRUEFSUMME :
record= c30d13d53977cecb87017896d72616ae
stdin = c7a17c2d8e7de65e82756527fe498c75

Note that normally you have to be the same user who burned the media in order to have access to the recorded checksums of that user. With option -auto and a backup written by scdbackup-0.8.4 or newer, you will not need those records.

If you want to verify at /dev/cdrecorder the first volume of the most recent three volume backup then execute
$ scdbackup_verify /dev/cdrecorder 1_3

More options of scdbackup_verify are explained by
$ scdbackup_verify -help
(or in file scdbackup_verify_help )


to scdbackup homepage

to top of page


Searching and restoring a particular file or directory


From time to time, i spoil some of my files. Linux is quite safe, i am not.
For example, i lost the file $HOME/gifs/milleniumbug.gif
But i have got my backup !

First i mount one of the CDs (it doesn't matter which one)
$ mount /cdrom
Since i have got the "exec" option in the /cdrom line of /etc/fstab, i can execute the ASKME script directly from CD.
$ /cdrom/ASKME gifs/milleniumbug.gif
5 gifs gifs/milleniumbug.gif
From this answer i learn that the whole directory "gifs" is on CD number 5.
Now i have to mount this CD and can restore my lost file.
$ umount /cdrom
I insert CD number 5 and execute
$ mount /cdrom
$ cp /cdrom/gifs/milleniumbug.gif $HOME/gifs
and the milleniumbug (75 K GIF movie) is back on my disk.

For some more info about ASKME see ASKME -help
For locating files within incremental backups you will either have to use scdbackup option -info_script -full at backup time or rely on the much more suitable command scdbackup_askme.


to scdbackup homepage

to top of page


Erasing backup media with confidential data


Backups often contain data which in no case shall be read by others. On a CD anybody can be superuser and read anything. Therefore it is desirable to make the stored data permanently unreadable before a media is reused or disposed.

Rewritable media with confidential data should be overwritten before they are refilled with new data and possibly handed to other people.
This is important especially with DVD because remainders of older backups can be read from the media without using any special means. With CD-RW this is not so easy but still not impossible.
The following command overwrites a DVD+RW with the amount of 4.7 billion bytes which is usually advertised on the DVD's label.
$ sdvdbackup_blank shred 4.7e9
sdvdbackup_blank 0.8.8 en
 
This program blanks a DVD-RW to make it ready for re-use.
After blanking, the data currently stored on the media will be unreadable.
Blanking DVD+RW makes only sense together with option "shred".
Note: DVD-RW media formatted in mode "Restricted Overwrite"
and DVD+RW media are rewriteable without blanking.
 
Beware: Blanking other media may produce any kind of damage.
 
insert media and press Enter
Press the Enter key.
evaluating media type ...
Detected media type : DVD+RW
 
Note: Blanking/formatting disabled because of media type DVD+RW
Note: Triggered by option "shred" the following will happen :
The media will get filled with 4.7e9 bytes of random
data in order to erase old content. You may verify these
data by sdvdbackup_verify.
The media will be overwriteable afterwards.
 
press Enter to continue
Press the Enter key again.
beginning to write random data
The usual messages of the DVD burn program follow.
After the writing is done, it is possible to check the result with command sdvdbackup_verify . This is a quality check for the media, too.

With CD-RW the capacity is advertised in binary MB (= 1024*1024) . So for the usual 700 MB media use :
$ scdbackup_blank shred 700m
scdbackup_blank 0.8.8 en
 
This program blanks a CD-RW to make it ready for re-use.
After blanking, the data currently stored on the media will be unreadable.
 
Beware: Blanking other media may produce any kind of damage.
 
insert media and press Enter
Press the Enter key.
evaluating media type ...
Detected media type : CD-RW
 
Note: Triggered by option "shred" the following will happen :
The media will get blanked by cdrecord option "blank=all".
The media will get filled with 700m bytes of random
data in order to erase old content. You may verify these
data by scdbackup_verify.
You will need another blanking run without "shred"
to make the media writable again.
press Enter to continue
Press the Enter key again.
beginning to blank media ...
...
beginning to write random data
You will see the usual messages of cdrecord blanking and writing the media.

After repeated treatment with these commands even first class equipment should have difficulties to read the original content.

With non rewritable media it is best to destroy them physically. Be aware that sophisticated precautions are made on the media for reconstruction of damaged data. Therefore i burn the recording layer and soften the media with a gas lighter, bend the disks and break them into pieces when cold again. (CAUTION : Hazard to burn or cut your fingers.)

to scdbackup homepage

to top of page


Incremental backups


If you got a substantial amount of data, it becomes annoying to backup the old data with each new run. To avoid this (within certain limitations) one may use an incremental backup scheme.
Be aware, though, that incremental backups increase complexity when it comes to restoring a whole backup area. So only decide to make incremental backups after you experience that full backups are unbearable. (See example below)

For an incremental backup a configuration needs to be created. We simply use the $HOME backup but one could also record an arbitrarily complicated set of data sources and options. After the configuration is created, only its name and the desired update level have to be given with the scdbackup command.
$ scdbackup_home \
-conf_dir $HOME/home_backup \
-level -create_configuration
This creates the directory $HOME/home_backup (and fails if it already exists), records the parameters of the backup (which are set within script scdbackup_home) and enables recording of the state of the backup area (i.e your $HOME with its sub directory trees).
Also it starts the initial base backup which is called -level 0. This backup is like any other run of scdbackup_home (see above) besides it may take quite some time to obtain CRC fingerprints from all the files in your $HOME tree. Also it eats some of your disk space by writing file $HOME/home_backup/level_0/content_file_list.gz
The difference comes with the update run a few days later:
$ scdbackup \
-conf_dir $HOME/home_backup \
-level 1
This command causes the backup configuration $HOME/home_backup to be read and updated on level 1. Level 1 backups all files that were changed or newly created since the last run of level 0. By comparing the current state with the one recorded from level 0, it also detects files which have vanished or got moved.

So from now on, you do base backups with -level 0. These base backups include all files without any time constraint. Updates of the base backups are done by -level 1 in regular intervals.
You may also do short term updates which refer to the last run of level 1 by running a -level 2 backup.
There is no upper limit of level numbers but you may only increase them by one with each run. That is, after a level 0 run was done, you cannot do a level 2 run but only level 1. The special level -next can be used to get the highest permittable level.

A backup scheme which from monday to saturday records only the daily changes and makes a full backup each sunday:
  Monday  :  -conf_dir $HOME/home_backup -level -next
  ...
  Saturday:  -conf_dir $HOME/home_backup -level -next
  Sunday  :  -conf_dir $HOME/home_backup -level 0
Now the media of monday to saturday may be recycled.

A scheme which makes quarterly (13 weeks) full backups, four-weekly and weekly updates (-conf_dir same as above). This scheme lets you retrieve old file versions for quite a long time.
  Week  1 :  -level 1
  Week  2 :  -level 2
  Week  3 :  -level 2
  Week  4 :  -level 2
  Week  5 :  -level 1
  Week  6 :  -level 2
  Week  7 :  -level 2
  Week  8 :  -level 2
  Week  9 :  -level 1
  Week 10 :  -level 2
  Week 11 :  -level 2
  Week 12 :  -level 2
  Week 13 :  -level 0
For week 14 hop to week 1.
It is advised to keep levels 0 and 1 for a longer time.

To combine both schemes shown, simply replace the sunday run of scheme 1 by the weekly run of scheme 2.

If you want to change the configuration parameters, you may either run the altered command with special level -update_configuration:
$ scdbackup_home \
-not trashdir newsdir \
-conf_dir $HOME/home_backup \
-update_configuration
or you may directly edit the parameter list file
$HOME/home_backup/specific_arguments
In this file, each single argument is recorded in a single line. The list begins with arguments automatically set by scdbackup and ends with arguments given by you (if any).
Be aware what the options mean before you delete or change them. Also perform a new -level 0 backup afterwards.

Incremental backups include the ASKME scripts of lower levels. If you use the ASKME script of level 2 it will try to execute the ASKMEs of level 1 and level 0 if it cannot find a file by itself. The option -levelall causes all levels to be searched in any case.
Since a large multilayer backup may produce very large and slow ASKME scripts there is a specialized binary program which can extract the information from the scripts and can perform searches much faster. (See scdbackup_askme_help )

These features are not restricted to scdbackup_home but may also be used with scdbackup and scdbackup_afio. One may have an abritrary number of configurations in distinct directories.

Now for some restrictions:

to scdbackup homepage

to top of page


Locating files in a multilevel backup and restoring them


Usually the ASKME scripts of an incremental backup can just tell their date, level and volume. Although they can at backup time be enabled for searching by option -info_script -full they are of very limited use when it comes to locating files in the heap of backup levels and volumes.
Much faster than ASKME is the command scdbackup_askme which uses a binary program to load and explore the content model of an incremental backup configuration. (It can also load ASKME scripts but in that case its search capabilities are nearly as limited as the ones of the script.)

The content model can be read from the -conf_dir directory of backup or it can be read from directory /added_by_scdbackup of the last volumes of the available backup levels. Beginning with scdbackup version 0.8.7, non-incremental ISO-9660 backups, too, have such a content list on the last volume.
Queries can be submitted either as program arguments or in dialog mode which usually offers (via libreadline) an editable input line with history. Since the load process lasts some time the dialog mode is to prefer if one does not have a clearly predefined set of queries.
$ scdbackup_askme $HOME/home_backup -dialog -page:20
scdbackup_askme 0.8.8 en
reading /home/thomas/home_backup/level_0/content_file_list.gz
        484262 lines read
reading /home/thomas/home_backup/level_0/complete_dir_list.gz
          2136 lines read
reading /home/thomas/home_backup/level_1/vanished_file_list.gz
         12224 lines read
reading /home/thomas/home_backup/level_1/content_file_list.gz
         61481 lines read
reading /home/thomas/home_backup/level_1/complete_dir_list.gz
          2254 lines read
reading /home/thomas/home_backup/level_2/vanished_file_list.gz
         10461 lines read
reading /home/thomas/home_backup/level_2/content_file_list.gz
         18790 lines read
reading /home/thomas/home_backup/level_2/complete_dir_list.gz
          2366 lines read
sorting directories of file tree model
        sorting done
sorting directories of file target model
        sorting done
The program starts up in mode -search_sh. Search patterns will work similar to the parser of the shell: one may use wildcards *,?,[] which cannot match character '/', nevertheless. Search patterns which begin with '/' describe absolute addresses. Other patterns get prepended the working directory, which was current at the time of -level -create_configuration.
Without further precautions, search is done on the source addresses.
=============================
enter option or search text :
-pwd
==============================================================
current working directory:
/home/thomas/
=============================
enter option or search text :
gifs/milleniumbug.gif
==============================================================
+++ on backup level 0
7 /gifs/milleniumbug.gif = /home/thomas/gifs/milleniumbug.gif
============================= 1 matches in 0.003 seconds
=============================
enter option or search text :
gifs
==============================================================
+++ on backup level 2
1 /gifs/ = /home/thomas/gifs
+++ on backup level 1
3 /gifs/ = /home/thomas/gifs
+++ on backup level 0
7 /gifs/ = /home/thomas/gifs
============================= 3 matches in 0.001 seconds
=============================
enter option or search text :
gifs/*
==============================================================
+++ on backup level 2
1 /gifs/best_before.gif = /home/thomas/gifs/best_before.gif
... further matches completing a 20 line page ...
 
.. [Press Enter to continue. @,Enter avoids further stops. @@@ aborts] ..
@
...
+++ on backup level 1
...
+++ on backup level 0
...
============================= 433 matches in 0.041 seconds
Let us assume we got no idea where our millenium thingee might have been stored. We only have a dim memory about its name.
Mode -search_shname will search on the single filenames only and ignore the preceeding paths.
=============================
enter option or search text :
-search_shname
==============================================================
=============================
enter option or search text :
*millenium*
+++ on backup level 2
3 /gifs/millenium.gif = /home/thomas/gifs/millenium.gif
+++ on backup level 0
7 /gifs/milleniumbug.gif = /home/thomas/gifs/milleniumbug.gif
10 /misc/milleniumparty/ = /home/thomas/misc/milleniumparty
11 /photo/happy_millenium.jpg = /home/thomas/photo/happy_millenium.jpg
============================= 3 matches in 2.324 seconds
(Run times are of a 32 bit AMD 2600+ working on a tree of 540,000 files.)
A description of the search rules and a complete list of options are provided by
=============================
enter option or search text :
-help
==============================================================
usage: scdbackup_askme infosource [options|searchtexts]
... further helptext...
It is possible to use scdbackup_askme for restoring data from ISO-9660 backups.
This offers the advantage that only those files are restored from older levels which are still valid. It is also more convenient and less error prone than restoring data by help of shell commands.
With long lasting restore operations, the user guidance and the ability to resume interrupted restore runs come in handy.

Initially the ability to restore is disabled in order to avoid inadverted mishaps. By option -restore_mode it can be enabled and restore rules can be set. It is also necessary to tell the mount point of the backup media.
=============================
enter option or search text :
-media_mode:load_eject:mount_umount
=============================
enter option or search text :
-restore_mode:overwrite_off:on_error_ignore
=============================
enter option or search text :
-restore_from:/cdrom
The following two options replace in the original addresses the start text /home/thomas by /home/test/thomas. After all it is only a test and by overwrite_off existing files are protected against overwriting.
Directory /home/test/thomas does already exist and and is owned by the user who runs this session. (Have your superuser create such a playground and chown it to you, or use as -restore_to: an empty directory within your own realm on the disk.)
=============================
enter option or search text :
-restore_skip:/home/thomas
=============================
enter option or search text :
-restore_to:/home/test/thomas
Now extract /home/thomas/gifs from the levels. Access permissions and adjustable timestamps will be restored, too.
=============================
enter option or search text :
-restore:/home/thomas/gifs
==============================================================
############################################################################
################## Please insert:  LEVEL   0   VOLUME   7  #################
############################################################################
.................. [ Press enter when done. @@@ = Abort ] ..................
Enter-key ... files get copied ...
############################################################################
################# Please insert:  LEVEL   1   VOLUME   3  ##################
############################################################################
.................. [ Press enter when done. @@@ = Abort ] ..................
Enter-key ... files get copied ...
############################################################################
################# Please insert:  LEVEL   2   VOLUME   1  ##################
############################################################################
.................. [ Press enter when done. @@@ = Abort ] ..................
Enter-key ... files get copied ...
Restored:                          732 file objects,  4152k data bytes
===============================      3 matches in    57.843 seconds
The quite long run time is due to three mount operations which with DVD may well last 10 to 15 seconds each.
Because -page:20 is still active, a prompt appears after a few files are copied and reported. It is ok to enter @ in order to disable this paging temporarily. One may also disable it permanently by -page:0.

to scdbackup homepage

to top of page


Full restore of a multilevel backup


There are moments when a backup area needs to be restored from scratch. Possibly on a completely new computer system.
In complicated disk situations you will have to prepare the mount tree below $HOME in advance. It is up to you to provide the disk space necessary to take the backup's content.

If it is possible to use scdbackup_askme then the restore can be done quite conveniently. If not, then one has to follow the somewhat more cumbersome path of restoring by hand using shell commands only.

The previous paragraph demonstrated how to load a backup configuration from a healthy disk and use it for search and restore. For this example it is assumed that this configuration directory is among the victims of the data accident. Nothing is left but the set of backup media.
The operating system and scdbackup are assumed operational, nevertheless.
$ scdbackup_askme /cdrom/added_by_scdbackup \
       -dialog \
       -logfile:I:/tmp/scdbackup_restore_log \
       -media_mode:load_eject:mount_umount \
       -restore_mode:overwrite_off:on_error_skip \
       -restore_from:/cdrom
The backup of /home/thomas consists of Level 0: 14 DVDs, Level 1: 3 DVDs und Level 2: 1 DVD .
At first the model of the backup area has to be read from the backup media. The necessary files are on the last volume of each level.
The program prompts:
#####################################################################
############# Please insert:  LEVEL   0   last VOLUME  ##############
#####################################################################
.......... [ Press enter when done. @@@ = All levels done ] .........
Insert DVD , Enter-key.
reading /cdrom/added_by_scdbackup/content_file_list.gz
        484262 lines read
reading /cdrom/added_by_scdbackup/complete_dir_list.gz
          2136 lines read
#####################################################################
############# Please insert:  LEVEL   1   last VOLUME  ##############
#####################################################################
.......... [ Press enter when done. @@@ = All levels done ] .........
I do this with the last DVDs of level 1 and 2 until finally i get asked for the non-existing level 3.
#####################################################################
############# Please insert:  LEVEL   3   last VOLUME  ##############
#####################################################################
.......... [ Press enter when done. @@@ = All levels done ] .........
@@@
... [@@@ = request to end registered. Really done reading ? (y/n) ] ..
j
--- ( request to end confirmed )
sorting directories of file tree model
        sorting done
sorting directories of file target model
        sorting done
Now the model is read in and can be exploited. My restore wish is a very general one:
=============================
enter option or search text :
-restore:/home/thomas
==============================================================
#####################################################################
############### Please insert:  LEVEL   0   VOLUME 1  ###############
#####################################################################
.............. [ Press enter when done. @@@ = Abort ] ...............
18 Enter-keys and six and a half hours later:
Restored:                       560464 file objects,  65.5g data bytes
===============================     18 matches in 20661.667 seconds
The plain restore time is 5.7 hours. About 45 minutes in total the program was waiting for me to put in the next volume.
After everything went well one may delete log file /tmp/scdbackup_restore_log.

scdbackup_askme will neither restore files below its own installation directory nor files below the eventually redirected directories conf, tmp, or logs. An attempt will cause messages but is not considered to be an error.

Resuming an interrupted restore run


With runs that last several hours one has to expect interruptions.
While restoring files the program is tracking the current search position within the scdbackup tmp directory. At the start of each restore run the complete setup status of the program is recorded.
To resume a restore run after the program ended more or less neatly it gets restarted with the pseudo information source "-resume".
$ scdbackup_askme -resume
The program will start the -restore run and eventually prompt for media.
If it is not desired to resume at the exact position where the last restore run stopped, one may add the level number and also a volume number:
$ scdbackup_askme -resume:1
$ scdbackup_askme -resume:1:4
The usual abort handling strives for leaving no incompletely restored file objects and to eventually reinstate the properties of existing directories in the path of the aborted file object.
So -resume usually can start restoring seamlessly.

If the previous restore run was interrupted very roughly or if abort handling was not able (or allowed) to clean up an incomplete restore, then it is possible that the affected file object on disk is in a questionable condition. It will then be recorded in the status files as "touched but not completed" unless the operating system failed to perform those status writes (e.g. because of a power failure).
If such an indication of an incomplete file restore is found then the program warns:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! WARNING: The last run left a file object in QUESTIONABLE state  !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
The address of the questionable file object is:
   /home/test/thomas/gifs/hastenichgesehn.gif
Some hints about potential problems and their potential solutions follow. If a dialog session is resumed then a prompt appears:
If everything is fixed, enter text "go on"
Fixing might mean to remove the file or to rectify directory permissions. One has to examine the particular situation.
go on
Restoring usually resumes with the file that was reported as questionable. Nevertheless one should check this file after the restore is done and eventually restore it as single file once again.

When resuming a run which had been performed in non-dialog mode, a questionable file causes abort in any case. The user gets gets told to resolve any problem and then to empty the file which caused the warning warning by this shell command: echo > $(scdbackup -where tmp)/askme_resume_state_"$UID"_dirty




Restoring by hand using shell commands only


This example describes how to restore a big $HOME tree by using a bash shell with standard commands and programs.


to top of page


CD mount options


The /cdrom line in my /etc/fstab looks like this
/dev/cdrom /cdrom iso9660 ro,user,noauto,unhide,exec,nosuid,nodev
Note that there are program interpreters which on their own account acknowledge the dangerous setuid-permissions and then act accordingly and irresponsibly. E.g. program suidperl. If you allow mount option exec you have to make sure that those interpreters are deactivated.


to top of page | to scdbackup homepage