Overwriting old/corrupted files in a backup

Overwriting old/corrupted files in a backup

[Linux for blind general discussion]

Okay, so using the cp command with the -Rn switch is convenient for
say copying my Music folder from my home directory to an external hard
drive or the SD card for my portable Media player when the destination
already has an older copy without needing to copy everything already
present in the old copy, manually determining what's been added and
copying manually, or dealing with a bunch of prompts. It also allows
an aborted copy to more or less be resumed from where it left off.

This method is simple enough to not require scripting or complex
command syntax, but it does have a few downsides:
1. It won't overwrite corrupted files left by an interrupted copy, and
such files are too rare for manual searching.
2. Files that have been altered don't get copied. unless they've
changed filename. Not a big issue for copying my Music folder since
those files are seldom altered, but copying say, my writing folder can
lead to the backup media containing only older drafts of some
documents.
3. It tells me nothing of files from an older copy that have been
deletd/renamed since the last copy. jdupes can find old files on the
destination media if the new file is just a rename, but it can't help
with files that have been altered as well as renamed.

I suppose what I'm looking for is a command line utility or script
that executes the following pseudo code:
Given directories source and destination:
for every file found in both source and destination:
if file.source != file.destination
prompt user whether to overwrite one version of the file with the
other or to add the files to a list for later examination.
For every file only in source:
search for match in destination.
If match found prompt user towhich to rename to match.
If not match found, copy to destination.
For every file only in destination:
search for match in source.
If match is found, prompt user to which file should be renamed.
If no match is found, prompt user whether to copy of delete.

If anyone knows of command line utilities to help with this task, it
would be greatly appreciated.

Re: Overwriting old/corrupted files in a backup

[Linux for blind general discussion]

[-- Attachment #1: Type: text/plain, Size: 3196 bytes --]

Hi,

I didn't know about 'cp -Rn'. I made friends with rsync
years ago, and use it for local copies as well as to/from
remote hosts over the network. 

It has flags for most of the behaviors you want.
To not clobber existing files during the copy you have:
--backup, --backup-dir and --suffix options.

To test before actually copying: --dry-run

I generally use -a and -v options (retain attributes, and
give verbose output).

For comparing to directory trees, there must be something
better available, however, I can offer a short perl script
(attached) takes two directory trees and lists differences
in size, modify date, owner, group, permission, etc. between
files; files only in source, or only in target. Not great
coding style (you shouldn't use $a and $b as variable names
because they have special meaning during sort).  

Hope this helps,

Joel

someone wrote:
> Okay, so using the cp command with the -Rn switch is convenient for
> say copying my Music folder from my home directory to an external hard
> drive or the SD card for my portable Media player when the destination
> already has an older copy without needing to copy everything already
> present in the old copy, manually determining what's been added and
> copying manually, or dealing with a bunch of prompts. It also allows
> an aborted copy to more or less be resumed from where it left off.
> 
> This method is simple enough to not require scripting or complex
> command syntax, but it does have a few downsides:
> 1. It won't overwrite corrupted files left by an interrupted copy, and
> such files are too rare for manual searching.
> 2. Files that have been altered don't get copied. unless they've
> changed filename. Not a big issue for copying my Music folder since
> those files are seldom altered, but copying say, my writing folder can
> lead to the backup media containing only older drafts of some
> documents.
> 3. It tells me nothing of files from an older copy that have been
> deletd/renamed since the last copy. jdupes can find old files on the
> destination media if the new file is just a rename, but it can't help
> with files that have been altered as well as renamed.
> 
> I suppose what I'm looking for is a command line utility or script
> that executes the following pseudo code:
> Given directories source and destination:
> for every file found in both source and destination:
> if file.source != file.destination
> prompt user whether to overwrite one version of the file with the
> other or to add the files to a list for later examination.
> For every file only in source:
> search for match in destination.
> If match found prompt user towhich to rename to match.
> If not match found, copy to destination.
> For every file only in destination:
> search for match in source.
> If match is found, prompt user to which file should be renamed.
> If no match is found, prompt user whether to copy of delete.
> 
> If anyone knows of command line utilities to help with this task, it
> would be greatly appreciated.
> 
> _______________________________________________
> Blinux-list mailing list
> Blinux-list@redhat.com
> https://www.redhat.com/mailman/listinfo/blinux-list

-- 
Joel Roth
  


[-- Attachment #2: treecmp --]
[-- Type: text/plain, Size: 1255 bytes --]

#!/usr/bin/env perl 
my ($a, $b) = @ARGV;
my (%a, %b); # hashes to hold stat data

my %stat_field =    ( 0 => 'dev',
					  1 => 'ino',
					  2 => 'mode',
					  3 => 'nlink',
					  4 => 'uid',
					  5 => 'gid',
					  6 => 'rdev',
					  7 => 'size',
					  8 => 'atime',
					  9 => 'mtime',
					  10 => 'ctime',
					  11 => 'blksize',
					  12 => 'blks');
my @target_fields = (2, 4, 5, 7, 9);
my %compared = ();

use File::Find;
find (\&want_a, $a );
sub want_a { $a{$File::Find::name} = [ stat $File::Find::name ]; }

find (\&want_b, $b);
sub want_b { $b{$File::Find::name} = [ stat $File::Find::name ]; }

#foreach my $k (keys %a) {print join (" ", $k, @{$a{$k}}, "\n" ); }  exit;
#foreach my $k (keys %b) {print join (" ", $k, @{$b{$k}}, "\n" ); }  exit;
# print join "\n", keys %b; exit;
foreach my $k (keys %a) {
    # print $k, "\n"; next;
    my $l = $k;
	$l =~ s/^$a/$b/;
	# print "$k $l\n";
    
    $b{$l} or print ("$k not in target\n"), next;
	my @diffs = ();
    foreach my $f (@target_fields) {  
    print(join " ", "$k differs in $stat_field{$f}:", $a{$k}->[$f], $b{$l}->[$f] ,"\n")
    	if ! $b{$l}->[$f] == $a{$k}->[$f];
	}
	$compared{$l}++;
}
foreach my $l (keys %b) {
    next if $compared{$l};
	print "$l not in source\n";
}

Re: Overwriting old/corrupted files in a backup

[Linux for blind general discussion]

Use rsync instead. You'll never look back.

Janina

Linux for blind general discussion writes:
> Okay, so using the cp command with the -Rn switch is convenient for
> say copying my Music folder from my home directory to an external hard
> drive or the SD card for my portable Media player when the destination
> already has an older copy without needing to copy everything already
> present in the old copy, manually determining what's been added and
> copying manually, or dealing with a bunch of prompts. It also allows
> an aborted copy to more or less be resumed from where it left off.
> 
> This method is simple enough to not require scripting or complex
> command syntax, but it does have a few downsides:
> 1. It won't overwrite corrupted files left by an interrupted copy, and
> such files are too rare for manual searching.
> 2. Files that have been altered don't get copied. unless they've
> changed filename. Not a big issue for copying my Music folder since
> those files are seldom altered, but copying say, my writing folder can
> lead to the backup media containing only older drafts of some
> documents.
> 3. It tells me nothing of files from an older copy that have been
> deletd/renamed since the last copy. jdupes can find old files on the
> destination media if the new file is just a rename, but it can't help
> with files that have been altered as well as renamed.
> 
> I suppose what I'm looking for is a command line utility or script
> that executes the following pseudo code:
> Given directories source and destination:
> for every file found in both source and destination:
> if file.source != file.destination
> prompt user whether to overwrite one version of the file with the
> other or to add the files to a list for later examination.
> For every file only in source:
> search for match in destination.
> If match found prompt user towhich to rename to match.
> If not match found, copy to destination.
> For every file only in destination:
> search for match in source.
> If match is found, prompt user to which file should be renamed.
> If no match is found, prompt user whether to copy of delete.
> 
> If anyone knows of command line utilities to help with this task, it
> would be greatly appreciated.
> 
> _______________________________________________
> Blinux-list mailing list
> Blinux-list@redhat.com
> https://www.redhat.com/mailman/listinfo/blinux-list

-- 

Janina Sajka,	Phone:	+1.443.300.2200
			sip:janina@asterisk.rednote.net
		Email:	janina@rednote.net

Linux Foundation Fellow
Executive Chair, Accessibility Workgroup:	http://a11y.org

The World Wide Web Consortium (W3C), Web Accessibility Initiative (WAI)
Chair, Accessible Platform Architectures	http://www.w3.org/wai/apa

Re: Overwriting old/corrupted files in a backup

[Linux for blind general discussion]

I agree. rsync is fast and extremely powerful.


If in the future you need something really sophisticated, like large 
companies use, try storeBackup.


Fernando



On 08/16/2017 09:57 AM, Linux for blind general discussion wrote:
> Use rsync instead. You'll never look back.
>
> Janina
>
> Linux for blind general discussion writes:
>> Okay, so using the cp command with the -Rn switch is convenient for
>> say copying my Music folder from my home directory to an external hard
>> drive or the SD card for my portable Media player when the destination
>> already has an older copy without needing to copy everything already
>> present in the old copy, manually determining what's been added and
>> copying manually, or dealing with a bunch of prompts. It also allows
>> an aborted copy to more or less be resumed from where it left off.
>>
>> This method is simple enough to not require scripting or complex
>> command syntax, but it does have a few downsides:
>> 1. It won't overwrite corrupted files left by an interrupted copy, and
>> such files are too rare for manual searching.
>> 2. Files that have been altered don't get copied. unless they've
>> changed filename. Not a big issue for copying my Music folder since
>> those files are seldom altered, but copying say, my writing folder can
>> lead to the backup media containing only older drafts of some
>> documents.
>> 3. It tells me nothing of files from an older copy that have been
>> deletd/renamed since the last copy. jdupes can find old files on the
>> destination media if the new file is just a rename, but it can't help
>> with files that have been altered as well as renamed.
>>
>> I suppose what I'm looking for is a command line utility or script
>> that executes the following pseudo code:
>> Given directories source and destination:
>> for every file found in both source and destination:
>> if file.source != file.destination
>> prompt user whether to overwrite one version of the file with the
>> other or to add the files to a list for later examination.
>> For every file only in source:
>> search for match in destination.
>> If match found prompt user towhich to rename to match.
>> If not match found, copy to destination.
>> For every file only in destination:
>> search for match in source.
>> If match is found, prompt user to which file should be renamed.
>> If no match is found, prompt user whether to copy of delete.
>>
>> If anyone knows of command line utilities to help with this task, it
>> would be greatly appreciated.
>>
>> _______________________________________________
>> Blinux-list mailing list
>> Blinux-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/blinux-list