How to use sed command to modify a file’s entry line by line in shell script?

by

This is my file1:

#$BQ
#{ VOL       @home    }
#database    daba
#relation    tcdeatid
#copy           1
#{ version        0 }
#opendb
#clear
#.lruno := 72
#.infno := 1
#.tid.noel := 101
#.tid.info := 64
#.tid.setnr := 1225              <--- (1225 number will be changed)
#.typeidm := 1
#.sourcetable := 2
#writedb     
#clear
#.lruno := 72
#.infno := 205
#.tid.noel := 101
#.tid.info := 76
#.tid.setnr := 5625              <--- (5625 number will be changed)
#.typeidm := 1
#.sourcetable := 2
#writedb
#$EOJ

In file2, I have the correct entries: 

#.tid.info := 3345              <--- (I want to put this number in file1 in place of 64)
#.tid.setnr := 1254              <--- (I want to put this number in file1 in place of 1225)
#.tid.info := 5567              <--- (I want to put this number in file1 in place of 76)
#.tid.setnr := 9056              <--- (I want to put this number in file1 in place of 5625)

I want to make changes in file1, directed by data from file2.

I have tried
sed "s/tid.setnr := 1225/tid.setnr := 1254/g" file1 > modified_file1

But instead of doing it manually, I want my script to read the new tid.info and tid.setnr values from file2 and modify the respective lines in file1. 
The first line in file2 should replace the entire first line in file1
that contains tid.info, the second line in file2 should replace
the second line in file1 that contains tid.setnr, and so on.

Contents hide
Answers:
Method 1
Method 2
Method 3
Method 4

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

With GNU sed’s R command:

sed -e 's/^#.tid.setnr :=.*//;tA;b;:A;R file2' -e 'd' file1

Method 2

One way to do this with sed is to generate a sed script:

tmpfile=/tmp/Nainita.$$
exec 3< file2
grep -n "tid.setnr" file1 | while IFS=: read n junk
do
        IFS= read -r line <&3  ||  break
        printf "%sc%sn%sn" "$n" '' "$line"
done > "$tmpfile"
exec 3<&-
sed -f "$tmpfile" file1 > modified_file1
rm -f "$tmpfile"
  • exec 3< file2 opens file2 for reading on file descriptor 3.
  • The grep -n finds the lines in file1 that contain tid.setnr,
    by line number.  This gets piped into a while loop.
  • while IFS=: read n junk
    • while … read … means read a line at a time, repeatedly,
      as long as there is information to read
      (i.e., stopping when it reaches the end of the data).
    • IFS=: read n junk means read everything up to the first :
      (which will be the line number from grep -n) into n,
      and the rest of the line (the old tid.setnr data line) into junk,
      which we ignore, because we don’t care about it.
  • IFS= read -r line <&3 reads a line from file descriptor 3 (file2),
    doing everything that we know how to do to tell the shell not to mangle it,
    and put it into a variable called line.
  • … || break says, if the above read fails
    (presumably due to end of file), break out of the loop.
  • The printf writes a sed change command,
    addressed to line number n, saying that that line should be replaced
    with the content on the line variable.
  • done > "$tmpfile" marks the end of the while loop
    and specifies that the standard output of the entire loop is $tmpfile. 
    This will end up looking like this: 
    13c
#.tid.setnr := 1254
22c
#.tid.setnr := 9056
    • Note that the while loop terminates
      as soon as it gets an EOF from either input. 
      Therefore, if file1 has more tid.setnr lines than file2,
      the extra ones will go unchanged
      (i.e., c commands for them will not be generated). 
      Similarly, if file2 has more tid.setnr lines than file1,
      the extra ones will be ignored. 
      Unfortunately, this discrepancy will not be reported,
      although adding that capability would not be very hard.
  • exec 3<&- closes file descriptor 3.
  • sed -f "$tmpfile" file1 > modified_file1 runs sed on file1,
    reading commands from $tmpfile and writing output to modified_file1.

This should do what you want.  Obviously, change the filenames if you want. 
You should run this “as is” once, and then look at the modified_file1 file
(or simply change the sed command not to redirect its output,
so it writes to the screen) and verify that the output is what you want. 
Then you can change the sed command to sed -i to edit file1 in place
(if that’s what you want to do).

If the sed command gives an error like “Too many line numbers”,
try splitting the script file ($tmpfile) into smaller files. 
I suggest starting with a size under 100 commands;
e.g., 90 commands, which is 180 lines, since each command is two lines. 
You can do this manually, with a text editor1,
but there’s a tool written specifically for this job. 
It is, intuitively, called split.  The command

split --lines=180 "$tmpfile"

will split the script into files in the current directory
called xaa, xab, xac, ….  The first n−1 will be 180 lines long;
the last one will be whatever it needs to be (≤ 180) to make up the total. 
For example, if you have 500 instances of tid.setnr,
then your script will be 1000 lines long, and you will get six x files —
xaa, xab, xac, xad, and xae will each have 180 lines,
and xaf will have 100.  Now do

sed -f xaa file1 > modified_file1aa

If you still get “Too many line numbers”,
go back and try it again with a smaller number of --lines. 
If it doesn’t report an error, look at modified_file1aa and see
whether it looks like the first 90 tid.setnr lines have been changed. 
If it looks OK, then do

sed -f xab modified_file1aa > modified_file1ab
sed -f xac modified_file1ab > modified_file1ac
sed -f xad modified_file1ac > modified_file1ad
sed -f xae modified_file1ad > modified_file1ae
sed -f xaf modified_file1ae > modified_file1af

modified_file1af is now your final modified_file1.

If you want to play experiment, you can

  • Try larger numbers of --lines until you find out what the maximum is.
  • Try 
    sed -f xaa -f xab -f xac -f xad -f xae -f xaf file1 > modified_file1_test

    but that probably won’t work.

  • If you do the above experiments, I encourage you to tell us the results.

If you only need to do this one time, you’re done. 
But, if you need to do this repeatedly,
change the last two lines of the code block at the top of this answer to

split --lines=180 "$tmpfile"            <--- (Using a number that works for you, of course)
cp file1 modified_file1
for f in x*
do
        sed -i -f "$f" modified_file1
done
rm -f "$tmpfile" x*

As I mentioned earlier,
the -i option tells sed to edit the specified file in place
(i.e., write the changes back out to the input file).  Or, even simpler,

split --lines=180 "$tmpfile"
for f in x*
do
        sed -i -f "$f" file1
done
rm -f "$tmpfile" x*

if you don’t need to keep your original file1 intact.

Note that the usage synopsis for split is

split [OPTION]… [INPUT [PREFIX]]

and its default behavior is to create files called PREFIXaa, PREFIXab, PREFIXac, etc. 
In other words, the default PREFIX is x. 
If you might have other files whose names begin with x,
you should define prefix to be something that will be unique
(e.g., prefix=Nainita.$$. or prefix=/tmp/Nainita.$$.)
and then change the above to

split --lines=180 "$tmpfile" "$prefix"
for f in "$prefix"*
do
        sed -i -f "$f" file1
done
rm -f "$tmpfile" "$prefix"*

______
1 If you’re a glutton for punishment,
you could do it with sed — but why play with fire?

Method 3

{   sed '/^#.tid.setnr/!d;=;g;G' |
    paste  -d'c\n' - - - ./addfile
}   <./mainfile | sed -f - ./mainfile

It just prepends the relevant line numbers from ./mainfile to each line in ./addfile as separated by the command string c and a <newline> before passing the results onto another sed as a script for editing ./mainfile. Given your example data, the script generated looks like:

13c
#.tid.setnr := 1254              <--- (I want to put this number in file1 in place of 1225)
22c
#.tid.setnr := 9056              <--- (I want to put this number in file1 in place of 5625)

…which instructs sed to change lines 13 and 22 of its output to match the appended lines for each command.

Method 4

I am found of perl. This script is a crude one that seems to solve the problem:

#!/usr/bin/perl
#
use strict;

# subroutine to load a test file into an array
sub load{
   my $file = shift;
   open my $in, "<", $file or die "unable to open $file : $!";
   my @data = <$in>;
   chomp @data;
   foreach (@data) { s/cM//g;}
   return @data;
}


my $file1 = shift || die "usage: $0 [file1] [file2]n";
my @file1_data = &load($file1);
my $file2 = shift || die "usage: $0 [file1] [file2]n";
my @file2_data = &load($file2);

my $i = 0;
foreach( @file1_data )
{
    if( $i < @file2_data ) 
    {
        my @s = split / /, $file2_data[$i];
        if( @s )
        {
            if( /^$s[0]/ )
            {
            $_ = $file2_data[$i++];
            }
        }
    }
    print "$_n";


All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0

0 0 votes
Article Rating
Subscribe
Notify of
guest

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x