I have an application which will produce a large amount of data which I do not wish to store onto the disk. The application mostly outputs data which I do not wish to use, but a set of useful information that must be split into separate files. For example, given the following output:
JUNK JUNK JUNK JUNK A 1 JUNK B 5 C 1 JUNK
I could run the application three times like so:
./app | grep A > A.out ./app | grep B > B.out ./app | grep C > C.out
This would get me what I want, but it would take too long. I also don’t want to dump all the outputs to a single file and parse through that.
Is there any way to combine the three operations shown above in such a way that I only need to run the application once and still get three separate output files?
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
If you have tee
./app | tee >(grep A > A.out) >(grep B > B.out) >(grep C > C.out) > /dev/null
(from here)
Method 2
You can use awk
./app | awk '/A/{ print > "A.out"}; /B/{ print > "B.out"}; /C/{ print > "C.out"}'
Method 3
You could also use your shell’s pattern matching abilities:
./app | while read line; do
[[ "$line" =~ A ]] && echo $line >> A.out;
[[ "$line" =~ B ]] && echo $line >> B.out;
[[ "$line" =~ C ]] && echo $line >> C.out;
done
Or even:
./app | while read line; do for foo in A B C; do
[[ "$line" =~ "$foo" ]] && echo $line >> "$foo".out;
done; done
A safer way that can deal with backslashes and lines starting with -:
./app | while IFS= read -r line; do for foo in A B C; do
[[ "$line" =~ "$foo" ]] && printf -- "$linen" >> "$foo".out;
done; done
As @StephaneChazelas points out in the comments, this is not very efficient. The best solution is probably @AurélienOoms’.
Method 4
If you have multiple cores and you want the processes to be in parallel, you can do:
parallel -j 3 -- './app | grep A > A.out' './app | grep B > B.out' './app | grep C > C.out'
This will spawn three processes in parallel cores. If you want there to be some output to the console, or a master file, it has the advantage of keeping the output in some order, rather that mixing it.
The gnu utility parallel from Ole Tange can be obtained from most repos under the name parallel or moreutils. Source can be obtained from Savannah.gnu.org. Also an introductory instructional video is here.
Addendum
Using the more recent version of parallel (not necessarily the version in your distribution repo), you can use the more elegant construct:
./app | parallel -j3 -k --pipe 'grep {1} >> {1}.log' ::: 'A' 'B' 'C'
Which achieves the result of running one ./app and 3 parallel grep processes in separate cores or threads (as determined by parallel itself, also consider the -j3 to be optional, but it is supplied in this example for instructive purposes).
The newer version of parallel can be obtained by doing:
wget http://ftpmirror.gnu.org/parallel/parallel-20131022.tar.bz2
Then the usual unpack, cd to parallel-{date}, ./configure && make, sudo make install. This will install parallel, man page parallel and man page parallel_tutorial.
Method 5
Here’s one in Perl:
./app | perl -ne 'BEGIN {open(FDA, ">A.out") and
open(FDB, ">B.out") and
open(FDC, ">C.out") or die("Cannot open files: $!n")}
print FDA $_ if /A/; print FDB $_ if /B/; print FDC $_ if /C/'
Method 6
sed -ne/A/w A.out -e/B/w B.out -e/C/p <in >C.out
…if <in is readable all three outfiles will be truncated before anything is written to them.
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0