Say, I have a command command which prints huge number of lines to stdout:
line1 line2 ..... lineN
I want to save the output to disk, but not as a single file, but as a sequence of files each having 1000 lines of stdout:
file0001.txt: ------------- line1 .... line1000 file0002.txt: ------------- line1001 .... line2000 etc
I’ve tried to google the answer, but every time google points me to tee command, which is useless in this situation. Probably, I’m entering wrong queries.
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
Once you are done saving the file, you could always split the file into file pieces or multiple files based on the number of lines.
split -l 1000 output_file
or even better just try
command | split -l 1000 -
This will split the output stream into files with each 1000 lines (default is 1000 lines without -l option).
The below command will give you additional flexibility to put or enforce a prefix to the filename that will be generated when the output is generated and splitted to store into the file.
command | split -l 1000 - small-
Method 2
You can use a bash script lines.bash
#!/bin/bash
a=0
while IFS='' read -r line
do
printf -v filename "%04d.txt" "$((a++/1000))"
echo "$line" >> $filename
done
and use it as:
cat long_file.txt | bash lines.bash
The only problem I noticed is with * sign in long_file.txt (somebody could correct it).
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0