I intend to pipe the output of a program into a while read VAR loop and break when a pattern is found, but it doesn’t.
Proof of concept:
inotifywait -qm -e create . | while read line; do echo $line; break; done ./ CREATE newfile
..
tail -f /var/log/syslog | while read line; do echo $line; break; done Nov 6 22:44:05 section9 ntpdate[2381]: adjust time server 91.189.89.199 offset 0.272779 sec
These never exit no matter what the source program outputs. Setting set -x beforehand suggests that the loop never iterates to the second read. $BASH_SUBSHELL is 1 in these examples.
Shouldn’t tail, inotifywait, etc. receive SIGPIPE and exit?
Note that process substitution (while read ... break; done < <(tail -f ...)) works OK. $BASH_SUBSHELL is 0 in this case.
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
The key is that, under bash (other may shells differ), a pipeline is not terminated until all commands in the pipeline are finished.
To understand, let’s consider:
inotifywait -qm -e create . | while read line; do echo $line; break; done
When read reads a line, it is echoed and then break is executed and the last process terminates. The first process, however, continues until a write to stdout fails. Thus, the loop will continue at least until inotifywait tries to write its second line of output. Because of the vagaries of buffering, it may not happen even then. It may take several lines before the discovery occurs. When that attempted write fails, then SIGPIPE is issued.
Now, consider the other case:
while read line; do echo $line; break; done < <(inotifywait -qm -e create .)
Here, there is no pipeline. When break is executed, the while loop is done.
Documentation
From the “Pipelines” section of man bash:
The shell waits for all commands in the pipeline to terminate….
This behavior is a choice made by bash. It is not mandated by POSIX. POSIX states:
If the pipeline is not in the background (see Asynchronous Lists), the shell
shall wait for the last command specified in the pipeline to complete, and
may also wait for all commands to complete.
Method 2
strace tail -f -n 5000 /var/log/messages |
while read line; do echo $line; break; done;
shows in detail what happens:
[...] write(1, "Nov 1 22:20:01 inno systemd[1]:"..., 4096) = 4096
i.e. tail writes up to 4096 bytes at once to the pipeline (that is what a pipeline can buffer). Usually there is not so much data at all for tail:
strace tail -f /var/log/messages | while read line; do echo $line; break; done;
shows
write(1, "Nov 7 07:00:02 inno systemd[1]:"..., 730) = 730
Thus tail would not try to write again until something is appended to the file.
I thought unbuffer would solve this problem. I am surprised that it doesn’t.
unbuffer tail -f file1 | while read line; do echo $line; break; done
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0