Is there a way to only execute a command after another is done without a temp file?
I have one longer running command and another command that formats the output and sends it to a HTTP server using curl.
If i just execute commandA | commandB, commandB will start curl, connect to the server and start sending data. Because commandAtakes so long, the HTTP server will timeout.
I can do what I want with commandA > /tmp/file && commandB </tmp/file && rm -f /tmp/file
Out of curiosity I want to know if there is a way to do it without the temp file.
I tried mbuffer -m 20M -q -P 100 but the curl process is still started right at the beginning. Mbuffer waits just until commandAis done with actually sending the data.
(The data itself is just a few hundred kb at max)
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
This is similar to a couple of the other answers.
If you have the “moreutils” package, you should have the sponge command. Try
commandA | sponge | { IFS= read -r x; { printf "%sn" "$x"; cat; } | commandB; }
The sponge command is basically a pass-through filter (like cat)
except that it does not start writing the output until it has read the entire input.
I.e., it “soaks up” the data, and then releases it when you squeeze it (like a sponge).
So, to a certain extent, this is “cheating” –
if there’s a non-trivial amount of data, sponge almost certainly uses a temporary file.
But it’s invisible to you; you don’t have to worry about housekeeping things
like choosing a unique filename and cleaning up afterwards.
The { IFS= read -r x; { printf "%sn" "$x"; cat; } | commandB; }
reads the first line of output from sponge.
Remember, this doesn’t appear until commandA has finished.
Then it fires up commandB, writes the first line to the pipe,
and invokes cat to read the rest of the output and write it to the pipe.
Method 2
Commands in pipe line are started concurrently, you need to store commandA output somewhere to use later. You can avoid temp file by using variable:
output=$(command A; echo A)
printf '%s' "${output%%A}" | commandB
Method 3
I don’t know of any standard UNIX utility that can address this issue. One option would be use awk to accumulate commandA output and flush it to commandB at one shot, like so
commandA | awk '{x = x ORS $0}; END{printf "%s", x | "commandB"}'
Beware that this could be memory intensive since awk is building up a string from its input.
Method 4
You can solve the requirement with a little script. This particular variant avoids the temporary file and potential memory hog at the expense of additional processes.
#!/bin/bash
#
IFS= read LINE
if test -n "$LINE"
then
test -t 2 && echo "Starting $*" >&2
(
echo "$LINE"
cat
) | "<a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="f0d4b0">[email protected]</a>"
else
exit 0
fi
If you were to call the script waituntil (and make it executable, put it in the PATH, etc), you’d use it like this
commandA... | waituntil commandB...
Example
( sleep 3 ; date ; id ) | waituntil nl
Method 5
Yet another option is tac;
| tac | tac
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0