Can’t redirect cut output

When I try to redirect the output of cut it always seems to be empty. If don’t redirect it, the output shows in terminal as expected. This is true for OS X 10.10 and Linux 4.1.6.

This works:

<a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="35475a5a41755e54475954">[email protected]</a>:~# nc 10.0.2.56 30003
[...] lots of lines [...]

This works:

<a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="30425f5f44705b51425c51">[email protected]</a>:~# nc 10.0.2.56 30003 | cat
[...] lots of lines [...]

This works:

<a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="f4869b9b80b49f95869895">[email protected]</a>:~# nc 10.0.2.56 30003 | cut -d, -f 15,16
[...] lots of lines [...]

This doesn’t

<a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="2755484853674c46554b46">[email protected]</a>:~# nc 10.0.2.56 30003 | cut -d, -f 15,16 | cat
[nothing]

This again DOES

<a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="82f0ededf6c2e9e3f0eee3">[email protected]</a>:~# cat messung1 | cut -d, -f15,16 | cat
[...] lots of lines [...]

This is not limited to cat after cut. grep, tee and standard redirection using > don’t work either.

What’s wrong there?

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

It’s not that much that there’s no output as that it’s coming in chunks.

Like many programs, when its output is no longer a terminal, cut buffers its output. That is, it only writes data when it has accumulated a buffer-full of it. Typically, something like 4 or 8 KiB though YMMV.

You can easily verify it by comparing:

(echo foo; sleep 1; echo bar) | cut -c2-

With:

(echo foo; sleep 1; echo bar) | cut -c2- | cat

In the first case, cut outputs oon and then arn one second later, while in the second case, cut outputs oonarn after 1 seconds, that is when it sees the end of its input and flushes its output upon exit.

In your case, since stdin is nc, it would only see the end of its input when the connection is closed, so it would only start outputting anything after it has accumulated 4KiB worth of data to write.

To work around that, several approaches are possible.

• On GNU or FreeBSD systems, you can use the stdbuf utility that can tweak the buffering behaviour of some commands (it doesn’t work for all as it uses a LD_PRELOAD hack to pre-configure the stdio buffering behaviour).

  ... | stdbuf -oL cut -d, -f15,16 | cat

  would tell cut to do a line-based buffering on its stdout.

• some commands like GNU grep have options to affect their buffering. (--line-buffered in the case of GNU grep).
• you can use a pseudo-tty wrapper to force the stdout of a command to be a terminal. However most of those solutions have some drawbacks and limitations. The unbuffer expect script for instance, often mentioned to address this kind of problem has a number of bugs for instance.

  One that doesn’t work too bad is when using socat as:

  ... | socat -u 'exec:"cut -d, -f15,16",pty,raw' -

• You can replace your text utility with a higher-level text processing tool that has support for unbuffered output.
  • GNU awk for instance has a fflush() function to flush its output. So your cut -d, -f15,16 could be written:

    awk -F, -vOFS=, '{print $15,$16;fflush()}'

  • if your awk lacks the fflush() function, you can use system("") instead. That’s normally the command to execute a command. Here, we would be executing an empty command, but actually using it for the fact that awk flushes its stdout before running the command.
  • or you can use perl:

    perl -F, -lane 'BEGIN{$,=",";$|=1} print @F[14..15]'

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0