Is there a tool that combines zcat and cat transparently?

When handling log files, some end up as gzipped files thanks to logrotate and others not. So when you try something like this:

$ zcat *

you end up with a command line like zcat xyz.log xyz.log.1 xyz.log.2.gz xyz.log.3.gz and then with:

gzip: xyz.log: not in gzip format

Is there a tool that will take the magic bytes, similar to how file works, and use zcat or cat depending on the outcome so that I can pipe the output to grep for example?

NB: I know I can script it, but I am asking whether there is a tool out there already.

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

Try it with -f or --force:

zcat -f -- *

Since zcat is just a simple script that runs

exec gzip -cd "<a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="694d29">[email protected]</a>"

with long options that would translate to

exec gzip --stdout --decompress "<a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="466206">[email protected]</a>"

and, as per the man gzip (emphasize mine):

-f --force
      Force compression or decompression even if the file has multiple links
      or the corresponding file already exists, or if the compressed data is
      read from or written to a terminal. If the input data is not in a format
      recognized by gzip, and if the option --stdout is also given, copy the
      input data without change to the standard output: let zcat behave as cat.

Also:

so that I can pipe the output to grep for example

You could use zgrep for that:

zgrep -- PATTERN *

though see Stéphane’s comment below.

Method 2

zless

It seems a pity about zcat, as libz has an API that supports reading from both compressed and uncompressed files transparently. But the manpage does say that zcat is equivalent to gunzip -c.

Method 3

I use exactly for the same purpose:

{ cat /var/log/messages ; zcat /var/log/messages*.gz ; }| grep something | grep "something else" ....

Method 4

There is a drop-in replacement for ztools (zcat, zgrep, ..) called zutils that unites all the decompression tools independently of the backend. So with the same command you can read plain, lzma, gzipped, xz files transparently.

It’s available in debian wheezy or newer, probably in redhat/centos too.

The project’s page is here nongnu.org

A blog post explaining the usage of the util here (noone.org)

Method 5

This works fine in RHEL 5.x where zcat is a binary. It fails in RHEL 6.x (and Ubuntu 12.x) where zcat is a script. This used to work fine.

I wouldn’t be using zcat at all but zgrep won’t properly handle uncompressed files either.

Method 6

What about wrapper?

$ cat xcat.sh 
#!/bin/bash

for i in <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="1c385c">[email protected]</a>;do 
        [ ! -z "$(file -i $i | grep "gzip")" ] && zcat $i || cat $i
done

$ bash xcat.sh plain.txt gzipped_text.gz

Method 7

Opens both compressed and non-compressed, in chronological order.

ls -v syslog* | tac | xargs zcat -f | less

Method 8

Building on @Ryan’s answer, the following will get the get all the ‘rolled’ files sorted alphabetically, then get the current file, uncompress them, if needed, and less them:

cat <(ls mylog.log-* | sort) <(ls mylog.log) | xargs zcat -f | less

or if you want to get them all as a continuous stream, you can tail them, and optionally pipe that to another process

cat <(ls mylog.log-* | sort | xargs zcat -f) <(tail -f -n +0 mylog.log)

I should note that this is designed for logs that are rotated daily with the date appended to the end of the file. If your logs us a different format, you’ll have to modify the first part of the cat statement to accommodate.

Method 9

There is a beautiful perl script exacly doing this. It’s logresolvemerge.pl from the awstats project: http://www.awstats.org/docs/awstats_tools.html

Logresolvemerge allows you to get one unique output log file, sorted on date, built from particular sources: