I have a text file in an unknown or mixed encoding. I want to see the lines that contain a byte sequence that is not valid UTF-8 (by piping the text file into some program). Equivalently, I want to filter out the lines that are valid UTF-8. In other words, I’m looking for grep [notutf8].
I have a text file encoded as following according to file:
I have several files with encoding issues in their file names (German umlauts, burned on CD with Windows, read by Windows and synced to Linux with Seafile. Something, somewhere went wrong…).
Bash and zsh only show “?” instead of umlauts, stat shows something like
I already know vim -b, however, depending on the locale used, it displays multi-byte characters (like UTF-8) as single letters.
I was about to diff a backup from it’s source to manually verify that the data is correct. Some chars, like åäö, is not shown correctly on the original data but as the clients (over samba) interpret it correctly it’s nothing to worry about. The data restored from backup shows the chars correctly, leading diff to not consider them to be the same files (with diffs, but rather completely different files).
Excel files can be converted to CSV using: