How to bulk convert all the file in a file system branch between Unix and Windows line break format?

Everybody knows 🙂 that in Windows plain text files lines are terminated with CR+LF, and in Unix&Linux – with LF only. How can I quickly convert all my source code files from one format to another and back?

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

That depends: if the files are under version control, this could be a rather unpopular history-polluting decision. Git has the option to automagically convert line endings on check-out.

If you do not care and want to quickly convert, there are programs like fromdos/todos and dos2unix/unix2dos that do this for you. You can use find: find . -type f -name '*.php' -exec dos2unix '{}' +.

Method 2

There are several dedicated programs, including

• dos2unix and unix2dos from the dos2unix tools
• todos and fromdos from the tofrodos tools

Simply pick the tool for the appropriate direction and pass the names of the files to convert on the command line.

If you don’t have either, but have Linux or Cygwin:

sed -i -e 's/r+$//' filename             # dos|unix -> unix
sed -i -e 's/r*$/r/' filename            # dos|unix -> dos

If you have perl:

perl -i -pe 's/r+$//' filename            # dos|unix -> unix
perl -i -pe 's/r*$/r/' filename          # dos|unix -> dos

With only POSIX tools (including BusyBox), to go from unix to dos, you’ll need to pass the CR character literally in the sed command.

cr=$(echo | tr 'n' 'r')
sed -e "s/$cr*$/$cr/" <filename >filename.dos
mv filename.dos filename

In the other direction, you can simply delete all CRs:

tr -d 'r' <filename >filename.dos
mv filename.dos filename

You can use wildcards to convert many files in the same directory at once, e.g.

sed -i -e 's/r+$//' *.txt

To convert all files in the current directory and its subdirectories, if your shell is zsh, you can use **/, e.g.

sed -i -e 's/r+$//' **/*.txt

You can use **/ in bash ≥4, but you need to run shopt -s globstar first (you can put this line in your ~/.bashrc). You can use **/ in ksh93, but you need to run set -o globstar first (you can put this line in your ~/.kshrc.

If you can only use the tools that require a redirection, use a for loop.

for x in *.txt; do
  tr -d 'r' <"$x" >"$x.dos"
  mv -- "$x.dos" "$x"
done

If you don’t have **/ or need more complex matching to select which files to convert, use the find command. Here’s a Linux/Cygwin example which converts all files under the current directory and its subdirectories recursively, except for files called .o and under subdirectories called bin.

find -name 'bin' -type d -prune -o 
     ! -name '*.o' 
     -exec sed -i -e 's/r+$//' {} +

Here’s a POSIX example. We tell find to start a shell that can perform the necessary redirection.

find -name 'bin' -type d -prune -o 
     ! -name '*.o' 
     -exec sh -c '
       tr -d 'r' <"$0" >"$0.dos"
       mv -- "$0.dos" "$0"
' {} ;

You can make the find method slightly faster, at the expense of more complex code, by using a loop in the shell command.

find -name 'bin' -type d -prune -o 
     ! -name '*.o' 
     -exec sh -c '
       for x; do
         tr -d 'r' <"$x" >"$x.dos"
         mv -- "$x.dos" "$x"
       done
' _ {} +

Method 3

winscp will also auto-convert, if told so.

If your files reside all in the same directory:

DIRECTORY=/your/directory
unix2dos $DIRECTORY/*

Method 4

Changing from ASCII text, with CRLF line terminators to ASCII text

VIM – Vi IMproved 8.0 (2016 Sep 12, compiled Apr 10 2018 21:31:58)

All files in current directory

for i in *; do vi -c "set fileformat=unix | wq" "${i}"; done

Specific mask (all filenames in current directory starting with word “file”)

for i in file*; do vi -c "set fileformat=unix | wq" "${i}"; done

Another mask (all filenames in current directory that contain the word “file”)

for i in *file*; do vi -c "set fileformat=unix | wq" "${i}"; done

To convert files back to CRLF just change fileformat=unix to fileformat=dos:

for i in *file*; do vi -c "set fileformat=dos | wq" "${i}"; done

This is the quickest way if using vim to convert fileformat.

Alternatively sed can also be used to remove all ^M by pressing CTRL+V and then CTRL+M

for i in *; do sed -i s/^M// "$i"; done

or

for i in *; do sed -i s/^M// "${i}"; done

This will also change all CRLF files in current directory to ASCII text.

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0