utf-16

Strange unicode characters when reading in file in node.js app

by

I am attempting to write a node app that reads in a set of files, splits them into lines, and puts the lines into an array. Pretty simple. It works on quite a few files except some SQL files that I am working with. For some reason I seem to be getting some kind of unicode output when I split the lines up. The app looks something like this:

Convert UTF-8 with BOM to UTF-8 with no BOM in Python

by

Two questions here. I have a set of files which are usually UTF-8 with BOM. I’d like to convert them (ideally in place) to UTF-8 with no BOM. It seems like codecs.StreamRecoder(stream, encode, decode, Reader, Writer, errors) would handle this. But I don’t really see any good examples on usage. Would this be the best way to handle this?