pandas DataFrame output end of csv

I wonder how to add new DataFrame data onto the end of an existing csv file? The to_csv doesn’t mention such functionality.

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

You can append using to_csv by passing a file which is open in append mode:

with open(file_name, 'a') as f:
    df.to_csv(f, header=False)

Use header=None, so as not to append the column names.

In fact, pandas has a wrapper to do this in to_csv using the mode argument (see Joe’s answer):

df.to_csv(f, mode='a', header=False)

Method 2

You can also pass the file mode as an argument to the to_csv method

df.to_csv(file_name, header=False, mode = 'a')

Method 3

A little helper function I use (based on Joe Hooper’s answer) with some header checking safeguards to handle it all:

def appendDFToCSV_void(df, csvFilePath, sep=","):
    import os
    if not os.path.isfile(csvFilePath):
        df.to_csv(csvFilePath, mode='a', index=False, sep=sep)
    elif len(df.columns) != len(pd.read_csv(csvFilePath, nrows=1, sep=sep).columns):
        raise Exception("Columns do not match!! Dataframe has " + str(len(df.columns)) + " columns. CSV file has " + str(len(pd.read_csv(csvFilePath, nrows=1, sep=sep).columns)) + " columns.")
    elif not (df.columns == pd.read_csv(csvFilePath, nrows=1, sep=sep).columns).all():
        raise Exception("Columns and column order of dataframe and csv file do not match!!")
    else:
        df.to_csv(csvFilePath, mode='a', index=False, sep=sep, header=False)

Method 4

Thank to Andy, the complete solution:

f = open(filename, 'a') # Open file as append mode
df.to_csv(f, header = False)
f.close()

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0