Panda’s Write CSV – Append vs. Write

I would like to use pd.write_csv to write “filename” (with headers) if “filename” doesn’t exist, otherwise to append to “filename” if it exists. If I simply use command:

     df.to_csv('filename.csv',mode = 'a',header ='column_names')

The write or append succeeds, but it seems like the header is written every time an append takes place.

How can I only add the header if the file doesn’t exist, and append without header if the file does exist?

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

Not sure there is a way in pandas but checking if the file exists would be a simple approach:

import os
# if file does not exist write header 
if not os.path.isfile('filename.csv'):
   df.to_csv('filename.csv', header='column_names')
else: # else it exists so append without writing the header
   df.to_csv('filename.csv', mode='a', header=False)

Method 2

with open(filename, 'a') as f:
    df.to_csv(f, mode='a', header=f.tell()==0)

it will add header when writes to the file first time

Method 3

In Pandas dataframe “to_csv” function, use header=False if csv file exists & append to existing file.

import os

hdr = False  if os.path.isfile('filename.csv') else True
df.to_csv('filename.csv', mode='a', header=hdr)

Method 4

The above solutions are great, but I have a moral obligation to include the pathlib solution here:

from pathlib import Path

file_path = Path(filename)
if file_path.exists():
   df.to_csv(file_path, header=False, mode='a')
else:
   df.to_csv(file_path, header=True, mode='w')

Alternatively (depending on your inlining preferences):

file_exists = file_path.exists()
df.to_csv(file_path, header=not file_exists, mode='a' if file_exists else 'w')

Method 5

Apart from file exist check, you can also check for non zero file size. Since it will make sense to add header if file exists but file size is zero i.e file without content. I find it helpful in some exceptional cases

import os.path
header_flag = False if (os.path.exists(fpath) and (os.path.getsize(fpath) > 0)) else True
df.to_csv(fpath, mode='a', index=False, header=header_flag)

Method 6

In case if you have dict() and want to write and append into CSV file :

import pandas as pd

file_name = 'data.csv'

my_dict = {"column_1":"Apple","column_2":"Mango"}

with open(file_name, 'a') as f:
  
  df = pd.DataFrame(my_dict)
  df.to_csv(f, mode='a', header=f.tell()==0)

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0