46

I wonder how to add newDataFrame data onto the end of an existing csv file? Theto_csv doesn't mention such functionality.

vvvvv's user avatar
vvvvv
32.9k19 gold badges70 silver badges103 bronze badges
askedJun 16, 2013 at 15:40
perigee's user avatar
1

4 Answers4

80

You can append usingto_csv by passing a file which isopen in append mode:

with open(file_name, 'a') as f:    df.to_csv(f, header=False)

Useheader=None, so as not to append the column names.

In fact, pandas has a wrapper to do this into_csv using themode argument (see Joe's answer):

df.to_csv(f, mode='a', header=False)
answeredJun 16, 2013 at 15:52
Andy Hayden's user avatar
Sign up to request clarification or add additional context in comments.

10 Comments

Also need to close the file by f.close(). Andy, you make my day. It works like a charm, I'm from c/c++ ethnic and need to learn the python philosophy. Any suggestion?
Andy, really appreciated :-D (cannot use @ symbol :-()
Bonus points that this closes the file after to_csv. I have some code that hits to_csv alot and was finding the files left open on later iterations.
@EzekielKruglick Were you passing an open file toto_csv or the filename? I recall a related issue wherenot closing the file led to a 99% speedup of their code (IIRC they were appending to the same file tens of thousands of times).
@perigee when "with" is used the file is closed automatically always.blog.lerner.co.il/dont-use-python-close-files-answer-depends
|
47

You can also pass the file mode as an argument to theto_csv method

df.to_csv(file_name, header=False, mode = 'a')
Leland Hepworth's user avatar
Leland Hepworth
1,01613 silver badges18 bronze badges
answeredJul 28, 2013 at 17:14
Joe Hooper's user avatar

Comments

4

A little helper function I use (based on Joe Hooper's answer) with some header checking safeguards to handle it all:

def appendDFToCSV_void(df, csvFilePath, sep=","):    import os    if not os.path.isfile(csvFilePath):        df.to_csv(csvFilePath, mode='a', index=False, sep=sep)    elif len(df.columns) != len(pd.read_csv(csvFilePath, nrows=1, sep=sep).columns):        raise Exception("Columns do not match!! Dataframe has " + str(len(df.columns)) + " columns. CSV file has " + str(len(pd.read_csv(csvFilePath, nrows=1, sep=sep).columns)) + " columns.")    elif not (df.columns == pd.read_csv(csvFilePath, nrows=1, sep=sep).columns).all():        raise Exception("Columns and column order of dataframe and csv file do not match!!")    else:        df.to_csv(csvFilePath, mode='a', index=False, sep=sep, header=False)
answeredMay 17, 2015 at 22:17
KCzar's user avatar

1 Comment

Is there an API setting for the 3rd test case, column order not matching between dataframe and csv? I want to write without headers, but have the columns be implicitly reordered.
3

Thank to Andy, the complete solution:

f = open(filename, 'a') # Open file as append modedf.to_csv(f, header = False)f.close()
answeredJun 16, 2013 at 16:21
perigee's user avatar

1 Comment

Just to mention, this is essentially equivalent to above but after this you're left with a closed file (f), whereas withwith it cleans up that for you. :)

Your Answer

Sign up orlog in

Sign up using Google
Sign up using Email and Password

Post as a guest

Required, but never shown

By clicking “Post Your Answer”, you agree to ourterms of service and acknowledge you have read ourprivacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.