pandas.DataFrame.drop_duplicates#

DataFrame.drop_duplicates(subset=None,*,keep='first',inplace=False,ignore_index=False)[source]#

Return DataFrame with duplicate rows removed.

Considering certain columns is optional. Indexes, including time indexesare ignored.

Parameters:

subsetcolumn label or sequence of labels, optional

Only consider certain columns for identifying duplicates, bydefault use all of the columns.

keep{‘first’, ‘last’,False}, default ‘first’

Determines which duplicates (if any) to keep.

• ‘first’ : Drop duplicates except for the first occurrence.

• ‘last’ : Drop duplicates except for the last occurrence.

• False : Drop all duplicates.

inplacebool, defaultFalse

Whether to modify the DataFrame rather than creating a new one.

ignore_indexbool, defaultFalse

IfTrue, the resulting axis will be labeled 0, 1, …, n - 1.

Returns:

DataFrame or None

DataFrame with duplicates removed or None ifinplace=True.

See also

DataFrame.value_counts

Count unique combinations of columns.

Examples

Consider dataset containing ramen rating.

>>>df=pd.DataFrame({...'brand':['Yum Yum','Yum Yum','Indomie','Indomie','Indomie'],...'style':['cup','cup','cup','pack','pack'],...'rating':[4,4,3.5,15,5]...})>>>df    brand style  rating0  Yum Yum   cup     4.01  Yum Yum   cup     4.02  Indomie   cup     3.53  Indomie  pack    15.04  Indomie  pack     5.0

By default, it removes duplicate rows based on all columns.

>>>df.drop_duplicates()    brand style  rating0  Yum Yum   cup     4.02  Indomie   cup     3.53  Indomie  pack    15.04  Indomie  pack     5.0

To remove duplicates on specific column(s), usesubset.

>>>df.drop_duplicates(subset=['brand'])    brand style  rating0  Yum Yum   cup     4.02  Indomie   cup     3.5

To remove duplicates and keep last occurrences, usekeep.

>>>df.drop_duplicates(subset=['brand','style'],keep='last')    brand style  rating1  Yum Yum   cup     4.02  Indomie   cup     3.54  Indomie  pack     5.0