How to see duplicated rows and how to remove them.¶

You have to think about your data and what you mean by duplicated rows.

• Do you consider a row to be duplicated based on every value of your columns or only a selection of your columns.
  

we have two method to work with here.¶

• duplicated() This method tells you which rows are duplicated, you can also choose which columns to look at while calculating the duplicated rows.
• drop_duplicates() This method removes duplicated rows, it also lets you choose which columns to look at whilst discovering the duplicated rows.
  by default both duplicated and drop_duplicates() look at all the columns when discovering duplicated values.
  
  
• if you wish to only look at subsection of your columns you should use subset parameter inside these methos.

Take a look at the data we are going to work with.

I used the method duplicated() below, which by default looks at every column and returns True if the rows is identical to any other row in the data and False if the row is unqiue.

• the keep = False parameter tells the method to keep all the record of the duplicated rows.
• you can also use keep = "first" which would then keep the first occurance of the duplicated rows, and discard the rest.
• or use keep = "last" to keep the last occurance of your duplicated rows.
  
  The keep parameter logic is the same in both duplicated() and drop_duplicates() methods.

As we have seen in the previous tutorials , when we want to filter our data we would pass in a boolean inside brackets or a loc method.

• The data will be filtered to show all the True value rows, which in this case is the duplicated rows.
• I used the sort_value method to put the duplicated rows on top of each other so we could see them better.

By default duplicated method looks at every column.

• What if we wanted to look at only some of the columns like distance and year when discovering duplicates?
  
• In this case we can make use of subset parameter which can take either a single column name or a list of column names and it will discover duplicated rows based on the given column(s).

duplicated() tells us which rows are duplicated, if we want to delete these rows, we should use drop_duplicates() which behaves the same way as duplicated when discovering duplicated rows.

Note:
By default all the columns are considered, if we don't pass in any arguments to the subset parameter.

We can drop duplicates by subsection of columns such as below:

Note that after removing duplicated rows we ended up with 679 rows as per above.
we can prove this by doing simple one line calculation shown below.

1. find the total number of rows in our Data using len method. ---> len(df)
2. find the number of duplicated rows based on distance and year columns. ---> df.duplicated(subset =["distance","year"]).sum()
3. subtract the duplicated rows from the total rows and sure enough we get 679 which is the same as above.

As we have seen with duplicated method , we have control over which of the duplicated row to keep and discard, via the keep parameter. to keep the first duplicated row we use keep = "first" such as below:

And to keep the last duplicated record(row) we use keep = "last".