Lesson 13

Removing Columns and Rows

Drop columns and rows you do not need, decide between a copy and an in-place change, and find and remove duplicate records.

Intermediate25 min

Removing Columns and Rows hero Module 24 Lesson 13

drop removes labels. Name columns with columns= and rows with index=. It returns a new DataFrame by default, so the original is safe until you assign the result back.

A table with the track column marked for removal

Drop columns

One name or a list of names. The result is a new table, so read the line as make me a copy without these columns.

python

import pandas as pd

learners = pd.DataFrame({
    "name": ["Avery", "Jordan", "Casey", "Riley", "Morgan", "Quinn"],
    "track": ["Python", "NumPy", "Python", "pandas", "NumPy", "pandas"],
    "city": ["Northgate", "Eastport", "Northgate", "Westfield", "Eastport", "Westfield"],
    "hours": [42, 31, 55, 24, 47, 38],
    "score": [88, 74, 91, 65, 82, 79],
})
print(learners.drop(columns="city").head(3))
print(learners.drop(columns=["city", "track"]).head(3))
print(learners.columns.tolist())

Output:

text

     name   track  hours  score
0   Avery  Python     42     88
1  Jordan   NumPy     31     74
2   Casey  Python     55     91
     name  hours  score
0   Avery     42     88
1  Jordan     31     74
2   Casey     55     91
['name', 'track', 'city', 'hours', 'score']

Drop rows

index= takes row labels. With the default index the labels are numbers, which look like positions but are not: after a filter or a sort the numbers no longer count from 0.

python

import pandas as pd

learners = pd.DataFrame({
    "name": ["Avery", "Jordan", "Casey", "Riley", "Morgan", "Quinn"],
    "track": ["Python", "NumPy", "Python", "pandas", "NumPy", "pandas"],
    "city": ["Northgate", "Eastport", "Northgate", "Westfield", "Eastport", "Westfield"],
    "hours": [42, 31, 55, 24, 47, 38],
    "score": [88, 74, 91, 65, 82, 79],
})
print(learners.drop(index=0))
print(learners.drop(index=[1, 3]))

Output:

text

     name   track       city  hours  score
1  Jordan   NumPy   Eastport     31     74
2   Casey  Python  Northgate     55     91
3   Riley  pandas  Westfield     24     65
4  Morgan   NumPy   Eastport     47     82
5   Quinn  pandas  Westfield     38     79
     name   track       city  hours  score
0   Avery  Python  Northgate     42     88
2   Casey  Python  Northgate     55     91
4  Morgan   NumPy   Eastport     47     82
5   Quinn  pandas  Westfield     38     79

Drop by position

When you know the position rather than the label, look the label up first. df.index[2] is the third row label and df.columns[1] is the second column name.

python

import pandas as pd

learners = pd.DataFrame({
    "name": ["Avery", "Jordan", "Casey", "Riley", "Morgan", "Quinn"],
    "track": ["Python", "NumPy", "Python", "pandas", "NumPy", "pandas"],
    "city": ["Northgate", "Eastport", "Northgate", "Westfield", "Eastport", "Westfield"],
    "hours": [42, 31, 55, 24, 47, 38],
    "score": [88, 74, 91, 65, 82, 79],
})
print(learners.drop(index=learners.index[[0, 2]]).head(3))
print(learners.drop(columns=learners.columns[1]).head(2))

Output:

text

     name   track       city  hours  score
1  Jordan   NumPy   Eastport     31     74
3   Riley  pandas  Westfield     24     65
4  Morgan   NumPy   Eastport     47     82
     name       city  hours  score
0   Avery  Northgate     42     88
1  Jordan   Eastport     31     74

Copy or in place

Assign the result back when you want to keep it, or pass inplace=True to change the table where it stands. Assigning is easier to follow and it chains, so prefer it unless the table is huge.

python

import pandas as pd

learners = pd.DataFrame({
    "name": ["Avery", "Jordan", "Casey", "Riley", "Morgan", "Quinn"],
    "track": ["Python", "NumPy", "Python", "pandas", "NumPy", "pandas"],
    "city": ["Northgate", "Eastport", "Northgate", "Westfield", "Eastport", "Westfield"],
    "hours": [42, 31, 55, 24, 47, 38],
    "score": [88, 74, 91, 65, 82, 79],
})
trimmed = learners.drop(columns=["city", "track"])
print(trimmed.columns.tolist())

learners.drop(columns="city", inplace=True)
print(learners.columns.tolist())

Output:

text

['name', 'hours', 'score']
['name', 'track', 'hours', 'score']

Find and remove duplicates

duplicated() marks every row that has already been seen, so the first copy is False. drop_duplicates() keeps that first copy. subset narrows the comparison to the columns that define a record, and keep="last" keeps the newest instead.

python

import pandas as pd

signups = pd.DataFrame({
    "name": ["Avery", "Jordan", "Avery", "Casey", "Jordan"],
    "track": ["Python", "NumPy", "Python", "pandas", "pandas"],
})
print(signups.duplicated())
print(signups.duplicated().sum())
print(signups.drop_duplicates())
print(signups.drop_duplicates(subset="name", keep="last"))

Output:

text

0    False
1    False
2     True
3    False
4    False
dtype: bool
1
     name   track
0   Avery  Python
1  Jordan   NumPy
3   Casey  pandas
4  Jordan  pandas
     name   track
2   Avery  Python
3   Casey  pandas
4  Jordan  pandas

Key takeaways

drop(columns=...) removes columns and drop(index=...) removes rows
drop returns a copy unless you pass inplace=True
Row labels are not positions, so look them up with df.index[...]
duplicated marks repeats and drop_duplicates keeps the first by default
Next: Sorting Data
Previous: Updating Values

Loading playground…

Check your understanding

Removing Columns and Rows Quiz

Which argument drops columns?

1 / 10

← Updating Values

Sorting Data →