Skip to main content
MahiDev — Learn. Build. Become AI-ready.

Lesson 10

Filtering Rows

Keep the rows you care about with a condition, combine conditions with and and or, and match against a list of values with isin and between.

Intermediate25 min
Filtering Rows hero Module 24 Lesson 10Filtering Rows hero Module 24 Lesson 10

A comparison on a column gives a Series of True and False, one per row. Hand that back to the DataFrame and you keep the True rows. This is the single most used move in pandas.

A four-row table with the two rows above forty hours highlightedA four-row table with the two rows above forty hours highlighted

A mask, then the rows

Look at the mask first. It is a Boolean Series that shares the index of the table, so pandas can line it up and drop the False rows.

python
import pandas as pd

learners = pd.DataFrame({
    "name": ["Avery", "Jordan", "Casey", "Riley", "Morgan", "Quinn"],
    "track": ["Python", "NumPy", "Python", "pandas", "NumPy", "pandas"],
    "city": ["Northgate", "Eastport", "Northgate", "Westfield", "Eastport", "Westfield"],
    "hours": [42, 31, 55, 24, 47, 38],
    "score": [88, 74, 91, 65, 82, 79],
})
busy = learners["hours"] > 40
print(busy)
print(learners[busy])
print(busy.sum())

Output:

text
0     True
1    False
2     True
3    False
4     True
5    False
Name: hours, dtype: bool
     name   track       city  hours  score
0   Avery  Python  Northgate     42     88
2   Casey  Python  Northgate     55     91
4  Morgan   NumPy   Eastport     47     82
3

sum() on a Boolean Series counts the True values.

Two conditions at once

Use & for and, | for or, and ~ for not. Each condition needs its own brackets, because & binds tighter than >. The Python keywords and and or do not work here, since they cannot look at a whole column at once.

python
import pandas as pd

learners = pd.DataFrame({
    "name": ["Avery", "Jordan", "Casey", "Riley", "Morgan", "Quinn"],
    "track": ["Python", "NumPy", "Python", "pandas", "NumPy", "pandas"],
    "city": ["Northgate", "Eastport", "Northgate", "Westfield", "Eastport", "Westfield"],
    "hours": [42, 31, 55, 24, 47, 38],
    "score": [88, 74, 91, 65, 82, 79],
})
print(learners[(learners["hours"] > 35) & (learners["score"] > 80)])
print(learners[(learners["track"] == "pandas") | (learners["score"] > 85)])
print(learners[~(learners["track"] == "Python")])

Output:

text
     name   track       city  hours  score
0   Avery  Python  Northgate     42     88
2   Casey  Python  Northgate     55     91
4  Morgan   NumPy   Eastport     47     82
    name   track       city  hours  score
0  Avery  Python  Northgate     42     88
2  Casey  Python  Northgate     55     91
3  Riley  pandas  Westfield     24     65
5  Quinn  pandas  Westfield     38     79
     name   track       city  hours  score
1  Jordan   NumPy   Eastport     31     74
3   Riley  pandas  Westfield     24     65
4  Morgan   NumPy   Eastport     47     82
5   Quinn  pandas  Westfield     38     79

Match a list or a range

isin tests membership against several values, which beats a chain of |. between covers a numeric range and includes both ends by default. Text columns get the .str helpers.

python
import pandas as pd

learners = pd.DataFrame({
    "name": ["Avery", "Jordan", "Casey", "Riley", "Morgan", "Quinn"],
    "track": ["Python", "NumPy", "Python", "pandas", "NumPy", "pandas"],
    "city": ["Northgate", "Eastport", "Northgate", "Westfield", "Eastport", "Westfield"],
    "hours": [42, 31, 55, 24, 47, 38],
    "score": [88, 74, 91, 65, 82, 79],
})
print(learners[learners["track"].isin(["NumPy", "pandas"])])
print(learners[learners["hours"].between(30, 45)])
print(learners[learners["city"].str.startswith("E")])

Output:

text
     name   track       city  hours  score
1  Jordan   NumPy   Eastport     31     74
3   Riley  pandas  Westfield     24     65
4  Morgan   NumPy   Eastport     47     82
5   Quinn  pandas  Westfield     38     79
     name   track       city  hours  score
0   Avery  Python  Northgate     42     88
1  Jordan   NumPy   Eastport     31     74
5   Quinn  pandas  Westfield     38     79
     name  track      city  hours  score
1  Jordan  NumPy  Eastport     31     74
4  Morgan  NumPy  Eastport     47     82

Filter and pick columns in one step

.loc takes a mask for the rows and names for the columns, so one call answers a real question instead of two.

python
import pandas as pd

learners = pd.DataFrame({
    "name": ["Avery", "Jordan", "Casey", "Riley", "Morgan", "Quinn"],
    "track": ["Python", "NumPy", "Python", "pandas", "NumPy", "pandas"],
    "city": ["Northgate", "Eastport", "Northgate", "Westfield", "Eastport", "Westfield"],
    "hours": [42, 31, 55, 24, 47, 38],
    "score": [88, 74, 91, 65, 82, 79],
})
print(learners.loc[learners["score"] > 80, ["name", "track", "score"]])
print(learners.loc[learners["hours"] < 35, "name"])

Output:

text
     name   track  score
0   Avery  Python     88
2   Casey  Python     91
4  Morgan   NumPy     82
1    Jordan
3     Riley
Name: name, dtype: object

query reads like a sentence

query takes the condition as text, so column names need no brackets and and and or are spelled out. It is easier on the eye for long conditions; the bracket form is easier when the condition is built from variables.

python
import pandas as pd

learners = pd.DataFrame({
    "name": ["Avery", "Jordan", "Casey", "Riley", "Morgan", "Quinn"],
    "track": ["Python", "NumPy", "Python", "pandas", "NumPy", "pandas"],
    "city": ["Northgate", "Eastport", "Northgate", "Westfield", "Eastport", "Westfield"],
    "hours": [42, 31, 55, 24, 47, 38],
    "score": [88, 74, 91, 65, 82, 79],
})
print(learners.query("hours > 35 and score > 80"))

floor = 80
print(learners.query("score >= @floor")[["name", "score"]])

Output:

text
     name   track       city  hours  score
0   Avery  Python  Northgate     42     88
2   Casey  Python  Northgate     55     91
4  Morgan   NumPy   Eastport     47     82
     name  score
0   Avery     88
2   Casey     91
4  Morgan     82

Inside query, @floor means the Python variable named floor.

Key takeaways

  • A comparison on a column gives a Boolean Series, one value per row

  • Combine with & | ~ and give every condition its own brackets

  • isin matches a list of values and between covers a range

  • .loc[mask, columns] filters and picks columns in one step

  • Next: Adding Columns and Rows

  • Previous: loc and iloc

Loading playground…

Check your understanding

Filtering Rows Quiz

What type is df["hours"] > 40?

1 / 10

Previous

← loc and iloc

Next

Adding Columns and Rows →