Skip to main content
MahiDev — Learn. Build. Become AI-ready.

Lesson 16

Working with Missing Data

Find the gaps in a table with isna, fill them with a sensible value or the value next door, and drop the rows or columns that are too empty to use.

Intermediate25 min
Working with Missing Data hero Module 24 Lesson 16Working with Missing Data hero Module 24 Lesson 16

Real data has holes. pandas shows them as NaN, and a missing value is not zero and not an empty string: it means nobody told us. Every summary skips them, so the first job is finding out how many there are.

A small table with its two missing cells highlightedA small table with its two missing cells highlighted

Find the gaps

isna() gives a table of True and False, and notna() is its opposite. Summing down the columns turns that into the number you actually want: how much is missing, and where.

python
import numpy as np
import pandas as pd

courses = pd.DataFrame({
    "course": ["Python", "NumPy", "pandas", "SQL"],
    "fee": [2400, np.nan, 2100, 1600],
    "days": [30, 45, np.nan, np.nan],
})
print(courses.isna())
print(courses.isna().sum())
print(courses["fee"].notna().sum())

Output:

text
   course    fee   days
0   False  False  False
1   False   True  False
2   False  False   True
3   False  False   True
course    0
fee       1
days      2
dtype: int64
3

isnull and notnull are older names for exactly the same two methods.

Fill with a value

fillna replaces every gap in the object you call it on. A dict gives each column its own filler, which matters because a sensible default for a fee is not a sensible default for a length.

python
import numpy as np
import pandas as pd

courses = pd.DataFrame({
    "course": ["Python", "NumPy", "pandas", "SQL"],
    "fee": [2400, np.nan, 2100, 1600],
    "days": [30, 45, np.nan, np.nan],
})
print(courses.fillna(0))
print(courses.fillna({"fee": courses["fee"].mean(), "days": 30}))

Output:

text
   course     fee  days
0  Python  2400.0  30.0
1   NumPy     0.0  45.0
2  pandas  2100.0   0.0
3     SQL  1600.0   0.0
   course          fee  days
0  Python  2400.000000  30.0
1   NumPy  2033.333333  45.0
2  pandas  2100.000000  30.0
3     SQL  1600.000000  30.0

Fill from the neighbours

In ordered data the value before or after a gap is often the best guess. ffill carries the last known value forward and bfill pulls the next one back. A leading gap has nothing before it, so ffill leaves it alone.

python
import numpy as np
import pandas as pd

readings = pd.Series([np.nan, 18.2, np.nan, np.nan, 19.6],
                     index=["Mon", "Tue", "Wed", "Thu", "Fri"])
print(readings.ffill())
print(readings.bfill())
print(readings.interpolate())

Output:

text
Mon     NaN
Tue    18.2
Wed    18.2
Thu    18.2
Fri    19.6
dtype: float64
Mon    18.2
Tue    18.2
Wed    19.6
Thu    19.6
Fri    19.6
dtype: float64
Mon          NaN
Tue    18.200000
Wed    18.666667
Thu    19.133333
Fri    19.600000
dtype: float64

interpolate walks a straight line between the two known values instead of repeating one of them.

Drop what you cannot use

dropna() removes any row with a gap, which is fine when they are rare and ruinous when they are not. how="all" only removes rows that are completely empty, subset looks at the columns you name, and axis=1 drops columns instead of rows.

python
import numpy as np
import pandas as pd

courses = pd.DataFrame({
    "course": ["Python", "NumPy", "pandas", "SQL"],
    "fee": [2400, np.nan, 2100, 1600],
    "days": [30, 45, np.nan, np.nan],
})
print(courses.dropna())
print(courses.dropna(subset="fee"))
print(courses.dropna(axis=1))
print(courses.dropna(thresh=3))

Output:

text
   course     fee  days
0  Python  2400.0  30.0
   course     fee  days
0  Python  2400.0  30.0
2  pandas  2100.0   NaN
3     SQL  1600.0   NaN
   course
0  Python
1   NumPy
2  pandas
3     SQL
   course     fee  days
0  Python  2400.0  30.0

thresh=3 keeps a row only if it has at least three real values.

Gaps and arithmetic

mean and sum skip the gaps, so the average of three values out of four is the average of three. count reports only the real values, while len reports every row. When that difference matters, say so in your result.

python
import numpy as np
import pandas as pd

fees = pd.Series([2400, np.nan, 2100, 1600])
print(fees.mean())
print(fees.sum())
print(fees.count(), len(fees))
print(fees.mean(skipna=False))

Output:

text
2033.3333333333333
6100.0
3 4
nan

Key takeaways

  • A gap means unknown, not zero, and prints as NaN

  • isna().sum() is the quickest picture of what is missing

  • fillna takes one value or a dict of one value per column

  • ffill and bfill copy a neighbour; dropna removes the unusable rows

  • Next: Iterating Rows and Columns

  • Previous: Column Types and astype

Loading playground…

Check your understanding

Working with Missing Data Quiz

What does NaN mean in a table?

1 / 10

Previous

← Column Types and astype

Next

Iterating Rows and Columns →