Lesson 13

Missing Values

Find gaps in a table, see why models refuse NaN, and fill them with SimpleImputer using the mean, median, or most frequent value per column.

Intermediate25 min

Real tables have holes: a learner skipped a question, a sensor dropped out. pandas shows them as NaN, and most scikit-learn models refuse to fit while any are present. An imputer fills each hole with a sensible stand-in, such as the column's average, learned from the training rows.

A small table where empty cells are filled with the column mean or most common value

Find the gaps

isna().sum() counts the missing cells in each column. Always look before you fit, because the count tells you how much guessing you are about to do.

python

import numpy as np
import pandas as pd

survey = pd.DataFrame({
    "hours": [5.0, np.nan, 7.5, 3.0, np.nan, 9.0],
    "practice": [12, 20, np.nan, 8, 15, 30],
    "track": ["Python", "Data", np.nan, "Web", "Data", "Data"],
})

print(survey)
print(survey.isna().sum())

Output:

text

   hours  practice   track
0    5.0      12.0  Python
1    NaN      20.0    Data
2    7.5       NaN     NaN
3    3.0       8.0     Web
4    NaN      15.0    Data
5    9.0      30.0    Data
hours       2
practice    1
track       1
dtype: int64

Models refuse NaN

LinearRegression stops at the first missing value. Some models such as histogram gradient boosting cope with gaps, but most do not.

python

import numpy as np
import pandas as pd

survey = pd.DataFrame({
    "hours": [5.0, np.nan, 7.5, 3.0, np.nan, 9.0],
    "practice": [12, 20, np.nan, 8, 15, 30],
    "track": ["Python", "Data", np.nan, "Web", "Data", "Data"],
})
from sklearn.linear_model import LinearRegression

try:
    LinearRegression().fit(survey[["hours", "practice"]], [50, 55, 62, 40, 58, 80])
except ValueError as error:
    print("ValueError:", "NaN" in str(error))

Output:

text

ValueError: True

Fill numbers with SimpleImputer

SimpleImputer learns one fill value per column during fit. The default strategy is the mean; median is safer when a few extreme values would drag the mean around. statistics_ shows what it learned.

python

import numpy as np
import pandas as pd

survey = pd.DataFrame({
    "hours": [5.0, np.nan, 7.5, 3.0, np.nan, 9.0],
    "practice": [12, 20, np.nan, 8, 15, 30],
    "track": ["Python", "Data", np.nan, "Web", "Data", "Data"],
})
from sklearn.impute import SimpleImputer

numbers = survey[["hours", "practice"]]
mean_imputer = SimpleImputer(strategy="mean")
print(mean_imputer.fit_transform(numbers))
print("means:", mean_imputer.statistics_.round(2))
print("medians:", SimpleImputer(strategy="median").fit(numbers).statistics_)

Output:

text

[[ 5.    12.   ]
 [ 6.125 20.   ]
 [ 7.5   17.   ]
 [ 3.     8.   ]
 [ 6.125 15.   ]
 [ 9.    30.   ]]
means: [ 6.12 17.  ]
medians: [ 6.25 15.  ]

Fill text with the most frequent value

Text has no average, so use most_frequent, or constant with a fill_value such as "unknown" when a gap means something on its own.

python

import numpy as np
import pandas as pd

survey = pd.DataFrame({
    "hours": [5.0, np.nan, 7.5, 3.0, np.nan, 9.0],
    "practice": [12, 20, np.nan, 8, 15, 30],
    "track": ["Python", "Data", np.nan, "Web", "Data", "Data"],
})
from sklearn.impute import SimpleImputer

common = SimpleImputer(strategy="most_frequent").fit_transform(survey[["track"]])
print(common.ravel())

labelled = SimpleImputer(strategy="constant", fill_value="unknown").fit_transform(survey[["track"]])
print(labelled.ravel())

Output:

text

['Python' 'Data' 'Data' 'Web' 'Data' 'Data']
['Python' 'Data' 'unknown' 'Web' 'Data' 'Data']

Remember where the gaps were

Sometimes the fact that a value is missing is itself useful. add_indicator=True appends a 0 or 1 column for every feature that had gaps, so the model can learn from that too.

python

import numpy as np
import pandas as pd

survey = pd.DataFrame({
    "hours": [5.0, np.nan, 7.5, 3.0, np.nan, 9.0],
    "practice": [12, 20, np.nan, 8, 15, 30],
    "track": ["Python", "Data", np.nan, "Web", "Data", "Data"],
})
from sklearn.impute import SimpleImputer

imputer = SimpleImputer(strategy="median", add_indicator=True)
print(imputer.fit_transform(survey[["hours", "practice"]]))
print(imputer.get_feature_names_out())

Output:

text

[[ 5.   12.    0.    0.  ]
 [ 6.25 20.    1.    0.  ]
 [ 7.5  15.    0.    1.  ]
 [ 3.    8.    0.    0.  ]
 [ 6.25 15.    1.    0.  ]
 [ 9.   30.    0.    0.  ]]
['hours' 'practice' 'missingindicator_hours' 'missingindicator_practice']

As with scalers, fit the imputer on training rows only and use it to transform the test rows, so test values never shape the fill numbers.

Key takeaways

isna().sum() counts missing cells per column
Most models raise an error when X contains NaN
SimpleImputer fills gaps with the mean, median, most frequent value, or a constant
statistics_ holds the fill value learned for each column
add_indicator=True keeps a record of where values were missing
Next: Pipelines
Previous: Encoding Categories

Loading playground…

Check your understanding

Missing Values Quiz

How does pandas show a missing number?

1 / 10

← Encoding Categories

Pipelines →