Remove rows meeting some criteria in a Pandas DataFrame

Python

Pandas

DataFrame

Remove specific rows

Luc B.

Python

Pandas

DataFrame

When doing data analysis, it's common to remove certain rows from a dataset to focus on a subset of the data. In Pandas, there are many ways to do this including indexing and the DataFrame.drop() method. While Pandas's indexing infrastruction is good enough for most selection use cases, more advanced selection operations occasionaly require DataFrame.drop().

Code Example

Use the Pandas indexing system to select for rows meeting some criteria. In this example, we narrow a dataset to just the students attending Hogwarts.

import pandas as pd
df = pd.read_csv('students.csv')
df
student GPA school
0 harry 3.1 hogwarts
1 victor 3.8 durmstrang
2 fluer 3.9 beauxbatons
3 hermione 4.0 hogwarts
4 ron 2.7 hogwarts
5 draco 3.5 hogwarts
6 max 2.8 durmstrang
7 sofia 2.9 beauxbatons
8 maxine 3.7 beauxbatons
9 neville 3.2 hogwarts
# Select for the Hogwarts students
df[df['school'] == 'hogwarts']
student GPA school
0 harry 3.1 hogwarts
3 hermione 4.0 hogwarts
4 ron 2.7 hogwarts
5 draco 3.5 hogwarts
9 neville 3.2 hogwarts

This works because the conditional code, df['school'] == 'hogwarts', creates a boolean array:

df['school'] == 'hogwarts'
0     True
1    False
2    False
3     True
4     True
5     True
6    False
7    False
8    False
9     True
Name: school, dtype: bool

And when you index into a DataFrame with a boolean array, all rows that correspond to a value of False are removed.

More Examples

Use DataFrame.drop()

For more advanced selections, it is sometimes necessary to use the DataFrame.drop() method, passing it the indicies of the rows you want to remove. In this example, we remove the two Hogwarts students with the highest GPAs from the DataFrame (Hermione and Draco).

# Remove the two Hogwarts students with the highest GPA's from the DataFrame
df.drop(
    df[df['school'] == 'hogwarts']
        .sort_values('GPA', ascending=False)[0:2]
        .index
)
student GPA school
0 harry 3.1 hogwarts
1 victor 3.8 durmstrang
2 fluer 3.9 beauxbatons
4 ron 2.7 hogwarts
6 max 2.8 durmstrang
7 sofia 2.9 beauxbatons
8 maxine 3.7 beauxbatons
9 neville 3.2 hogwarts