Pandas Selecting and Filtering

Selecting and filtering means picking out only the rows and columns you need from a DataFrame.

The Example Data

All examples in this lesson use this small table of orders:

Example

python

import pandas as pd

df = pd.DataFrame({
    "city": ["Karachi", "Lahore", "Karachi", "Islamabad", "Karachi", "Lahore"],
    "product": ["Cover", "Charger", "Earbuds", "Cover", "Charger", "Earbuds"],
    "qty": [2, 1, 1, 3, 2, 1],
    "price": [450, 1200, 3500, 450, 1200, 3500]
})
print(df)

Output

        city  product  qty  price
0    Karachi    Cover    2    450
1     Lahore  Charger    1   1200
2    Karachi  Earbuds    1   3500
3  Islamabad    Cover    3    450
4    Karachi  Charger    2   1200
5     Lahore  Earbuds    1   3500

loc: Select by Label

df.loc[rows, columns] uses index labels and column names. With loc, the end of a slice is included.

Example

python

print(df.loc[0, "city"])
print(df.loc[1:3, ["city", "qty"]])

Output

Karachi
        city  qty
1     Lahore    1
2    Karachi    1
3  Islamabad    3

iloc: Select by Position

df.iloc[rows, columns] uses position numbers. Like Python slices, the end is not included.

Example

python

print(df.iloc[0, 1])
print(df.iloc[0:2, 0:2])
print(df.iloc[-1])

Output

Cover
      city  product
0  Karachi    Cover
1   Lahore  Charger
city        Lahore
product    Earbuds
qty              1
price         3500
Name: 5, dtype: object

	`loc`	`iloc`
Works with	Labels and names	Position numbers
Slice end	Included	Not included
Example	`df.loc[0:2, "city"]`	`df.iloc[0:2, 0]`

Filtering Rows With a Condition

Put a condition inside the brackets. Pandas keeps the rows where it is True:

Example

python

print(df[df["city"] == "Karachi"])
print(df[df["price"] > 1000])

Output

      city  product  qty  price
0  Karachi    Cover    2    450
2  Karachi  Earbuds    1   3500
4  Karachi  Charger    2   1200
      city  product  qty  price
1   Lahore  Charger    1   1200
2  Karachi  Earbuds    1   3500
4  Karachi  Charger    2   1200
5   Lahore  Earbuds    1   3500

Combining Conditions

Use & for "and", the pipe sign for "or" and ~ for "not". Wrap each condition in brackets.

Example

python

print(df[(df["city"] == "Karachi") & (df["qty"] >= 2)])
print(df[~(df["city"] == "Karachi")])

Output

      city  product  qty  price
0  Karachi    Cover    2    450
4  Karachi  Charger    2   1200
        city  product  qty  price
1     Lahore  Charger    1   1200
3  Islamabad    Cover    3    450
5     Lahore  Earbuds    1   3500

The brackets around each condition are required. Without them, Python mixes up the order of operations and gives an error.

isin(): Match Any of Several Values

Example

python

print(df[df["city"].isin(["Lahore", "Islamabad"])])

Output

        city  product  qty  price
1     Lahore  Charger    1   1200
3  Islamabad    Cover    3    450
5     Lahore  Earbuds    1   3500

Filter and Pick Columns Together

loc can filter rows and choose columns in one step:

Example

python

print(df.loc[df["product"] == "Earbuds", ["city", "price"]])

Output

      city  price
2  Karachi   3500
5   Lahore   3500

query(): Filter With a Text Expression

query() lets you write the condition as a string, which some people find easier to read:

Example

python

print(df.query("qty > 1 and price < 1000"))

Output

        city product  qty  price
0    Karachi   Cover    2    450
3  Islamabad   Cover    3    450

Exercise

Which code keeps only the rows where the column "city" is "Lahore"?

df["city" == "Lahore"]df[df["city"] == "Lahore"]df.loc["Lahore"]df.filter("Lahore")