SciPy Statistics
The scipy.stats module helps you describe data and calculate probabilities.
Describing Data
stats.describe() gives several summary numbers in one call. Here are the test marks of ten students:
Example
import numpy as np
from scipy import stats
marks = np.array([65, 72, 78, 80, 85, 88, 90, 92, 72, 95])
d = stats.describe(marks)
print("Count:", d.nobs)
print("Min and max:", d.minmax)
print("Mean:", d.mean)
print("Variance:", round(d.variance, 2))Count: 10 Min and max: (np.int64(65), np.int64(95)) Mean: 81.7 Variance: 98.46
| Measure | What it tells you |
|---|---|
| Mean | The average |
| Variance | How spread out the values are (in squared units) |
| Skewness | Whether the data leans to one side |
| Kurtosis | How heavy the tails are compared to a normal curve |
Mode, Skewness and Spread
Example
import numpy as np
from scipy import stats
marks = np.array([65, 72, 78, 80, 85, 88, 90, 92, 72, 95])
print(stats.mode(marks).mode)
print(round(stats.skew(marks), 3))
print(stats.iqr(marks))
print(round(stats.sem(marks), 3))72 -0.268 16.0 3.138
- The mode is 72, the most common mark
- The skewness is slightly negative, so a few low marks pull the tail to the left
- The IQR (interquartile range) is the spread of the middle 50% of marks
- The SEM (standard error of the mean) shows how precise the average is
Z-Scores and Percentiles
A z-score tells you how many standard deviations a value is above or below the mean. It lets you compare values from different tests.
Example
import numpy as np
from scipy import stats
scores = np.array([60, 70, 80, 90])
print(np.round(stats.zscore(scores), 2))
marks = np.array([65, 72, 78, 80, 85, 88, 90, 92, 72, 95])
print(stats.percentileofscore(marks, 85))[-1.34 -0.45 0.45 1.34] 60.0
A mark of 85 is at the 60th percentile, so it is equal to or better than 60% of the class.
Probability Distributions
A distribution describes how likely different values are. SciPy has more than 100 of them. The most famous is the normal distribution, the bell curve.
| Method | Question it answers |
|---|---|
pdf(x) | How dense the probability is at x (continuous data) |
pmf(k) | Probability of exactly k (counting data) |
cdf(x) | Probability of a value less than or equal to x |
ppf(p) | The value below which a fraction p of the data falls |
rvs(size) | Random values from the distribution |
The Normal Distribution
Say the heights of adult men in a city have a mean of 168 cm and a standard deviation of 7 cm. What fraction is shorter than 175 cm? And what height do only 5% of men exceed?
Example
from scipy.stats import norm
print(round(norm.cdf(175, loc=168, scale=7), 4))
print(round(1 - norm.cdf(180, loc=168, scale=7), 4))
print(round(norm.ppf(0.95, loc=168, scale=7), 1))0.8413 0.0432 179.5
About 84% are shorter than 175 cm, only about 4% are taller than 180 cm, and the top 5% start at about 179.5 cm.
loc is the mean and scale is the standard deviation. Without them, SciPy uses the standard normal curve, with mean 0 and standard deviation 1.
The Binomial Distribution
Use the binomial distribution for "how many successes in n tries". A batsman scores a boundary off a ball with probability 0.2. In an over of 6 balls, what is the chance of exactly 2 boundaries, and of at least 1?
Example
from scipy.stats import binom
print(round(binom.pmf(2, n=6, p=0.2), 4))
print(round(1 - binom.pmf(0, n=6, p=0.2), 4))0.2458 0.7379
Exercise
Exercise
Which method gives the probability that a value is less than or equal to x?