Univariate Linear Regression in Python

Introduction

Does x predict y?

This is the basic question that linear regression aims to answer, or at least give a hint about. Technically speaking, linear regression is a way of establishing if two variables are related.

In this post we need to be familiar with the idea of both dependent and independent variables. Generally, the dependent variable or "y" is the variable that we are measuring (it can help also to frame this as the outcome). The independent variable or "x" is the variable that is modified or changed. If these two variables are at all correlated, a change in the independent variable should result in a somewhat reliable change in the dependent variable.

For example, lets say we are interested in how rainfall effects umbrella usage, we could hypothesize that the more it rains, the more likely people are to use an umbrella. In this case our independent variable is rainfall lets quantify that as mm per day. Our dependent variable is umbrella usage  and we can choose to quantify this as number of umbrellas seen per day on some particular street. It seems obvious that people would be much more likely to use an umbrella when it is raining, but lets say you actually ran this study for a month and eventually you came up with some data and plotted it on a scatterplot like so:

As you can clearly see, more rain generally equals more umbrellas. We could easily state that a correlation exists between rainfall and umbrella usage. 

If an increase in the independent variable leads to an increase in the dependent variable, it is called a positive correlation. If an increase in the independent variable leads to a decrease in the dependent variable it is called a negative correlation.

This is all fine and dandy when dealing with two variables that are so obviously related. However, when dealing with larger, less clearly related data, researchers needed a way to quantify this relationship, and objectively test if two variables are truly correlated. That is why many techniques including linear regression were developed.

Rather than writing at length about the specific details of how linear regression works, something that has been explained on countless other websites, I would encourage you to read about it somewhere online, say here and then return to the tutorial once you have a solid understanding.

Writing Code

To begin, we will need some example data to work with. Ive chosen the cricket chirps vs degrees Fahrenheit from this lovely website. All I have done is download the excel spreadsheet version and saved it as a .csv file with the name cricket_data.csv and made the headings a tad more informative. You can download the sample data here

It is generally a good idea when working with any new dataset to first visualize it. This could easily be done in excel, but lets instead do it in Python.

	import pandas
	import matplotlib.pyplot as plt
	# import the data into a pandas dataframe
	data = pandas.read_csv('cricket_data.csv')
	# print the first for headings of the data to terminal
	print(data.head())
	# set x to the chirps per second column
	x = data['chirps_per_second']
	# set y to the temperature column
	y = data['temperature']
	# graph as a scatterplot
	plt.scatter(x,y)
	# set axis labels
	plt.xlabel('chirps_per_second')
	plt.ylabel('temperature')
	# display plot
	plt.show()

view raw Visualize_Data.py hosted with ❤ by GitHub

Result:

It certainly looks like there is a positive correlation between these two variables, so lets import sk-learn and try to quantify this relationship with linear regression.

	from sklearn import linear_model as lm
	# create our linear regression model
	classifier = lm.LinearRegression()
	# fit our model
	classifier.fit(x,y)
	# get r^2 score
	score = classifier.score(x,y)
	#print r^2 score
	print("R^2: " + score.__str__())

view raw further_linear_regression.py hosted with ❤ by GitHub

After running the above code it reported an R^2 score of 0.69 .  Pretty good considering the scale of the data. You can read more about R^2 or the coefficient of determination here. 

Getting More Information

We could easily stop here, but for completeness-sake lets gather some more information about the regression.

Coefficient

Getting the coefficient is simple but a little bit hidden in the way sklearn implements linear regression:

	# fit our model
	results = classifier.fit(x,y)
	# store coefficient
	coefficient = results.coef_

view raw save_coef.py hosted with ❤ by GitHub

Line of Best-Fit

If we want to plot the line of best fit we will need two values, the coefficient and the intercept:

	import math
	# fit our model
	results = classifier.fit(x,y)
	# store coefficient
	coefficient = results.coef_
	# store intercept
	intercept = results.intercept_
	# get min and max values
	x_min = math.floor(x.min())
	x_max = math.ceil(x.max()) + 1
	# get y values for best fit line
	best_fit_line = [coefficient * i + intercept for i in range(x_min,x_max)]
	# plot the original x values and the best fit y values
	plt.plot(range(x_min,x_max),best_fit_line, 'r')
	plt.show()

view raw line_of_best_fit.py hosted with ❤ by GitHub

Result:

Pearson Correlation Coefficient

We will calculate the Pearson Correlation Coefficient and the p-value (odds of obtaining this result by chance) in Scipy as sklearn does not support it.

	from scipy import stats as sp
	# convert dataframes to lists
	x_list = list(x.values.flatten())
	y_list = list(y.values.flatten())
	# get pearson results
	pearson_results = sp.pearsonr(x_list, y_list)
	# extract correlation coefficient
	cor_coef = pearson_results[0]
	# extract p-value
	p_value = pearson_results[1]

view raw pearson_values.py hosted with ❤ by GitHub

Conclusion

In this blog post we learned the basics of single-variable regression. Much of this translates closely to multivariate regression which I will post about if their is significant interest.

Source Code and Project Repository:

Available here