Distance Decay | Lenka's blog

Proximity and Distance Decay, chapter in GIS&T Body of Knowledge

Wed, 18 May 2022 00:00:00 +0000

Modelling Urban Flows, Spatial Effects in Origin-Destination data

Wed, 22 Jul 2020 00:00:00 +0000

Application of XGBoost regression for Spatial Interaction of Urban flow

Wed, 27 May 2020 00:00:00 +0000

Introduction

This notebook provides a simple example of the application of XGBoost Spatial Interaction model in python. For the purposes of this tutorial I will expect you to have some knowledge of Spatial Interaction Models and their purpose as well as some python skills. Yet, if you want to refresh a little or wish to learn, here is a list of relevant sources.

Watch Maarten Vanhoof talking about what are the spatial Interaction Models and how they can be used here (6 min)
Have a look at a list of some relevant literature in different fields here
Look at the example of Spatial Interaction Model application in R from Adam Dennett Part1 and Part2
Look at how you can apply the the same method in python with SpInt package

So what is the XGBoost? and how does it work?

XGBoost stands for “Extreme Gradient Boosting”, where the term “Gradient Boosting” originates from the paper Greedy Function Approximation: A Gradient Boosting Machine, by Friedman. Please have a look at this A Gentle Introduction to XGBoost for Applied Machine Learning and the Full documentation to XGBoost that also provides a tutorials. If you are like me and need something more visual and efficient explanation of the how the model works, I would highly recommend YouTube videos from Josh Starmer; Gradient Boost Part 1: Regression Main Ideas, Gradient Boost Part 2: Regression Details.

So after you gone through all the resources you know that there are two types of the model depending on what do you want to do with them. You can use them as a classifier, similar to decision trees, or as a regressor. Using the method as a classifier, we can predict discrete and categorical variables, and using it as a regressors, continuous variables can be predicted. Here we will be talking solely about Gradient Boosting as a regressor, because we are trying to model Urban flow, number of people that moves from one point to another.

What is an Urban flow here?

In this example, I am looking at patients Urban flow. to be specific, this is a patients that are being treated at particular GP, receive a prescription and then go to one of the pharmacies to pick up the prescribed drug. This process of prescribing and dispensing a drug is closely monitored by the National Health Service, although the precise information about the patients is disclosed, the NHS Digital provides completely anonymised data of number of the prescriptions that has been prescribed and dispensed between each pair of GP and Pharmacy each month. Those can be translated as traces of patients movement between two points and together creates rich network of patients flow.

For further description of the data follow up on this notebook.

Cross validation challenge

With any ordinary data, we would usually divide the data into testing set and training set by 80/20 split or any other proportion. However, we face more difficult case here.

The Urban flow networks are very particular case of network, which is basically a representation of the human behaviour in a space. And this is related to anything that affect our decision to choose a destination and the journey to it. Starting from maps, road or public transport structure and the type of vehicle, ending with our personal preference, gender, or family status. The network that the humans created by acting in it has very complex pattern in which the existence of a destination and its location matters.

Now, imagine we would divide the dataset into training and testing, how do we decide which locations will be for training and which ones will be for testing if each location is unique? For example, if we train the model on the flow data in a Scotland, we cannot test the same model on the data from London as people in both places has different behavioural patterns. There is more to it, read through this study that provides nice overview.

Luckily, the data I collected ranges from 2013 to mid-2019, which means that we can divide the data into blocks with the same timeframe. In this case I will use the data from 2017 for training and data from 2018 for testing. This block cross validation is only possible thanks to the consistent data.

Let's get started

Load the libraries

import xgboost as xgb # xgboost model library
import numpy as np # array manipulation
import pandas as pd # dataframe manipulation library
import geopandas as gpd # spatial data frame manipulatioon library
import statsmodels.api as sm
from sklearn.model_selection import train_test_split  # take out the validation packages from scikit.learn
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import mean_squared_error
import matplotlib.pyplot as plt # plotting 
import seaborn as sns

Load the 2017 train data

# define the path
file = r'./../.././data/NHS/working/year_data_flow/predicted_flow_2017.csv'
# read the flow file
fl17 = pd.read_csv(file, index_col=None)

# define the path
file = r'./../.././data/NHS/working/year_data_flow/complete_predicted_flow_2017.csv'
# read the flow file
cfl17 = pd.read_csv(file, index_col=None)

fl17.head()

	P_Code	P_Name_x	P_Postcode	D_Code_x	D_Name_x	D_Postcode	N_Items	Date	loop	...	P_full_address_y	P_Status_Code_y	P_Sub_type_y	P_Setting_y	P_X_y	P_Y_y	CODE	POSTCODE	NUMBER_OF_PATIENTS	yhat
0	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FA251	HORIZON PHARMACY LTD	BS34 6AS	130.0	2017-01-01	False	...	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18363	0.044989
1	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FA613	DOWNHAM FB LTD	BS7 9JT	39.0	2017-01-01	False	...	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18363	0.240777
2	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FCL91	MYGLOBE LTD	BS15 4ND	12.0	2017-01-01	False	...	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18363	0.001761
3	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FCT35	LLOYDS PHARMACY LTD	BS20 7QA	661.0	2017-01-01	False	...	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18363	2393.873899
4	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FDD77	JOHN WARE LIMITED	BS20 6LT	4058.0	2017-01-01	False	...	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18363	2149.320471

5 rows × 77 columns

Load the 2018 test data

# first get the 018 data in, those will be the test data
file = r'./../.././data/NHS/working/year_data_flow/predicted_flow_2018.csv'
# read the flow file
fl18 = pd.read_csv(file, index_col=None)

# define the path
file = r'./../.././data/NHS/working/year_data_flow/complete_predicted_flow_2018.csv'
# read the flow file
cfl18 = pd.read_csv(file, index_col=None)

cfl18.head()

	P_Code	D_Code_new	Date	Density	NUMBER_OF_PATIENTS	N_Items	Euc_distance	great_circle_dist	predicted
0	L81004	FA696	2018-01-01	36.696000	18615	8.0	0.224446	16.560437	0.000777
1	L81004	FCT35	2018-01-01	34.918367	18615	21.0	0.009198	0.653229	1963.680982
2	L81004	FDD77	2018-01-01	29.092308	18615	5309.0	0.014101	0.983999	1673.379704
3	L81004	FD125	2018-01-01	28.670000	18615	26.0	0.177172	12.816035	0.030142
4	L81004	FEA59	2018-01-01	30.833333	18615	1.0	0.292934	20.302017	0.000028

I have already selected the columns we need for the analysis but let me provide you with some explanation.

The gravity equation says that the flow, can be explained by gravitational power of two masses on origin and destination, and inversely proportional to the distance between them.

$$ F_{od} = k\frac{M_o M_d}{D_{od}} $$

Where $F_{od}$ is the flow mass, $M_o$ is the mass on the origin, $M_d$ is mass on the destination and $D_{od}$ is a distance between them.

In our case the flow is the number of patient's between the origin (General Practitioner) and the destination (Pharmacy). The mass of the origin is represented by the number of patients registered at that particular GP in that year (mid-year estimate). The mass of the destination is the average daytime population in 1km radius from the pharmacy. And the distance is a great circle distance between the origin and destination. All of those has been calculated previously and I'm going to bore you with it another time.

In our datasets this is;

	Variable
$F_{od}$	‘N_Items’
$M_o$	‘NUMBER_OF_PATIENTS’
$M_d$	‘Density’
$D_{od}$	‘great_circle_dist’

There are different spatial interaction models but for this purpose we only want to stick with the very basic form, one based on gravity model. We just need to translate the formula into an regression, Poisson regression to be more specific, because the flow is a count of the patients and skewed dis

Spatial Interaction model definition

$$F_{od} = \exp( k + \mu \ln{M_o} + \alpha \ln{M_d} − \beta \ln{ D_{od}} )$$

Where $\lambda_{od}$ is the flow from $o$ to $d$, logarithmically linked or modelled by a linear combination of the logged independent vars. $\kappa ; \mu ; \alpha$ and $\beta$ are parameters to be estimated, $M_o$ is the mass on origin, $M_d$ is the mass on destination and $D_{od}$ is the distance between origin and destination.

this is in log-normal equation;

$$\ln{F_{od}} = k + \mu \ln{M_d} + \alpha \ln{M_d} − \beta \ln{ D_{od}}$$

and Poisson equation, which should have the same result;

$$F_{od} = k + \mu \ln{M_o} + \alpha \ln{M_d} − \beta \ln{ D_{od}}$$

Let's try both

log-normal model
Poisson model, correct definition

The XGBOOST offers

objective = “reg:squarederror”
objective = “count:poisson”

Note the regressor takes a all the data from data frame as they are, so we need to apply the transformations before we build to model as I understand.

# log all the variables
fl17['log_N_Items'] = np.log(fl17['N_Items'])
fl18['log_N_Items'] = np.log(fl18['N_Items'])

# log all the variables
fl17['1*great_circle_dist'] = 1*fl17['great_circle_dist']
fl18['1*great_circle_dist'] = 1*fl18['great_circle_dist']

# Do we need to do the 1-Distance???

fl17['log_NUMBER_OF_PATIENTS'] = np.log(fl17['NUMBER_OF_PATIENTS'])
fl18['log_NUMBER_OF_PATIENTS'] = np.log(fl18['NUMBER_OF_PATIENTS'])

fl17['log_Density'] = np.log(fl17['Density'])
fl18['log_Density'] = np.log(fl18['Density'])

# do the same thing for the complete data

# log all the variables
cfl17['1*great_circle_dist'] = 1*cfl17['great_circle_dist']
cfl18['1*great_circle_dist'] = 1*cfl18['great_circle_dist']

# Do we need to do the 1-Distance???

cfl17['log_NUMBER_OF_PATIENTS'] = np.log(cfl17['NUMBER_OF_PATIENTS'])
cfl18['log_NUMBER_OF_PATIENTS'] = np.log(cfl18['NUMBER_OF_PATIENTS'])

cfl17['log_Density'] = np.log(cfl17['Density'])
cfl18['log_Density'] = np.log(cfl18['Density'])

fl17log = fl17.iloc[:,[77,78,79,80]]
fl18log = fl18.iloc[:,[77,78,79,80]]

fl18log

	log_N_Items	1*great_circle_dist	log_NUMBER_OF_PATIENTS	log_Density
0	2.079442	16.560437	9.831723	3.602668
1	3.044522	0.653229	9.831723	3.553013
2	8.577159	0.983999	9.831723	3.370474
3	3.258097	12.816035	9.831723	3.355851
4	0.000000	20.302017	9.831723	3.428596
...	...	...	...	...
71729	0.000000	0.973280	6.907755	5.144711
71730	0.000000	1.015876	6.907755	4.824431
71731	0.693147	1.507605	6.907755	4.806714
71732	1.098612	1.739077	6.907755	4.549942
71733	0.693147	1.578409	6.907755	4.358013

71734 rows × 4 columns

fl17log.hist(figsize = (10,10));

Log-normal XGBoost SIM model

Parameters

booster = ‘dart’

XGBoost mostly combines a huge number of regression trees with a small learning rate. In this situation, trees added early are significant and trees added late are unimportant. Vinayak and Gilad-Bachrach proposed a new method to add dropout techniques from the deep neural net community to boosted trees and reported better results in some situations. Drop trees to solve the over-fitting. Trivial trees (to correct trivial errors) may be prevented. Because of the randomness introduced in the training, expect the following few differences: Training can be slower than gbtree because the random dropout prevents usage of the prediction buffer. The early stop might not be stable, due to the randomness.

Follow here to see all the parameters.

I also found this tutorial which has used the grid search from ‘scikit’ cross validation to test several versions of the model and it's parameters. This comes very handy, as we can instantly find the best performing model in instance. Maybe this can help to explain a bit more about that.

Learning rate = Step size shrinkage used in update to prevents overfitting. After each boosting step, we can directly get the weights of new features, and eta shrinks the feature weights to make the boosting process more conservative.

max_depth = Maximum depth of a tree. Increasing this value will make the model more complex and more likely to overfit. 0 is only accepted in lossguided growing policy when tree_method is set as hist and it indicates no limit on depth. Beware that XGBoost aggressively consumes memory when training a deep tree. Range: [0,∞] (0 is only accepted in lossguided growing policy when tree_method is set as hist)

Note

Below, you can see an example for grid search for the best model within certain parameters. It was very efficient when I was working with the log-normal model only, however, using it within the Poisson version, the model converges after third of all the runs (depends how many options you give it)

# define the training and testing dataset
X_train, y_train = fl17log.loc[:,fl17log.columns != 'log_N_Items'], fl17log.loc[:,['log_N_Items']]
X_test, y_test = fl18log.loc[:,fl18log.columns != 'log_N_Items'], fl18log.loc[:,['log_N_Items']]

# transfer those into matrix as recommended class by in the package
DM_train = xgb.DMatrix(data = X_train, 
                       label = y_train)
					   
DM_test =  xgb.DMatrix(data = X_test,
                       label = y_test)

# define the parameters that we want to inspect
gbm_param_grid = {
     'colsample_bytree': np.linspace(0.5, 0.9), # define fairly wide range
     'n_estimators':[15, 20], # try different numbers
     'max_depth': [ 15, 20] # different depths
}

# define the regressor model
gbm = xgb.XGBRegressor(objective = "reg:squarederror", booster = 'dart')

# define the grid search cross validation 
grid_mse = GridSearchCV(estimator = gbm, param_grid = gbm_param_grid, scoring = 'neg_mean_squared_error', cv = 5, verbose = 1)

# fit the models and go through all the parameters
grid_mse.fit(X_train, y_train)

# print the best results
print("Best parameters found: ",grid_mse.best_params_)
print("Lowest RMSE found: ", np.sqrt(np.abs(grid_mse.best_score_)))

# define the training and testing dataset
X_train, y_train = fl17log.loc[:,fl17log.columns != 'log_N_Items'], fl17log.loc[:,['log_N_Items']]
X_test, y_test = fl18log.loc[:,fl18log.columns != 'log_N_Items'], fl18log.loc[:,['log_N_Items']]

# define the model (I already tested several version and stacked with the default setting and learning rate 0.4, the grid search is work in progress)
gbm = xgb.XGBRegressor(objective = "reg:squarederror", learning_rate= 0.4).fit(X_train, y_train)

pred = gbm.predict(X_test)

print("Root mean square error for test dataset: {}".format(np.round(np.sqrt(mean_squared_error(np.exp(y_test), np.exp(pred))), 2)))

Root mean square error for test dataset: 949.98

test = pd.DataFrame({"prediction": np.exp(pred), "observed": np.exp(y_test['log_N_Items'])})

lowess = sm.nonparametric.lowess

z = lowess(np.exp(pred).flatten(), np.exp(y_test['log_N_Items']))


test.plot(figsize = [14,8],
          x ="prediction", y = "observed", kind = "scatter", color = 'darkred')

plt.title("Extreme Gradient Boosting: Prediction Vs Test Data", fontsize = 18, color = "darkgreen")

plt.xlabel("Predicted Power Output", fontsize = 18) 

plt.ylabel("Observed Power Output", fontsize = 18)

plt.plot(z[:,0], z[:,1], color = "blue", lw= 3)

plt.show()

fl17['log'] =  np.exp(gbm.predict(X_train))

fl18['log'] =  np.exp(gbm.predict(X_test))

Poisson XGBoost SIM model

# create new dataset for the Poisson
# this will be logged masses, 1*dist and no change to y
fl17poi = fl17.iloc[:,[6,78,79,80]]
fl18poi = fl18.iloc[:,[6,78,79,80]]
cfl17poi = cfl17.iloc[:,[9,10,11,5]]
cfl18poi = cfl18.iloc[:,[9,10,11,5]]

fl17poi.hist(figsize = (10,10));
cfl17poi.hist(figsize = (10,10));

For published data only

# define the training and testing dataset
X_train2, y_train2 = fl17poi.loc[:,fl17poi.columns != 'N_Items'], fl17poi.loc[:,['N_Items']]
X_test2, y_test2 = fl18poi.loc[:,fl18poi.columns != 'N_Items'], fl18poi.loc[:,['N_Items']]

# define the model
gbm2 = xgb.XGBRegressor(objective = "count:poisson",  learning_rate= 0.4).fit(X_train2, y_train2)

For complete data with all the 0 flows in

# define the training and testing dataset
X_train3, y_train3 = cfl17poi.loc[:,cfl17poi.columns != 'N_Items'], cfl17poi.loc[:,['N_Items']]
X_test3, y_test3 = cfl18poi.loc[:,cfl18poi.columns != 'N_Items'], cfl18poi.loc[:,['N_Items']]

gbm3 = xgb.XGBRegressor(objective = "count:poisson",  learning_rate= 0.4).fit(X_train3, y_train3)

Best parameters found: {‘colsample_bytree’: 0.8, ‘max_depth’: 40, ‘n_estimators’: 30} Lowest RMSE found: 1159.4761503247805

pred2 = gbm2.predict(X_test2)

print("Root mean square error for test dataset: {}".format(np.round(np.sqrt(mean_squared_error(y_test2, pred2)), 2)))

pred3 = gbm3.predict(X_test3)

print("Root mean square error for test dataset: {}".format(np.round(np.sqrt(mean_squared_error(y_test3, pred3)), 2)))

Root mean square error for test dataset: 704.17
Root mean square error for test dataset: 279.73

sns.regplot(np.exp(pred), np.exp(y_test['log_N_Items']), color = 'blue',line_kws = {'color':'red'})
plt.title('Predicted vs actual, Log-normal XGBOOST, published data only', fontsize=20);

sns.regplot(pred2, y_test2['N_Items'], color = 'blue',line_kws = {'color':'red'})
plt.title('Predicted vs actual, Poisson XGBOOST, published data only', fontsize=20);

sns.regplot(pred3, y_test3['N_Items'], color = 'blue',line_kws = {'color':'red'})
plt.title('Predicted vs actual, Poisson XGBOOST, complete data', fontsize=20);

The Poisson seems to be better when using the default parameter setting and learning rate 0.4

We already have the in sample and out of sample predictions from gravity model in the data, so now we need to add the in and out of sample prediction from the XGB model

# published
fl17['xgb'] =  gbm2.predict(X_train2)

fl18['xgb'] =  gbm2.predict(X_test2)

# complete
cfl17['xgb'] =  gbm2.predict(X_train3)

cfl18['xgb'] =  gbm2.predict(X_test3)

fl18.head()

	P_Code	P_Name_x	P_Postcode	D_Code_x	D_Name_x	D_Postcode	N_Items	Date	loop	...	CODE	POSTCODE	NUMBER_OF_PATIENTS	predicted	log_N_Items	1*great_circle_dist	log_NUMBER_OF_PATIENTS	log_Density	log	xgb
0	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FA696	THE BULLEN HEALTHCARE GROUP LIMITED	BS32 4JT	8.0	2018-01-01	False	...	L81004	BS20 6AQ	18615	0.021203	2.079442	16.560437	9.831723	3.602668	3.172393	2.952730
1	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FCT35	LLOYDS PHARMACY LTD	BS20 7QA	21.0	2018-01-01	False	...	L81004	BS20 6AQ	18615	2417.864485	3.044522	0.653229	9.831723	3.553013	163.713943	861.965210
2	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FDD77	JOHN WARE LIMITED	BS20 6LT	5309.0	2018-01-01	False	...	L81004	BS20 6AQ	18615	2170.860226	8.577159	0.983999	9.831723	3.370474	833.031494	3730.304199
3	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FD125	BOOTS UK LIMITED	BS34 5UP	26.0	2018-01-01	False	...	L81004	BS20 6AQ	18615	0.390401	3.258097	12.816035	9.831723	3.355851	6.255939	5.463578
4	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FEA59	LLOYDS PHARMACY LTD	BS16 7AE	1.0	2018-01-01	False	...	L81004	BS20 6AQ	18615	0.001570	0.000000	20.302017	9.831723	3.428596	3.263482	1.575869

5 rows × 83 columns

The accuracy measure

Lets organize this into a table;

	R2 Gravity SIM	R2 XGBOOST SIM	RMSE Gravity SIM	RMSE XGBOOST SIM
Models based on published data	PGIr2	PXIr2	PGIrs	PXIrs
	PGOr2	PXOr2	PGOrs	PXOrs
Models based on complete data	CGIr2	CXIr2	CGIrs	CXIrs
	CGOr2	CXOr2	CGOrs	CXOrs

# define R2 function
def get_r2_numpy_corrcoef(x, y):
    return np.corrcoef(x, y)[0, 1]**2

# define RMSE function
def rmse(predictions, targets):
    return np.sqrt(((predictions - targets) ** 2).mean())

Other evaluations?

Some is available at Robinsons github, but requires matrices.

See to definition of the Common Part of Commuters measure. Source article

def cpc(actual, predicted):
    
    # reshape the variables 
    predicted = np.reshape(np.array(predicted), (-1, 1))
    actual = np.reshape(np.array(actual),(-1, 1))
    # stack them together
    YYhat = np.hstack([actual, predicted])
    # define sum of the minimym of the predicted var
    NCC = np.sum(np.min(YYhat, axis=1)) # , axis=1
    # number of observations
    N = len(actual)
    # sum of all the actual numbers (reshaped)
    NCY = np.sum(actual)
    # sum of the predicted numbers (reshaped)
    NCYhat = np.sum(predicted)
    
    numerator = (N * NCC)
    denominator = (NCY + NCYhat)
    
    return numerator/denominator

$$ \frac{ N_i * \sum \min(ij) }{ \sum_i + \sum_j } $$

def sorensen(actual, predicted):
    # reshape the variables
    predicted = np.reshape(np.array(predicted), (-1, 1))
    actual = np.reshape(np.array(actual),(-1, 1))
    # define length of the variables
    N =  len(actual)
    # stack them together in array
    YYhat = np.hstack([actual, predicted])
    # define 2 times minimum from both
    num = 2.0 * np.min(YYhat, axis=1)
    # define the differnece
    den =  predicted + actual
           
    return (1.0 / N) * (np.sum(num.reshape((-1, 1)) / den.reshape((-1, 1))))

$$ \frac{1}{N} * \frac{\sum{2 * \min(i,j)} }{j-i} $$

Let's do the result tables again with those measures

	R2 Gravity SIM	R2 XGBOOST SIM	RMSE Gravity SIM	RMSE XGBOOST SIM	CPC Gravity SIM	CPC XGBOOST SIM	SORENSEN I Gravity	SORENSEN I XGBOOST
Models based on published data	PGIr2	PXIr2	PGIrs	PXIrs	PGIcpc	PXIcpc	PGIsn	PXIsn
	PGOr2	PXOr2	PGOrs	PXOrs	PGOcpc	PXOcpc	PGOsn	PXOsn
Models based on complete data	CGIr2	CXIr2	CGIrs	CXIrs	CGIcpc	CXIcpc	CGIsn	CXIsn
	CGOr2	CXOr2	CGOrs	CXOrs	CGOcpc	CXOcpc	CGOsn	CXOsn

# R2
# published data
PGIr2 = round(get_r2_numpy_corrcoef(fl17['N_Items'], fl17['yhat']),3)
PGOr2 = round(get_r2_numpy_corrcoef(fl18['N_Items'], fl18['predicted']),3)

PXIr2 = round(get_r2_numpy_corrcoef(fl17['N_Items'], fl17['xgb']),3)
PXOr2 = round(get_r2_numpy_corrcoef(fl18['N_Items'], fl18['xgb']),3)

# complete data
CGIr2 = round(get_r2_numpy_corrcoef(cfl17['N_Items'], cfl17['yhat']),3)
CGOr2 = round(get_r2_numpy_corrcoef(cfl18['N_Items'], cfl18['predicted']),3)

CXIr2 = round(get_r2_numpy_corrcoef(cfl17['N_Items'], cfl17['xgb']),3)
CXOr2 = round(get_r2_numpy_corrcoef(cfl18['N_Items'], cfl18['xgb']),3)

# RMSE

PGIrs = round(rmse(fl17['N_Items'], fl17['yhat']),1)
PGOrs = round(rmse(fl18['N_Items'], fl18['predicted']),1)

PXIrs = round(rmse(fl17['N_Items'], fl17['xgb']),1)
PXOrs = round(rmse(fl18['N_Items'], fl18['xgb']),1)

CGIrs = round(rmse(cfl17['N_Items'], cfl17['yhat']),1)
CGOrs = round(rmse(cfl18['N_Items'], cfl18['predicted']),1)

CXIrs = round(rmse(cfl17['N_Items'], cfl17['xgb']),1)
CXOrs = round(rmse(cfl18['N_Items'], cfl18['xgb']),1)

# CPC

PGIcpc = round(cpc(fl17['N_Items'], fl17['yhat']),1)
PGOcpc = round(cpc(fl18['N_Items'], fl18['predicted']),1)

PXIcpc = round(cpc(fl17['N_Items'], fl17['xgb']),1)
PXOcpc = round(cpc(fl18['N_Items'], fl18['xgb']),1)

CGIcpc = round(cpc(cfl17['N_Items'], cfl17['yhat']),1)
CGOcpc = round(cpc(cfl18['N_Items'], cfl18['predicted']),1)

CXIcpc = round(cpc(cfl17['N_Items'], cfl17['xgb']),1)
CXOcpc = round(cpc(cfl18['N_Items'], cfl18['xgb']),1)

# Sorensen 

PGIsn = round(sorensen(fl17['N_Items'], fl17['yhat']),3)
PGOsn = round(sorensen(fl18['N_Items'], fl18['predicted']),3)

PXIsn = round(sorensen(fl17['N_Items'], fl17['xgb']),3)
PXOsn = round(sorensen(fl18['N_Items'], fl18['xgb']),3)

CGIsn = round(sorensen(cfl17['N_Items'], cfl17['yhat']),3)
CGOsn = round(sorensen(cfl18['N_Items'], cfl18['predicted']),3)

CXIsn = round(sorensen(cfl17['N_Items'], cfl17['xgb']),3)
CXOsn = round(sorensen(cfl18['N_Items'], cfl18['xgb']),3)

# everything for the log-normal model
LXIr2 = round(get_r2_numpy_corrcoef(fl17['N_Items'], fl17['log']),3)
LXOr2 = round(get_r2_numpy_corrcoef(fl18['N_Items'], fl18['log']),3)

LXIrs = round(rmse(fl17['N_Items'], fl17['log']),1)
LXOrs = round(rmse(fl18['N_Items'], fl18['log']),1)

LXIcpc = round(cpc(fl17['N_Items'], fl17['log']),1)
LXOcpc = round(cpc(fl18['N_Items'], fl18['log']),1)

LXIsn = round(sorensen(fl17['N_Items'], fl17['log']),3)
LXOsn = round(sorensen(fl18['N_Items'], fl18['log']),3)

model_name = ['In-sample','Out-sample']
mG = [PGIr2, PGOr2]
mX = [PXIr2, PXOr2]
mF = [PGIrs, PGOrs]
mH = [PXIrs, PXOrs]
mD1 = [PGIcpc, PGOcpc]
mD2 = [PXIcpc, PXOcpc]
mK1 = [PGIsn, PGOsn]
mK2 = [PXIsn, PXOsn]

cols = {'Poisson models based on published data': model_name,'R2 Gravity SIM': mG, 'R2 XGBOOST SIM': mX, 'RMSE Gravity SIM': mF, 'RMSE XGBOOST SIM': mH, 'CPC Gravity SIM': mD1,
      'CPC XGBOOST SIM': mD2,  'SORENSEN I Gravity': mK1, 'SORENSEN I XGBOOST': mK2}

table_published = pd.DataFrame(cols).set_index('Poisson models based on published data')

model_name = ['In-sample','Out-sample']
G = [CGIr2, CGOr2]
X = [CXIr2, CXOr2]
mF = [CGIrs, CGOrs]
mH = [CXIrs, CXOrs]
mD1 = [CGIcpc, CGOcpc]
mD2 = [CXIcpc, CXOcpc]
mK1 = [CGIsn, CGOsn]
mK2 = [CXIsn, CXOsn]

cols = {'Poisson models based on complete data': model_name,'R2 Gravity SIM': G, 'R2 XGBOOST SIM': X, 'RMSE Gravity SIM': mF, 'RMSE XGBOOST SIM': mH,  'CPC Gravity SIM': mD1,
      'CPC XGBOOST SIM': mD2,  'SORENSEN I Gravity': mK1, 'SORENSEN I XGBOOST': mK2}

table_complete = pd.DataFrame(cols).set_index('Poisson models based on complete data')

NA = 'NA'
model_name = ['In-sample','Out-sample']
mG = [NA, NA]
mX = [LXIr2, LXOr2]
mF = [NA, NA]
mH = [LXIrs, LXOrs]
mD1 = [NA, NA]
mD2 = [LXIcpc, LXOcpc]
mK1 = [NA, NA]
mK2 = [LXIsn, LXOsn]

cols = {'Log-normal models based on published data': model_name,'R2 Gravity SIM': mG, 'R2 XGBOOST SIM': mX, 'RMSE Gravity SIM': mF, 'RMSE XGBOOST SIM': mH, 'CPC Gravity SIM': mD1,
      'CPC XGBOOST SIM': mD2,  'SORENSEN I Gravity': mK1, 'SORENSEN I XGBOOST': mK2}

table_log = pd.DataFrame(cols).set_index('Log-normal models based on published data')

table_log

	R2 Gravity SIM	R2 XGBOOST SIM	RMSE Gravity SIM	RMSE XGBOOST SIM	CPC Gravity SIM	CPC XGBOOST SIM	SORENSEN I Gravity	SORENSEN I XGBOOST
Log-normal models based on published data
In-sample	NA	0.560	NA	809.9	NA	25504.1	NA	0.686
Out-sample	NA	0.381	NA	950.0	NA	21690.5	NA	0.620

# models based on published data only
table_published

	R2 Gravity SIM	R2 XGBOOST SIM	RMSE Gravity SIM	RMSE XGBOOST SIM	CPC Gravity SIM	CPC XGBOOST SIM	SORENSEN I Gravity	SORENSEN I XGBOOST
Poisson models based on published data
In-sample	0.321	0.736	976.2	611.1	17813.1	27736.8	0.318	0.513
Out-sample	0.330	0.653	981.4	704.2	17207.5	24790.0	0.314	0.490

# models based on complete data with all the zero flows included
table_complete

	R2 Gravity SIM	R2 XGBOOST SIM	RMSE Gravity SIM	RMSE XGBOOST SIM	CPC Gravity SIM	CPC XGBOOST SIM	SORENSEN I Gravity	SORENSEN I XGBOOST
Poisson models based on complete data
In-sample	0.287	0.691	380.6	249.9	106667.5	168990.5	0.045	0.070
Out-sample	0.297	0.605	385.3	288.8	100976.7	145788.9	0.046	0.068

x = table_log.append(table_published)
x = x.append(table_complete)
x['model_type'] = ['log-normal', 'log-normal', 'Poisson', 'Poisson', 'Poisson', 'Poisson']
x['data_type'] = ['published_only', 'published_only', 'published_only', 'published_only', 'complete', 'complete']
x['CV'] = ['In-sample', 'Out-sample','In-sample', 'Out-sample','In-sample', 'Out-sample']
x = x.set_index(['model_type', 'data_type'])
x

		R2 Gravity SIM	R2 XGBOOST SIM	RMSE Gravity SIM	RMSE XGBOOST SIM	CPC Gravity SIM	CPC XGBOOST SIM	SORENSEN I Gravity	SORENSEN I XGBOOST	CV
model_type	data_type
log-normal	published_only	NA	0.560	NA	809.9	NA	25504.1	NA	0.686	In-sample
published_only	NA	0.381	NA	950.0	NA	21690.5	NA	0.620	Out-sample
Poisson	published_only	0.321	0.736	976.2	611.1	17813.1	27736.8	0.318	0.513	In-sample
published_only	0.33	0.653	981.4	704.2	17207.5	24790.0	0.314	0.490	Out-sample
complete	0.287	0.691	380.6	249.9	106668	168990.5	0.045	0.070	In-sample
complete	0.297	0.605	385.3	288.8	100977	145788.9	0.046	0.068	Out-sample

x.iloc[[0,2,4],:]['R2 XGBOOST SIM'].plot() 
x.iloc[[1,3,5],:]['R2 XGBOOST SIM'].plot()
plt.title('R2 XGBOOST', fontsize=20)
plt.legend(['In-sample', 'Out-sample'])
plt.xlabel(' ');

x.iloc[[0,2,4],:]['RMSE XGBOOST SIM'].plot()
x.iloc[[1,3,5],:]['RMSE XGBOOST SIM'].plot()
plt.title('RMSE  XGBOOST', fontsize=20)
plt.legend(['In-sample', 'Out-sample'])
plt.xlabel(' ');

x.iloc[[0,2,4],:]['CPC XGBOOST SIM'].plot()
x.iloc[[1,3,5],:]['CPC XGBOOST SIM'].plot()
plt.title('CPC XGBOOST', fontsize=20)
plt.legend(['In-sample', 'Out-sample'])
plt.xlabel(' ');

x.iloc[[0,2,4],:]['SORENSEN I XGBOOST'].plot()
x.iloc[[1,3,5],:]['SORENSEN I XGBOOST'].plot()
plt.title('SORENSEN XGBOOST', fontsize=20)
plt.legend(['In-sample', 'Out-sample'])
plt.xlabel(' ');

Sorensen similarity Index is bounded between values of 0 and 1 with values closer to 1 indicating a better model fit.

How come that the complete dataset has exceptionally low index, because of the zero issue?

cfl17.loc[:,['N_Items', 'xgb']]

	N_Items	xgb
0	130.0	11.200489
1	39.0	7.248779
2	12.0	3.174812
3	661.0	861.965210
4	4058.0	3730.304199
...	...	...
543739	0.0	8.880725
543740	0.0	5.127942
543741	0.0	5.786361
543742	0.0	7.492306
543743	0.0	18.535973

543744 rows × 2 columns

Prob it is caused by the excessive zeros, we would probably need to introduce an ensemble model to deal with it.

Some other is available from Tylor Oshan He has CPC as well as Sorensen index

Comparing the graph structures of the patients flow

Thu, 26 Mar 2020 00:00:00 +0000

# we will first import all necessary libraries
import pandas as pd
import os
import numpy as np
import glob
from datetime import datetime
import geopandas as gpd
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px
from shapely.geometry import Point, LineString
#from mpl_toolkits.basemap import Basemap
import osmnx as ox
import networkx as nx
import matplotlib as mpl
from networkx.algorithms import bipartite as bi
import statsmodels.api as sm
from scipy import stats
from statsmodels.formula.api import glm

wgs84 = {'init':'EPSG:4326'}
bng = {'init':'EPSG:27700'}

Prescription flow model validation

Using graphs to detect changes in spatial structure of the prescription flow

Using a graphs to look at the properties of the flow network, we need to first think about what kind of graph the network actually is and what kind of connections it has.

Based on the number of edges between two points we recognize

Simple graph - Undirected graph with no loops and one edge between two points
Multiple graph - Undirected graph with no loops
Pseudograph - Graph with loops

Based on the type of the link we recognise;

Undirected graph - the flow does not have a direction
Directed graph - the flow have a direction, flow from point A to point B

We also recognize;

Incomplete graph - graph in which at least one set of nodes is missing connection
Complete graph - graph in which all the nodes are connected to all the other nodes

Lastly, we can also recognize very special case of a graph called Bipartite. What is Bipartiv graph? Recently, “bipartivity” has been proposed as an important topological characteristic of complex networks. A network (graph) G = (V,E) is called bipartite if its vertex set V can be partitioned into two subsets V1 and V2 such that all edges have one endpoint in V1 and the other in V2.

The GP and pharmacy flow are in nature Directed Incomplete pseudograph. that is because the flow goes from GP to pharmacy, always one direction, it does not contain the flows between all the pairs and also contains some loops; Gp can also dispense small amount odf prescriptions. Nevertheles, as the loops are very small proportoin of the data and we can benefit from the graph as a bipartite, I have attached dummy GPs into a pharmacy points so we can treat the loops as normal flows.

NOTE: what is the implication of this? We are loosing the information about connection of dome flows to origin, however, we are gaining an information on the process which reminds zero-inflated or hurdle process;

Step 1 - patient come to GP but leaves without a prescriprion
Step 2 - patient come to GP get the prescription virtually and recieve the drug from GP directly
Step 3 - patient come to GP, get prescription, go to nearest pharmacy to get the drug
Step 4 - patient come to GP, get the prescription and go to pharmacy close to shopping opportunity to get the drug
Step 5 - patient come to GP, get the prescription and go to other pharmacy to get the drug

How do we represent the graph in Networkx?

Here are types of graphs in the networkx package

Networkx Class	Type	Self-loops allowed	Parallel edges allowed
Graph	undirected	Yes	No
DiGraph	directed	Yes	No
MultiGraph	undirected	Yes	Yes
MultiDiGraph	directed	Yes	Yes

We know our graph is directed so Digraph should serve the purpose. However, the data consist from monthly prescription flow so for most of the nodes we can find 12 edges throughout the year, so we need to work with MultiDiGraph to make sure we can account for all of those.

1. Load the data

# load the output of the gravity models, the flow data
# set the path
file = r'./../../data/NHS/working/predicted_flow_17.csv'
# read the file
flow17 = pd.read_csv(file, index_col=None)
# plot the top of the table
flow17.head()

	P_Code	P_Name_x	P_Postcode	D_Code_x	D_Name_x	D_Postcode	N_Items	Date	loop	...	D_Left_y	D_X_y	D_Y_y	D_E_y	D_N_y	D_Setting_y	residents	Density	predict_items	predict_items_doubly
0	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FA251	HORIZON PHARMACY LTD	BS34 6AS	130.0	2017-01-01	False	...	20180115.0	-2.562191	51.539195	361105.872423	182404.906640	50	456.517857	36.592857	29.316590	10.581379
1	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FA613	DOWNHAM FB LTD	BS7 9JT	39.0	2017-01-01	False	...	NaN	-2.582219	51.480445	359664.895464	175881.865127	50	249.144737	54.493421	29.473430	2.853510
2	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FCL91	MYGLOBE LTD	BS15 4ND	12.0	2017-01-01	False	...	NaN	-2.478870	51.459944	366826.940029	173549.785593	50	249.128205	30.569231	24.155556	2.061345
3	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FCT35	LLOYDS PHARMACY LTD	BS20 7QA	661.0	2017-01-01	False	...	20180131.0	-2.758272	51.484474	347443.775309	176441.826480	50	369.081633	34.918367	1648.893267	1078.144522
4	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FDD77	JOHN WARE LIMITED	BS20 6LT	4058.0	2017-01-01	False	...	NaN	-2.781349	51.481409	345837.744901	176117.839266	50	391.076923	29.092308	1194.123265	3570.014826

5 rows × 72 columns

# create a lists of the unique codes so we can subset the GP and PH point data
list_flow_GP = list(flow17['P_Code'].unique())
list_flow_PH = list(flow17['D_Code_new'].unique())

# get the pharmacies in
fl = './../../data/NHS/edispensary/PH_joined_daytimepop.geojson'
PH = gpd.read_file(fl)
PH = PH.to_crs(wgs84)
PH = PH[PH['D_Code_new'].isin(list_flow_PH)]
PH.plot();

C:\Users\tk18583\AppData\Local\Continuum\anaconda3\envs\geonet\lib\site-packages\pyproj\crs\crs.py:53: FutureWarning:

'+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6

# get the GP's points in
GP = gpd.read_file("./../../data/NHS/epraccur/GP_AVON.geojson")
GP = GP[['P_Code','P_Name', 'full_address', 'P_Postcode', 'Open_date', 
                                         'Close_date', 'Status_Code', 'Sub_type', 'Join', 'left', 'Setting', 'X', 'Y', 'geometry']]
GP = GP.rename(columns={'P_Postcode':'P_Postcode2','Open_date': 'P_Open_date', 'Close_date':'P_Close_date','full_address':'P_full_address' ,'Status_Code':'P_Status_Code', 'Sub_type':'P_Sub_type', 'Join':'P_Join', 'left':'P_Left', 'Setting':'P_Setting', 'X':'P_X', 'Y':'P_Y'}, errors="raise")
GP = GP.to_crs(wgs84)
GP = GP[GP['P_Code'].isin(list_flow_GP)]
GP.plot();

C:\Users\tk18583\AppData\Local\Continuum\anaconda3\envs\geonet\lib\site-packages\pyproj\crs\crs.py:53: FutureWarning:

'+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6

2. Create a graph

We now know that the graph is bipartite, has multiple edges and has a directions, which means that we need to create MultiDiGraph. We also have weight on edge, that is the number of prescriptions on flow and predicted items on flow for both of the used models.

G1 = nx.from_pandas_edgelist(flow17, source='P_Code', target='D_Code_new', edge_attr=['N_Items'], create_using=nx.MultiDiGraph())
print('Is the graph bipirtite?   ' + str(nx.is_bipartite(G1)))
print('Is the graph directed??   ' + str(nx.is_directed(G1)))

Is the graph bipirtite?   True
Is the graph directed??   True

G2 = nx.from_pandas_edgelist(flow17, source='P_Code', target='D_Code_new', edge_attr=['predict_items'], create_using=nx.MultiDiGraph())
print('Is the graph bipirtite?   ' + str(nx.is_bipartite(G2)))
print('Is the graph directed??   ' + str(nx.is_directed(G2)))

Is the graph bipirtite?   True
Is the graph directed??   True

G3 = nx.from_pandas_edgelist(flow17, source='P_Code', target='D_Code_new', edge_attr=['predict_items_doubly'], create_using=nx.MultiDiGraph())
print('Is the graph bipirtite?   ' + str(nx.is_bipartite(G3)))
print('Is the graph directed??   ' + str(nx.is_directed(G3)))

Is the graph bipirtite?   True
Is the graph directed??   True

3. Look at the properties of the graph

# Look at number on graph properties
nb_nodes = len(G1.nodes())
nb_arr = len(G1.edges())
print("Number of nodes : " + str(nb_nodes))
print("Number of edges : " + str(nb_arr))

Number of nodes : 485
Number of edges : 74668

degree_freq = np.array(nx.degree_histogram(G1)).astype('float')
plt.figure(figsize=(12, 8))
plt.stem(degree_freq)
plt.ylabel("Frequence")
plt.xlabel("Degre")
plt.show();

C:\Users\tk18583\AppData\Local\Continuum\anaconda3\envs\geonet\lib\site-packages\ipykernel_launcher.py:3: UserWarning:

In Matplotlib 3.3 individual lines on a stem plot will be added as a LineCollection instead of individual lines. This significantly improves the performance of a stem plot. To remove this warning and switch to the new behaviour, set the "use_line_collection" keyword argument to True.

#degree_freq[degree_freq > 0]
x = degree_freq[degree_freq < 80]

plt.figure(figsize=(12, 8))
plt.stem(x)
plt.ylabel("Frequence")
plt.xlabel("Degre")
plt.show();

C:\Users\tk18583\AppData\Local\Continuum\anaconda3\envs\geonet\lib\site-packages\ipykernel_launcher.py:2: UserWarning:

In Matplotlib 3.3 individual lines on a stem plot will be added as a LineCollection instead of individual lines. This significantly improves the performance of a stem plot. To remove this warning and switch to the new behaviour, set the "use_line_collection" keyword argument to True.


plt.figure(figsize=(14,14))
nx.draw_networkx(G1, with_labels=True, width=0.02)

4. Define the functions to calculate centrality and snap the resluts to the point data

The Bipirtate graph is very specific cas of graph and has its own dedicated centraliy functions that can be calculated with networkx package.

This is the closeness and degree centrality, however none of those contain any weight, they are only based on the number of flows coming into the nodes.

No centrality can be measured in weighted bipirtite graph

Other measures we can look at are the Link Analysis measures. Those are Page Rank and HIIT (hubs and authorities). However, only Page Rank include a weight on edges,

Page Rank is the only measure we can apply on weighted directed bipartite graph!

def close_bi(graph, nodes, name, joinsA):
    # calculate the closenest
    close = bi.closeness_centrality(G = graph, nodes = nodes)
    # put it into a dataframe 
    x = pd.DataFrame.from_dict(close,  orient='index')
    x = x.reset_index()
    x = x.rename({0: name},axis = 1)

    df = joinsA.merge(x, left_on = 'P_Code', right_on = 'index', how = 'left')
    return df

def degree_bi(graph, nodes, name, joinsA, joinsB):
    # calculate the closenest
    close = bi.degree_centrality(G = graph, nodes = nodes)
    # put it into a dataframe 
    x = pd.DataFrame.from_dict(close,  orient='index')
    x = x.reset_index()
    x = x.rename({0: name},axis = 1)

    df = joinsA.merge(x, left_on = 'P_Code', right_on = 'index', how = 'left')
    df2 = joinsB.merge(x, left_on = 'D_Code_new', right_on = 'index', how = 'left')
    return df, df2

def pgr(graph, name, joinsA, joinsB, weight):
    # calculate the closenest
    close = nx.pagerank_scipy(G = graph, weight = weight, max_iter = 50)
    # put it into a dataframe 
    x = pd.DataFrame.from_dict(close,  orient='index')
    x = x.reset_index()
    x = x.rename({0: name},axis = 1)

    df = joinsA.merge(x, left_on = 'P_Code', right_on = 'index', how = 'left')
    df2 = joinsB.merge(x, left_on = 'D_Code_new', right_on = 'index', how = 'left')
    return df, df2

def pgr(graph, name, joinsA, joinsB, weight):
    # calculate the closenest
    close = nx.pagerank_scipy(G = graph, weight = weight, max_iter = 50)
    # put it into a dataframe 
    x = pd.DataFrame.from_dict(close,  orient='index')
    x = x.reset_index()
    x = x.rename({0: name},axis = 1)

    df = joinsA.merge(x, left_on = 'P_Code', right_on = 'index', how = 'left')
    df2 = joinsB.merge(x, left_on = 'D_Code_new', right_on = 'index', how = 'left')
    return df, df2

# eigenvector centrality does not work for multidigraphs
def eigen_ce(graph, name, joinsA, joinsB, weight):
    # calculate the closenest
    close = nx.eigenvector_centrality(G = graph, max_iter = 100, weight = weight)
    # put it into a dataframe 
    x = pd.DataFrame.from_dict(close,  orient='index')
    x = x.reset_index()
    x = x.rename({0: name},axis = 1)

    df = joinsA.merge(x, left_on = 'P_Code', right_on = 'index', how = 'left')
    df2 = joinsB.merge(x, left_on = 'D_Code_new', right_on = 'index', how = 'left')
    return df, df2

5. Apply the functions for Bipartate graph

GP2 = close_bi(graph = G1, nodes = list_flow_GP, name = 'close_G1', joinsA = GP)
GP2 = close_bi(graph = G2, nodes = list_flow_GP, name = 'close_G2', joinsA = GP2)
GP2 = close_bi(graph = G3, nodes = list_flow_GP, name = 'close_G3', joinsA = GP2)

GP2, PH2 = degree_bi(graph = G1, nodes = list_flow_GP, name = 'degree_G1', joinsA = GP2, joinsB = PH)
GP2, PH2 = degree_bi(graph = G2, nodes = list_flow_GP, name = 'degree_G2', joinsA = GP2, joinsB = PH2)
GP2, PH2 = degree_bi(graph = G3, nodes = list_flow_GP, name = 'degree_G3', joinsA = GP2, joinsB = PH2)

GP2, PH2 =  pgr(graph = G1, name = 'pgr_G1', weight = 'N_Items', joinsA = GP2, joinsB = PH2)
GP2, PH2 =  pgr(graph = G2, name = 'pgr_G2', weight = 'predict_items', joinsA = GP2, joinsB = PH2)
GP2, PH2 =  pgr(graph = G3, name = 'pgr_G3', weight = 'predict_items_doubly', joinsA = GP2, joinsB = PH2)

# eigenvector centrality does not work for multidigraphs
#GP2, PH2 =  eigen_ce(graph = G1, name = 'eig_G1', weight = 'N_Items', joinsA = GP2, joinsB = PH2)
#GP2, PH2 =  eigen_ce(graph = G2, name = 'eig_G2', weight = 'predict_items', joinsA = GP2, joinsB = PH2)
#GP2, PH2 =  eigen_ce(graph = G3, name = 'eig_G3', weight = 'predict_items_doubly', joinsA = GP2, joinsB = PH2)

GP2.drop(['index_x','index_y' ,'index'], axis=1, inplace=True)
PH2.drop(['index_x','index_y'], axis=1, inplace=True)

PH2.info()

<class 'geopandas.geodataframe.GeoDataFrame'>
Int64Index: 357 entries, 0 to 356
Data columns (total 25 columns):
 #   Column          Non-Null Count  Dtype   
---  ------          --------------  -----   
 0   D_Code          357 non-null    object  
 1   D_Name          357 non-null    object  
 2   D_full_address  357 non-null    object  
 3   D_Postcode2     357 non-null    object  
 4   D_Open_date     357 non-null    int64   
 5   D_Close_date    27 non-null     float64 
 6   D_Status_Code   357 non-null    object  
 7   D_Sub_type      357 non-null    object  
 8   D_Join          356 non-null    float64 
 9   D_Left          27 non-null     float64 
 10  D_X             357 non-null    float64 
 11  D_Y             357 non-null    float64 
 12  D_E             357 non-null    float64 
 13  D_N             357 non-null    float64 
 14  D_Setting       357 non-null    int64   
 15  D_Code_new      357 non-null    object  
 16  residents       357 non-null    float64 
 17  Density         357 non-null    float64 
 18  geometry        357 non-null    geometry
 19  degree_G1       357 non-null    float64 
 20  degree_G2       357 non-null    float64 
 21  degree_G3       357 non-null    float64 
 22  pgr_G1          357 non-null    float64 
 23  pgr_G2          357 non-null    float64 
 24  pgr_G3          357 non-null    float64 
dtypes: float64(15), geometry(1), int64(2), object(7)
memory usage: 72.5+ KB

# look at the values
GP2.iloc[:,14:23].head()

	close_G1	close_G2	close_G3	degree_G1	degree_G2	degree_G3	pgr_G1	pgr_G2	pgr_G3
0	1.262397	1.262397	1.262397	1.456583	1.456583	1.456583	0.001684	0.001684	0.001684
1	1.262397	1.262397	1.262397	2.207283	2.207283	2.207283	0.001684	0.001684	0.001684
2	1.262397	1.262397	1.262397	1.834734	1.834734	1.834734	0.001684	0.001684	0.001684
3	1.262397	1.262397	1.262397	1.733894	1.733894	1.733894	0.001684	0.001684	0.001684
4	1.262397	1.262397	1.262397	0.806723	0.806723	0.806723	0.001684	0.001684	0.001684

PH2.iloc[:,19:25].head()

	degree_G1	degree_G2	degree_G3	pgr_G1	pgr_G2	pgr_G3
0	2.140625	2.140625	2.140625	0.002018	0.001812	0.001996
1	1.687500	1.687500	1.687500	0.002194	0.001957	0.002179
2	0.648438	0.648438	0.648438	0.001689	0.001774	0.001689
3	2.937500	2.937500	2.937500	0.002471	0.002330	0.002497
4	3.773438	3.773438	3.773438	0.001700	0.001867	0.001718

6. Changing graph structure

We can see that the degree centrality and closeness centraility is the same for all the graphs, because the wight is not included.

The Pagerank for the GP's is not changing either as the flows are leaving them and page rank focuses on the incoming flows only.

The Page Rank algorithm computes a ranking of the nodes in the graph G based on the structure of the incoming links, so we can only observe the importance of pharmacies.

plt.figure(figsize=(30, 6))
# Degree Centrality
f, axarr = plt.subplots(1,3, num=1)

plt.sca(axarr[0])
PH2['pgr_G1'].plot.hist(grid=True, bins=20, rwidth=0.9, color='#607c8e')
axarr[0].set_title('Page Rank Graph 1', size=16)

plt.sca(axarr[1])
PH2['pgr_G2'].plot.hist(grid=True, bins=20, rwidth=0.9, color='#607c8e')
axarr[1].set_title('Page Rank Graph 2', size=16)

plt.sca(axarr[2])
PH2['pgr_G3'].plot.hist(grid=True, bins=20, rwidth=0.9, color='#607c8e')
axarr[2].set_title('Page Rank Graph 3', size=16);

# which one is the most important? the one that sticks out
PH2[PH2['pgr_G1'] > 0.0044]

	D_Code	D_Name	D_full_address	D_Postcode2	D_Open_date	D_Close_date	D_Status_Code	D_Sub_type	D_Join	D_Left	...	D_Code_new	residents	Density	geometry	degree_G1	degree_G2	degree_G3	pgr_G1	pgr_G2	pgr_G3
112	FLQ56	BOOTS UK LIMITED	BOOTS UK LIMITED,59 BROADMEAD,BRISTOL,nan,BS1 3ED	BS1 3ED	19480101	NaN	A	1	20040401.0	NaN	...	FLQ56	1021.5	202.028571	POINT (-2.58972 51.45774)	8.898438	8.898438	8.898438	0.004406	0.002691	0.003961

1 rows × 25 columns

ax = plt.figure(figsize=(10, 10))
plt.axis([PH2['pgr_G1'].min(), PH2['pgr_G1'].max(), PH2['pgr_G1'].min(), PH2['pgr_G1'].max()])
# plots
plt.scatter(PH2['pgr_G1'],PH2['pgr_G1'])
plt.scatter(PH2['pgr_G1'],PH2['pgr_G2'])
plt.plot(PH2['pgr_G1'].mean(),PH2['pgr_G1'].mean(), c = 'r', marker = 'X', markersize = 30)
# regression plots
sns.regplot(PH2['pgr_G1'],PH2['pgr_G1'])
sns.regplot(PH2['pgr_G1'],PH2['pgr_G2'])

plt.legend(['Mean','Page Rank from data', 'Page Rank from base gravity']);

PH2['pgr_G1'].mean()

0.002197291434997281

ax = plt.figure(figsize=(10, 10))
plt.axis([PH2['pgr_G1'].min(), PH2['pgr_G1'].max(), PH2['pgr_G1'].min(), PH2['pgr_G1'].max()])
# plots
plt.scatter(PH2['pgr_G1'],PH2['pgr_G1'])
plt.scatter(PH2['pgr_G1'],PH2['pgr_G3'])
plt.plot(PH2['pgr_G1'].mean(),PH2['pgr_G1'].mean(), c = 'r', marker = 'X', markersize = 30)
# regression plots
sns.regplot(PH2['pgr_G1'],PH2['pgr_G1'])
sns.regplot(PH2['pgr_G1'],PH2['pgr_G3'])

plt.legend(['Mean','Page Rank from data', 'Page Rank from doubly gravity model' ]);

Results

What we can see on the figure above is how the node centrality in the graph conctructed from prediction relates to a node centrality in the graph constructed from raw data. In ideal situation, the line of best fit and the observations would follow the blue line, however, this is not the case. From the first figure, which looks at the importance of the nodes based on base gravity model (very simple one), you can observe that where the actual importannce of the node is high, the importance based on predicted flows drop down drastically, and where the actual importance of the node is low, the importance based on predicted flows is higher. You can also notice line of zeros trying to hide behind the y axes. So although the model predicts flow with certain accuracy, th quality of the flow is not the same as from the data. More specifically the model is not really able to see the hierarchical structure of the nodes and creates predictions that are close to mean for all nodes.

Not suprisingly this is not the case for more complex gravity model such as doubly constrained gravity model. Here the hierarchy of the nodes is sustained by the prediction, however, that is expected as each node has it's own parameter in the model. Seems like a simple and smart solution, however, you need to remember the with doubly constrained model, where each node is considered as unique, predictions can be made only to existing nodes. So question like ‘What will happen with the flows if one of the GP's closes’ are hardly answerable. It's like a cheeky shortcut; if I am not able to meodel complicated flow structure, let's model each node separately.

Drop me a message on twitter if you have any comments :)

Inter-National Migration; Part 2 - Calculating Distances

Thu, 09 Jan 2020 00:00:00 +0000

This is a second part of Intra-National Migration data exploration. This post will also construt distance variables for the data and crack on with some Spatial Interction Modeling. This is a contribution to my PhD research that examines Origin-Destination and its spatial aspects.

Introduction

UK intra-migration data are open source data that can be downloaded from ONS. ONS uses the NHS Patient Register Data Service (PRDS) to find out the changes in patients adresses. Since most people change their address with their doctor soon after moving, these data are considered to provide a good proxy indicator of migration.

The migration data is published every year and consist of Origin Local Authority ID, Destination Local Authority ID, sex, age and movement factor field. This movement factor is based on resscaled number of people on the flow.

Please follow ONS to find more information about the methodology.

Follow Part 1 to see some basic graphs of the data.

Recap

I'm just going to quickly recap what we know alrady about the data.

The origins and destinations are the same, UK Local Authorities.
The flow volume is represented by movement factor wich is already scaled variable and is extremely skewed to the left.
The year 2011 has very different nature from all the other years.
Because we are not able to judstify the difference in nature of the 2011, we are going to work with the later years only.

In this post…

I want to calculate distances between the Originn and Destination.

Now, it might sound easy, but the theory behind this is more colmpicated than you would think. Read through my other post that explains most of the theory behind this.

I need to;

Calculate the Eclidian Distance between the Origins and Destinations
Calculate the Great Circle Distance between the Origins and Destinations
Calculate the Network Distance (walking path along the streets)

1. Calculate the Euclidian Distance between the Origins and Destinations

To do that, we first need to load the libraries and the data we mungedin the post 1

# we will first import all necessary libraries
import pandas as pd
import os
import numpy as np
import glob
from datetime import datetime
import geopandas as gpd
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px
from shapely.geometry import Point, LineString

# set the path
file = r'./../data/migration/full_mig.csv'
# read the file
df = pd.read_csv(file, index_col=None, low_memory=False)
# plot the top of the table
df.head()

	OutLA	InLA	Age	Sex	Moves	Date	sex_bin
0	E09000001	E09000002	27	2	0.0002	2011-01-01	2
1	E09000001	E09000002	36	1	0.0004	2011-01-01	1
2	E09000001	E09000002	26	2	0.0004	2011-01-01	2
3	E09000001	E09000002	25	2	0.0005	2011-01-01	2
4	E09000001	E09000002	34	1	0.0014	2011-01-01	1

# set the path
file = r'./../data/migration/munged/LAcentrooids.shp'
# read the file in
la = gpd.read_file(file)
# show the map
la.plot();

Now I can create lat and long columns in the centrooid data which I'll merge to the flow data twice. First based on the Out-flow LA ID and second based on In-flow LA ID.

# create new callumn x and y wit lat and long coordinates
la['x'] = la.geometry.x
la['y'] = la.geometry.y
la.head()

	objectid	lad17cd	lad17nm	geometry	x	y
0	1	E06000001	Hartlepool	POINT (-1.259479894328456 54.66947264221945)	-1.259480	54.669473
1	2	E06000002	Middlesbrough	POINT (-1.222265950359884 54.54199624640567)	-1.222266	54.541996
2	3	E06000003	Redcar and Cleveland	POINT (-1.020534867661266 54.55159000727666)	-1.020535	54.551590
3	4	E06000004	Stockton-on-Tees	POINT (-1.332307567288941 54.56157879321239)	-1.332308	54.561579
4	5	E06000005	Darlington	POINT (-1.552585765196361 54.54869927572448)	-1.552586	54.548699

# lets get rid of the rows we dont have spatial information for
# list of Unique ID's
d = list(la['lad17cd'].unique())
# new dataset with just those that are in a list
df2 = df.loc[df['OutLA'].isin(d)]
df3 = df2.loc[df2['InLA'].isin(d)]
# see how many rows we lost
len(df) - len(df3)

# create two datasets from the la's base on Outflow and Inflow
# drop th geometries and create dataframes
outla = pd.DataFrame(la)
inla = pd.DataFrame(la)
# select just what we need
outla = outla.loc[:,['lad17cd','x','y']]
inla = inla.loc[:,['lad17cd','x','y']]
# rename the columns
outla.rename(columns = {'x':'out_x','y':'out_y'}, inplace = True)
inla.rename(columns = {'x':'in_x','y':'in_y'}, inplace = True)
# look at what we have
outla.head()

	lad17cd	out_x	out_y
0	E06000001	-1.259480	54.669473
1	E06000002	-1.222266	54.541996
2	E06000003	-1.020535	54.551590
3	E06000004	-1.332308	54.561579
4	E06000005	-1.552586	54.548699

# join the spatial points to the data, first for the OutLA and secondly for the InLA
df4 = df3.merge(outla, how = 'left', left_on = 'OutLA', right_on = 'lad17cd')
df4 = df4.merge(inla, how = 'left', left_on = 'InLA', right_on = 'lad17cd')
df4.tail()

	OutLA	InLA	Age	Sex	Moves	Date	sex_bin	lad17cd_x	out_x	out_y	lad17cd_y	in_x	in_y
10701098	W06000024	W06000023	67	F	1.1416	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737
10701099	W06000024	W06000023	73	F	0.9519	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737
10701100	W06000024	W06000023	74	M	1.0020	2018-01-01	1	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737
10701101	W06000024	W06000023	74	F	0.9866	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737
10701102	W06000024	W06000023	77	F	3.0000	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737

# make sure there is no missing values
x = df4.isna()
x.loc[x['out_x'] == True]

	OutLA	InLA	Age	Sex	Moves	Date	sex_bin	lad17cd_x	out_x	out_y	lad17cd_y	in_x	in_y

That looks like exactly what we need!

Next step will be calculating the Euclidian distance between the Drigin and the Destination.

There are actually quite few way to do this.

1. You can just use pythagoras function to do that

test = df4
test['Euc_distance'] = (((test.out_x - test.in_x) ** 2) + (test.out_y - test.in_y) ** 2) ** .5
test.head()

	OutLA	InLA	Age	Sex	Moves	Date	sex_bin	lad17cd_x	out_x	out_y	lad17cd_y	in_x	in_y	Euc_distance
0	E09000001	E09000002	27	2	0.0002	2011-01-01	2	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735
1	E09000001	E09000002	36	1	0.0004	2011-01-01	1	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735
2	E09000001	E09000002	26	2	0.0004	2011-01-01	2	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735
3	E09000001	E09000002	25	2	0.0005	2011-01-01	2	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735
4	E09000001	E09000002	34	1	0.0014	2011-01-01	1	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735

Very simple, we just need to pay attention to the units of the distance which is in degrees (coordinates of the points in degrees).

2. You can use scipy.spatial.distance.cdist() function that will basically do the same job, however needs more data manipulation

# import the library
# this could be done different ways too so feel free to do what suits you
import scipy.spatial.distance as ssd
# importing the library you use the the function as x=ssd.cdist()
from scipy.spatial import distance
# then use the function as x=distance.cdist()
import scipy
# then use the function as x=scipy.spatial.distance.cdist()

# Make selection
# I'll test this on selection of points as I did not find out why it does not like my vector of size 11575714...to big for this I guess
selection = test.iloc[0:5,:]

# to use this function, we neet to create numpy arrays from the coordinates, as the cdist() requires
coords_out = np.array(selection[['out_x','out_y']])
coords_in = np.array(selection[['in_x','in_y']])
# note that this function return a matrix
mat = distance.cdist(coords_out, coords_in, 'euclidean')
mat

array([[0.22773512, 0.22773512, 0.22773512, 0.22773512, 0.22773512],
       [0.22773512, 0.22773512, 0.22773512, 0.22773512, 0.22773512],
       [0.22773512, 0.22773512, 0.22773512, 0.22773512, 0.22773512],
       [0.22773512, 0.22773512, 0.22773512, 0.22773512, 0.22773512],
       [0.22773512, 0.22773512, 0.22773512, 0.22773512, 0.22773512]])

Then you need to convert the matrix into the collumn of the dataframe… I did not worked that one out, so let me know if you know how O:-)

However, BIG ADVANTAGE of this function is that it has predefined 24 different calculation of distances. Check out the documentation, and because it's based on C++ its quite fast.

2. Calculate the Great Circle Distance between the Origins and Destinations

This could be also done several ways.

The easiest way is using the GeoPy library, but if you find a faster way, let me know!

To do this we need to get the coordinates in one column as it was in original point dataset just stripped out of the geometry column.

from geopy.distance import geodesic, great_circle
import itertools

# create stripped column of coordinates 
la['xy'] = la.geometry.apply(lambda x: [x.y, x.x])

# turn it into dataframe
points = pd.DataFrame(la)
# select just what we need
points1 = points.loc[:,['lad17cd','xy']]
points2 = points.loc[:,['lad17cd','xy']]
# rename the columns
points1.rename(columns = {'xy':'xy_1'}, inplace = True)
points2.rename(columns = {'xy':'xy_2'}, inplace = True)
# join the spatial points to the data, first for the OutLA and secondly for the InLA
test2 = test.merge(points1, how = 'left', left_on = 'OutLA', right_on = 'lad17cd')
test2 = test2.merge(points2, how = 'left', left_on = 'InLA', right_on = 'lad17cd')
test2.tail()

	OutLA	InLA	Age	Sex	Moves	Date	sex_bin	lad17cd_x	out_x	out_y	lad17cd_y	in_x	in_y	Euc_distance	lad17cd_x	xy_1	lad17cd_y	xy_2
10701098	W06000024	W06000023	67	F	1.1416	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737	0.584839	W06000024	[51.74094175880734, -3.361056696671829]	W06000023	[52.323736681870706, -3.4099121042932894]
10701099	W06000024	W06000023	73	F	0.9519	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737	0.584839	W06000024	[51.74094175880734, -3.361056696671829]	W06000023	[52.323736681870706, -3.4099121042932894]
10701100	W06000024	W06000023	74	M	1.0020	2018-01-01	1	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737	0.584839	W06000024	[51.74094175880734, -3.361056696671829]	W06000023	[52.323736681870706, -3.4099121042932894]
10701101	W06000024	W06000023	74	F	0.9866	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737	0.584839	W06000024	[51.74094175880734, -3.361056696671829]	W06000023	[52.323736681870706, -3.4099121042932894]
10701102	W06000024	W06000023	77	F	3.0000	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737	0.584839	W06000024	[51.74094175880734, -3.361056696671829]	W06000023	[52.323736681870706, -3.4099121042932894]

# now calculate the great circle distance
# this takes good 15 min
test2['great_circle_dist'] = test2.apply(lambda x: great_circle(x.xy_1, x.xy_2).km, axis=1)
test2.head()

	OutLA	InLA	Age	Sex	Moves	Date	sex_bin	lad17cd_x	out_x	out_y	lad17cd_y	in_x	in_y	Euc_distance	lad17cd_x	xy_1	lad17cd_y	xy_2	great_circle_dist
0	E09000001	E09000002	27	2	0.0002	2011-01-01	2	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735	E09000001	[51.51484485100944, -0.09217115162643398]	E09000002	[51.54527247765296, 0.13352209887032826]	15.97471
1	E09000001	E09000002	36	1	0.0004	2011-01-01	1	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735	E09000001	[51.51484485100944, -0.09217115162643398]	E09000002	[51.54527247765296, 0.13352209887032826]	15.97471
2	E09000001	E09000002	26	2	0.0004	2011-01-01	2	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735	E09000001	[51.51484485100944, -0.09217115162643398]	E09000002	[51.54527247765296, 0.13352209887032826]	15.97471
3	E09000001	E09000002	25	2	0.0005	2011-01-01	2	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735	E09000001	[51.51484485100944, -0.09217115162643398]	E09000002	[51.54527247765296, 0.13352209887032826]	15.97471
4	E09000001	E09000002	34	1	0.0014	2011-01-01	1	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735	E09000001	[51.51484485100944, -0.09217115162643398]	E09000002	[51.54527247765296, 0.13352209887032826]	15.97471

YAAAAAAAAY

We have got a distance in kilometres!

Ok it's a mess, let's clean the dataset a little bit as there is several extra columns.

# create back up first
test3 = test2
test3 = test3[['OutLA','InLA','Age','Moves','Date','sex_bin', 'Euc_distance','great_circle_dist','out_x', 'out_y','in_x','in_y', 'xy_1','xy_2']]
test3.head()

	OutLA	InLA	Age	Moves	Date	sex_bin	Euc_distance	great_circle_dist	out_x	out_y	in_x	in_y	xy_1	xy_2
0	E09000001	E09000002	27	0.0002	2011-01-01	2	0.227735	15.97471	-0.092171	51.514845	0.133522	51.545272	[51.51484485100944, -0.09217115162643398]	[51.54527247765296, 0.13352209887032826]
1	E09000001	E09000002	36	0.0004	2011-01-01	1	0.227735	15.97471	-0.092171	51.514845	0.133522	51.545272	[51.51484485100944, -0.09217115162643398]	[51.54527247765296, 0.13352209887032826]
2	E09000001	E09000002	26	0.0004	2011-01-01	2	0.227735	15.97471	-0.092171	51.514845	0.133522	51.545272	[51.51484485100944, -0.09217115162643398]	[51.54527247765296, 0.13352209887032826]
3	E09000001	E09000002	25	0.0005	2011-01-01	2	0.227735	15.97471	-0.092171	51.514845	0.133522	51.545272	[51.51484485100944, -0.09217115162643398]	[51.54527247765296, 0.13352209887032826]
4	E09000001	E09000002	34	0.0014	2011-01-01	1	0.227735	15.97471	-0.092171	51.514845	0.133522	51.545272	[51.51484485100944, -0.09217115162643398]	[51.54527247765296, 0.13352209887032826]

3. Calculating the network distances

I will demonstrate this on random points on Bristol as our migration data are not that relevant to a network distance. We just need to upload two libraries osmnx and networkx

There is nothing easier than get your network from osmnx, no need for looking for data and extra faf

import osmnx as ox
import networkx as nx

# define the place
place_name = "Bristol, United Kingdom"
# define the graph
graph = ox.graph_from_place(place_name, network_type='drive')

Now get some random points from Bristol

# 51.453700, -2.595778 # Dominos pizza in Bristol city centre
# 51.456294, -2.605236 # Bristol Gallery and Museum
# 51.468233, -2.588812 # Montpelier rail station
# 51.456569, -2.626453 # Clifton observatory

# create the list of coordinates
lat_list = [51.453700, 51.456294, 51.468233, 51.456569]
lon_list = [-2.595778, -2.605236, -2.588812, -2.626453]

#put it together
random_points = zip(lon_list, lat_list)

# create dataframe
random_points = pd.DataFrame(random_points)
# change the names of columns
random_points.rename(columns = {0:'x',1:'y'}, inplace = True)

# create spatial dataframe from the dataframe
random_points_gpd = gpd.GeoDataFrame(
    random_points, geometry=gpd.points_from_xy(random_points.x, random_points.y))
# assign coordinates
random_points_gpd.crs = la.crs

Now we need to convert everything to UTM as we want to get the lenght in metres not degrees

# define the crs
utm = "+proj=utm +zone=33 +ellps=WGS84 +datum=WGS84 +units=m +no_defs"
# points to utm
random_points_gpd = random_points_gpd.to_crs(utm)
# convert the graph into utm...projct_graph function does it all
graph = ox.project_graph(graph, to_crs = utm)

# lets just see how it looks like
fig, ax = ox.plot_graph(graph_proj)
plt.tight_layout()

<Figure size 432x288 with 0 Axes>

# separate the nodes and edges from the graph
nodes_proj, edges_proj = ox.graph_to_gdfs(graph_proj, nodes=True, edges=True)
# look at the crs just to check
print("Coordinate system:", edges_proj.crs)

Coordinate system: +proj=utm +zone=33 +ellps=WGS84 +datum=WGS84 +units=m +no_defs

# create nearest nodes for each point

#get the coordinates only into dataframe
random_points_gpd['a'] = random_points_gpd.geometry.y
random_points_gpd['b'] = random_points_gpd.geometry.x
target_xy = pd.DataFrame(random_points_gpd[['a','b']])

# get the nearest node id for each row
def nearest_node(Lat,Lon):
    nearest_node,dist=ox.get_nearest_node(graph, (Lat,Lon), return_dist=True)  
    return nearest_node 

# apply the function
target_xy['node'] = np.vectorize(nearest_node)(target_xy['a'],target_xy['b'])

Now we create the Origin-Destination data this is going to be little fidly, indeed, but at least we have just a sample dataset

# separate just the node id
target2 = target_xy.iloc[:,[2]]

# double this for destinations
target3 = target_xy.iloc[:,[2]]
# reset the index and reverse the order
target3 = target3.sort_index(ascending=False).reset_index().reset_index()
target2 = target2.reset_index()

# merge those two node datasets into OD data
OD = target2.merge(target3, left_on='index', right_on= 'level_0')

# get rid of unnecessary columns
OD = OD.loc[:,['node_x','node_y']]
OD.head()

	node_x	node_y
0	6502760749	26127008
1	5432376184	358536159
2	358536159	5432376184
3	26127008	6502760749

# define a function that calculates shortest path between the nodes on graph
def short_path_length(row):
    return nx.shortest_path_length(graph, row['node_x'], row['node_y'], weight='length')

# apply the function to our OD data.....Note: as we are working with UTM projection, the units should be metres
OD['short_path_length'] = OD.apply(short_path_length, axis=1)
OD.head()

	node_x	node_y	short_path_length
0	6502760749	26127008	4537.728
1	5432376184	358536159	4031.132
2	358536159	5432376184	4039.488
3	26127008	6502760749	4611.492

Bare in mind I calculated the network distances on sample points in small area. That is because graph of whole UK is just huuuge and the computation would be time-expensive. In my opinion, we do not need the network distance for the migration data, however it could be usefull to know how to do that, so let me know if you found any quick and efficient way to do that.