Python | Lenka's blog

The Spatial Mismatch of Empty Homes and Housing Pressure

Thu, 07 May 2026 00:00:00 +0000

I've recently read Andrew Brooks article in the Geographical, which provides a convincing argument in today’s debate on UK empty homes. The argument resonated with me, as the pressure to build new houses, both from the public and the government, has never quite sat well with me. I suppose this may be influenced by coming from the Czech Republic, where 50–60% of the population lives in flats, which is a stark contrast to the UK, where only around 20% do. Moreover, many people in the Czech Republic would rather own a flat than a house, perhaps due to better transport options and lower maintenance requirements. Looking at national statistics, one might not initially assume this to be a problem. With a total population of 69.3 million and roughly 30.4 million dwellings, this equates to around 2.2 people per dwelling. Theoretically, this appears entirely reasonable.

Setting that aside, however, the reality is that housing demand is generally much higher than supply, especially in some locations. This is not simply due to a lack of homes; rather, it is likely driven by a number of underlying factors such as the increasing number of empty homes, the increasing number of lone households, the increasing number of retirees living in otherwise empty family housing, and at the same time the increasing number of empty retirement properties, among others. Additionally, each of these factors has its own underlying causes, which often circle back to the broader demand–supply dynamic. (I'm not a housing expert however, so don't quote me on this and let me know whats your theory!) This complete picture is much more nuanced (as Brooks suggests), and the complexity of the situation is difficult to fully understand, let alone quantify. However, we can still examine some aspects of Brooks’s claim:

“What is called a ‘housing crisis’ is really an ‘empty room crisis’, caused by vacant properties and uninhabited bedrooms. There is not a problem of housing stock, but of distribution”

So let’s take a closer look at what this looks like on a map (a phrase geographers tend to use at every opportunity, a slightly toxic trait, if you will).

The available data

I use the latest (2025) data on the number of empty homes in England, derived from council tax records (Council Taxbase 2025). This dataset is only available at the local authority level, so the analysis cannot be more granular. However, it also includes information on second homes and a breakdown by tax band.

The second dataset is occupancy data from the latest Census (2021), obtained via Nomis. This shows how many households have either unoccupied bedrooms or are overcrowded in each local authority.

I complement this with data on population size for each local authority, as well as urban–rural classification and the latest IMD data (2025), to provide additional context. Finally, I include geographical boundaries for local authorities to enable mapping.

import pandas as pd 
import numpy as np 
import geopandas as gpd
import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1.inset_locator import inset_axes
import matplotlib.ticker as mtick
import matplotlib.patches as mpatches
import seaborn as sns
from libpysal.weights import Queen
from esda.moran import Moran_Local_BV
import numpy as np
import folium
import warnings
warnings.filterwarnings("ignore")

# Load data
# technical note: you may need to make some manual adjustments to certain sheets,
# such as removing the first or last few rows and converting object columns to numeric values
# You may also need to use LAD lookup tables to correct issues with Sheffield and Barnsley LAD codes
empty_homes = pd.read_csv('./empty_homes_2025.csv')
occupancy = pd.read_csv('./occupancy2021.csv').rename(columns={'mnemonic': 'LADCD'})
ruc = pd.read_csv('./RUC.csv').iloc[:, [0,2,4]]
imd = pd.read_csv('./IMD.csv').loc[:, ['LAD24CD','IMD_decile(10 least deprived)']].groupby('LAD24CD').mean().reset_index()
pop = pd.read_csv('./pop.csv').rename(columns={'2021': 'population', 'mnemonic': 'LADCD'})
LA = gpd.read_file('./Local_Authority_Districts_May_2024_Boundaries_UK_BFE_7458506961569058424.geojson')

# Merge datasets together
empty_homes.rename(columns={'ONS Code': 'LADCD'}, inplace=True)
data = empty_homes.merge(occupancy, on='LADCD', how='left'
                         ).merge(ruc, left_on='LADCD', right_on='LAD24CD', how='left'
                                 ).merge(pop, on='LADCD', how='left').merge(imd, left_on='LADCD', right_on='LAD24CD', how='left')
data = LA.merge(data, right_on='LADCD', left_on='LAD24CD', how='right')
data.columns = data.columns.str.strip()

# let's just keep the columns we need
data = data.loc[~data['LAD24NM'].isna(), ['LADCD','LAD24NM','Region','geometry', 
       'Total empty homes', 'all_empty_small', 'all_empty_medium',
       'all_empty_large', 'Total homes empty more than 6 months',
       'long_empty_small', 'long_empty_medium', 'long_empty_large', 'Total_dwellings',
       'Total_small', 'Total_medium', 'Total_large', 'Second_homes',
       'second_small', 'second_medium', 'second_large',
       'Total: All households', 'Occupancy rating of bedrooms: +2 or more',
       'Occupancy rating of bedrooms: +1', 'Occupancy rating of bedrooms: 0',
       'Occupancy rating of bedrooms: -1',
       'Occupancy rating of bedrooms: -2 or less', 'RUC21CD',
       'Urban_rural_flag','IMD_decile(10 least deprived)',
       'population']].rename(columns={'Total_dwellings': 'Total homes'})
data.head(5)

We now convert the core variables into more digestible measures, expressed as rates.

data["long_empty_rate"] = (data["Total homes empty more than 6 months"] / data["Total homes"])
data["second_homes_rate"] = (data['Second_homes'] / data["Total homes"])
data["all_empty_rate"] = (data["Total empty homes"] / data["Total homes"])
data["all_unoccupied_rate"] = ((data["Total empty homes"]+ data['Second_homes']) / data["Total homes"])

data["overcrowded_rate"] = ((data["Occupancy rating of bedrooms: -2 or less"]+ data['Occupancy rating of bedrooms: -1']) /data["Total: All households"])
data["underoccupancy_rate"] = ((data['Occupancy rating of bedrooms: +2 or more'] + data['Occupancy rating of bedrooms: +1']) /data["Total: All households"])

# look into the size of the bedrooms
data["all_unoccupied_small_rate"] = ((data["Total empty homes"]+data["Total homes empty more than 6 months"]+ data['Second_homes']) / data["Total homes"])

data["all_unoccupied_small_rate"] = ((data['all_empty_small']  + data['second_small'])/ data['Total_small'])
data["all_unoccupied_medium_rate"] = ((data['all_empty_medium']  + data['second_medium'])/ data['Total_medium'])
data["all_unoccupied_large_rate"] = ((data['all_empty_large']  + data['second_large'])/ data['Total_large'])

# predefines some figures
def map_with_london(data, column, title, ax, percentage=False):

    data.plot(column=column,ax=ax, legend=True)
    
    if percentage:
        cbar = ax.get_figure().axes[-1]
        cbar.yaxis.set_major_formatter(mtick.PercentFormatter(xmax=1))

    ax.set_title(title)
    ax.axis("off")

    # london
    london = data[data["Region"] == "L"]
    axins = inset_axes(ax, width="40%", height="40%", loc="center left")
    london.plot(column=column, ax=axins, legend=False)
    axins.set_title("London", fontsize=8)
    axins.axis("off")


def top_bottom_plot(data, column, ax, name_col="LAD24NM", title = None):
    
    top5 = data.sort_values(column, ascending=False).head(5)
    bottom5 = data.sort_values(column, ascending=True).head(5)

    cmap = plt.get_cmap("viridis")
    top5["color"] = [cmap(0.95)] * len(top5)      
    bottom5["color"] = [cmap(0.05)] * len(bottom5)  

    combined = pd.concat([bottom5, top5])
    combined = combined.sort_values(column, ascending=True)

    ax.barh(combined[name_col],combined[column],color=combined["color"])

    ax.set_title(f"Top 5 and Bottom 5 areas: {column}" if title is None else title)
    ax.set_xlabel("Rate")
    ax.xaxis.set_major_formatter(mtick.PercentFormatter(1))

fig, axes = plt.subplots(1, 2, figsize=(22, 6))
map_with_london(data, "all_empty_rate", 
                "All empty homes \n(percentage of all homes left empty for more than 6 months)", ax = axes[0], percentage=True)
top_bottom_plot(data, "all_empty_rate", ax = axes[1], title="Top 5 and Bottom 5 areas")
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(22, 6))
map_with_london(data, "long_empty_rate", 
                "Long term empty homes \n(percentage of all homes left empty for more than 6 months)", ax = axes[0], percentage=True)
top_bottom_plot(data, "long_empty_rate", ax = axes[1], title="Top 5 and Bottom 5 areas")
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(22, 6))
map_with_london(data, "second_homes_rate", 
                "Second homes \n(percentage of all homes that are second homes)", ax = axes[0], percentage=True)
top_bottom_plot(data, "second_homes_rate", ax = axes[1], title="Top 5 and Bottom 5 areas")
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(22, 6))
map_with_london(data, "all_unoccupied_rate", 
                "All unoccupied homes \n(percentage of all homes that are unoccupied \nat least portion of the year, including second homes)", ax = axes[0], percentage=True)
top_bottom_plot(data, "all_unoccupied_rate", ax = axes[1], title="Top 5 and Bottom 5 areas")
plt.show()

The proportion of empty homes ranges between 0% and 4.4%, with an average of around 2%. The highest value is observed in the Isles of Scilly, while the lowest is in Havant. Within these, long-term empty properties range between 1.1% and 3.3%, with similar areas appearing in both the top and bottom five.

However, these are not the only types of empty properties people tend to notice in their neighbourhoods. In fact, when discussing empty homes, people often refer to second homes, technically not vacant properties but still visibly udnerused ones. I therefore extracted this information from the same dataset. The rate of second homes ranges from 0% to 24%, with an average of around 1%. This represents a much wider range than empty homes, but it is also highly skewed: most areas have very low rates, while only a few have very high concentrations. On a map, the City of London immediately stands out as having the highest rate of second homes.

As a result, a combined measure of empty homes and second homes is more strongly influenced by the distribution of second homes than by empty properties alone.

Let’s now take a look at the size distribution.

fig, axes = plt.subplots(1, 2, figsize=(22, 6))
map_with_london(data, "all_unoccupied_small_rate", 
                "Small size unoccupied homes \n(percentage of all homes left unoccupied)", ax = axes[0], percentage=True)
top_bottom_plot(data, "all_unoccupied_small_rate", ax = axes[1], title="Top 5 and Bottom 5 areas")
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(22, 6))
map_with_london(data, "all_unoccupied_medium_rate", 
                "Medium size unoccupied homes \n(percentage of all homes that are unoccupied)", ax = axes[0], percentage=True)
top_bottom_plot(data, "all_unoccupied_medium_rate", ax = axes[1], title="Top 5 and Bottom 5 areas")
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(22, 6))
map_with_london(data, "all_unoccupied_large_rate", 
                "Large size unoccupied homes \n(percentage of all homes that are unoccupied)", ax = axes[0], percentage=True)
top_bottom_plot(data, "all_unoccupied_large_rate", ax = axes[1], title="Top 5 and Bottom 5 areas")
plt.show()

Before looking into the empty bedrooms data, let us first examine how many households live in appropriately sized properties.

data['balance_rate'] = data['Occupancy rating of bedrooms: 0'] / data['Total: All households']

fig, axes = plt.subplots(1, 2, figsize=(15, 6))

map_with_london(data, 'balance_rate', 
                "Balanced households \n(percentage of household living in 'appropriate' accommodation)", ax = axes[0], percentage=True)
top_bottom_plot(data, 'balance_rate', ax = axes[1], title="Top 5 and Bottom 5 areas")

plt.show()

The average rate of balanced households across England is 25%, with a maximum of 63% and a minimum of 12%. Interestingly, the top five areas with the highest rates are all located in London.

fig, axes = plt.subplots(1, 2, figsize=(18, 6))
map_with_london(data, "overcrowded_rate", 
                "Overcrowded households \n(percentage of household with 1 or less bedrooms than needed)", ax = axes[0], percentage=True)
top_bottom_plot(data, "overcrowded_rate", ax = axes[1], title="Top 5 and Bottom 5 areas")
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(16, 6))
map_with_london(data, "underoccupancy_rate", 
                "Underoccupied households \n(percentage of household with 1+ 'empty' bedrooms)", ax = axes[0], percentage=True)
top_bottom_plot(data, "underoccupancy_rate", ax = axes[1], title="Top 5 and Bottom 5 areas")
plt.show()

# reshape
df_long = data.loc[:, ["all_empty_rate", "long_empty_rate", "second_homes_rate", "all_unoccupied_rate",
                'balance_rate', "overcrowded_rate", 'underoccupancy_rate']].melt(
    var_name="variable",
    value_name="rate")

# labels
label_map = {"all_empty_rate": "All empty homes",
    "long_empty_rate": "Long-term empty",
    "second_homes_rate": "Second homes",
    "all_unoccupied_rate": "All empty and second homes",
    "balance_rate": "Balanced",
    "overcrowded_rate": "Overcrowded",
    "underoccupancy_rate": "Underoccupancy"}

df_long["variable"] = df_long["variable"].map(label_map)

# plot
fig, ax = plt.subplots(figsize=(10, 6))
sns.boxplot(data=df_long, x="variable", y="rate", ax=ax, color="orange")
ax.set_ylabel("Rate")
ax.set_xlabel("")
ax.yaxis.set_major_formatter(mtick.PercentFormatter(1))
ax.set_title("Distribution of empty housing rates and occupation levels across local authorities")
plt.xticks(rotation=20)
plt.tight_layout()
plt.show()

Interestingly, the percentage of households with underoccupied bedrooms is much higher than the percentage of overcrowded households. In some local authorities, up to 85% of households have two or more spare bedrooms, with an average of around 70%, while maximum overcrowding reaches approximately 21%, with an average of just 3%. This is an important finding.

It is far more common in England to have the luxury of a spare bedroom (at least 80% of households have at least one) than to experience overcrowding (around 20%). Across local authorities, we observe empty property rates ranging between 0% and 3%, with an average of around 1%. This comparison depends on how we define “extra space”—if we take any spare bedroom as surplus.

We therefore adopt more flexible definition. One spare bedroom can often serve practical purposes (such as a home office, storage, or a child’s room), and similarly, a shortfall of one bedroom may reflect temporary life circumstances, such as a growing family. For this reason, we focus only on households with two or more spare bedrooms when identifying underoccupation.

data['balance_rate'] = (data['Occupancy rating of bedrooms: 0']+ data['Occupancy rating of bedrooms: +1']+ data['Occupancy rating of bedrooms: -1']) / data['Total: All households']
data["overcrowded_rate"] = (data["Occupancy rating of bedrooms: -2 or less"] /data["Total: All households"])
data["underoccupancy_rate"] = (data['Occupancy rating of bedrooms: +2 or more'] /data["Total: All households"])

fig, axes = plt.subplots(1, 2, figsize=(16, 6))
map_with_london(data, "balance_rate", 
                "Balanced households \n(percentage of household living in 'appropriate' accommodation)", ax = axes[0], percentage=True)
top_bottom_plot(data, "balance_rate", ax = axes[1], title="Top 5 and Bottom 5 areas")
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(18, 6))
map_with_london(data, "overcrowded_rate", 
                "Overcrowded households \n(percentage of household with 2 or less bedrooms than needed)", ax = axes[0], percentage=True)
top_bottom_plot(data, "overcrowded_rate", ax = axes[1], title="Top 5 and Bottom 5 areas")
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(16, 6))
map_with_london(data, "underoccupancy_rate", 
                "Underoccupied households \n(percentage of household with 2 or more spare bedrooms)", ax = axes[0], percentage=True)
top_bottom_plot(data, "underoccupancy_rate", ax = axes[1], title="Top 5 and Bottom 5 areas")
plt.show()

# reshape
df_long = data.loc[:, ["all_empty_rate", "long_empty_rate", "second_homes_rate", "all_unoccupied_rate",
                'balance_rate', "overcrowded_rate", 'underoccupancy_rate']].melt(
    var_name="variable",
    value_name="rate")

# labels
label_map = {"all_empty_rate": "All empty homes",
    "long_empty_rate": "Long-term empty",
    "second_homes_rate": "Second homes",
    "all_unoccupied_rate": "All empty and second homes",
    "balance_rate": "Balanced",
    "overcrowded_rate": "Overcrowded",
    "underoccupancy_rate": "Underoccupancy"}

df_long["variable"] = df_long["variable"].map(label_map)

# plot
fig, ax = plt.subplots(figsize=(10, 6))
sns.boxplot(data=df_long, x="variable", y="rate", ax=ax, color="orange")
ax.set_ylabel("Rate")
ax.set_xlabel("")
ax.yaxis.set_major_formatter(mtick.PercentFormatter(1))
ax.set_title("Distribution of empty housing rates and occupation levels across local authoritie")
plt.xticks(rotation=20)
plt.tight_layout()
plt.show()

Adjusting our definition of what counts as “spare” results in a less extreme view of the situation. On average, 62% of households in each area are considered balanced, although this ranges from 45% to 91%. Among the areas with the lowest levels of balanced accommodation are Rutland, Rushcliffe, and the Cotswolds; at the same time, these areas also have some of the highest proportions of households with underoccupied bedrooms.

In contrast, the areas with the highest levels of balanced housing, such as the City of London, Tower Hamlets, and Islington, also have the lowest rates of underoccupation.

Interestingly, none of these areas appear among those with the highest or lowest levels of overcrowding. The proportion of households experiencing overcrowding averages around 0.5%, ranging from 0% to 5%. This indicates a strong skew: most areas experience very low levels of overcrowding, while only a few exhibit relatively higher rates. All of the top five areas are London boroughs, with Newham having the highest rate.

To provide some context, Newham, despite having the highest overcrowding rate at 5%, still performs relatively well on other measures: around 80% of households are balanced (above average), approximately 12% experience underoccupation (well below average), and only about 1% of homes are empty (also below average). This suggests that while overcrowding is an issue, the overall housing situation is not uniformly poor.

Let’s now take a closer look at how these variables vary across the urban–rural classification, as the overcrowding map already hints at some differences between these contexts.

fig, axes = plt.subplots(1, 4, figsize=(16, 5), sharey=False)

for i, var in enumerate(["all_unoccupied_rate", "balance_rate", "overcrowded_rate", "underoccupancy_rate"]):
    sns.violinplot(data=data, x="Urban_rural_flag", y=var,
        hue="Urban_rural_flag",
        palette={"Urban": "grey", "Rural": "darkgreen"},
        inner="quartile",
        legend=False,
        ax=axes[i]
    )
    titles = ["All Empty and Second Homes","Balanced Housing","Overcrowding","Under-occupation"]
    axes[i].set_title(titles[i])
    axes[i].set_xlabel("")
    axes[i].set_ylabel("Rate")
    axes[i].yaxis.set_major_formatter(mtick.PercentFormatter(1))

plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 4, figsize=(16, 5), sharey=False)

for i, var in enumerate(["all_unoccupied_rate", "balance_rate", "overcrowded_rate", "underoccupancy_rate"]):
    sns.regplot(data=data, x="IMD_decile(10 least deprived)", y=var,
        ax=axes[i],
        scatter_kws={"alpha": 0.5}
    )
    titles = ["All Empty and Second Homes","Balanced Housing","Overcrowding","Under-occupation"]
    axes[i].set_title(titles[i])
    axes[i].set_xlabel("IMD Decile (10 = least deprived)")
    axes[i].set_ylabel("Rate")
    axes[i].yaxis.set_major_formatter(mtick.PercentFormatter(1))

plt.tight_layout()
plt.show()

So where is the mismatch?

Looking across the maps and variables, several patterns become clear:

Empty properties are distributed relatively evenly across both urban and rural areas, suggesting there is no strong spatial (e.g. polycentric) pattern.
At both national and local authority scales, overcrowding appears less common than might be expected. It is important to note that this dataset does not capture the full extent of overcrowding (it only identifies households with two or fewer bedrooms). However, where overcrowding is present, it is almost entirely concentrated in urban areas.
Underoccupation is widespread across England. Over 40% of local authorities have at least 40% of households with two or more spare bedrooms. It is present in both urban and rural areas, although it is slightly more common in rural areas.

It therefore appears that the relationship between these three variables within a single area is not straightforward. It is not immediately clear where empty homes or spare bedrooms could be used to alleviate overcrowding, beyond the general observation that spare bedrooms are widely available. To address this, we can apply some simple feature engineering to better capture potential mismatches.

Finding areas of mismatch

Let us consider the “empty bedroom problem” as a single-variable issue. By combining the relevant variables, we can construct an index of “occupational pressure” by subtracting the number of households with spare bedrooms from those lacking sufficient bedrooms. For example:

$$ \text{Pressure Index} = \frac{(\text{-2 or less}) - (\text{+2 or more})}{\text{Total households}} $$

This produces a variable that ranges from negative values (indicating excess capacity) to positive values (indicating pressure from overcrowding), with values around zero suggesting a balance between spare bedrooms and overcrowding.

This is, of course, a simplification. Areas that appear “balanced” may not be evenly distributed in reality, but the index provides a useful proxy for internal housing pressure and helps highlight potential mismatches.

data["pressure_index"] = (data["Occupancy rating of bedrooms: -2 or less"] - data["Occupancy rating of bedrooms: +2 or more"]) / data["Total: All households"]
data["pressure_index"].hist();

fig, axes = plt.subplots(1, 2, figsize=(10, 5), sharey=False)

for i, var in enumerate(["all_unoccupied_rate", "pressure_index"]):
    sns.violinplot(data=data, x="Urban_rural_flag", y=var,
        hue="Urban_rural_flag",
        palette={"Urban": "grey", "Rural": "darkgreen"},
        inner="quartile",
        legend=False,
        ax=axes[i]
    )
    titles = ["All Empty and Second Homes","Occupancy Pressure Index"]
    axes[i].set_title(titles[i])
    axes[i].set_xlabel("")
    axes[i].set_ylabel("Rate")
    axes[i].yaxis.set_major_formatter(mtick.PercentFormatter(1))

plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(10, 5), sharey=False)

for i, var in enumerate(["all_unoccupied_rate", "pressure_index"]):
    sns.regplot(data=data, x="IMD_decile(10 least deprived)", y=var,
        ax=axes[i],
        scatter_kws={"alpha": 0.5}
    )
    titles = ["All Empty and Second Homes","Occupancy Pressure Index"]
    axes[i].set_title(titles[i])
    axes[i].set_xlabel("IMD Decile (10 = least deprived)")
    axes[i].set_ylabel("Rate")
    axes[i].yaxis.set_major_formatter(mtick.PercentFormatter(1))

plt.tight_layout()
plt.show()

The new pressure index is overwhelmingly negative, reflecting what we observed in the maps above: the issue of underoccupation is far more widespread and pronounced than overcrowding. This creates many areas with potential “pressure release” in terms of available space.

Let's now examine the relationship between these variables using a scatter plot, highlighting observations that fall into the extremes. To do this, we define the top and bottom 20% of observations within each variable.

low_empty = data["all_unoccupied_rate"].quantile(0.2)
high_empty = data["all_unoccupied_rate"].quantile(0.8)

low_pressure = data["pressure_index"].quantile(0.2)
high_pressure = data["pressure_index"].quantile(0.8)



fig, ax = plt.subplots(figsize=(8, 8))

sns.scatterplot(data=data, x="all_unoccupied_rate", y="pressure_index",
    hue="Urban_rural_flag", palette={"Urban": "grey","Rural": "darkgreen"}, alpha=0.8, ax=ax)
xlim = ax.get_xlim()
ylim = ax.get_ylim()

# colored boxes
ax.fill_between([xlim[0], low_empty], high_pressure, ylim[1], color="orange", alpha=0.15)
ax.fill_between([high_empty, xlim[1]], high_pressure, ylim[1], color="red", alpha=0.15)
ax.fill_between([high_empty, xlim[1]], ylim[0], low_pressure, color="blue", alpha=0.15)
ax.fill_between([xlim[0], low_empty], ylim[0], low_pressure, color="green", alpha=0.15)

# lines
ax.axvline(low_empty, color="black", linestyle="--", linewidth=1)
ax.axvline(high_empty, color="black", linestyle="--", linewidth=1)
ax.axhline(low_pressure, color="black", linestyle="--", linewidth=1)
ax.axhline(high_pressure, color="black", linestyle="--", linewidth=1)

# axes ticks
ax.set_xticks([low_empty, data["all_unoccupied_rate"].median(), high_empty])
ax.set_xticklabels(["Low", "Average", "High"], rotation=60)
ax.set_yticks([low_pressure, data["pressure_index"].median(), high_pressure])
ax.set_yticklabels(["Low", "Average", "High"])

# labels
ax.set_xlabel("Empty and Second Homes")
ax.set_ylabel("Occupancy Pressure Index")
ax.text(xlim[0], ylim[1], "Property Shortage", color="orange", ha="left", va="top")
ax.text(xlim[1], ylim[1], "High pressure", color="red", ha="right", va="top")
ax.text(xlim[1], ylim[0], "Underused spaces", color="blue", ha="right", va="bottom")
ax.text(xlim[0], ylim[0], "Balanced living", color="green", ha="left", va="bottom")

ax.spines["top"].set_visible(False)
ax.spines["right"].set_visible(False)
plt.tight_layout()
plt.show()

This allows us to clearly distinguish between different types of areas:

Red are areas that experience both a high number of empty properties and high occupancy pressure. In these places, local populations may face higher levels of crowding while being surrounded by unused dwellings.
Orange are areas with low levels of empty properties but high occupancy pressure. Here, people may experience crowding, but there are few empty properties available locally.
Blue are areas with a high number of empty homes and substantial spare bedroom capacity. These could be described as approaching “ghost town” conditions, where housing space is significantly underused.
Green are areas with low levels of empty homes and, at the same time, a high availability of spare bedrooms, perhaps the most balanced housing environments.

Let’s now look at how these patterns appear on a map.

def classify(row):
    if row["all_unoccupied_rate"] <= low_empty and row["pressure_index"] >= high_pressure:
        return "Property Shortage"
    elif row["all_unoccupied_rate"] >= high_empty and row["pressure_index"] >= high_pressure:
        return "High pressure"
    elif row["all_unoccupied_rate"] >= high_empty and row["pressure_index"] <= low_pressure:
        return "Underused spaces"
    elif row["all_unoccupied_rate"] <= low_empty and row["pressure_index"] <= low_pressure:
        return "Balanced living"
    else:
        return "Middle"  # optional: non-extreme areas

data["housing_type"] = data.apply(classify, axis=1)

color_map = {
    "Property Shortage": "gold",    
    "High pressure": "#d73027",    
    "Underused spaces": "#4575b4",   
    "Balanced living": "#1a9850",     
    "Middle": "#d9d9d9"       
}

data["color"] = data["housing_type"].map(color_map)
pop_by_type = data.groupby("housing_type")["population"].sum()

def fmt_pop(x):
    if x >= 1_000_000:
        return f"{x/1_000_000:.1f}M"
    elif x >= 1_000:
        return f"{x/1_000:.0f}K"
    else:
        return str(int(x))
    

ig, ax = plt.subplots(figsize=(10, 10))

data.plot(color=data["color"], linewidth=0.2, edgecolor="white", ax=ax)

ax.set_axis_off()

legend_handles = [
    mpatches.Patch(
        color="gold",
        label=f"Property Shortage ({fmt_pop(pop_by_type.get('Property Shortage', 0))})"
    ),
    mpatches.Patch(
        color="#d73027",
        label=f"High pressure ({fmt_pop(pop_by_type.get('High pressure', 0))})"
    ),
    mpatches.Patch(
        color="#4575b4",
        label=f"Underused spaces ({fmt_pop(pop_by_type.get('Underused spaces', 0))})"
    ),
    mpatches.Patch(
        color="#1a9850",
        label=f"Balanced living ({fmt_pop(pop_by_type.get('Balanced living', 0))})"
    ),
]

ax.legend(
    handles=legend_handles,
    loc="lower right",
    frameon=False,
    title="The Extreme (Total area population size)"
)
ax.set_title("Housing Pressure (Extreme Conditions)", fontsize=14)

plt.tight_layout()
plt.show()

The areas at the extremes are distributed across the UK. Orange and blue areas are those with relatively high numbers of empty homes; however, in blue areas there is also a high level of spare bedroom capacity, while red areas are those experiencing overcrowding. In these red areas, policies aimed at increasing property occupancy and reducing bedroom underoccupation could be particularly beneficial.

Orange areas, on the other hand, experience higher levels of household crowding but have very low levels of empty homes.

So far, this interpretation has been fairly straightforward, each area is considered in isolation. In reality, neighbouring areas are often closely connected. If one area experiences high overcrowding, surrounding areas are likely to face similar housing pressures. This suggests that housing pressure is not simply a local issue, but a spatial one. Let’s now explore whether we can account for this by incorporating spatial statistics.

Accounting for spatial relations



# spatial weights 
w = Queen.from_dataframe(data)
w.transform = "r"

# variables
x = data["pressure_index"].values
y = data["all_unoccupied_rate"].values

# identify islands 
data["is_island"] = False
data.loc[w.islands, "is_island"] = True

# bivariate local Moran
lisa = Moran_Local_BV(x, y, w)

# set islands to NaN for cluster type 
data.loc[data["is_island"], "lisa_cluster"] = np.nan

# cluster type
data["lisa_cluster"] = lisa.q
data["p_value"] = lisa.p_sim

data["lisa_5"] = np.where(data["p_value"] < 0.05, data["lisa_cluster"], 0)

label_map = {
    1: "High pressure",        # HH
    2: "Underused spaces",     # LH
    3: "Balanced living",      # LL
    4: "Property Shortage",    # HL
    0: "Middle"}

data["lisa_5_label"] = data["lisa_5"].map(label_map)

Here are the final variables and the number of areas in each combination.


data.loc[:,['housing_type','lisa_5_label']].value_counts()

housing_type       lisa_5_label     
Middle             Middle               201
                   Balanced living       24
Balanced living    Middle                11
High pressure      Middle                10
Middle             Property Shortage      9
Property Shortage  Middle                 7
Underused spaces   Middle                 7
Middle             High pressure          7
                   Underused spaces       5
High pressure      High pressure          5
Balanced living    Balanced living        4
High pressure      Property Shortage      3
Property Shortage  High pressure          1
                   Property Shortage      1
Underused spaces   Underused spaces       1
Name: count, dtype: int64


# different colour schema
color_map2 = {
    "High pressure + high nearby empties": "#850000",     # muted red
    "High pressure + low nearby empties": "#B37100",  # muted yellow
    "Low pressure + high nearby empties": "#002487",      # muted blue
    "Balanced living + low empties": "#006622",        # muted green
    "Not significant": "lightgray"
}

# new labels
lisa_labels = {
    "High pressure": "High pressure + high nearby empties",
    "Property Shortage": "High pressure + low nearby empties",
    "Underused spaces": "Low pressure + high nearby empties",
    "Balanced living": "Balanced living + low empties",
    "Middle": "Not significant"
}

data["lisa_readable"] = data["lisa_5_label"].map(lisa_labels)

# plot
fig, ax = plt.subplots(figsize=(10, 15), constrained_layout=True)
data.plot(color=data["lisa_readable"].map(color_map2), ax=ax)

ax.set_title("Spatial clusters of housing pressure and empty homes (LISA 5%)", fontsize = 14)
ax.axis("off")

legend_items = [
    ("High pressure + high nearby empties", "High occupancy, surrounded by empty homes"),
    ("High pressure + low nearby empties", "High occupancy, little nearby empty stock"),
    ("Low pressure + high nearby empties", "Spare capacity, surrounded by empty homes"),
    ("Balanced living + low empties", "Spare capacity, little nearby empty stock"),
    ("Not significant", "No clear spatial pattern")
]

handles = [
    mpatches.Patch(color=color_map2[k], label=v)
    for k, v in legend_items
]

ax.legend(handles=handles, frameon=False, loc="center left", title="Cluster type")

plt.show()

The map shows clusters of the two variables. Specifically:

Red areas are local authorities with relatively high occupancy pressure (in lived-in households) that are surrounded by areas with high rates of empty dwellings.
Yellow areas are local authorities with relatively high occupancy pressure, but are surrounded by areas with little to no empty dwellings.
Blue areas are local authorities with low occupancy pressure (i.e. plenty of spare bedrooms), surrounded by areas with high rates of empty dwellings—whether these are second homes or genuinely empty properties.
Green areas are perhaps the most balanced, characterised by low occupancy pressure, ample spare bedrooms, and very low levels of empty homes in surrounding areas.

You can explore both layers in the interactive map below:

first_layer = data.loc[data['housing_type'] != 'Middle',['LADCD', 'LAD24NM', 'geometry','housing_type','Urban_rural_flag','IMD_decile(10 least deprived)','population']
                       ].rename(columns={'housing_type': 'Housing Type'})
second_layer = data.loc[data['lisa_readable'] != 'Not significant',['LADCD', 'LAD24NM', 'geometry','lisa_readable','Urban_rural_flag','IMD_decile(10 least deprived)','population']
                        ].rename(columns={'lisa_readable': 'LISA Cluster'})

# base map
m = folium.Map(location=[54.5, -2.5], zoom_start=6, tiles="cartodbpositron")

# style functions
def style_function_factory(column, cmap, opacity=0.5):
    def style_function(feature):
        val = feature["properties"][column]
        color = cmap.get(val, "#d9d9d9")

        return {
            "fillColor": color,
            "color": "black",
            "weight": 0.3,
            "fillOpacity": opacity
        }
    return style_function

# layers
folium.GeoJson(
    first_layer,
    name="Housing Pressure Extreme",
    style_function=style_function_factory('Housing Type', color_map),
    tooltip=folium.GeoJsonTooltip(fields=["LAD24NM", "Housing Type",'Urban_rural_flag','population','IMD_decile(10 least deprived)'])
).add_to(m)


folium.GeoJson(
    second_layer,
    name="Spatial cluster (LISA 5%)",
    style_function=style_function_factory("LISA Cluster", color_map2),
    tooltip=folium.GeoJsonTooltip(fields=["LAD24NM", "LISA Cluster",'Urban_rural_flag','population','IMD_decile(10 least deprived)'])
).add_to(m)

folium.LayerControl(collapsed=False).add_to(m)
m.save("map.html")

Summary

Is this a problem of distribution rather than a shortage of housing? The maps above suggest YES. There appears to be a substantial amount of underoccupied housing within UK local authorities. In fact, in around 40% of local authorities, at least 40% of households have two or more spare bedrooms, which is more common in rural local authorities.

However, the patterns suggest that underoccupation is more likely to occur in rural and less deprived areas, while overcrowding is concentrated almost entirely in urban, and often more deprived, areas. Moreover, although the high-pressure areas appear relatively small on the map, they contain roughly twice the population of the other extreme categories because they are predominantly urban and have much higher population densities.

Perhaps this is one of the reasons why the “not enough homes” debate is far more visible and widely discussed than the issue of “ghost towns” and underused housing space, it is simply a experienced by more people.

There appear to be areas where both housing pressure and opportunity coexist, and where targeted policy interventions could help rebalance local housing markets and mitigate future pressures.

That said, there are several important limitations and unknowns:

There is no clear definition of what constitutes an acceptable level of underoccupation or overcrowding.
Similarly, the “acceptable” rate of empty homes is unclear—one could argue that any empty home represents inefficient use of resources.
The levels of overcrowding observed in the UK may be relatively low compared to those in other countries.
The methodology may overlook areas where underoccupation and overcrowding offset one another.
Housing mobility is constrained by financial, social, and temporal factors, meaning that reallocating space is a complex and long-term process for individuals and families. Some degree of empty or underused housing is likely unavoidable.

In future work, it would be useful to examine the size and type of empty homes, as affordability may be a key factor. As some sources suggest, “it is not the lack of housing, but the lack of affordable housing.” Additionally, incorporating a temporal dimension could provide further insight, particularly given indications that the situation may be worsening over time.

Retrieving custom networks from OSM using Pyrosm and translating to Pandana and NetworkX

Wed, 22 Jun 2022 00:00:00 +0000

Introduction

Pyrosm is a very clever python library, written by Henrikki Tenkane, that allows users to quickly and effectively retrieve data from OpenStreetMap, using Cython and Protocolbuffer Binary Format -files (PBFs). It is ultimately the most useful tool to retrieve OSM data within python and is fairly well documented.

Yet, there were few places which can be confusing and where extended description and tutorial is needed. This is specifically;

retrieving custom network (cycling example)
converting custom network to Pandana/Networkx graph

In this tutorial, I'll also repeat the distances calculation with pandana from this post and update on any changes.

I would have like to contribute to the Pyrosm library with the first point, however, Cython is not amongst my skills right now. So if someone knows how to do that, feel free.

Retrieving custom driving/walking/cycling network

The Pyrosm uses custom_filter to define what features should be extracted from the PBF, however, this filter is not available for the networks. Instead the Pyrosm offers get_data_by_custom_criteria() and keep vs exclude feature which allows you to define which keys to keep or exclude from the OSM. This function also allows you to choose if you want to keep the nodes in the data, but those are not the network nodes, those are point features that has the tags defined in your custom filter. This means, that you can get custom network, but only the edge list. However, in order to construct graph in Pandana we need both edge list and node list. Now you could just get the edge list, create a graph in Networkx and then translate to Pandana by exporting edgelist and node list out, but let's be realistic. If there is a way to avoid multiple conversions and installing another package, I would be much happier.

I found an easy way to retrieve both nodes and edges for custom created network. The process is as follows;

Retrieve complete network for the selected area
Filter out the edges features based on selected keys from OSM
List all the existing nodes in the edge dataframe
Filter the nodes by the list from the previous step

# import the necessary packages
import geopandas as gpd
import pandas as pd

# import pyrosm functions
from pyrosm import OSM
from pyrosm import get_data
# network libraries
import pandana
import networkx as nx

def get_network_custom(place,filter_type):
    
    """
    place: string 
        name of the pbf to be downloaded 
        
    filter_type: string
        'cycling' only for now
    """
    # connect to the data
    osm = OSM(get_data(place))
    
    ## if you have already downloaded pbf file and it’s in your file directory use following
    ## osm = OSM(place)
    ## where place is 'filepath.pbf'
    
    # retrieve the complete network 
    nodes,edges = osm.get_network(nodes=True, network_type="all")
    
    
    # define the filter
    if filter_type == 'cycling':
        highway = ['primary',
                'primary_link',
                'trunk',
                'trunk_link',
                'secondary',
                'secondary_link',
                'tertiary',
                'tertiary_link',
                'unclassified',
                'residential',
                'living_street', 
                'road',   
                'service',
                'track',
                'path',
                'pedestrian',
                'footway',
                'bridleway',
                'cycleway',
                'busway']
    else: raise ValueError('try cycling')
    # you could define other profiles such as driving or walking
       
    # choose only those edges that are inside the filter
    edges = edges[edges['highway'].isin(highway)]
    
    # list origin nodes
    list_origins = list(edges.u.unique())
    
    # list destination nodes
    list_destinations = list(edges.v.unique())
    
    # combine the to lists to see all nodes that exist in filtered network 
    list_nodes = list_origins + list_destinations
    
    # cut the nodes by the existing list
    nodes_gdf = nodes[nodes.id.isin(list_nodes)] 
            
    return nodes_gdf,edges

nodes, edges = get_network_custom('Bristol','cycling')

ax = edges.plot()
nodes.plot(ax = ax, color = 'orange', markersize = 2);

Converting custom network to pandana/networkx graph

Converting the two geodataframes to graphs is then dead easy.

nodes.info()

<class 'geopandas.geodataframe.GeoDataFrame'>
Int64Index: 515871 entries, 0 to 522320
Data columns (total 8 columns):
 #   Column     Non-Null Count   Dtype   
---  ------     --------------   -----   
 0   lon        515871 non-null  float64 
 1   lat        515871 non-null  float64 
 2   tags       20935 non-null   object  
 3   timestamp  515871 non-null  int8    
 4   version    515871 non-null  int8    
 5   changeset  515871 non-null  int8    
 6   id         515871 non-null  int64   
 7   geometry   515871 non-null  geometry
dtypes: float64(2), geometry(1), int64(1), int8(3), object(1)
memory usage: 25.1+ MB

edges.info()

<class 'geopandas.geodataframe.GeoDataFrame'>
Int64Index: 551813 entries, 0 to 560619
Data columns (total 43 columns):
 #   Column          Non-Null Count   Dtype   
---  ------          --------------   -----   
 0   access          45542 non-null   object  
 1   area            10881 non-null   object  
 2   bicycle         40145 non-null   object  
 3   bridge          2960 non-null    object  
 4   busway          84 non-null      object  
 5   cycleway        7300 non-null    object  
 6   est_width       785 non-null     object  
 7   foot            87694 non-null   object  
 8   footway         5027 non-null    object  
 9   highway         551813 non-null  object  
 10  int_ref         0 non-null       object  
 11  junction        6390 non-null    object  
 12  lanes           30165 non-null   object  
 13  lit             49202 non-null   object  
 14  maxspeed        97150 non-null   object  
 15  motorcar        672 non-null     object  
 16  motorroad       8 non-null       object  
 17  motor_vehicle   8631 non-null    object  
 18  name            202888 non-null  object  
 19  oneway          49142 non-null   object  
 20  overtaking      61 non-null      object  
 21  path            835 non-null     object  
 22  passing_places  47 non-null      object  
 23  psv             873 non-null     object  
 24  ref             25646 non-null   object  
 25  service         45451 non-null   object  
 26  segregated      13048 non-null   object  
 27  sidewalk        38667 non-null   object  
 28  smoothness      2099 non-null    object  
 29  surface         145345 non-null  object  
 30  tracktype       5161 non-null    object  
 31  tunnel          1281 non-null    object  
 32  turn            7 non-null       object  
 33  width           3221 non-null    object  
 34  id              551813 non-null  int64   
 35  timestamp       551813 non-null  int64   
 36  version         551813 non-null  int8    
 37  tags            183067 non-null  object  
 38  osm_type        551813 non-null  object  
 39  geometry        551813 non-null  geometry
 40  u               551813 non-null  int64   
 41  v               551813 non-null  int64   
 42  length          551813 non-null  float64 
dtypes: float64(1), geometry(1), int64(4), int8(1), object(36)
memory usage: 181.6+ MB

# This might be necessary if you get the error 
## ValueError: Buffer dtype mismatch, expected 'long' but got 'double'
nodes2 = nodes.set_index('id')

# create pandana graph
pandana_graph = pandana.Network(nodes2['lon'], 
                           nodes2['lat'], 
                           edges['u'], 
                           edges['v'], 
                           edges[['length']],
                           twoway=[False,True])

# create networkx graph
networkx_graph = nx.from_pandas_edgelist(edges,source = 'u', target = 'v', edge_attr = 'length')

Calculating shortest distance between set of points using pandana

pois = gpd.read_file("./points.geojson")
pois2 = pois.rename(columns = {'ID_code':'code_ID'})

import itertools

def get_shortest_path(points, polygons, point_id, poly_id, Graph):
    # check the projections
    if points.crs != 'EPSG:4326':
        raise ValueError("The point CRS is not Mercator, Use gpd.to_crs('EPSG4326') to transform the CRS")

    if polygons.crs != 'EPSG:4326':
        raise ValueError("The polygon CRS is not Mercator, Use gpd.to_crs('EPSG4326') to transform the CRS")

        """
        Parses graph and two sets of locational datasets and returns dataframe with Origin destination ids, distances and times along the network.
        
        Parameters
        ----------
        point : point geodataframe 
        polygons : polygon or second point geodataframe
        point_id : str
            unique ID column of points
        poly_id : str
            unique ID column of polygons or the second points
         Graph: object 
            pandana.network
        
        Returns
        -------
        dataframe with distances and times between each pair of points
        
        Return type
        -----------
        pandas.dataframe
        """


    # define the unique origins and destinations
    origins = list(points[point_id].unique())
    destinations = list(polygons[poly_id].unique())
    # create OD data
    od_data = pd.DataFrame(list(itertools.product(origins,destinations))).rename(columns = {0:point_id,1:poly_id})
    # add coordinates to the data
    # if the data is other then points, it would take centrooid of the geometry
    
    if points.type[0] == 'Point':
        point_xy = pd.DataFrame({point_id: points[point_id], 'X':points.geometry.x, 'Y':points.geometry.y}).reset_index(drop=True)
    elif points.type[0] == 'MultiPoint':
        point_xy = pd.DataFrame({point_id: points[point_id], 'X':points.geometry.x, 'Y':points.geometry.y}).reset_index(drop=True)
    else: point_xy = pd.DataFrame({point_id: points[point_id], 'X':points.geometry.centroid.x, 'Y':points.geometry.centroid.y}).reset_index(drop=True)
    
    
    if polygons.type[0] == 'Point':
        poly_xy = pd.DataFrame({poly_id: polygons[poly_id], 'X':polygons.geometry.x, 'Y':polygons.geometry.y}).reset_index(drop=True)
    elif polygons.type[0] == 'MultiPoint':
        poly_xy = pd.DataFrame({poly_id: polygons[poly_id], 'X':polygons.geometry.x, 'Y':polygons.geometry.y}).reset_index(drop=True)
    else: poly_xy = pd.DataFrame({poly_id: polygons[poly_id], 'X':polygons.geometry.centroid.x, 'Y':polygons.geometry.centroid.y}).reset_index(drop=True)
    
    # build origin destination data    
    od_data = od_data.merge(point_xy, on=point_id, how = 'left').merge(poly_xy, on = poly_id, how = 'left')

    # find nodes on the network
    origin_nodes = Graph.get_node_ids(od_data.X_x, od_data.Y_x).values
    dests_nodes = Graph.get_node_ids(od_data.X_y, od_data.Y_y).values

    # find the distances and times
    od_data['distance'] = pd.Series(Graph.shortest_path_lengths(origin_nodes, dests_nodes, 'length'))
            
   
    return od_data

import time

start = time.time()

dists = get_shortest_path(points = pois, polygons = pois2, point_id = 'ID_code', poly_id = 'code_ID', Graph = pandana_graph)

end = time.time()
print(end - start )

9.235082149505615

dists.distance.hist();

Notes: You can see that there is a suspicious distance that are bigger than 300 000 meters. This happens when the graph can't find the route for set of two points. This is most likely because one of the nodes is on disconnected graphs within the network and can't reach the other ones, or they could be outside the graph extent or something else. It would be smart to investigate the graph, but that’s not part of this post :)

Google Summer of Code 2021

Sat, 29 May 2021 00:00:00 +0000

Overview

Introduction

Welcome!

My name is Lenka, I am currently working on my PhD in Advanced Quantitative Geography at Bristol University , and over the summer 2021 I worked alongside Dr Levi Wolf and Dr Taylor Oshan on implementation of Competing Destination models for SpInt python package, as part of Google Summer of Code 2021.

Links & references

If you are future GSoC participant and you are interested in seeing my proposal, it is still living on my google drive.
To see all the work and commits, see this Github pull request. Or find my working repository here.
If you want to know how the work progressed step by step, keep reading this blog. Each entry has title, so you can jump to the most recent entry in the table of contents.
If you are interested what other projects look like, have a look at blogs of Noah and Germano who also completed their projects for Pysal.

State of the project

I am happy to say that everything stated in proposal has been completed. We have managed to create accessibility function, working test and notebook with an example. We also managed to optimize the function for big datasets and significantly speed up the calculation. Link for the pull request including the whole code is above. The blog documenting tho whole progress is just below, so keep reading if you are interested in that.

Summary & Final thoughts

We had a lot of discussion about the form of the accessibility. This is a duifficult one as there is many forms and ways it can be calculated, and incorporating them all into this the module would be close to impossible. By defining the function as generally as we did in this project allows the users to interchange wight variables, use different masses and even calculate accessibility for bipartite or tripirtite interaction systems. We hope this will be usefull to many researchers and python users who are looking for Competiong Destination replication.

I loved to be part of GSoC project, as well as the Pysal community. This experience gave me a lot of insight into python developement and especially into the community proccess of developement. This is something that I would not be able to experience otherwise.

Blog

1. Community Bonding, week 24th May till 1th June

Meeting the team

On the first meeting, I have been able to connect with my mentors Levi, who is also my PhD supervisor, and Taylor, who is an author of the SpInt package. It has been such a pleasure to connect with Taylor who lives roughly 8.5 thousand kilometres (yes I work in km's and I'm not aiming to change that, so deal with it) away from Bristol where me and Levi are based. We chat about their own experiences with GSoC, as they both have been a participants and now mentors, I was interested in their perspective.
We have also briefly chat about how the SpInt package is structured, so I get better idea of what will the task entail, and what the first steps are.

Meeting the Pysal community and other students with their project.

The pysal community has a meeting once a month, where they discuss current problems, updates and future improvements. One of the main things discussed on the meeting was the problem of fragmented graph packages and their inability to cater for translation of graphs from other packages such as street network graphs from osmnx. This is very relevant to my work as I have bumped into this issue in March this year. I was trying to speed up the computation of distances along the road for bunch of points and I found that osmnx has the best function for loading street network based on bounding boxes, but pandana has the fastest computation of shortest paths along the graph (the package is build on C++). To combine these two methods, we just need to translate the graph from osmnx graph to pandana graph. Check out this post to see how this is done and how you can calculate shortest paths using pandana.

It's been good to see how the community takes care of the open-source software, as this part is well hidden from an average user.

2. First Coding period. WEEK 1

Project introduction

For the general introduction into the project please read the proposal, which provides an overview for the problem.

There is one main deliverable for this project and that is a function that calculate Accessibility term for competing destination models. This can be broken down into X steps.

Create initial accessibility function
Create functional test to validate the function and its future versions
Research how the function could be speeded up

What is the accessibility function

According the initial paper from Fotheringham (1983) the accessibility $A_{ij}$ represents the accessibility of destination $j$ to all other destinations available to origin $i$ as perceived by the residents of origin $i$ and is defined as

This can be little confusing, so let's break it down.

Imagine a migration between 5 following English cities, where people from Liverpool migrate to South Hampton, Bristol and London, and people from London migrate to Brighton. If we want to calculate the accessibility term for the flow Liverpool to Bristol, we consider following; The accessibility of $\text{flow}_{Li,B}$ is the sum of the connections between the destination (Bristol) to all other destinations. This connection or accessibility between Bristol and other destinations can be then calculated as the distance between Bristol and destination times the mass of the destination.

In other words;

$$ \text{Accessibility for the flow between Liverpool and Bristol} = \sum \text{population SH * distance1, population Lo * distance2} $$

Important: We don't necessarily know if this is meant as an accessibility of all possible destinations in a system, or only those that actually exist for given origin.

I could design 2 functions

1. Accessibility of flow taking all potential destinations in the system 
2. Accessibility of flow taking only existing destinations

In those two functions, we would assume that the input is just one Data frame: Data Frame of all possible flows in system with distances between them, masses on destinations and the flow volumes between existing flows (0 where no flow exists). First I draw the second function which accounts only for those connections that exists in flow network.

Initial accessibility function

# coding=utf-8
# 3. AFED = Accessibility of flow taking existing destinations

def AFED(flow_df, row_index): # AFAPF
    
    # rename teh columns so we can call them 
    flow_df = flow_df.rename(columns = {flow_df.columns[0]:'origin_ID', 
                                            flow_df.columns[1]:'dest_ID', 
                                            flow_df.columns[2]:'dist', 
                                            flow_df.columns[3]:'weight', 
                                            flow_df.columns[4]:'dest_mass'})
    # define O and D for each row the variables
    D = flow_df['dest_ID'][row_index]
    O = flow_df['origin_ID'][row_index]
    
    # get the list of possible destinations
    all_dest = (flow_df.query('origin_ID == @O')
                .query('weight > 0')
                ['dest_ID']
                .unique()
               )    
    
    # Create all destination flows 
    x1 = pd.DataFrame({'D': np.array([D]*len(all_dest), dtype=object), 
                       'dests':all_dest}).merge(flow_df, how='left', left_on=['D','dests'], right_on=['origin_ID','dest_ID'])
    
    # merge with the distances and masses 
    
    # Delete the flow to origin
    x1 = x1[~x1.dests.isin(list(O))]    

    # calculate the accessibility
    A = (x1['dist']*x1['dest_mass']).sum()

    return A

Further notes on accessibility function

I have described accessibility function defined by Fotheringham (1983) as very generic function of distances and masses, however, more specific definition with different weighting could be considered for specific situations. For example, some could opt to substitute distance for some more specific measure of separation, such as travel time or journey cost. Similarly, there is many things that could be considered as a mass on the destination.

Note: it would be good to discuss the potential of the function to cater for alternative definitions of accessibility

3. First Coding period. WEEK 2 and 3

Extending the accessibility calculation, opt for all possible connections or just existing connections

As I said last week (few lines above), you can consider either all potential destinations or the existing ones.

Let me clarify this a bit. Estimating accessibility in a very generic way that we do here, we do in 3 steps.

For each unique combination of origin A and destination B;

find all the potential or all existing destinations to origin A
find the distances between all those destinations and destination B
compute (distance*destination mass) for all those connections from B to other destinations
deduct the (distance*destination mass) which belongs to destination B and origin A

In point 1, all potential destinations are those that exist in a Spatial Interaction system. Even though there is no flow between our origin A and destination X, it is available for people from Origin A.

However, if there is no flow between origin A and destination X in a system, then we could assume that people from that origin just do not consider that destination desirable. Thus, as competing destinations we could consider these, that receive at least some flow from origin A (flow volume > 0).

Have a look at this example:

In the updated function, I include an if statement which either constructs the competing destinations based on all potential flows or just the existing ones. This is controllable by the argument all_destination = False, where False (include only the existing flows) is the default option, while True will include all the potential destinations regardless of the flow volume.

# coding=utf-8
# 3. FlowAccessibility = Accessibility of flow taking existing destinations

def AFED(flow_df, row_index, all_destinations=False): 
    
    # rename teh columns so we can call them 
    flow_df = flow_df.rename(columns = {flow_df.columns[0]:'origin_ID', 
                                            flow_df.columns[1]:'dest_ID', 
                                            flow_df.columns[2]:'dist', 
                                            flow_df.columns[3]:'weight', 
                                            flow_df.columns[4]:'dest_mass'})
    # define O and D for each row the variables
    D = flow_df['dest_ID'][row_index]
    O = flow_df['origin_ID'][row_index]
    
    
    # get the list of possible destinations 
    if all_destinations:
         all_dest = (flow_df.query('origin_ID == @O')
                ['dest_ID']
                .unique()
               )
    else:
        all_dest = (flow_df.query('origin_ID == @O')
                .query('weight > 0')
                ['dest_ID']
                .unique()
               )

        
    # Create all destination flows 
    x1 = pd.DataFrame({'D': np.array([D]*len(all_dest), dtype=object), 
                       'dests':all_dest}).merge(flow_df, how='left', left_on=['D','dests'], right_on=['origin_ID','dest_ID'])
    
    # merge with the distances and masses 
    
    # Delete the flow to origin
    x1 = x1[~x1.dests.isin(list(O))]    

    # calculate the accessibility
    A = (x1['dist']*x1['dest_mass']).sum()

    return A

Application of Accessibility function for whole dataset

The function defined above calculates accessibility for a one row of data, for one connection in the interaction system. In next step, I need to do this for all rows in the dataset. This is probably not the most efficient way, but we need to start somewhere.

Below, I define another function that takes the accessibility computation, loops it over the full dataset supplied, and tells you how long it took.

def Accessibility(flow_df, function, all_destinations=False):
    start = timer()

    A_ij = []

    for idx in flow_df.index:
       
            if all_destinations: 
                A = function(flow_df=flow_df, row_index=idx, all_destinations=True)
            else:
                A = function(flow_df=flow_df, row_index=idx, all_destinations=False)
            A_ij.append(A)
                
    A_ij = pd.Series(A_ij)
    end = timer()

    print('time elapsed: ' + str(end - start))
    return A_ij

Contributing to GitHub

So, now is the time to get these online into a repo!

If you are not familiar with a GitHub yet, here are some useful links

Here is a video with the most effective explanation of what is GitHub.
This video explains and demonstrates forking, branching merging, pull requests and even more. I like this video and the presenter because he is very humanly and slightly quirky.
On the same channel, you can also find branching video and [Intro to GitHub video](](https://www.youtube.com/watch?v=_NrSWLQsDL4&t=491s).
This video provides a tutorial for the command line fundamentals. You can find more specific step-by-step guide here.

Lastly, here is a cheat-sheet that I created for myself. I am one of these people who do not hold in memory things that I can easily find online, however, that can be sometimes a bit lengthy. So, cheat-sheets are the best way I can quickly access things that I already did, but don't hold them in memory.

To contribute to a package, I did this;

fork the pysal/spint package = create copied-repository on my own GitHub
commit changes, new codes and fixes into this repo over the period of this project
once this project is getting to the end, and everybody is happy with the contribution, I will create pull request to send the code from my repo to the official pysal repo.

4. First Coding period. WEEK 4 and 5

Testing and validation

the next step is to create an efficient test, which will be automatically checking if the version of the function in the package is correct. To do this, I need to;

create a simple dummy dataset for which I manually calculate the desired accessibility values
create test_accessibility.py script that will use the function from the package to calculate accessibility for dummy data and check if it equals to manually calculated values.
make sure this test runs correctly within the framework

Create dummy dataset

The simplest way to test if function works is to see if its output matches the expected output. Nothing scary, nothing revolutionary. Simple yet effective.

Below you can see simple data set that I use for testing. It only has 5 points in space, but it has all possible types of connections and has realistic masses and distances.

Create test

I was not familiar with testing in python until now, so I needed to do some studying for this. I found these two links very useful;

There are two modules used for testing in python; pytest and unittest. Pytest is the newer one and is super easy to use. Check out this quick tutorial. While unittest is the more established module, that is already used within the pysal/spint.

"""
Tests for Accessibility function.

The correctness of the function is veryfied by matching the output to manually calculated values, using very simple dummy dataset.

"""

__author__ = 'Lenka Hasova haska.lenka@gmail.com'

import unittest
import numpy as np
from generate_dummy import generate_dummy_flows # this is a function that generates dummy flow
from ..flow_accessibility import Accessibility # This is the function that applies accessibility to whole dataset
from ..flow_accessibility import AFED # Accessibility function


class AccessibilityTest(unittest.TestCase):
    
    def test_accessibility(self):
        flow = generate_dummy_flows()
        flow = flow.loc[:,['origin_ID', 'destination_ID','distances', 'volume_in_unipartite','dest_masses','results_all=False']]
        flow['acc_uni'] = Accessibility(flow_df = flow, all_destinations=False, function = AFED)
        
        self.assertEqual(flow['results_all=False'].all(), flow['acc_uni'].all())

if __name__ == '__main__':
    unittest.main()

To run the test on your local machine, just open your anaconda prompt, navigate to directory where all the tests are and run following

$ python -m unittest test_accessibility.py

Or simply just push the code online and login to Travis.Cl. Travis runs all the test every time changes are made in your repo. Don't forget to signup to Travis and connect it to your GitHub. So cool!

5. What's next?

With the test that works, most of the important work on this project is basically ready to be submitted. However, because the function compromises 2 for loops, it is more likely to be very slow when you apply it on the big data. In real word research, you don't just have 5 points in space that creates $5*5=25$ flows. Often you have tens, hundreds and even thousands of points. For example, if you would have to compute accessibility for all the flows between all UK train stations (there is 2,563 of them) you would have 6,568,969 flows.

The rest of the GSoC project I will dedicate to trying to find out how fast is the current function, what is the best way to speed it up and how that can be implemented in practice.

6. Second coding period, making the code faster

The function I wrote initially can be quite slow. We can already tell that from the code, where we use for loop to calculate accessibility for each row of your dataset. for loops are not very efficient way to compute things, unless you have very few jobs.

There are few ways the speed up your code. Here are the two most appropriate options:

If I would really need the loops, Numba is a really good way to speed it up by allowing for parralel processing. But sometimes the loops are not that necessary and can be rewriten more efficiently using arrays and matrices. So lets try to do that!

The code bellows shows you 3 functions and compares their speed on real data (set of points located within Bristol).

orginal function from above (AFED)
function that loops over set of arrays (AFED_MAT)
function that multiplies 3-d arrays (no loops) (AFED_MAT_2)

import numpy as np
import pandas as pd
import geopandas as gpd
import itertools
import libpysal
from timeit import default_timer as timer
import matplotlib.pyplot as plt

from geopy.distance import geodesic, great_circle
from pyproj import CRS

Define the functions

def AFED(flow_df, row_index, all_destinations=False): 
    
    # rename teh columns so we can call them 
    flow_df = flow_df.rename(columns = {flow_df.columns[0]:'origin_ID', 
                                            flow_df.columns[1]:'dest_ID', 
                                            flow_df.columns[2]:'dist', 
                                            flow_df.columns[3]:'weight', 
                                            flow_df.columns[4]:'dest_mass'})

    
    flow_df['dist'] = flow_df['dist'].astype(int)
    flow_df['weight'] = flow_df['weight'].astype(int)
    flow_df['dest_mass'] = flow_df['dest_mass'].astype(int)
    
        # define O and D for each row the variables
    D = flow_df['dest_ID'][row_index]
    O = flow_df['origin_ID'][row_index]
    
    
    # get the list of possible destinations 
    if all_destinations:
         all_dest = (flow_df.query('origin_ID == @O')
                ['dest_ID']
                .unique()
               )
    else:
        all_dest = (flow_df.query('origin_ID == @O')
                .query('weight > 0')
                ['dest_ID']
                .unique()
               )

        
    # Create all destination flows 
    x1 = pd.DataFrame({'D': np.array([D]*len(all_dest), dtype=object
                                    ), 'dests':all_dest}
                     ).merge(flow_df, how='left', left_on=['D','dests'], right_on=['origin_ID','dest_ID'])
    
    # merge with the distances and masses 
    
    # Delete the flow to origin
    x1 = x1[~x1.dests.isin([O])]    

    # calculate the accessibility
    A = (x1['dist']*x1['dest_mass']).sum()

    return A

import numpy as np
import pandas as pd
from timeit import default_timer as timer
import itertools


def AFED_MAT(flow_df, all_destinations=False):
    
     # rename teh columns so we can call them 
    flow_df = flow_df.rename(columns = {flow_df.columns[0]:'origin_ID', 
                                            flow_df.columns[1]:'dest_ID', 
                                            flow_df.columns[2]:'dist', 
                                            flow_df.columns[3]:'weight', 
                                            flow_df.columns[4]:'dest_mass'})
    
    flow_df['dist'] = flow_df['dist'].astype(int)
    flow_df['weight'] = flow_df['weight'].astype(int)
    flow_df['dest_mass'] = flow_df['dest_mass'].astype(int)
    
    # create binary for weight
    flow_df['v_bin'] = 1
    flow_df['v_bin'][(flow_df['weight'].isna() | flow_df['weight'] == 0 ) ] = 0
    
    # define base matrices
    distance = flow_df.pivot_table(values='dist', index='dest_ID', columns="origin_ID")
    mass = flow_df.pivot_table(values='dest_mass', columns='dest_ID', index="origin_ID")
    exists = flow_df.pivot_table(values='v_bin', index='dest_ID', columns="origin_ID")
    
    # lists of O-D
    list_c = list(itertools.product(exists.columns,exists.index))
    df = pd.DataFrame(list_c).rename(columns = {0:'origin_ID', 1:'dest_ID'})
    
    # Start function
    array = []
    start = timer()
    for i,j in list_c:
        
        if all_destinations:
            f = exists.loc[i,:]
            f[0] =  1
            f[f.index.isin([i])] = 0
        else:
            f = exists.loc[i,:]
            f[f.index.isin([i])] = 0
            
        empty = pd.DataFrame(np.zeros((len(exists.index),len(exists.columns))), index = exists.index, columns = exists.columns)
        empty.loc[j,:] = f
        A_ij = empty *  (distance * mass)
        A_ij = A_ij.sum().sum()
        array.append(A_ij)
    
    # Get the result back to original dataframe
    df['A_ij'] = array
    flow_df = flow_df.merge(df, how = 'left', on = ['origin_ID','dest_ID'])

    #how long did this took?
    end = timer()
    print('time elapsed: ' + str(end - start))
    return(flow_df)

def AFED_VECTOR(flow_df, all_destinations=False):
    start = timer()
     # rename teh columns so we can call them 
    flow_df = flow_df.rename(columns = {flow_df.columns[0]:'origin_ID', 
                                            flow_df.columns[1]:'dest_ID', 
                                            flow_df.columns[2]:'dist', 
                                            flow_df.columns[3]:'weight', 
                                            flow_df.columns[4]:'dest_mass'})
    
    flow_df['dist'] = flow_df['dist'].astype(int)
    flow_df['weight'] = flow_df['weight'].astype(int)
    flow_df['dest_mass'] = flow_df['dest_mass'].astype(int)
    # create binary for weight
    flow_df['v_bin'] = 1
    flow_df.loc[flow_df['weight'].isna(),'v_bin'] = 0
    flow_df.loc[flow_df['weight'] <= 0,'v_bin'] = 0
    
    # define the base matrices
    distance = np.array(flow_df.pivot_table(values='dist', index='origin_ID', columns="dest_ID"))
    mass = np.array(flow_df.pivot_table(values='dest_mass', columns="origin_ID", index='dest_ID'))
    exists = np.array(flow_df.pivot_table(values='v_bin', index='dest_ID', columns="origin_ID"))
    
    # define the base 3d array
    nrows= len(exists)
    ones = np.ones((nrows,len(flow_df.origin_ID.unique()),len(flow_df.dest_ID.unique())))
    
    # define the identity array
    idn = np.identity(nrows) 
    idn = np.where((idn==1), 0, 1)
    idn = np.concatenate(nrows * [idn.ravel()], axis = 0).reshape(nrows,nrows,nrows).T

    # multiply the distance by mass
    ard = np.array(distance)*np.array(mass)
    
    # combine all into and calculate the output
    
    if all_destinations:
        output = np.array(idn) * (np.array(nrows * [ard]
                                              )
                                     )
        
    else:
        output = (np.concatenate(nrows * [exists], axis = 0
                                ).reshape(nrows,nrows,nrows
                                         ).T 
                 ) * np.array(idn) * (np.array(nrows * [ard]
                                              )
                                     )
    
    # get the sum and covert to series
    g = pd.DataFrame((
        np.sum(output,axis = 1
                           )  ).reshape(1,len(flow_df)
                                                    ).T
                    )
    
    end = timer()
    print('time elapsed: ' + str(end - start))
    return g[0]

Define the data sets for testing

pois = gpd.read_file("./../points.geojson")
pois.info()

<class 'geopandas.geodataframe.GeoDataFrame'>
RangeIndex: 855 entries, 0 to 854
Data columns (total 4 columns):
 #   Column    Non-Null Count  Dtype   
---  ------    --------------  -----   
 0   ID_code   855 non-null    object  
 1   X         855 non-null    float64 
 2   Y         855 non-null    float64 
 3   geometry  855 non-null    geometry
dtypes: float64(2), geometry(1), object(1)
memory usage: 26.8+ KB

# strip
pois_all = pois.iloc[:,[0,3]]

# Add random masses
pois_all['mass'] = pd.Series( np.random.randint(0,1500, size=len(pois_all))
                            )
# create stripped column of coordinates 
pois_all['xy'] = pois_all.geometry.apply(lambda x: [x.y, x.x])

# get all unique combinations of all the origins and destinations
flow_all = pd.DataFrame( list( itertools.product( pois_all['ID_code'].unique(),pois_all['ID_code'].unique())
                             )
                       ).rename(columns = {0:'origin',
                                        1:'destination'})

# joining the xy to flows and create distances
distances_all = flow_all.merge(pois_all.loc[:,['ID_code','xy']], how = 'left', left_on = 'origin', right_on = 'ID_code' 
                        ).merge(pois_all.loc[:,['ID_code','xy']], how = 'left', left_on = 'destination', right_on = 'ID_code'
                               )


# calculate distances
distances_all['great_circle_dist'] = distances_all.apply(lambda x: great_circle(x.xy_x, x.xy_y).km, axis=1)
distances_all = distances_all.loc[:,['origin','destination','great_circle_dist']]

# list points
list5 = list(pois_all['ID_code'][0:6])
list10 = list(pois_all['ID_code'][0:11])
list20 = list(pois_all['ID_code'][0:21])
list40 = list(pois_all['ID_code'][0:41])
list80 = list(pois_all['ID_code'][0:81])
list160 = list(pois_all['ID_code'][0:161])

# Define different number of points
pois_all_s = pd.DataFrame(pois_all.loc[:,['ID_code','mass']])
pois5 = pois_all_s[pois_all_s['ID_code'].isin(list5)]
pois10 = pois_all_s[pois_all_s['ID_code'].isin(list10)]
pois20 = pois_all_s[pois_all_s['ID_code'].isin(list20)]
pois40 = pois_all_s[pois_all_s['ID_code'].isin(list40)]
pois80 = pois_all_s[pois_all_s['ID_code'].isin(list80)]
pois160 = pois_all_s[pois_all_s['ID_code'].isin(list160)]

flow5 = flow_all[flow_all['origin'].isin(list5) & flow_all['destination'].isin(list5)].reset_index(drop=True)
flow10 = flow_all[flow_all['origin'].isin(list10) & flow_all['destination'].isin(list10)].reset_index(drop=True)
flow20 = flow_all[flow_all['origin'].isin(list20) & flow_all['destination'].isin(list20)].reset_index(drop=True)
flow40 = flow_all[flow_all['origin'].isin(list40) & flow_all['destination'].isin(list40)].reset_index(drop=True)
flow80 = flow_all[flow_all['origin'].isin(list80) & flow_all['destination'].isin(list80)].reset_index(drop=True)
flow160 = flow_all[flow_all['origin'].isin(list160) & flow_all['destination'].isin(list160)].reset_index(drop=True)

distances5 = distances_all[distances_all['origin'].isin(list5) & distances_all['destination'].isin(list5)].reset_index(drop=True)
distances10 = distances_all[distances_all['origin'].isin(list10) & distances_all['destination'].isin(list10)].reset_index(drop=True)
distances20 = distances_all[distances_all['origin'].isin(list20) & distances_all['destination'].isin(list20)].reset_index(drop=True)
distances40 = distances_all[distances_all['origin'].isin(list40) & distances_all['destination'].isin(list40)].reset_index(drop=True)
distances80 = distances_all[distances_all['origin'].isin(list80) & distances_all['destination'].isin(list80)].reset_index(drop=True)
distances160 = distances_all[distances_all['origin'].isin(list160) & distances_all['destination'].isin(list160)].reset_index(drop=True)

li_ommited = [1,4,5,9,13,15,17,26,27,28,34]

flow5_ommited = flow5.drop(flow5.index[li_ommited]).reset_index(drop=True)
flow10_ommited = flow10.drop(flow10.index[li_ommited]).reset_index(drop=True)
flow20_ommited = flow20.drop(flow20.index[li_ommited]).reset_index(drop=True)
flow40_ommited = flow40.drop(flow40.index[li_ommited]).reset_index(drop=True)
flow80_ommited = flow80.drop(flow80.index[li_ommited]).reset_index(drop=True)
flow160_ommited = flow160.drop(flow160.index[li_ommited]).reset_index(drop=True)

# combine the points and distances
flow_integrated = distances_all.merge(pois_all_s, how = 'left', left_on=['destination'], right_on=['ID_code'])

#generate some weights
flow_integrated['weight'] = pd.Series( np.random.randint(10,1500, size=len(flow_integrated)) # here - and 0 represent flows that does not exist
                                     )
# reorder
flow_integrated = flow_integrated.loc[:,['origin','destination','great_circle_dist','weight','mass']]

# build short datasets
flow5_integrated = flow_integrated[flow_integrated['origin'].isin(list5) & flow_integrated['destination'].isin(list5)].reset_index(drop=True)
flow10_integrated = flow_integrated[flow_integrated['origin'].isin(list10) & flow_integrated['destination'].isin(list10)].reset_index(drop=True)
flow20_integrated = flow_integrated[flow_integrated['origin'].isin(list20) & flow_integrated['destination'].isin(list20)].reset_index(drop=True)
flow40_integrated = flow_integrated[flow_integrated['origin'].isin(list40) & flow_integrated['destination'].isin(list40)].reset_index(drop=True)
flow80_integrated = flow_integrated[flow_integrated['origin'].isin(list80) & flow_integrated['destination'].isin(list80)].reset_index(drop=True)
flow160_integrated = flow_integrated[flow_integrated['origin'].isin(list160) & flow_integrated['destination'].isin(list160)].reset_index(drop=True)

flow5_integrated['great_circle_dist'] = round(flow5_integrated['great_circle_dist'],0)
flow10_integrated['great_circle_dist'] = round(flow10_integrated['great_circle_dist'],0)
flow20_integrated['great_circle_dist'] = round(flow20_integrated['great_circle_dist'],0)
flow40_integrated['great_circle_dist'] = round(flow40_integrated['great_circle_dist'],0)
flow80_integrated['great_circle_dist'] = round(flow80_integrated['great_circle_dist'],0)
flow160_integrated['great_circle_dist'] = round(flow160_integrated['great_circle_dist'],0)

flow5_integrated.head()

	origin	destination	great_circle_dist	weight	mass
0	A91120	A91120	0.0	906	1039
1	A91120	A99931	5.0	585	993
2	A91120	A99986	3.0	264	600
3	A91120	L81002	30.0	1495	1408
4	A91120	L81004	15.0	1135	762

Apply the functions


# for 05 dataset function 3
list_AFED = []

# 5
start = timer()
A_ij = []
for idx in flow5_integrated.index:
    A = AFED(flow_df=flow5_integrated, row_index=idx)
    A_ij.append(A)
end = timer()
t = (end - start)
list_AFED.append(t)

# 10
start = timer()
A_ij = []
for idx in flow10_integrated.index:
    A = AFED(flow_df=flow10_integrated, row_index=idx)
    A_ij.append(A)
end = timer()
t = (end - start)
list_AFED.append(t)

# 20
start = timer()
A_ij = []
for idx in flow20_integrated.index:
    A = AFED(flow_df=flow20_integrated, row_index=idx)
    A_ij.append(A)
end = timer()
t = (end - start)
list_AFED.append(t)

# 40
start = timer()
A_ij = []
for idx in flow40_integrated.index:
    A = AFED(flow_df=flow40_integrated, row_index=idx)
    A_ij.append(A)
end = timer()
t = (end - start)
list_AFED.append(t)

# 80
start = timer()
A_ij = []
for idx in flow80_integrated.index:
    A = AFED(flow_df=flow80_integrated, row_index=idx)
    A_ij.append(A)
end = timer()
t = (end - start)
list_AFED.append(t)

# 160
start = timer()
A_ij = []
for idx in flow160_integrated.index:
    A = AFED(flow_df=flow160_integrated, row_index=idx)
    A_ij.append(A)
end = timer()
t = (end - start)
list_AFED.append(t)

list_AFED

[0.26563450000000444,
 0.8232417000000112,
 3.175610599999999,
 12.360262700000007,
 54.0198017,
 303.65919140000005]


# for 05 dataset function 3
list_AFED_MAT = []

# 5
start = timer()
flow5_mat = AFED_MAT(flow5_integrated)
end = timer()
t = (end - start)
list_AFED_MAT.append(t)

# 10
start = timer()
A_ij = []
flow10_mat = AFED_MAT(flow10_integrated)
end = timer()
t = (end - start)
list_AFED_MAT.append(t)

# 20
start = timer()
A_ij = []
flow20_mat = AFED_MAT(flow20_integrated)
end = timer()
t = (end - start)
list_AFED_MAT.append(t)

# 40
start = timer()
A_ij = []
flow40_mat = AFED_MAT(flow40_integrated)
end = timer()
t = (end - start)
list_AFED_MAT.append(t)

# 80
start = timer()
A_ij = []
flow80_mat = AFED_MAT(flow80_integrated)
end = timer()
t = (end - start)
list_AFED_MAT.append(t)

# 160
start = timer()
A_ij = []
flow160_mat = AFED_MAT(flow160_integrated)
end = timer()
t = (end - start)
list_AFED_MAT.append(t)

time elapsed: 0.06575129999998808
time elapsed: 0.2165177999999628 
time elapsed: 0.8499411000000237   
time elapsed: 3.23107200000004   
time elapsed: 14.276894200000015
time elapsed: 93.39352159999999


# for 05 dataset function 3
list_AFED_VECTOR = []

# 5
start = timer()
flow5_mat = AFED_VECTOR(flow5_integrated)
end = timer()
t = (end - start)
list_AFED_VECTOR.append(t)

# 10
start = timer()
A_ij = []
flow10_mat = AFED_VECTOR(flow10_integrated)
end = timer()
t = (end - start)
list_AFED_VECTOR.append(t)

# 20
start = timer()
A_ij = []
flow20_mat = AFED_VECTOR(flow20_integrated)
end = timer()
t = (end - start)
list_AFED_VECTOR.append(t)

# 40
start = timer()
A_ij = []
flow40_mat = AFED_VECTOR(flow40_integrated)
end = timer()
t = (end - start)
list_AFED_VECTOR.append(t)

# 80
start = timer()
A_ij = []
flow80_mat = AFED_VECTOR(flow80_integrated)
end = timer()
t = (end - start)
list_AFED_VECTOR.append(t)

# 160
start = timer()
A_ij = []
flow160_mat = AFED_VECTOR(flow160_integrated)
end = timer()
t = (end - start)
list_AFED_VECTOR.append(t)

time elapsed: 0.023291800000038165
time elapsed: 0.023622900000077607
time elapsed: 0.022046499999987645
time elapsed: 0.026168299999994815
time elapsed: 0.04007190000004357
time elapsed: 0.15891480000004776

Evaluate

times = pd.DataFrame({'original': pd.Series(list_AFED),
                     'matrices': pd.Series(list_AFED_MAT),
                    'vectorized': pd.Series(list_AFED_VECTOR)})
times

	original	matrices	vectorized
0	0.265635	0.093522	0.023567
1	0.823242	0.240578	0.024067
2	3.175611	0.879755	0.022347
3	12.360263	3.258259	0.026568
4	54.019802	14.310321	0.041264
5	303.659191	93.476836	0.166047

fig, ax = plt.subplots(figsize=(15, 10))
(times.iloc[:,0:3]/60).plot(ax = ax,  linewidth=5)
plt.xlabel("Number of observations/flows",size=15)
plt.ylabel("Minutes",size=15)
plt.xticks(times.index, (str(len(flow5)), str(len(flow10)), str(len(flow20)), str(len(flow40)),str(len(flow80)), str(len(flow160))),size=15)
plt.yticks(size=15)
plt.legend( prop={'size': 15})
plt.show();

From the graph above you can see that with each new finction the speed improved. There is very little difference in the speed if your data has less then 200 observations/flows (or 14 origins and destinations). However, when your data sets are bigger, the speed becomes significantly better with the vectorized function, then the original one. For example, 25,921 observations/flows (161 origins and destinations will take only 17 seconds to compute by vectorized function, in compare with the original one that takes close to 6 minutes to compute.

7. Changes after review

After the mentors reviewed my code, there was few places for improvements.

Instead of feeding dataframe in, we can just feed numpy arrays. This simply makes things easier for the user as less thing can go wrong with the input.
We can create an error message which checks that all the inputs are the same length. This will validate that the input is correct.
There is also some unnecessary stuff to delete.
The test could use more appropriate function.

Final version of function

def Accessibility(nodes, distances, weights, masses, all_destinations=False):
    
    # convert numbers to integers
    distances = np.array(distances.astype(int))
    weights = np.array(weights.astype(int))
    masses = np.array(masses.astype(int))
    nodes = np.array(nodes)
    
    # define error
    if len(distances) != len(weights) != len(masses) != len(nodes):
        raise ValueError("One of the input array is different length then the others, but they should all be the same length. See notebook example if you are unsure what the input should look like ")
    
    # define number of rows
    nrows= len(nodes)
    uniques = len(np.unique(np.array(nodes)))
    
    # create binary for weight
    v_bin =  np.ones(nrows)
    weights[np.isnan(weights)] = 0
    v_bin[weights <= 0] = 0
    
    # define the base matrices
    distance = distances.reshape(uniques,uniques)
    mass =masses.reshape(uniques,uniques).T
    exists = v_bin.reshape(uniques,uniques)
    
      
    # define the identity array
    idn = np.identity(uniques) 
    idn = np.where((idn==1), 0, 1)
    idn = np.concatenate(uniques * [idn.ravel()], axis = 0
                        ).reshape(uniques,uniques,uniques
                                 ).T

    # multiply the distance by mass
    dm = distance * mass
    
    # combine all matrices for either all or existing destinations
    if all_destinations:
        output = idn * (nrows * [dm])
        
    else:
        output = (np.concatenate(uniques * [exists], axis = 0
                                ).reshape(uniques,uniques,uniques
                                         ).T
                 ) * idn * (uniques * [dm]
                           )
    
    # get the sum and covert to series
    output = (np.sum(output,axis = 1
                    ) 
             ).reshape(nrows
                      ).T
    
    
    return output

Final version of test

class AccessibilityTest(unittest.TestCase):
    
    def test_accessibility(self):
        flow = _generate_dummy_flows()
        flow = flow.loc[:,['origin_ID', 'destination_ID','distances', 'volume_in_unipartite','dest_masses','results_all=False']]
        flow['acc_uni'] = Accessibility(nodes = flow['origin_ID'],  distances = flow['distances'], weights = flow['volume_in_unipartite'], masses = flow['dest_masses'], all_destinations=False)
        
        np.testing.assert_array_equal(flow['results_all=False'].all(), flow['acc_uni'].all())

if __name__ == '__main__':
    unittest.main()

Calculating Walking distance in Python. Networkx vs Pandana.

Fri, 12 Mar 2021 00:00:00 +0000

Introduction

Let's say you have spatial point data and you want to calculate walking distances between all combination of the points. For example, you have data of local bars and you want to find out all the walking distances between them, so you can include them in your regression model (or anything else), for example as a variable that defines how accessible two pubs are to each other.

This post will show you how you can do that in python using

osmnx package to generate OSM road network
networkx package to find the nearest point on that network and calulate the walking distance between them
pandana package to make this significantly faster!

Data

Availiable here

Background

You might need some essential information that will help you to understand what we are doing here. Feel free to skip this part and go straight to the code.

What is the walking distance?

It's exactly what it says. Walking distance from one place to another. In other words, how far is the place from where I stand along the walking path. This information is crucial for estimating how long is it going to take me to go there, how much effort or money will this cost me. Those are all variables that contribute to estimations of accessibility, connectivity, mobility or others.

Estimating the walking distance.

The distance between two places/nodes is then the sum of the weights/lengths of all the paths/roads/edges that connects them.

This can be also referred to as the Shortest Path Problem, in which we are looking for the most efficient paths between nodes.

In this example, the shortest path between Node 1 and Node 2 is the $$SUM(Weight 3, Weight 2, Weight 4)$$. So if the weights represent a number of minutes (how long it takes to walk that path) then we would walk for 7 minutes. $$ 1+2+4=7$$

If would choose to go the other way, which is seemingly shorter, $$SUM(Weight 1, Weight 4)$$ $$8+4=12$$ We would walk for 12 minutes. Therefore the first option is the most efficient one.

Find the Nearest Neighbour

In any scenarios where the places you want to use for your analysis are not directly placed on the road or path, we need to connect the place to the network. We do this by finding the Nearest neighbouring node on the graph/network to the specific place. Both packages (networkx and pandana) have their own functions that can be used for these purposes.

Load the packages

import pandas as pd
import numpy as np
import geopandas as gpd
import matplotlib.pyplot as plt
import osmnx as ox # install osmnx first, it will download appropriate version of networkx
import networkx as nx
from pyproj import CRS
import itertools

import pandana
print(pandana.__version__)

# define coordinates if you need them
wgs84 = CRS(4326)
bng = CRS(27700)
print(wgs84)

epsg:4326

Load the point data and create flows

pois = gpd.read_file("./points.geojson")
#pois.crs # check the CRS

pois.head()

	ID_code	X	Y	geometry
0	A91120	-2.559063	51.502830	POINT (-2.55906 51.50283)
1	A99931	-2.596140	51.459182	POINT (-2.59614 51.45918)
2	A99986	-2.538833	51.483143	POINT (-2.53883 51.48314)
3	L81002	-2.930352	51.360207	POINT (-2.93035 51.36021)
4	L81004	-2.767317	51.482805	POINT (-2.76732 51.48281)

# tale first 5 points and find all possinle combinations
pois = pois.iloc[0:6,:]

# create a list of all the ID's
p_li = list(pois['ID_code'].unique())

# get all unique combinations of all the origins and destinations
flows = pd.DataFrame(list(itertools.product(p_li,p_li))).rename(columns = {0:'origin',1:'destination'})

flows.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 36 entries, 0 to 35
Data columns (total 2 columns):
 #   Column       Non-Null Count  Dtype 
---  ------       --------------  ----- 
 0   origin       36 non-null     object
 1   destination  36 non-null     object
dtypes: object(2)
memory usage: 704.0+ bytes

Using Networkx

Load Graph

The osmnx package has very useful function ‘graph_from_bbox’ which loads the osm roads graph quite quickly. Here I am loading walking paths for whola area of Avon where my points are located, which is quite a big area so expect this to take few minutes.

# create bounding box for our data 
bbox = [51.623985, 51.291124,  -2.272797, -3.029480] 

# generate graph based on the bounding box
avon = ox.graph_from_bbox(bbox[0], bbox[1], bbox[2], bbox[3], 
                               retain_all=False, 
                               truncate_by_edge=True, 
                               simplify=False,
                               network_type='walk')

# get the nodes and the edges into geopandas
nodes, edges = ox.graph_to_gdfs(avon, nodes=True, edges=True)

Find the nearest node to each point with osmnx.get_nearest_node()

# define function for the nearest neighbour
def nearest_node(a,b):
    nearest_node,dist=ox.get_nearest_node(avon, (a,b), return_dist=True, method = 'euclidean')  
    return nearest_node

# apply the function
pois['NX_node'] = np.vectorize(nearest_node)(pois['Y'],pois['X'])
pois.head()

	ID_code	X	Y	geometry	NX_node
0	A91120	-2.559063	51.502830	POINT (-2.55906 51.50283)	2914306154
1	A99931	-2.596140	51.459182	POINT (-2.59614 51.45918)	1859320333
2	A99986	-2.538833	51.483143	POINT (-2.53883 51.48314)	2094196650
3	L81002	-2.930352	51.360207	POINT (-2.93035 51.36021)	317399984
4	L81004	-2.767317	51.482805	POINT (-2.76732 51.48281)	2488740793

# create list of the node id's and check if they exist and if they are on the right place
nodelist = list(pois['NX_node'].unique())

nodes[nodes.index.isin(nodelist)]

	y	x	street_count	highway	ref	geometry
osmid
317399984	51.360246	-2.930589	2	NaN	NaN	POINT (-2.93059 51.36025)
1859320333	51.459442	-2.596420	2	NaN	NaN	POINT (-2.59642 51.45944)
2094196650	51.482695	-2.538694	2	NaN	NaN	POINT (-2.53869 51.48270)
2424544775	51.413646	-2.571651	2	NaN	NaN	POINT (-2.57165 51.41365)
2488740793	51.483134	-2.767173	2	NaN	NaN	POINT (-2.76717 51.48313)
2914306154	51.502744	-2.558880	1	NaN	NaN	POINT (-2.55888 51.50274)

# get the node id's to flow data
# create two seperate data the target origins and target destinations with the XY coordinates
flows_full = flows.merge(pois.loc[:, pois.columns != 'geometry'], left_on = 'origin', right_on = 'ID_code', how = 'left')

# lets just rename everything for the sake of clarity
flows_full = flows_full.merge(pois.loc[:, pois.columns != 'geometry'], left_on = 'destination', right_on = 'ID_code', how = 'left').rename(columns = {'NX_node_x':'node_o','NX_node_y':'node_d', 'X_x':'X_o','Y_x':'Y_o', 'X_y':'X_d','Y_y':'Y_d', 'ID_code_x':'ID_code_o', 'ID_code_y':'ID_code_d'})

flows_full.head()

	origin	destination	ID_code_o	X_o	Y_o	node_o	ID_code_d	X_d	Y_d	node_d
0	A91120	A91120	A91120	-2.559063	51.50283	2914306154	A91120	-2.559063	51.502830	2914306154
1	A91120	A99931	A91120	-2.559063	51.50283	2914306154	A99931	-2.596140	51.459182	1859320333
2	A91120	A99986	A91120	-2.559063	51.50283	2914306154	A99986	-2.538833	51.483143	2094196650
3	A91120	L81002	A91120	-2.559063	51.50283	2914306154	L81002	-2.930352	51.360207	317399984
4	A91120	L81004	A91120	-2.559063	51.50283	2914306154	L81004	-2.767317	51.482805	2488740793

Check that the nodes are on the corect place

# check that the nodes are close to the points
plt.figure( figsize=(10,10))
fig = plt.plot()
ax = plt.axes()

nodes[nodes.index.isin(nodelist)].plot(ax=ax, markersize = 60, color="b" )
pois.plot(ax=ax, markersize = 20, color="r")

plt.show();

Looks fine to me

Apply the networkx.shortest_path_length

# define a function that calculates shortest path between the nodes on graph
def path_length(row):
    return nx.shortest_path_length(avon, row['node_o'], row['node_d'], weight='length')

# apply the function to our OD data
%timeit flows_full['path_length'] = flows_full.apply(path_length, axis=1)

26.8 s ± 1.58 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

# This is approximately
print(str(round((26.8*(7*1))/60,2)) + ' minutes')

3.13 minutes

flows_full.head()

	origin	destination	ID_code_o	X_o	Y_o	node_o	ID_code_d	X_d	Y_d	node_d	path_length
0	A91120	A91120	A91120	-2.559063	51.50283	2914306154	A91120	-2.559063	51.502830	2914306154	0.000
1	A91120	A99931	A91120	-2.559063	51.50283	2914306154	A99931	-2.596140	51.459182	1859320333	6960.514
2	A91120	A99986	A91120	-2.559063	51.50283	2914306154	A99986	-2.538833	51.483143	2094196650	4333.457
3	A91120	L81002	A91120	-2.559063	51.50283	2914306154	L81002	-2.930352	51.360207	317399984	36334.316
4	A91120	L81004	A91120	-2.559063	51.50283	2914306154	L81004	-2.767317	51.482805	2488740793	19754.732

Using Pandana

Load the graph

Listen, I have tried to use the osm loader inside the pandana, but it took ages. after 2 hours of loading, I decided it will be easier to get the previously loaded osm graph with osmnx and use that as a base for the pandana graph. You can just simply use the geodataframes of nodes and flows for the base of your graph.

# reset index so our origins and destinations are not in idnex
edges = edges.reset_index()

# create network with pandana
avon_pan = pandana.Network(nodes['x'], nodes['y'], 
                          edges['u'], edges['v'], edges[['length']])

Find the nearest nodes to our points in pandana graph

# we are going to define the origins and destinations separately as alist
# get_node_ids uses kdtree writen in c++, google it its like a magic

origin_nodes = avon_pan.get_node_ids(flows_full.X_o, flows_full.Y_o).values


dests_nodes = avon_pan.get_node_ids(flows_full.X_d, flows_full.Y_d).values

flows_full.head()

	origin	destination	ID_code_o	X_o	Y_o	node_o	ID_code_d	X_d	Y_d	node_d	path_length
0	A91120	A91120	A91120	-2.559063	51.50283	2914306154	A91120	-2.559063	51.502830	2914306154	0.000
1	A91120	A99931	A91120	-2.559063	51.50283	2914306154	A99931	-2.596140	51.459182	1859320333	6960.514
2	A91120	A99986	A91120	-2.559063	51.50283	2914306154	A99986	-2.538833	51.483143	2094196650	4333.457
3	A91120	L81002	A91120	-2.559063	51.50283	2914306154	L81002	-2.930352	51.360207	317399984	36334.316
4	A91120	L81004	A91120	-2.559063	51.50283	2914306154	L81004	-2.767317	51.482805	2488740793	19754.732

Calculate distances using pandana ‘shortest_path_lengths’

%%time
flows_full['distances'] = pd.Series(avon_pan.shortest_path_lengths(origin_nodes, dests_nodes))

Wall time: 144 ms

# How much faster is this from networkx
(((3.13*60)*1000))/(144)

1304.1666666666665

This is more than 1000times faster!

flows_full.head()

	origin	destination	ID_code_o	X_o	Y_o	node_o	ID_code_d	X_d	Y_d	node_d	path_length	distances
0	A91120	A91120	A91120	-2.559063	51.50283	2914306154	A91120	-2.559063	51.502830	2914306154	0.000	0.000
1	A91120	A99931	A91120	-2.559063	51.50283	2914306154	A99931	-2.596140	51.459182	1859320333	6960.514	6960.508
2	A91120	A99986	A91120	-2.559063	51.50283	2914306154	A99986	-2.538833	51.483143	2094196650	4333.457	4333.454
3	A91120	L81002	A91120	-2.559063	51.50283	2914306154	L81002	-2.930352	51.360207	317399984	36334.316	36334.302
4	A91120	L81004	A91120	-2.559063	51.50283	2914306154	L81004	-2.767317	51.482805	2488740793	19754.732	19754.729

The distances seems to be satisfying but are we sure that the packages picked up the same nodes?

Check that the nodes are the same as those from NetworkX

x = origin_nodes == flows_full['node_o']
x.describe()

count       36
unique       1
top       True
freq        36
Name: node_o, dtype: object

y = dests_nodes == flows_full['node_d']
y.describe()

count       36
unique       1
top       True
freq        36
Name: node_d, dtype: object

This looks spot on.

Appendix

Google Colab

If you want to run this on Google Colab, you need to install

Geopandas
OsmnX
Matplotlib

# install Geopandas
# Important library for many geopython libraries
!apt install gdal-bin python-gdal python3-gdal 
# Install rtree - Geopandas requirment
!apt install python3-rtree 
# Install Geopandas
!pip install git+git://github.com/geopandas/geopandas.git
# Install descartes - Geopandas requirment
!pip install descartes 
# Install Folium for Geographic data visualization
!pip install folium
# Install plotlyExpress
!pip install plotly_express# install packages

# osmnx
# this gives errors but works ...no idea
!apt-get -qq install -y libspatialindex-dev && pip install -q -U osmnx 
ox.config(use_cache=True, log_console=True)

!python -m pip uninstall matplotlib
!pip install matplotlib==3.1.3# mount the drive

Environment set up

name: graphs
channels:
  - conda-forge
  - defaults
dependencies:
  - _r-mutex=1.0.1=anacondar_1
  - anyio=2.2.0=py39hcbf5309_0
  - argon2-cffi=20.1.0=py39hb82d6ee_2
  - async_generator=1.10=py_0
  - attrs=20.3.0=pyhd3deb0d_0
  - babel=2.9.0=pyhd3deb0d_0
  - backcall=0.2.0=pyh9f0ad1d_0
  - backports=1.0=py_2
  - backports.functools_lru_cache=1.6.1=py_0
  - bleach=3.3.0=pyh44b312d_0
  - blosc=1.21.0=h0e60522_0
  - boost-cpp=1.74.0=h54f0996_2
  - branca=0.4.2=pyhd8ed1ab_0
  - brotlipy=0.7.0=py39hb82d6ee_1001
  - bzip2=1.0.8=h8ffe710_4
  - ca-certificates=2020.12.5=h5b45459_0
  - cairo=1.16.0=hba8bd2f_1007
  - certifi=2020.12.5=py39hcbf5309_1
  - cffi=1.14.5=py39h0878f49_0
  - cfitsio=3.470=h0af3d06_7
  - chardet=4.0.0=py39hcbf5309_1
  - click=7.1.2=pyh9f0ad1d_0
  - click-plugins=1.1.1=py_0
  - cligj=0.7.1=pyhd8ed1ab_0
  - colorama=0.4.4=pyh9f0ad1d_0
  - cryptography=3.4.6=py39hd8d06c1_0
  - curl=7.75.0=hf1763fc_0
  - cycler=0.10.0=py_2
  - decorator=4.4.2=py_0
  - defusedxml=0.7.1=pyhd8ed1ab_0
  - descartes=1.1.0=py_4
  - entrypoints=0.3=pyhd8ed1ab_1003
  - expat=2.2.10=h39d44d4_0
  - fiona=1.8.18=py39h9f1b043_1
  - folium=0.12.0=pyhd8ed1ab_0
  - fontconfig=2.13.1=h1989441_1004
  - freetype=2.10.4=h546665d_1
  - freexl=1.0.5=hd288d7e_1002
  - gdal=3.2.2=py39h6795fcd_0
  - geopandas=0.9.0=pyhd8ed1ab_0
  - geos=3.9.1=h39d44d4_2
  - geotiff=1.6.0=h8e90983_5
  - gettext=0.19.8.1=h1a89ca6_1005
  - hdf4=4.2.13=h0e5069d_1004
  - hdf5=1.10.6=nompi_h5268f04_1114
  - icu=68.1=h0e60522_0
  - idna=2.10=pyh9f0ad1d_0
  - importlib-metadata=3.7.2=py39hcbf5309_0
  - intel-openmp=2020.3=h57928b3_311
  - ipykernel=5.5.0=py39h832f523_1
  - ipython=7.21.0=py39h832f523_0
  - ipython_genutils=0.2.0=py_1
  - jedi=0.18.0=py39hcbf5309_2
  - jinja2=2.11.3=pyh44b312d_0
  - joblib=1.0.1=pyhd8ed1ab_0
  - jpeg=9d=h8ffe710_0
  - json5=0.9.5=pyh9f0ad1d_0
  - jsonschema=3.2.0=pyhd8ed1ab_3
  - jupyter-packaging=0.7.12=pyhd8ed1ab_0
  - jupyter_client=6.1.11=pyhd8ed1ab_1
  - jupyter_core=4.7.1=py39hcbf5309_0
  - jupyter_server=1.4.1=py39hcbf5309_0
  - jupyterlab=3.0.10=pyhd8ed1ab_0
  - jupyterlab_pygments=0.1.2=pyh9f0ad1d_0
  - jupyterlab_server=2.3.0=pyhd8ed1ab_0
  - kealib=1.4.14=h96bfa42_2
  - kiwisolver=1.3.1=py39h2e07f2f_1
  - krb5=1.17.2=hbae68bd_0
  - lcms2=2.12=h2a16943_0
  - libblas=3.9.0=8_mkl
  - libcblas=3.9.0=8_mkl
  - libcurl=7.75.0=hf1763fc_0
  - libffi=3.3=h0e60522_2
  - libgdal=3.2.2=hbe61683_0
  - libglib=2.66.7=h1e62bf3_1
  - libiconv=1.16=he774522_0
  - libkml=1.3.0=h02ac0ef_1012
  - liblapack=3.9.0=8_mkl
  - libnetcdf=4.7.4=nompi_h3a9aa94_107
  - libpng=1.6.37=h1d00b33_2
  - libpq=13.1=h4f54205_2
  - librttopo=1.1.0=hb340de5_6
  - libsodium=1.0.18=h8d14728_1
  - libspatialindex=1.9.3=h39d44d4_3
  - libspatialite=5.0.1=h6b539a6_4
  - libssh2=1.9.0=h680486a_6
  - libtiff=4.2.0=hc10be44_0
  - libwebp-base=1.2.0=h8ffe710_0
  - libxml2=2.9.10=hf5bbc77_3
  - lz4-c=1.9.3=h8ffe710_0
  - m2w64-bwidget=1.9.10=2
  - m2w64-bzip2=1.0.6=6
  - m2w64-expat=2.1.1=2
  - m2w64-fftw=3.3.4=6
  - m2w64-flac=1.3.1=3
  - m2w64-gcc-libgfortran=5.3.0=6
  - m2w64-gcc-libs=5.3.0=7
  - m2w64-gcc-libs-core=5.3.0=7
  - m2w64-gettext=0.19.7=2
  - m2w64-gmp=6.1.0=2
  - m2w64-gsl=2.1=2
  - m2w64-libiconv=1.14=6
  - m2w64-libjpeg-turbo=1.4.2=3
  - m2w64-libogg=1.3.2=3
  - m2w64-libpng=1.6.21=2
  - m2w64-libsndfile=1.0.26=2
  - m2w64-libtiff=4.0.6=2
  - m2w64-libvorbis=1.3.5=2
  - m2w64-libwinpthread-git=5.0.0.4634.697f757=2
  - m2w64-libxml2=2.9.3=3
  - m2w64-mpfr=3.1.4=4
  - m2w64-pcre2=10.34=0
  - m2w64-speex=1.2rc2=3
  - m2w64-speexdsp=1.2rc3=3
  - m2w64-tcl=8.6.5=3
  - m2w64-tk=8.6.5=3
  - m2w64-tktable=2.10=5
  - m2w64-wineditline=2.101=5
  - m2w64-xz=5.2.2=2
  - m2w64-zlib=1.2.8=10
  - markupsafe=1.1.1=py39hb82d6ee_3
  - matplotlib-base=3.3.4=py39h581301d_0
  - mistune=0.8.4=py39hb82d6ee_1003
  - mkl=2020.4=hb70f87d_311
  - mock=4.0.3=py39hcbf5309_1
  - msys2-conda-epoch=20160418=1
  - munch=2.5.0=py_0
  - nbclassic=0.2.6=pyhd8ed1ab_0
  - nbclient=0.5.3=pyhd8ed1ab_0
  - nbconvert=6.0.7=py39hcbf5309_3
  - nbformat=5.1.2=pyhd8ed1ab_1
  - nest-asyncio=1.4.3=pyhd8ed1ab_0
  - networkx=2.5=py_0
  - notebook=6.2.0=py39hcbf5309_0
  - numexpr=2.7.3=py39h2e25243_0
  - numpy=1.20.1=py39h6635163_0
  - olefile=0.46=pyh9f0ad1d_1
  - openjpeg=2.4.0=h48faf41_0
  - openssl=1.1.1j=h8ffe710_0
  - osmnx=1.0.1=pyhd3deb0d_0
  - packaging=20.9=pyh44b312d_0
  - pandana=0.6=py39h2e25243_0
  - pandas=1.2.3=py39h2e25243_0
  - pandoc=2.12=h8ffe710_0
  - pandocfilters=1.4.2=py_1
  - parso=0.8.1=pyhd8ed1ab_0
  - pcre=8.44=ha925a31_0
  - pickleshare=0.7.5=py_1003
  - pillow=8.1.2=py39h1a9d4f7_0
  - pip=21.0.1=pyhd8ed1ab_0
  - pixman=0.40.0=h8ffe710_0
  - poppler=21.03.0=h9ff6ed8_0
  - poppler-data=0.4.10=0
  - postgresql=13.1=h0f1a9bc_2
  - proj=8.0.0=h1cfcee9_0
  - prometheus_client=0.9.0=pyhd3deb0d_0
  - prompt-toolkit=3.0.16=pyha770c72_0
  - pycparser=2.20=pyh9f0ad1d_2
  - pygments=2.8.1=pyhd8ed1ab_0
  - pyopenssl=20.0.1=pyhd8ed1ab_0
  - pyparsing=2.4.7=pyh9f0ad1d_0
  - pyproj=3.0.1=py39h1007a03_1
  - pyrsistent=0.17.3=py39hb82d6ee_2
  - pysocks=1.7.1=py39hcbf5309_3
  - pytables=3.6.1=py39h42e6cd8_3
  - python=3.9.2=h7840368_0_cpython
  - python-dateutil=2.8.1=py_0
  - python_abi=3.9=1_cp39
  - pytz=2021.1=pyhd8ed1ab_0
  - pywin32=300=py39hb82d6ee_0
  - pywinpty=0.5.7=py39hde42818_1
  - pyzmq=22.0.3=py39he46f08e_1
  - r-base=4.0.3=hddad469_7
  - r-iterators=1.0.13=r40h142f84f_0
  - r-itertools=0.1_3=r40_1003
  - requests=2.25.1=pyhd3deb0d_0
  - rtree=0.9.7=py39h09fdee3_1
  - scikit-learn=0.24.1=py39he931e04_0
  - scipy=1.6.0=py39hc0c34ad_0
  - send2trash=1.5.0=py_0
  - setuptools=49.6.0=py39hcbf5309_3
  - shapely=1.7.1=py39h90c6b7e_4
  - six=1.15.0=pyh9f0ad1d_0
  - sniffio=1.2.0=py39hcbf5309_1
  - sqlite=3.34.0=h8ffe710_0
  - terminado=0.9.2=py39hcbf5309_0
  - testpath=0.4.4=py_0
  - threadpoolctl=2.1.0=pyh5ca1d4c_0
  - tiledb=2.2.4=hddc2a84_2
  - tk=8.6.10=h8ffe710_1
  - tornado=6.1=py39hb82d6ee_1
  - traitlets=5.0.5=py_0
  - tzdata=2021a=he74cb21_0
  - urllib3=1.26.3=pyhd8ed1ab_0
  - vc=14.2=hb210afc_4
  - vs2015_runtime=14.28.29325=h5e1d092_4
  - wcwidth=0.2.5=pyh9f0ad1d_2
  - webencodings=0.5.1=py_1
  - wheel=0.36.2=pyhd3deb0d_0
  - win_inet_pton=1.1.0=py39hcbf5309_2
  - wincertstore=0.2=py39hcbf5309_1006
  - winpty=0.4.3=4
  - xerces-c=3.2.3=h0e60522_2
  - xz=5.2.5=h62dcd97_1
  - zeromq=4.3.4=h0e60522_0
  - zipp=3.4.1=pyhd8ed1ab_0
  - zlib=1.2.11=h62dcd97_1010
  - zstd=1.4.9=h6255e5f_0

Distribution of Children homes in Czech Republic

Fri, 01 Jan 2021 00:00:00 +0000

Introduction

My friend, that is currently completing Phd in Social studies and pedagogy, has asked me to create some maps for her thesis. Primary topic for these should be an institutional care in Czech Republic

Distribution of Children homes across Czech
Choropleth map of relative distribution (per capita) of the homes across Czech districts.

Data

Data needed to complete those maps are

Register of schools and school facilities in CR - XML database
Population in municipalities - as of 1.1.2020 - CSV table
A set of administrative boundaries and boundaries of cadastral territories of the Czech Republic - shp spatial layer in WGS84 projection

Notes

Be aware that Czech republic has its own projection; JTSK (EPSG:2065), in which all the maps are drawn. We might use it or not, just giving you heads up.

Preparation

Packages

# Necessary for munging the files
import xml.etree.cElementTree as ET # XLM manipulation package
import requests
import pandas as pd
import numpy as np
import geopandas as gpd
import logging
import time
# Necessary for geocoding
from geopy.geocoders import Nominatim
from shapely.geometry import Point
import matplotlib.patheffects as pe
import getpass
from geopy.extra.rate_limiter import RateLimiter
# plotting
import folium
from folium.plugins import MarkerCluster
from folium import IFrame
import pysal as ps
import mapclassify
import libpysal as lp
import matplotlib.pyplot as plt

Projection shortcuts

wgs84 = {'init':'EPSG:4326'}
jtsk = {'init':'EPSG:2065'}

Data loading

# population csv
popul = pd.read_excel('./pocet_obyvatel.xlsx', sheet_name='List1') 
# spatial layer of districts
distr = gpd.read_file('./SPH_SHP_WGS84/WGS84/SPH_OKRES.shp')
distr.crs = wgs84
region = gpd.read_file('./SPH_SHP_WGS84/WGS84/SPH_KRAJ.shp')
region.crs = wgs84
country = gpd.read_file('./SPH_SHP_WGS84/WGS84/SPH_STAT.shp')
country.crs = wgs84
# data about institutions
xml_str = requests.get('https://rejstriky.msmt.cz/opendata/vrejcelk.xml').text

Data preview

popul.head()

	ID	okres	pocet obyvatel	kraj
0	CZ0100	Hlavní město Praha	1324277	Středočeský kraj
1	CZ0201	Benešov	99414	Středočeský kraj
2	CZ0202	Beroun	95058	Středočeský kraj
3	CZ0203	Kladno	166483	Středočeský kraj
4	CZ0204	Kolín	102623	Středočeský kraj

country.plot()
region.plot()
distr.plot()

xml_str[0:500]

'<?xml version="1.0" encoding="utf-8"?>\r\n<ExportDat>\r\n  <DatumVystupu>2021-01-05T21:30:01.2452138+01:00</DatumVystupu>\r\n  <PravniSubjekt>\r\n    <RedIzo>600000206</RedIzo>\r\n    <ICO>49625918</ICO>\r\n    <Reditelstvi>\r\n      <RedPlnyNazev>Mateřská škola sv. Voršily v Praze</RedPlnyNazev>\r\n      <RedZkracenyNazev>Mateřská škola sv. Voršily v Praze</RedZkracenyNazev>\r\n      <RedRUAINKod>21702101</RedRUAINKod>\r\n      <RedAdresa1>Ostrovní 139/11</RedAdresa1>\r\n      <RedAdresa2>Nové Město</RedAdresa2>\r\n  '

Extracting data from XML using ElementTree package

How do I know what does the tree looks like?

Fisrtly, the source should always give you a metadata about the structure of the data, as it is in this case. However, you can find out by inspecting the file and the individual children of the tree.

# pass the string to elemt tree
etree = ET.fromstring(xml_str)
# inspect the childern
etree.getchildren()[1]

<Element 'PravniSubjekt' at 0x000001A475D1D950>

etree.getchildren()[1].getchildren()

[<Element 'RedIzo' at 0x000001A475D1D090>,
 <Element 'ICO' at 0x000001A475D1D180>,
 <Element 'Reditelstvi' at 0x000001A475D20770>,
 <Element 'SkolyZarizeni' at 0x000001A475BED9A0>]

etree.getchildren()[1].getchildren()[3].getchildren()

[<Element 'SkolaZarizeni' at 0x000001A475BED8B0>,
 <Element 'SkolaZarizeni' at 0x000001A475BA9720>]

How do I extract some of the data and get then into a dataframe

In this case we know that all the information is part of one child and should have the same number of records, so we can just iterate through the tree, list all the text and assign it to dataframe. You can do this only if you are sure that the order of the data and their number matches.

ID = []
for typ in etree.iter('IDMista'):
    ID.append(typ.text)    
    
ruian = []
for typ in etree.iter('MistoRUAINKod'):
    ruian.append(typ.text)    
    
druh = []
for typ in etree.iter('MistoDruhTyp'):
    druh.append(typ.text) 

address_1=[]

for typ in etree.iter('MistoAdresa1'):
    address_1.append(typ.text)
    
address_2=[]

for typ in etree.iter('MistoAdresa2'):
    address_2.append(typ.text)
    
address_3=[]

for typ in etree.iter('MistoAdresa3'):
    address_3.append(typ.text)

print(len(ID))
print(len(ruian))
print(len(druh))
print(len(address_1))
print(len(address_2))
print(len(address_3))

Perfect!

df = pd.DataFrame(np.column_stack((ID, ruian, druh, address_1, address_2, address_3)), columns = ['ID', 'ruian','druh','address_1', 'address_2', 'address_3'])
df

	ID	ruian	druh	address_1	address_2	address_3
0	049625918	21702101	Mateřská škola	Ostrovní 139/11	Nové Město	110 00 Praha 1
1	102413096	21702101	Školní jídelna	Ostrovní 139/11	Nové Město	110 00 Praha 1
2	107500884	21852201	Mateřská škola	Ke Kamýku 686/2	Kamýk	142 00 Praha 4
3	161102263	21852201	Školní jídelna - výdejna	Ke Kamýku 686/2	Kamýk	142 00 Praha 4
4	110034384	21851018	Mateřská škola pro děti se zdravotním postižením	Smolkova 567/2	Kamýk	142 00 Praha 4
...	...	...	...	...	...	...
38290	181118084	None	Lesní mateřská škola	pozemky p. č. 546/1, 481/40, 2292,568/56	k. ú. Třebechovice pod Orebem	None
38291	181118106	3147045	Mateřská škola	Bohumínská 37/158	Muglinov	712 00 Ostrava
38292	181118114	3147045	Školní jídelna	Bohumínská 37/158	Muglinov	712 00 Ostrava
38293	181118190	2509008	Mateřská škola	č.p. 77	281 63 Oplany	None
38294	181118203	2509008	Školní jídelna - výdejna	č.p. 77	281 63 Oplany	None

38295 rows × 6 columns

Filter out only children homes

dd = ['Dětský domov']
df2 = df[df['druh'].isin(dd)]
df2.head()

	ID	ruian	druh	address_1	address_2	address_3
4902	150069812	3089011	Dětský domov	Přestavlky 1	257 91 Sedlec-Prčice	None
4911	150018312	2929724	Dětský domov	Racek 1	256 01 Chlístov	None
4928	151038147	5936675	Dětský domov	Mládeže 1102/8	Beroun-Město	266 01 Beroun
4974	048706302	1377931	Dětský domov	č.p. 55	273 05 Ledce	None
5027	150036787	11647388	Dětský domov	Poštovní 593	285 22 Zruč nad Sázavou	None

More filters

We know that some of the homes are duplicated as they might have two buldings on the same postodes but one is the actual home and the second is for example a food providing facility. To avoid duplicated places, we need to separate the postcode from adress and select only the first from duplicated postcodes.

# split the adress string, the postcode is always in the first 6 caracters
df2['address_4'] = df2['address_2'].str[0:6]
df2['address_5'] = df2['address_3'].str[0:6]
# find rows in second part of adress that does not have a postcode
df2['is_5'] = df2['address_5'].isnull()
# logical argument
# if the postcode is in the second part of adress, fill column with it, if its not, fill it with te first part adress postcode
df2['postcode'] = np.where( (df2['is_5'] == False) , df2['address_5'], df2['address_4'])
# voala
df2

	ID	ruian	druh	address_1	address_2	address_3	address_4	address_5	is_5	postcode
4902	150069812	3089011	Dětský domov	Přestavlky 1	257 91 Sedlec-Prčice	None	257 91	None	True	257 91
4911	150018312	2929724	Dětský domov	Racek 1	256 01 Chlístov	None	256 01	None	True	256 01
4928	151038147	5936675	Dětský domov	Mládeže 1102/8	Beroun-Město	266 01 Beroun	Beroun	266 01	False	266 01
4974	048706302	1377931	Dětský domov	č.p. 55	273 05 Ledce	None	273 05	None	True	273 05
5027	150036787	11647388	Dětský domov	Poštovní 593	285 22 Zruč nad Sázavou	None	285 22	None	True	285 22
...	...	...	...	...	...	...	...	...	...	...
30762	150066147	11165901	Dětský domov	č.p. 204	750 02 Stará Ves	None	750 02	None	True	750 02
30763	181078538	11164361	Dětský domov	č.p. 47	750 02 Stará Ves	None	750 02	None	True	750 02
30769	150008660	22885871	Dětský domov	Lazec 2	261 01 Příbram	None	261 01	None	True	261 01
33321	151019142	22855521	Dětský domov	č.p. 22	261 01 Dubenec	None	261 01	None	True	261 01
33382	151035351	22551166	Dětský domov	Čs. tankistů 277/11	Dolní Měcholupy	109 00 Praha 10	Dolní	109 00	False	109 00

243 rows × 10 columns

# drop all the postcode duplicates and keep the first records for the duplicates
df2.drop_duplicates(subset ="postcode", 
                     keep = 'first', inplace = True)
df2.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 167 entries, 4902 to 33382
Data columns (total 10 columns):
 #   Column     Non-Null Count  Dtype 
---  ------     --------------  ----- 
 0   ID         167 non-null    object
 1   ruian      165 non-null    object
 2   druh       167 non-null    object
 3   address_1  166 non-null    object
 4   address_2  167 non-null    object
 5   address_3  47 non-null     object
 6   address_4  167 non-null    object
 7   address_5  47 non-null     object
 8   is_5       167 non-null    bool  
 9   postcode   167 non-null    object
dtypes: bool(1), object(9)
memory usage: 13.2+ KB

We now have 167 records of the Childrens home, we know it is suppose to be 137 so there is still 30 places extra. Those could be separate buildings with different postcodes but belonging to one institution. We have no information about this so lets just leave it for now and get to the geocoding.

Geocoding the addresses

# clean the data a little
df3 = df2.iloc[:,[0,1,2,3,4,9]]
# create column with full address
df3['plna'] =  df3['address_1'].map(str) + ' , ' + df3['address_2'].map(str) 
df3.head()

	ID	ruian	druh	address_1	address_2	postcode	plna
4902	150069812	3089011	Dětský domov	Přestavlky 1	257 91 Sedlec-Prčice	257 91	Přestavlky 1 , 257 91 Sedlec-Prčice
4911	150018312	2929724	Dětský domov	Racek 1	256 01 Chlístov	256 01	Racek 1 , 256 01 Chlístov
4928	151038147	5936675	Dětský domov	Mládeže 1102/8	Beroun-Město	266 01	Mládeže 1102/8 , Beroun-Město
4974	048706302	1377931	Dětský domov	č.p. 55	273 05 Ledce	273 05	č.p. 55 , 273 05 Ledce
5027	150036787	11647388	Dětský domov	Poštovní 593	285 22 Zruč nad Sázavou	285 22	Poštovní 593 , 285 22 Zruč nad Sázavou

Using GoogleV3 engine to ceocode the addresses

# define the engine and limiter
geolocator = Nominatim(user_agent="GoogleV3")
geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1)
# Define Google API key 
key = getpass.getpass('Password:')
# pasweord can be found in your google acount under API & Services. if you dont have any active one, you need to create one
#https://console.developers.google.com/apis/credentials?project=annular-garden-259610

Password:

# create a column with geocoded address
df3['location'] = df3['plna'].apply(geocode)
# create a column with point coordinates 
df3['point'] = df3['location'].apply(lambda loc: tuple(loc.point) if loc else None)

df3.head()

	ID	ruian	druh	address_1	address_2	postcode	plna
4902	150069812	3089011	Dětský domov	Přestavlky 1	257 91 Sedlec-Prčice	257 91	Přestavlky 1 , 257 91 Sedlec-Prčice
4911	150018312	2929724	Dětský domov	Racek 1	256 01 Chlístov	256 01	Racek 1 , 256 01 Chlístov
4928	151038147	5936675	Dětský domov	Mládeže 1102/8	Beroun-Město	266 01	Mládeže 1102/8 , Beroun-Město
4974	048706302	1377931	Dětský domov	č.p. 55	273 05 Ledce	273 05	č.p. 55 , 273 05 Ledce
5027	150036787	11647388	Dětský domov	Poštovní 593	285 22 Zruč nad Sázavou	285 22	Poštovní 593 , 285 22 Zruč nad Sázavou

We can see some of the places has not been found so lets do one more round but with partial address only (postcode, city).

Once that is done, there is actually some residual of addresses that was not geocoded at all so let me just skip ahead and add them in manually, right now.

df3['address_2'][df3['ID'] == '110023587'] = '49.154827045632665, 17.996702689654494'
df3['address_2'][df3['ID'] == '150070446'] = '49.715585698531825, 17.902557782916908'
df3['address_2'][df3['ID'] == '110005414'] = '49.70128584452374, 17.077698497646164'
df3['address_2'][df3['ID'] == '150068999'] = '49.828035620591606, 17.77214007066865'
df3['address_2'][df3['ID'] == '150070411'] = '49.975334977613926, 17.73716291115126'
df3['address_2'][df3['ID'] == '150012012'] = '49.88712758017286, 16.867843041835485'
df3['address_2'][df3['ID'] == '102175594'] = '49.53657949442161, 15.351256882293946'
df3['address_2'][df3['ID'] == '102464537'] = '49.79357507981935, 12.624905338126101'
df3['address_2'][df3['ID'] == '181056381'] = '50.48084538132141, 13.426100477617592'
df3['address_2'][df3['ID'] == '102177899'] = '50.72867195615058, 15.184360530266257'
df3['address_2'][df3['ID'] == '048159638'] = '50.072083156427496, 15.986443039170952'
df3['address_2'][df3['ID'] == '102406227'] = '50.33954862443628, 16.314161709055785'
df3['address_2'][df3['ID'] == '181103532'] = '49.70512665455396, 16.342441590148137'
df3['address_2'][df3['ID'] == '102591831'] = '49.47082213504419, 17.01841195530714'
df3['address_2'][df3['ID'] == '181058260'] = '48.932739996226346, 15.686178770625474'
df3['address_2'][df3['ID'] == '108035727'] = '49.79288628247885, 17.62378088230638'
df3['address_2'][df3['ID'] == '102692424'] = '50.31112466112202, 17.155736543399872'
df3['address_2'][df3['ID'] == '110250958'] = '49.26050205879534, 16.036339638623517'
df3['address_2'][df3['ID'] == '110301021'] = '49.462917407166046, 17.116385426470682'
df3['address_2'][df3['ID'] == '110400542'] = '49.16950427831417, 13.885607270636793'
df3['address_2'][df3['ID'] == '110400691'] = '50.82836600146331, 14.702413510732878'
df3['address_2'][df3['ID'] == '110501161'] = '49.75405044016851, 18.33950724056137'
df3['address_2'][df3['ID'] == '000412074'] = '51.011154195511004, 14.359466135386068'

df3['location2'] = df3['address_2'].apply(geocode)

df3['point2'] = df3['location2'].apply(lambda loc: tuple(loc.point) if loc else None)
df3.head()

# now we jus need to create a final column with te point
# ths is logical statement, if point1 is None than fill with point 2, otherwise fill with point1
df3['point_final'] = np.where(df3['point'].isna(),df3['point2'],df3['point'])

df3['point_final'].isna().describe()

count       167
unique        1
top       False
freq        167
Name: point_final, dtype: object

Success, all the addresses are filled.

Making the points spatial

We know this is all in WGS84 so we just need to convert the dataset into spatial geopandas dataset.

df3.head()

	ID	ruian	druh	address_1	address_2	postcode	plna	location	point	location2	point2	point_final
4901	150069812	3089011	Dětský domov	Přestavlky 1	257 91 Sedlec-Prčice	257 91	Přestavlky 1 , 257 91 Sedlec-Prčice	(u Černčického rybníka, Dlážděná, Sedlec u Vot...	(49.5702975, 14.5327368, 0.0)	(Sedlec-Prčice, okres Příbram, Střední Čechy, ...	(49.5726082, 14.5326911, 0.0)	(49.5702975, 14.5327368, 0.0)
4910	150018312	2929724	Dětský domov	Racek 1	256 01 Chlístov	256 01	Racek 1 , 256 01 Chlístov	(ev.1, Racek, Chlístov, okres Benešov, Střední...	(49.8051895, 14.6632285, 0.0)	(Chlístov, okres Benešov, Střední Čechy, Česko...	(49.8015212, 14.6547167, 0.0)	(49.8051895, 14.6632285, 0.0)
4927	151038147	5936675	Dětský domov	Mládeže 1102/8	Beroun-Město	266 01	Mládeže 1102/8 , Beroun-Město	(1102/8, Mládeže, Beroun-Město, Beroun, Jarov,...	(49.9543712, 14.0503775, 0.0)	(Beroun-Město, Beroun, okres Beroun, Střední Č...	(49.9616066, 14.0675721, 0.0)	(49.9543712, 14.0503775, 0.0)
4973	048706302	1377931	Dětský domov	č.p. 55	273 05 Ledce	273 05	č.p. 55 , 273 05 Ledce	None	None	(Ledce, okres Brno-venkov, Jihomoravský kraj, ...	(49.0515031, 16.5570155, 0.0)	(49.0515031, 16.5570155, 0.0)
5026	150036787	11647388	Dětský domov	Poštovní 593	285 22 Zruč nad Sázavou	285 22	Poštovní 593 , 285 22 Zruč nad Sázavou	(Poštovní, Domahoř, Zruč nad Sázavou, okres Ku...	(49.7442708, 15.0883736, 0.0)	(Zruč nad Sázavou, okres Kutná Hora, Střední Č...	(49.740296, 15.106074, 0.0)	(49.7442708, 15.0883736, 0.0)

# clean
df4 = df3.iloc[:,[0,1,2,3,4,6,11]].rename(columns={"point_final": "point"})
df4.head()

	ID	ruian	druh	address_1	address_2	plna	point
4901	150069812	3089011	Dětský domov	Přestavlky 1	257 91 Sedlec-Prčice	Přestavlky 1 , 257 91 Sedlec-Prčice	(49.5702975, 14.5327368, 0.0)
4910	150018312	2929724	Dětský domov	Racek 1	256 01 Chlístov	Racek 1 , 256 01 Chlístov	(49.8051895, 14.6632285, 0.0)
4927	151038147	5936675	Dětský domov	Mládeže 1102/8	Beroun-Město	Mládeže 1102/8 , Beroun-Město	(49.9543712, 14.0503775, 0.0)
4973	048706302	1377931	Dětský domov	č.p. 55	273 05 Ledce	č.p. 55 , 273 05 Ledce	(49.0515031, 16.5570155, 0.0)
5026	150036787	11647388	Dětský domov	Poštovní 593	285 22 Zruč nad Sázavou	Poštovní 593 , 285 22 Zruč nad Sázavou	(49.7442708, 15.0883736, 0.0)

# get and XY columns and clean
df4['point'] = df4['point'].astype(str)
df4[['x','y', 'z']] = df4.point.str.split(n=3, expand=True)
df4['x'] = df4['x'].str[1:-1]
df4['y'] = df4['y'].str[:-1]
df5 = df4.iloc[:,[0,1,2,3,4,5,6,7,8]]
df5.head()

	ID	ruian	druh	address_1	address_2	plna	point	x	y
4901	150069812	3089011	Dětský domov	Přestavlky 1	257 91 Sedlec-Prčice	Přestavlky 1 , 257 91 Sedlec-Prčice	(49.5702975, 14.5327368, 0.0)	49.5702975	14.5327368
4910	150018312	2929724	Dětský domov	Racek 1	256 01 Chlístov	Racek 1 , 256 01 Chlístov	(49.8051895, 14.6632285, 0.0)	49.8051895	14.6632285
4927	151038147	5936675	Dětský domov	Mládeže 1102/8	Beroun-Město	Mládeže 1102/8 , Beroun-Město	(49.9543712, 14.0503775, 0.0)	49.9543712	14.0503775
4973	048706302	1377931	Dětský domov	č.p. 55	273 05 Ledce	č.p. 55 , 273 05 Ledce	(49.0515031, 16.5570155, 0.0)	49.0515031	16.5570155
5026	150036787	11647388	Dětský domov	Poštovní 593	285 22 Zruč nad Sázavou	Poštovní 593 , 285 22 Zruč nad Sázavou	(49.7442708, 15.0883736, 0.0)	49.7442708	15.0883736

homes = gpd.GeoDataFrame(df5, geometry=gpd.points_from_xy(df5.y, df5.x))

homes.plot();

Munging data for choropleth map

# Load checked homes
homes2 = pd.read_excel('./dohledat_Ver.xlsx', sheet_name='Sheet1') 
homes2.head()

	ID_left	druh	address_1	address_2	plna	x	y	geometry	okres	Kraj	Existuje	check
0	150018312	Dětský domov	Racek 1	256 01 Chlístov	Racek 1 , 256 01 Chlístov	49.805190	14.663229	POINT (14.6632285 49.8051895)	Benešov	Středočeský kraj	ANO	1
1	108003990	Dětský domov	Senohrabská 112	251 67 Pyšely	Senohrabská 112 , 251 67 Pyšely	49.878053	14.679728	POINT (14.6797284 49.8780534)	Benešov	Středočeský kraj	ANO	1
2	102238324	Dětský domov	Benešovská 7	285 06 Sázava	Benešovská 7 , 285 06 Sázava	49.871872	14.896189	POINT (14.8961892 49.8718717)	Benešov	Středočeský kraj	ano	1
3	151038147	Dětský domov	Mládeže 1102/8	Beroun-Město	Mládeže 1102/8 , Beroun-Město	49.954371	14.050377	POINT (14.0503775 49.9543712)	Beroun	Středočeský kraj	ANO	1
4	150008732	Dětský domov	Štefanikova 2344/2b	680 01 Boskovice	Štefanikova 2344/2b , 680 01 Boskovice	49.495124	16.660245	POINT (16.660245 49.4951238)	Blansko	Jihomoravský kraj	ANO	1

homes = gpd.GeoDataFrame(homes2, geometry=gpd.points_from_xy(homes2.y, homes2.x))
homes.loc[homes['check'] == 1]

	ID_left	druh	address_1	address_2	plna	x	y	geometry	okres	Kraj	Existuje	check
0	150018312	Dětský domov	Racek 1	256 01 Chlístov	Racek 1 , 256 01 Chlístov	49.805190	14.663229	POINT (14.66323 49.80519)	Benešov	Středočeský kraj	ANO	1
1	108003990	Dětský domov	Senohrabská 112	251 67 Pyšely	Senohrabská 112 , 251 67 Pyšely	49.878053	14.679728	POINT (14.67973 49.87805)	Benešov	Středočeský kraj	ANO	1
2	102238324	Dětský domov	Benešovská 7	285 06 Sázava	Benešovská 7 , 285 06 Sázava	49.871872	14.896189	POINT (14.89619 49.87187)	Benešov	Středočeský kraj	ano	1
3	151038147	Dětský domov	Mládeže 1102/8	Beroun-Město	Mládeže 1102/8 , Beroun-Město	49.954371	14.050377	POINT (14.05038 49.95437)	Beroun	Středočeský kraj	ANO	1
4	150008732	Dětský domov	Štefanikova 2344/2b	680 01 Boskovice	Štefanikova 2344/2b , 680 01 Boskovice	49.495124	16.660245	POINT (16.66024 49.49512)	Blansko	Jihomoravský kraj	ANO	1
...	...	...	...	...	...	...	...	...	...	...	...	...
170	151001308	Dětský domov	3. května 528	763 12 Vizovice	3. května 528 , 763 12 Vizovice	49.215309	17.849604	POINT (17.84960 49.21531)	Zlín	Zlínský kraj	ANO	1
171	150070462	Dětský domov	Lazy VI 3695	760 01 Zlín	Lazy VI 3695 , 760 01 Zlín	49.219676	17.681226	POINT (17.68123 49.21968)	Zlín	Zlínský kraj	ANO	1
172	110023587	Dětský domov	Smolina 16	49.154827045632665, 17.996702689654494	Smolina 16 , 766 01 Valašské Klobouky	49.154843	17.996750	POINT (17.99675 49.15484)	Zlín	Zlínský kraj	ANO	1
173	102331146	Dětský domov	Lazy1	760 01 Zlín	760 01 Zlín	49.219029	17.676235	POINT (17.67624 49.21903)	Zlín	Zlínský kraj	ANO	1
174	102867232	Dětský domov	Hakenova 716/18	669 02 Znojmo	Hakenova 716/18 , 669 02 Znojmo	48.851656	16.065432	POINT (16.06543 48.85166)	Znojmo	Jihomoravský kraj	ANO	1

137 rows × 12 columns

homes.crs = wgs84

gdf1 = gpd.sjoin(homes, distr, how="inner”, op='intersects’) #gdf1.to_excel(‘dohledat.xlsx’)

distr.head()

	ID	KOD_NUTS3	KOD_LAU1	NAZEV_LAU1	geometry
0	3100.0	CZ010	CZ0100	Hlavní město Praha	POLYGON ((14.52826 49.99984, 14.52812 49.99975...
1	3201.0	CZ020	CZ0201	Benešov	POLYGON ((14.42653 49.81924, 14.42654 49.81976...
2	3202.0	CZ020	CZ0202	Beroun	POLYGON ((13.79700 49.82979, 13.79763 49.83013...
3	3203.0	CZ020	CZ0203	Kladno	POLYGON ((13.92115 50.21131, 13.92117 50.21135...
4	3204.0	CZ020	CZ0204	Kolín	POLYGON ((15.41584 50.10747, 15.41511 50.10730...

#Aggregate the institution to districts
gdf1 = gpd.sjoin(homes, distr, how="inner", op='intersects')
gdf1 = gdf1[['ID_left','KOD_LAU1']]
df8 = gdf1.groupby(by=["KOD_LAU1"]).count().rename(columns={"ID_left": "count"})
df8.head()

	count
KOD_LAU1
CZ0100	7
CZ0201	3
CZ0202	1
CZ0203	2
CZ0205	1

# double merge all the data together
distr_new = distr.merge(df8, on = 'KOD_LAU1')
distr_new = distr_new.merge(popul, left_on = 'KOD_LAU1', right_on = 'ID')
distr_new.head()

	ID_x	KOD_NUTS3	KOD_LAU1	NAZEV_LAU1	geometry	count	ID_y	okres	pocet obyvatel	kraj
0	3100.0	CZ010	CZ0100	Hlavní město Praha	POLYGON ((14.52826 49.99984, 14.52812 49.99975...	7	CZ0100	Hlavní město Praha	1324277	Hlavní město Praha
1	3201.0	CZ020	CZ0201	Benešov	POLYGON ((14.42653 49.81924, 14.42654 49.81976...	3	CZ0201	Benešov	99414	Středočeský kraj
2	3202.0	CZ020	CZ0202	Beroun	POLYGON ((13.79700 49.82979, 13.79763 49.83013...	1	CZ0202	Beroun	95058	Středočeský kraj
3	3203.0	CZ020	CZ0203	Kladno	POLYGON ((13.92115 50.21131, 13.92117 50.21135...	2	CZ0203	Kladno	166483	Středočeský kraj
4	3205.0	CZ020	CZ0205	Kutná Hora	POLYGON ((15.37964 50.02393, 15.38011 50.02372...	1	CZ0205	Kutná Hora	75828	Středočeský kraj

#Aggregate the institution to districts
gdf1 = gpd.sjoin(homes, region, how="inner", op='intersects')
gdf1 = gdf1[['ID_left','KOD_NUTS3']]
df8 = gdf1.groupby(by=["KOD_NUTS3"]).count().rename(columns={"ID_left": "count"})
df8.head()

	count
KOD_NUTS3
CZ010	7
CZ020	19
CZ031	11
CZ032	13
CZ041	10

popul2 = popul[['kraj','pocet obyvatel']].groupby(['kraj']).sum()
popul2

	pocet obyvatel
kraj
Hlavní město Praha	1324277
Jihomoravský kraj	1191989
Jihočeský kraj	644083
Karlovarský kraj	294664
Kraj Vysočina	509813
Královéhradecký kraj	551647
Liberecký kraj	443690
Moravskoslezský kraj	1200539
Olomoucký kraj	632015
Pardubický kraj	522662
Plzeňský kraj	589899
Středočeský kraj	1385141
Zlínský kraj	582555
Ústecký kraj	820965

# double merge all the data together
region_new = region.merge(df8, on = 'KOD_NUTS3')
region_new = region_new.merge(popul2, left_on = 'NAZEV_NUTS', right_on = 'kraj')
region_new.head()

	ID	KOD_NUTS3	NAZEV_NUTS	geometry	count	pocet obyvatel
0	19.0	CZ010	Hlavní město Praha	POLYGON ((14.52826 49.99984, 14.52812 49.99975...	7	1324277
1	27.0	CZ020	Středočeský kraj	POLYGON ((13.99190 50.30468, 13.99190 50.30468...	19	1385141
2	35.0	CZ031	Jihočeský kraj	POLYGON ((14.66161 49.52495, 14.66296 49.52522...	11	644083
3	43.0	CZ032	Plzeňský kraj	POLYGON ((13.79700 49.82979, 13.79704 49.82967...	13	589899
4	51.0	CZ041	Karlovarský kraj	POLYGON ((13.24542 50.12623, 13.24539 50.12605...	10	294664

# clean
distr_new2 = distr_new.iloc[[1,2,3,4,5,8,9]]
distr_new2.head()

# calculate homes per capita
# Per capita = measurement/number of people in a population
distr_new['per_thousand_people'] = (distr_new['count']/(distr_new['pocet obyvatel']/1000))
region_new['per_thousand_people'] = (region_new['count']/(region_new['pocet obyvatel']/1000))
distr_new.head()

	ID_x	KOD_NUTS3	KOD_LAU1	NAZEV_LAU1	geometry	count	ID_y	okres	pocet obyvatel	kraj	per_thousand_people
0	3100.0	CZ010	CZ0100	Hlavní město Praha	POLYGON ((14.52826 49.99984, 14.52812 49.99975...	7	CZ0100	Hlavní město Praha	1324277	Hlavní město Praha	0.005286
1	3201.0	CZ020	CZ0201	Benešov	POLYGON ((14.42653 49.81924, 14.42654 49.81976...	3	CZ0201	Benešov	99414	Středočeský kraj	0.030177
2	3202.0	CZ020	CZ0202	Beroun	POLYGON ((13.79700 49.82979, 13.79763 49.83013...	1	CZ0202	Beroun	95058	Středočeský kraj	0.010520
3	3203.0	CZ020	CZ0203	Kladno	POLYGON ((13.92115 50.21131, 13.92117 50.21135...	2	CZ0203	Kladno	166483	Středočeský kraj	0.012013
4	3205.0	CZ020	CZ0205	Kutná Hora	POLYGON ((15.37964 50.02393, 15.38011 50.02372...	1	CZ0205	Kutná Hora	75828	Středočeský kraj	0.013188

Plot choropleth map

fig, ax = plt.subplots(figsize=(20,13), subplot_kw={'aspect':'equal'})
country.plot(ax=ax, facecolor="lightgrey", edgecolor="black", linewidth = 2)
distr_new.plot(column='per_thousand_people', scheme='Quantiles', k=4, legend=True,legend_kwds={'fontsize':15},  ax=ax, 
        cmap='Blues')
region_new.geometry.boundary.plot(color = None,edgecolor="green", linewidth = 0.5, ax=ax)
ax.set_title('Number of Children homes per thousand people in Czech districts. Each Class contain 25% of all homes. Grey indicates no homes in the district', fontsize = 15)
ax.text(16,48.5, 'Source: rejstriky.msmt.cz/opendata/vrejcelk, Author: Lenka Hasova, MSc, lenkahas.com', fontsize=10)
region_new.apply(lambda x: ax.annotate(s=x.NAZEV_NUTS, xy=x.geometry.centroid.coords[0], ha='center',color= 'black', size = 12, path_effects=[pe.withStroke(linewidth=0.5, foreground="white")]),axis=1)
ax.axis('off');

fig, ax = plt.subplots(figsize=(20,13), subplot_kw={'aspect':'equal'})
country.plot(ax=ax, facecolor="lightgrey", edgecolor="black", linewidth = 2)
region_new.plot(column='per_thousand_people', scheme='Quantiles', k=4, legend=True,legend_kwds={'fontsize':15},  ax=ax, 
        cmap='Blues')
ax.set_title('Number of Children homes per thousand people in Czech regions. Each Class contain 25% of all homes.', fontsize = 15)
ax.text(16,48.5, 'Source: rejstriky.msmt.cz/opendata/vrejcelk, Author: Lenka Hasova, MSc, lenkahas.com', fontsize=10)
region_new.apply(lambda x: ax.annotate(s=x.NAZEV_NUTS, xy=x.geometry.centroid.coords[0], ha='center',color= 'black', size = 12, path_effects=[pe.withStroke(linewidth=0.5, foreground="white")]),axis=1)
ax.axis('off');

Plot markers

fig, ax = plt.subplots(figsize=(20,130), subplot_kw={'aspect':'equal'})
distr_new.plot(edgecolor = 'grey', facecolor='none',  ax=ax)
ax.set_title('Distribution of Children homes in Czech Republic', fontsize = 18)
country.plot(ax=ax, facecolor="none", edgecolor="black")
homes.plot(color = 'blue', ax = ax, markersize = 35)
ax.text(16,48.5,  'Source: rejstriky.msmt.cz/opendata/vrejcelk, Author: Lenka Hasova MSc, lenkahas.com', fontsize=10)
ax.axis('off');

Making interactive maps

As you can see, the interactive map wioth folium did not work as expected. Forstly the popups are not working and secondly, the choropleth maps is not showing correctly. I have no idea about the first one, but secodn might be due to a wrong definition of geometrical layer for the choropleth. Please let me know if you spot the issue.

Happy Coding!

# get centrooids for the map
x_map=homes.centroid.x.mean()
y_map=homes.centroid.y.mean()
print(x_map,y_map)

15.47750331531277 49.827483377983235

# source http://andrewgaidus.com/leaflet_webmaps_python/
def add_point_clusters(mapobj, gdf, popup_field_list):
    #Create empty lists to contain the point coordinates and the point pop-up information
    coords, popups = [], [] 
    #Loop through each record in the GeoDataFrame
    for i, row in gdf.iterrows():
        #Append lat and long coordinates to "coords" list
        coords.append([row.geometry.y, row.geometry.x])
        #Create a string of HTML code used in the IFrame popup
        #Join together the fields in "popup_field_list" with a linebreak between them
        label = '<br>'.join([row[field] for field in popup_field_list])
        #Append an IFrame that uses the HTML string to the "popups" list 
        popups.append(IFrame(label, width = 400, height =100))
        
    #Create a Folium feature group for this layer, since we will be displaying multiple layers
    pt_lyr = folium.FeatureGroup(name = 'Domovy')
    
    #Add the clustered points of crime locations and popups to this layer
    pt_lyr.add_children(MarkerCluster(locations = coords, popups = popups))
    
    #Add this point layer to the map object
    mapobj.add_children(pt_lyr)
    return mapobj

def add_choropleth(mapobj, gdf, id_field, value_field, fill_color = 'YlOrRd', fill_opacity = 0.6, 
                    line_opacity = 0.2, num_classes = 5, classifier = 'Fisher_Jenks'):
    #Allow for 3 Pysal map classifiers to display data
    #Generate list of breakpoints using specified classification scheme. List of breakpoint will be input to choropleth function
    if classifier == 'Fisher_Jenks':
        threshold_scale=mapclassify.FisherJenks(gdf[value_field], k = num_classes).bins.tolist()
    if classifier == 'Equal_Interval':
        threshold_scale=mapclassify.EqualInterval(gdf[value_field], k = num_classes).bins.tolist()
    if classifier == 'Quantiles':
        threshold_scale=mapclassify.Quantiles(gdf[value_field], k = num_classes).bins.tolist()
    
    #Convert the GeoDataFrame to WGS84 coordinate reference system
    gdf_wgs84 = gdf.to_crs({'init': 'epsg:4326'})
    
    #Call Folium choropleth function, specifying the geometry as a the WGS84 dataframe converted to GeoJSON, the data as 
    #the GeoDataFrame, the columns as the user-specified id field and and value field.
    #key_on field refers to the id field within the GeoJSON string
    mapobj.choropleth(geo_str = gdf_wgs84.to_json(), data = gdf,
                columns = [id_field, value_field], key_on = 'feature.properties.{}'.format(id_field),
                fill_color = fill_color, fill_opacity = fill_opacity, line_opacity = line_opacity,  
                threshold_scale = threshold_scale)
    return mapobj

mymap = folium.Map(location=[y_map, x_map], zoom_start=7,tiles=None)
folium.TileLayer('CartoDB positron',name="Light Map",control=False).add_to(mymap)


#Update choropleth with point clusters
mymap = add_point_clusters(mymap, homes, [ 'plna','druh'])
#Update basemap with choropleth
folium.Choropleth(
    geo_data=distr_new,
    name='choropleth',
    data=distr_new,
    columns=['NAZEV_LAU1', 'per_thousand_people'],
    key_on='feature.id',
    fill_color='YlGn',
    fill_opacity=0.7,
    line_opacity=0.2,
    legend_name='per_thousand_people'
).add_to(mymap)

folium.LayerControl().add_to(mymap)

mymap

Make this Notebook Trusted to load map: File -> Trust Notebook

Global Food Hazard Network

Sun, 30 Aug 2020 00:00:00 +0000

Food hazards from around the world

This post is based on the initiative from the Jean Golding Institute (JGI) and Food Standard Agency (FSA). The challenge is to provide a visualization that would potentially identify hazards before they can be bought by consumers.

From the challenge description, I got tempted by one of the questions ‘Can we detect any consistent patterns in the way alerts move across countries?’ I will give you some basic information about the phenomena, the data, and guide you through my thinking process behind the visualizations I created.

According to USDA, food hazard ‘is defined by FSIS HACCP as any biological, chemical, or physical property that may cause a food to be unsafe for human consumption.' The FSA data is a collection of global food hazards since 2015, and contains much information such as dates, places, types and further description. Most importantly, the data is in a form of flows from origins to destinations. Here origins are the countries in which the food hazard sources and destinations are countries in which the food hazard was identified. This way I could construct a spatial flows of food hazards.

I noticed that a big amount of food hazards originates and is identified within the same country. The first graph shows you the proportion of those flows (55.3%) as well as those that are identified by other countries (39.5%) or by unknown identifier (5.1%), which could be just an anonymous identifier.. I am really interested in what does this really mean?

On the next graph, you can see the exact numbers of food hazard sourcing in a specific country and fall into one of the previous categories. Interestingly, some countries have a higher proportion of identified food hazards by unknown identifier than the others. The United States, for example. Food hazards originating in the UK are mainly identified in the country.

This can be compared with the next graph that shows the other side, the countries in which the food hazard was identified and the proportions of the same classes. In this case, food hazards have an unknown origin. UK has a higher proportion of identified food hazards from other countries than originating food hazards identified by others. It also identified 17 food hazards that do not have known origin.

Finally, I have constructed a network from the food hazard flows, in mathematics, this is called a graph. Graphs/ networks can be also used for the analysis of its components. I have looked at one of those and defined the importance of each country for the food hazard network. You can see a Rank of importance I included in the popup for the countries. You can also compare this rank with the Food Safety and Quality Rank based on GFSI (Global Food Security Index).

Interestingly, several countries stand out with their pattern. For example, have a look at China, which seems to originate very little food hazards, given the population size, the food safety and quality rank (interactive map) that is lower than you would expect in that case and the common knowledge connected to China. I mean, we all saw several instances in past decades when there was a disease outbreak originating in China, caused by wild animal consumption (aka situation around coronavirus). The Food hazards that originate in China are mainly only identified by other countries. Furthermore, China has no records of identifying global food hazards. I would be interested in further investigation of this because there could be multiple explanations. Most likely, it could be a result of the political system and relaxed law and culture, that could be either not publishing the internal food hazards or simply just do not even record them. It could also be that the food safety system is not on a level that they are able to detect global food hazards. I am only saying this, because several countries follow the same pattern, such as Turkey, Brazil, India, Vietnam or Thailand. However, those are only conspiracies. Drop me a message on twitter if you know of any articles or books that can explain such phenomena.

There is something suspicious also on the USA, just look at the number of food hazards originating in the USA and having an unknown identifier, like someone does not want to be seen as the enemy. Well, drop me a message again if you have any thoughts on that.

Last note, this post is built with plotly, which is an interactive graph library and kepler.gl, which is an open source geospatial analysis tool for large-scale data sets. Kepler.gl is super easy to work with, you can use it on it's own, within a python or build from scratch. This way, I am trying to show that valuable insights into the data, does not need a complicated code and that it can be pretty fun to make them.

Thank for reading, I will try to post the code for this soon, so you can see what has been done.

Application of XGBoost regression for Spatial Interaction of Urban flow

Wed, 27 May 2020 00:00:00 +0000

Introduction

This notebook provides a simple example of the application of XGBoost Spatial Interaction model in python. For the purposes of this tutorial I will expect you to have some knowledge of Spatial Interaction Models and their purpose as well as some python skills. Yet, if you want to refresh a little or wish to learn, here is a list of relevant sources.

Watch Maarten Vanhoof talking about what are the spatial Interaction Models and how they can be used here (6 min)
Have a look at a list of some relevant literature in different fields here
Look at the example of Spatial Interaction Model application in R from Adam Dennett Part1 and Part2
Look at how you can apply the the same method in python with SpInt package

So what is the XGBoost? and how does it work?

XGBoost stands for “Extreme Gradient Boosting”, where the term “Gradient Boosting” originates from the paper Greedy Function Approximation: A Gradient Boosting Machine, by Friedman. Please have a look at this A Gentle Introduction to XGBoost for Applied Machine Learning and the Full documentation to XGBoost that also provides a tutorials. If you are like me and need something more visual and efficient explanation of the how the model works, I would highly recommend YouTube videos from Josh Starmer; Gradient Boost Part 1: Regression Main Ideas, Gradient Boost Part 2: Regression Details.

So after you gone through all the resources you know that there are two types of the model depending on what do you want to do with them. You can use them as a classifier, similar to decision trees, or as a regressor. Using the method as a classifier, we can predict discrete and categorical variables, and using it as a regressors, continuous variables can be predicted. Here we will be talking solely about Gradient Boosting as a regressor, because we are trying to model Urban flow, number of people that moves from one point to another.

What is an Urban flow here?

In this example, I am looking at patients Urban flow. to be specific, this is a patients that are being treated at particular GP, receive a prescription and then go to one of the pharmacies to pick up the prescribed drug. This process of prescribing and dispensing a drug is closely monitored by the National Health Service, although the precise information about the patients is disclosed, the NHS Digital provides completely anonymised data of number of the prescriptions that has been prescribed and dispensed between each pair of GP and Pharmacy each month. Those can be translated as traces of patients movement between two points and together creates rich network of patients flow.

For further description of the data follow up on this notebook.

Cross validation challenge

With any ordinary data, we would usually divide the data into testing set and training set by 80/20 split or any other proportion. However, we face more difficult case here.

The Urban flow networks are very particular case of network, which is basically a representation of the human behaviour in a space. And this is related to anything that affect our decision to choose a destination and the journey to it. Starting from maps, road or public transport structure and the type of vehicle, ending with our personal preference, gender, or family status. The network that the humans created by acting in it has very complex pattern in which the existence of a destination and its location matters.

Now, imagine we would divide the dataset into training and testing, how do we decide which locations will be for training and which ones will be for testing if each location is unique? For example, if we train the model on the flow data in a Scotland, we cannot test the same model on the data from London as people in both places has different behavioural patterns. There is more to it, read through this study that provides nice overview.

Luckily, the data I collected ranges from 2013 to mid-2019, which means that we can divide the data into blocks with the same timeframe. In this case I will use the data from 2017 for training and data from 2018 for testing. This block cross validation is only possible thanks to the consistent data.

Let's get started

Load the libraries

import xgboost as xgb # xgboost model library
import numpy as np # array manipulation
import pandas as pd # dataframe manipulation library
import geopandas as gpd # spatial data frame manipulatioon library
import statsmodels.api as sm
from sklearn.model_selection import train_test_split  # take out the validation packages from scikit.learn
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import mean_squared_error
import matplotlib.pyplot as plt # plotting 
import seaborn as sns

Load the 2017 train data

# define the path
file = r'./../.././data/NHS/working/year_data_flow/predicted_flow_2017.csv'
# read the flow file
fl17 = pd.read_csv(file, index_col=None)

# define the path
file = r'./../.././data/NHS/working/year_data_flow/complete_predicted_flow_2017.csv'
# read the flow file
cfl17 = pd.read_csv(file, index_col=None)

fl17.head()

	P_Code	P_Name_x	P_Postcode	D_Code_x	D_Name_x	D_Postcode	N_Items	Date	loop	...	P_full_address_y	P_Status_Code_y	P_Sub_type_y	P_Setting_y	P_X_y	P_Y_y	CODE	POSTCODE	NUMBER_OF_PATIENTS	yhat
0	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FA251	HORIZON PHARMACY LTD	BS34 6AS	130.0	2017-01-01	False	...	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18363	0.044989
1	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FA613	DOWNHAM FB LTD	BS7 9JT	39.0	2017-01-01	False	...	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18363	0.240777
2	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FCL91	MYGLOBE LTD	BS15 4ND	12.0	2017-01-01	False	...	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18363	0.001761
3	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FCT35	LLOYDS PHARMACY LTD	BS20 7QA	661.0	2017-01-01	False	...	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18363	2393.873899
4	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FDD77	JOHN WARE LIMITED	BS20 6LT	4058.0	2017-01-01	False	...	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18363	2149.320471

5 rows × 77 columns

Load the 2018 test data

# first get the 018 data in, those will be the test data
file = r'./../.././data/NHS/working/year_data_flow/predicted_flow_2018.csv'
# read the flow file
fl18 = pd.read_csv(file, index_col=None)

# define the path
file = r'./../.././data/NHS/working/year_data_flow/complete_predicted_flow_2018.csv'
# read the flow file
cfl18 = pd.read_csv(file, index_col=None)

cfl18.head()

	P_Code	D_Code_new	Date	Density	NUMBER_OF_PATIENTS	N_Items	Euc_distance	great_circle_dist	predicted
0	L81004	FA696	2018-01-01	36.696000	18615	8.0	0.224446	16.560437	0.000777
1	L81004	FCT35	2018-01-01	34.918367	18615	21.0	0.009198	0.653229	1963.680982
2	L81004	FDD77	2018-01-01	29.092308	18615	5309.0	0.014101	0.983999	1673.379704
3	L81004	FD125	2018-01-01	28.670000	18615	26.0	0.177172	12.816035	0.030142
4	L81004	FEA59	2018-01-01	30.833333	18615	1.0	0.292934	20.302017	0.000028

I have already selected the columns we need for the analysis but let me provide you with some explanation.

The gravity equation says that the flow, can be explained by gravitational power of two masses on origin and destination, and inversely proportional to the distance between them.

$$ F_{od} = k\frac{M_o M_d}{D_{od}} $$

Where $F_{od}$ is the flow mass, $M_o$ is the mass on the origin, $M_d$ is mass on the destination and $D_{od}$ is a distance between them.

In our case the flow is the number of patient's between the origin (General Practitioner) and the destination (Pharmacy). The mass of the origin is represented by the number of patients registered at that particular GP in that year (mid-year estimate). The mass of the destination is the average daytime population in 1km radius from the pharmacy. And the distance is a great circle distance between the origin and destination. All of those has been calculated previously and I'm going to bore you with it another time.

In our datasets this is;

	Variable
$F_{od}$	‘N_Items’
$M_o$	‘NUMBER_OF_PATIENTS’
$M_d$	‘Density’
$D_{od}$	‘great_circle_dist’

There are different spatial interaction models but for this purpose we only want to stick with the very basic form, one based on gravity model. We just need to translate the formula into an regression, Poisson regression to be more specific, because the flow is a count of the patients and skewed dis

Spatial Interaction model definition

$$F_{od} = \exp( k + \mu \ln{M_o} + \alpha \ln{M_d} − \beta \ln{ D_{od}} )$$

Where $\lambda_{od}$ is the flow from $o$ to $d$, logarithmically linked or modelled by a linear combination of the logged independent vars. $\kappa ; \mu ; \alpha$ and $\beta$ are parameters to be estimated, $M_o$ is the mass on origin, $M_d$ is the mass on destination and $D_{od}$ is the distance between origin and destination.

this is in log-normal equation;

$$\ln{F_{od}} = k + \mu \ln{M_d} + \alpha \ln{M_d} − \beta \ln{ D_{od}}$$

and Poisson equation, which should have the same result;

$$F_{od} = k + \mu \ln{M_o} + \alpha \ln{M_d} − \beta \ln{ D_{od}}$$

Let's try both

log-normal model
Poisson model, correct definition

The XGBOOST offers

objective = “reg:squarederror”
objective = “count:poisson”

Note the regressor takes a all the data from data frame as they are, so we need to apply the transformations before we build to model as I understand.

# log all the variables
fl17['log_N_Items'] = np.log(fl17['N_Items'])
fl18['log_N_Items'] = np.log(fl18['N_Items'])

# log all the variables
fl17['1*great_circle_dist'] = 1*fl17['great_circle_dist']
fl18['1*great_circle_dist'] = 1*fl18['great_circle_dist']

# Do we need to do the 1-Distance???

fl17['log_NUMBER_OF_PATIENTS'] = np.log(fl17['NUMBER_OF_PATIENTS'])
fl18['log_NUMBER_OF_PATIENTS'] = np.log(fl18['NUMBER_OF_PATIENTS'])

fl17['log_Density'] = np.log(fl17['Density'])
fl18['log_Density'] = np.log(fl18['Density'])

# do the same thing for the complete data

# log all the variables
cfl17['1*great_circle_dist'] = 1*cfl17['great_circle_dist']
cfl18['1*great_circle_dist'] = 1*cfl18['great_circle_dist']

# Do we need to do the 1-Distance???

cfl17['log_NUMBER_OF_PATIENTS'] = np.log(cfl17['NUMBER_OF_PATIENTS'])
cfl18['log_NUMBER_OF_PATIENTS'] = np.log(cfl18['NUMBER_OF_PATIENTS'])

cfl17['log_Density'] = np.log(cfl17['Density'])
cfl18['log_Density'] = np.log(cfl18['Density'])

fl17log = fl17.iloc[:,[77,78,79,80]]
fl18log = fl18.iloc[:,[77,78,79,80]]

fl18log

	log_N_Items	1*great_circle_dist	log_NUMBER_OF_PATIENTS	log_Density
0	2.079442	16.560437	9.831723	3.602668
1	3.044522	0.653229	9.831723	3.553013
2	8.577159	0.983999	9.831723	3.370474
3	3.258097	12.816035	9.831723	3.355851
4	0.000000	20.302017	9.831723	3.428596
...	...	...	...	...
71729	0.000000	0.973280	6.907755	5.144711
71730	0.000000	1.015876	6.907755	4.824431
71731	0.693147	1.507605	6.907755	4.806714
71732	1.098612	1.739077	6.907755	4.549942
71733	0.693147	1.578409	6.907755	4.358013

71734 rows × 4 columns

fl17log.hist(figsize = (10,10));

Log-normal XGBoost SIM model

Parameters

booster = ‘dart’

XGBoost mostly combines a huge number of regression trees with a small learning rate. In this situation, trees added early are significant and trees added late are unimportant. Vinayak and Gilad-Bachrach proposed a new method to add dropout techniques from the deep neural net community to boosted trees and reported better results in some situations. Drop trees to solve the over-fitting. Trivial trees (to correct trivial errors) may be prevented. Because of the randomness introduced in the training, expect the following few differences: Training can be slower than gbtree because the random dropout prevents usage of the prediction buffer. The early stop might not be stable, due to the randomness.

Follow here to see all the parameters.

I also found this tutorial which has used the grid search from ‘scikit’ cross validation to test several versions of the model and it's parameters. This comes very handy, as we can instantly find the best performing model in instance. Maybe this can help to explain a bit more about that.

Learning rate = Step size shrinkage used in update to prevents overfitting. After each boosting step, we can directly get the weights of new features, and eta shrinks the feature weights to make the boosting process more conservative.

max_depth = Maximum depth of a tree. Increasing this value will make the model more complex and more likely to overfit. 0 is only accepted in lossguided growing policy when tree_method is set as hist and it indicates no limit on depth. Beware that XGBoost aggressively consumes memory when training a deep tree. Range: [0,∞] (0 is only accepted in lossguided growing policy when tree_method is set as hist)

Note

Below, you can see an example for grid search for the best model within certain parameters. It was very efficient when I was working with the log-normal model only, however, using it within the Poisson version, the model converges after third of all the runs (depends how many options you give it)

# define the training and testing dataset
X_train, y_train = fl17log.loc[:,fl17log.columns != 'log_N_Items'], fl17log.loc[:,['log_N_Items']]
X_test, y_test = fl18log.loc[:,fl18log.columns != 'log_N_Items'], fl18log.loc[:,['log_N_Items']]

# transfer those into matrix as recommended class by in the package
DM_train = xgb.DMatrix(data = X_train, 
                       label = y_train)
					   
DM_test =  xgb.DMatrix(data = X_test,
                       label = y_test)

# define the parameters that we want to inspect
gbm_param_grid = {
     'colsample_bytree': np.linspace(0.5, 0.9), # define fairly wide range
     'n_estimators':[15, 20], # try different numbers
     'max_depth': [ 15, 20] # different depths
}

# define the regressor model
gbm = xgb.XGBRegressor(objective = "reg:squarederror", booster = 'dart')

# define the grid search cross validation 
grid_mse = GridSearchCV(estimator = gbm, param_grid = gbm_param_grid, scoring = 'neg_mean_squared_error', cv = 5, verbose = 1)

# fit the models and go through all the parameters
grid_mse.fit(X_train, y_train)

# print the best results
print("Best parameters found: ",grid_mse.best_params_)
print("Lowest RMSE found: ", np.sqrt(np.abs(grid_mse.best_score_)))

# define the training and testing dataset
X_train, y_train = fl17log.loc[:,fl17log.columns != 'log_N_Items'], fl17log.loc[:,['log_N_Items']]
X_test, y_test = fl18log.loc[:,fl18log.columns != 'log_N_Items'], fl18log.loc[:,['log_N_Items']]

# define the model (I already tested several version and stacked with the default setting and learning rate 0.4, the grid search is work in progress)
gbm = xgb.XGBRegressor(objective = "reg:squarederror", learning_rate= 0.4).fit(X_train, y_train)

pred = gbm.predict(X_test)

print("Root mean square error for test dataset: {}".format(np.round(np.sqrt(mean_squared_error(np.exp(y_test), np.exp(pred))), 2)))

Root mean square error for test dataset: 949.98

test = pd.DataFrame({"prediction": np.exp(pred), "observed": np.exp(y_test['log_N_Items'])})

lowess = sm.nonparametric.lowess

z = lowess(np.exp(pred).flatten(), np.exp(y_test['log_N_Items']))


test.plot(figsize = [14,8],
          x ="prediction", y = "observed", kind = "scatter", color = 'darkred')

plt.title("Extreme Gradient Boosting: Prediction Vs Test Data", fontsize = 18, color = "darkgreen")

plt.xlabel("Predicted Power Output", fontsize = 18) 

plt.ylabel("Observed Power Output", fontsize = 18)

plt.plot(z[:,0], z[:,1], color = "blue", lw= 3)

plt.show()

fl17['log'] =  np.exp(gbm.predict(X_train))

fl18['log'] =  np.exp(gbm.predict(X_test))

Poisson XGBoost SIM model

# create new dataset for the Poisson
# this will be logged masses, 1*dist and no change to y
fl17poi = fl17.iloc[:,[6,78,79,80]]
fl18poi = fl18.iloc[:,[6,78,79,80]]
cfl17poi = cfl17.iloc[:,[9,10,11,5]]
cfl18poi = cfl18.iloc[:,[9,10,11,5]]

fl17poi.hist(figsize = (10,10));
cfl17poi.hist(figsize = (10,10));

For published data only

# define the training and testing dataset
X_train2, y_train2 = fl17poi.loc[:,fl17poi.columns != 'N_Items'], fl17poi.loc[:,['N_Items']]
X_test2, y_test2 = fl18poi.loc[:,fl18poi.columns != 'N_Items'], fl18poi.loc[:,['N_Items']]

# define the model
gbm2 = xgb.XGBRegressor(objective = "count:poisson",  learning_rate= 0.4).fit(X_train2, y_train2)

For complete data with all the 0 flows in

# define the training and testing dataset
X_train3, y_train3 = cfl17poi.loc[:,cfl17poi.columns != 'N_Items'], cfl17poi.loc[:,['N_Items']]
X_test3, y_test3 = cfl18poi.loc[:,cfl18poi.columns != 'N_Items'], cfl18poi.loc[:,['N_Items']]

gbm3 = xgb.XGBRegressor(objective = "count:poisson",  learning_rate= 0.4).fit(X_train3, y_train3)

Best parameters found: {‘colsample_bytree’: 0.8, ‘max_depth’: 40, ‘n_estimators’: 30} Lowest RMSE found: 1159.4761503247805

pred2 = gbm2.predict(X_test2)

print("Root mean square error for test dataset: {}".format(np.round(np.sqrt(mean_squared_error(y_test2, pred2)), 2)))

pred3 = gbm3.predict(X_test3)

print("Root mean square error for test dataset: {}".format(np.round(np.sqrt(mean_squared_error(y_test3, pred3)), 2)))

Root mean square error for test dataset: 704.17
Root mean square error for test dataset: 279.73

sns.regplot(np.exp(pred), np.exp(y_test['log_N_Items']), color = 'blue',line_kws = {'color':'red'})
plt.title('Predicted vs actual, Log-normal XGBOOST, published data only', fontsize=20);

sns.regplot(pred2, y_test2['N_Items'], color = 'blue',line_kws = {'color':'red'})
plt.title('Predicted vs actual, Poisson XGBOOST, published data only', fontsize=20);

sns.regplot(pred3, y_test3['N_Items'], color = 'blue',line_kws = {'color':'red'})
plt.title('Predicted vs actual, Poisson XGBOOST, complete data', fontsize=20);

The Poisson seems to be better when using the default parameter setting and learning rate 0.4

We already have the in sample and out of sample predictions from gravity model in the data, so now we need to add the in and out of sample prediction from the XGB model

# published
fl17['xgb'] =  gbm2.predict(X_train2)

fl18['xgb'] =  gbm2.predict(X_test2)

# complete
cfl17['xgb'] =  gbm2.predict(X_train3)

cfl18['xgb'] =  gbm2.predict(X_test3)

fl18.head()

	P_Code	P_Name_x	P_Postcode	D_Code_x	D_Name_x	D_Postcode	N_Items	Date	loop	...	CODE	POSTCODE	NUMBER_OF_PATIENTS	predicted	log_N_Items	1*great_circle_dist	log_NUMBER_OF_PATIENTS	log_Density	log	xgb
0	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FA696	THE BULLEN HEALTHCARE GROUP LIMITED	BS32 4JT	8.0	2018-01-01	False	...	L81004	BS20 6AQ	18615	0.021203	2.079442	16.560437	9.831723	3.602668	3.172393	2.952730
1	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FCT35	LLOYDS PHARMACY LTD	BS20 7QA	21.0	2018-01-01	False	...	L81004	BS20 6AQ	18615	2417.864485	3.044522	0.653229	9.831723	3.553013	163.713943	861.965210
2	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FDD77	JOHN WARE LIMITED	BS20 6LT	5309.0	2018-01-01	False	...	L81004	BS20 6AQ	18615	2170.860226	8.577159	0.983999	9.831723	3.370474	833.031494	3730.304199
3	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FD125	BOOTS UK LIMITED	BS34 5UP	26.0	2018-01-01	False	...	L81004	BS20 6AQ	18615	0.390401	3.258097	12.816035	9.831723	3.355851	6.255939	5.463578
4	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FEA59	LLOYDS PHARMACY LTD	BS16 7AE	1.0	2018-01-01	False	...	L81004	BS20 6AQ	18615	0.001570	0.000000	20.302017	9.831723	3.428596	3.263482	1.575869

5 rows × 83 columns

The accuracy measure

Lets organize this into a table;

	R2 Gravity SIM	R2 XGBOOST SIM	RMSE Gravity SIM	RMSE XGBOOST SIM
Models based on published data	PGIr2	PXIr2	PGIrs	PXIrs
	PGOr2	PXOr2	PGOrs	PXOrs
Models based on complete data	CGIr2	CXIr2	CGIrs	CXIrs
	CGOr2	CXOr2	CGOrs	CXOrs

# define R2 function
def get_r2_numpy_corrcoef(x, y):
    return np.corrcoef(x, y)[0, 1]**2

# define RMSE function
def rmse(predictions, targets):
    return np.sqrt(((predictions - targets) ** 2).mean())

Other evaluations?

Some is available at Robinsons github, but requires matrices.

See to definition of the Common Part of Commuters measure. Source article

def cpc(actual, predicted):
    
    # reshape the variables 
    predicted = np.reshape(np.array(predicted), (-1, 1))
    actual = np.reshape(np.array(actual),(-1, 1))
    # stack them together
    YYhat = np.hstack([actual, predicted])
    # define sum of the minimym of the predicted var
    NCC = np.sum(np.min(YYhat, axis=1)) # , axis=1
    # number of observations
    N = len(actual)
    # sum of all the actual numbers (reshaped)
    NCY = np.sum(actual)
    # sum of the predicted numbers (reshaped)
    NCYhat = np.sum(predicted)
    
    numerator = (N * NCC)
    denominator = (NCY + NCYhat)
    
    return numerator/denominator

$$ \frac{ N_i * \sum \min(ij) }{ \sum_i + \sum_j } $$

def sorensen(actual, predicted):
    # reshape the variables
    predicted = np.reshape(np.array(predicted), (-1, 1))
    actual = np.reshape(np.array(actual),(-1, 1))
    # define length of the variables
    N =  len(actual)
    # stack them together in array
    YYhat = np.hstack([actual, predicted])
    # define 2 times minimum from both
    num = 2.0 * np.min(YYhat, axis=1)
    # define the differnece
    den =  predicted + actual
           
    return (1.0 / N) * (np.sum(num.reshape((-1, 1)) / den.reshape((-1, 1))))

$$ \frac{1}{N} * \frac{\sum{2 * \min(i,j)} }{j-i} $$

Let's do the result tables again with those measures

	R2 Gravity SIM	R2 XGBOOST SIM	RMSE Gravity SIM	RMSE XGBOOST SIM	CPC Gravity SIM	CPC XGBOOST SIM	SORENSEN I Gravity	SORENSEN I XGBOOST
Models based on published data	PGIr2	PXIr2	PGIrs	PXIrs	PGIcpc	PXIcpc	PGIsn	PXIsn
	PGOr2	PXOr2	PGOrs	PXOrs	PGOcpc	PXOcpc	PGOsn	PXOsn
Models based on complete data	CGIr2	CXIr2	CGIrs	CXIrs	CGIcpc	CXIcpc	CGIsn	CXIsn
	CGOr2	CXOr2	CGOrs	CXOrs	CGOcpc	CXOcpc	CGOsn	CXOsn

# R2
# published data
PGIr2 = round(get_r2_numpy_corrcoef(fl17['N_Items'], fl17['yhat']),3)
PGOr2 = round(get_r2_numpy_corrcoef(fl18['N_Items'], fl18['predicted']),3)

PXIr2 = round(get_r2_numpy_corrcoef(fl17['N_Items'], fl17['xgb']),3)
PXOr2 = round(get_r2_numpy_corrcoef(fl18['N_Items'], fl18['xgb']),3)

# complete data
CGIr2 = round(get_r2_numpy_corrcoef(cfl17['N_Items'], cfl17['yhat']),3)
CGOr2 = round(get_r2_numpy_corrcoef(cfl18['N_Items'], cfl18['predicted']),3)

CXIr2 = round(get_r2_numpy_corrcoef(cfl17['N_Items'], cfl17['xgb']),3)
CXOr2 = round(get_r2_numpy_corrcoef(cfl18['N_Items'], cfl18['xgb']),3)

# RMSE

PGIrs = round(rmse(fl17['N_Items'], fl17['yhat']),1)
PGOrs = round(rmse(fl18['N_Items'], fl18['predicted']),1)

PXIrs = round(rmse(fl17['N_Items'], fl17['xgb']),1)
PXOrs = round(rmse(fl18['N_Items'], fl18['xgb']),1)

CGIrs = round(rmse(cfl17['N_Items'], cfl17['yhat']),1)
CGOrs = round(rmse(cfl18['N_Items'], cfl18['predicted']),1)

CXIrs = round(rmse(cfl17['N_Items'], cfl17['xgb']),1)
CXOrs = round(rmse(cfl18['N_Items'], cfl18['xgb']),1)

# CPC

PGIcpc = round(cpc(fl17['N_Items'], fl17['yhat']),1)
PGOcpc = round(cpc(fl18['N_Items'], fl18['predicted']),1)

PXIcpc = round(cpc(fl17['N_Items'], fl17['xgb']),1)
PXOcpc = round(cpc(fl18['N_Items'], fl18['xgb']),1)

CGIcpc = round(cpc(cfl17['N_Items'], cfl17['yhat']),1)
CGOcpc = round(cpc(cfl18['N_Items'], cfl18['predicted']),1)

CXIcpc = round(cpc(cfl17['N_Items'], cfl17['xgb']),1)
CXOcpc = round(cpc(cfl18['N_Items'], cfl18['xgb']),1)

# Sorensen 

PGIsn = round(sorensen(fl17['N_Items'], fl17['yhat']),3)
PGOsn = round(sorensen(fl18['N_Items'], fl18['predicted']),3)

PXIsn = round(sorensen(fl17['N_Items'], fl17['xgb']),3)
PXOsn = round(sorensen(fl18['N_Items'], fl18['xgb']),3)

CGIsn = round(sorensen(cfl17['N_Items'], cfl17['yhat']),3)
CGOsn = round(sorensen(cfl18['N_Items'], cfl18['predicted']),3)

CXIsn = round(sorensen(cfl17['N_Items'], cfl17['xgb']),3)
CXOsn = round(sorensen(cfl18['N_Items'], cfl18['xgb']),3)

# everything for the log-normal model
LXIr2 = round(get_r2_numpy_corrcoef(fl17['N_Items'], fl17['log']),3)
LXOr2 = round(get_r2_numpy_corrcoef(fl18['N_Items'], fl18['log']),3)

LXIrs = round(rmse(fl17['N_Items'], fl17['log']),1)
LXOrs = round(rmse(fl18['N_Items'], fl18['log']),1)

LXIcpc = round(cpc(fl17['N_Items'], fl17['log']),1)
LXOcpc = round(cpc(fl18['N_Items'], fl18['log']),1)

LXIsn = round(sorensen(fl17['N_Items'], fl17['log']),3)
LXOsn = round(sorensen(fl18['N_Items'], fl18['log']),3)

model_name = ['In-sample','Out-sample']
mG = [PGIr2, PGOr2]
mX = [PXIr2, PXOr2]
mF = [PGIrs, PGOrs]
mH = [PXIrs, PXOrs]
mD1 = [PGIcpc, PGOcpc]
mD2 = [PXIcpc, PXOcpc]
mK1 = [PGIsn, PGOsn]
mK2 = [PXIsn, PXOsn]

cols = {'Poisson models based on published data': model_name,'R2 Gravity SIM': mG, 'R2 XGBOOST SIM': mX, 'RMSE Gravity SIM': mF, 'RMSE XGBOOST SIM': mH, 'CPC Gravity SIM': mD1,
      'CPC XGBOOST SIM': mD2,  'SORENSEN I Gravity': mK1, 'SORENSEN I XGBOOST': mK2}

table_published = pd.DataFrame(cols).set_index('Poisson models based on published data')

model_name = ['In-sample','Out-sample']
G = [CGIr2, CGOr2]
X = [CXIr2, CXOr2]
mF = [CGIrs, CGOrs]
mH = [CXIrs, CXOrs]
mD1 = [CGIcpc, CGOcpc]
mD2 = [CXIcpc, CXOcpc]
mK1 = [CGIsn, CGOsn]
mK2 = [CXIsn, CXOsn]

cols = {'Poisson models based on complete data': model_name,'R2 Gravity SIM': G, 'R2 XGBOOST SIM': X, 'RMSE Gravity SIM': mF, 'RMSE XGBOOST SIM': mH,  'CPC Gravity SIM': mD1,
      'CPC XGBOOST SIM': mD2,  'SORENSEN I Gravity': mK1, 'SORENSEN I XGBOOST': mK2}

table_complete = pd.DataFrame(cols).set_index('Poisson models based on complete data')

NA = 'NA'
model_name = ['In-sample','Out-sample']
mG = [NA, NA]
mX = [LXIr2, LXOr2]
mF = [NA, NA]
mH = [LXIrs, LXOrs]
mD1 = [NA, NA]
mD2 = [LXIcpc, LXOcpc]
mK1 = [NA, NA]
mK2 = [LXIsn, LXOsn]

cols = {'Log-normal models based on published data': model_name,'R2 Gravity SIM': mG, 'R2 XGBOOST SIM': mX, 'RMSE Gravity SIM': mF, 'RMSE XGBOOST SIM': mH, 'CPC Gravity SIM': mD1,
      'CPC XGBOOST SIM': mD2,  'SORENSEN I Gravity': mK1, 'SORENSEN I XGBOOST': mK2}

table_log = pd.DataFrame(cols).set_index('Log-normal models based on published data')

table_log

	R2 Gravity SIM	R2 XGBOOST SIM	RMSE Gravity SIM	RMSE XGBOOST SIM	CPC Gravity SIM	CPC XGBOOST SIM	SORENSEN I Gravity	SORENSEN I XGBOOST
Log-normal models based on published data
In-sample	NA	0.560	NA	809.9	NA	25504.1	NA	0.686
Out-sample	NA	0.381	NA	950.0	NA	21690.5	NA	0.620

# models based on published data only
table_published

	R2 Gravity SIM	R2 XGBOOST SIM	RMSE Gravity SIM	RMSE XGBOOST SIM	CPC Gravity SIM	CPC XGBOOST SIM	SORENSEN I Gravity	SORENSEN I XGBOOST
Poisson models based on published data
In-sample	0.321	0.736	976.2	611.1	17813.1	27736.8	0.318	0.513
Out-sample	0.330	0.653	981.4	704.2	17207.5	24790.0	0.314	0.490

# models based on complete data with all the zero flows included
table_complete

	R2 Gravity SIM	R2 XGBOOST SIM	RMSE Gravity SIM	RMSE XGBOOST SIM	CPC Gravity SIM	CPC XGBOOST SIM	SORENSEN I Gravity	SORENSEN I XGBOOST
Poisson models based on complete data
In-sample	0.287	0.691	380.6	249.9	106667.5	168990.5	0.045	0.070
Out-sample	0.297	0.605	385.3	288.8	100976.7	145788.9	0.046	0.068

x = table_log.append(table_published)
x = x.append(table_complete)
x['model_type'] = ['log-normal', 'log-normal', 'Poisson', 'Poisson', 'Poisson', 'Poisson']
x['data_type'] = ['published_only', 'published_only', 'published_only', 'published_only', 'complete', 'complete']
x['CV'] = ['In-sample', 'Out-sample','In-sample', 'Out-sample','In-sample', 'Out-sample']
x = x.set_index(['model_type', 'data_type'])
x

		R2 Gravity SIM	R2 XGBOOST SIM	RMSE Gravity SIM	RMSE XGBOOST SIM	CPC Gravity SIM	CPC XGBOOST SIM	SORENSEN I Gravity	SORENSEN I XGBOOST	CV
model_type	data_type
log-normal	published_only	NA	0.560	NA	809.9	NA	25504.1	NA	0.686	In-sample
published_only	NA	0.381	NA	950.0	NA	21690.5	NA	0.620	Out-sample
Poisson	published_only	0.321	0.736	976.2	611.1	17813.1	27736.8	0.318	0.513	In-sample
published_only	0.33	0.653	981.4	704.2	17207.5	24790.0	0.314	0.490	Out-sample
complete	0.287	0.691	380.6	249.9	106668	168990.5	0.045	0.070	In-sample
complete	0.297	0.605	385.3	288.8	100977	145788.9	0.046	0.068	Out-sample

x.iloc[[0,2,4],:]['R2 XGBOOST SIM'].plot() 
x.iloc[[1,3,5],:]['R2 XGBOOST SIM'].plot()
plt.title('R2 XGBOOST', fontsize=20)
plt.legend(['In-sample', 'Out-sample'])
plt.xlabel(' ');

x.iloc[[0,2,4],:]['RMSE XGBOOST SIM'].plot()
x.iloc[[1,3,5],:]['RMSE XGBOOST SIM'].plot()
plt.title('RMSE  XGBOOST', fontsize=20)
plt.legend(['In-sample', 'Out-sample'])
plt.xlabel(' ');

x.iloc[[0,2,4],:]['CPC XGBOOST SIM'].plot()
x.iloc[[1,3,5],:]['CPC XGBOOST SIM'].plot()
plt.title('CPC XGBOOST', fontsize=20)
plt.legend(['In-sample', 'Out-sample'])
plt.xlabel(' ');

x.iloc[[0,2,4],:]['SORENSEN I XGBOOST'].plot()
x.iloc[[1,3,5],:]['SORENSEN I XGBOOST'].plot()
plt.title('SORENSEN XGBOOST', fontsize=20)
plt.legend(['In-sample', 'Out-sample'])
plt.xlabel(' ');

Sorensen similarity Index is bounded between values of 0 and 1 with values closer to 1 indicating a better model fit.

How come that the complete dataset has exceptionally low index, because of the zero issue?

cfl17.loc[:,['N_Items', 'xgb']]

	N_Items	xgb
0	130.0	11.200489
1	39.0	7.248779
2	12.0	3.174812
3	661.0	861.965210
4	4058.0	3730.304199
...	...	...
543739	0.0	8.880725
543740	0.0	5.127942
543741	0.0	5.786361
543742	0.0	7.492306
543743	0.0	18.535973

543744 rows × 2 columns

Prob it is caused by the excessive zeros, we would probably need to introduce an ensemble model to deal with it.

Some other is available from Tylor Oshan He has CPC as well as Sorensen index

Sensitivity analysis

Thu, 21 May 2020 00:00:00 +0000

Load the data and the libraries

import pandas as pd
import os
import numpy as np
import glob
import geopandas as gpd
from shapely.geometry import Point
import matplotlib.pyplot as plt
import datetime as dt
import seaborn as sns
import plotly.express as px
import itertools
from scipy import stats
import sys
sys.path.append('/Users/toshan/dev/pysal/pysal/contrib/spint')
from spint.gravity import Gravity, BaseGravity
import spint.gravity
#from utils import sorensen
import statsmodels.formula.api as smf
from statsmodels.api import families as families

wgs84 = {'init':'EPSG:4326'}
bng = {'init':'EPSG:27700'}

file = './../../data/NHS/working/year_data_flow/flow_2018.csv'
fl18 = pd.read_csv(file, index_col=None)

file = './../../data/NHS/working/year_data_flow/flow_2017.csv'
fl17 = pd.read_csv(file, index_col=None)

file = './../../data/NHS/working/year_data_flow/flow_2016.csv'
fl16 = pd.read_csv(file, index_col=None)

file = './../../data/NHS/working/year_data_flow/flow_2015.csv'
fl15 = pd.read_csv(file, index_col=None)

# plot the top of the table
fl18.head()

	P_Code	P_Name_x	P_Postcode	D_Code_x	D_Name_x	D_Postcode	N_Items	Date	loop	...	P_Name	P_full_address_y	P_Status_Code_y	P_Sub_type_y	P_Setting_y	P_X_y	P_Y_y	CODE	POSTCODE	NUMBER_OF_PATIENTS
0	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FA696	THE BULLEN HEALTHCARE GROUP LIMITED	BS32 4JT	8.0	2018-01-01	False	...	PORTISHEAD MEDICAL GROUP	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18615
1	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FCT35	LLOYDS PHARMACY LTD	BS20 7QA	21.0	2018-01-01	False	...	PORTISHEAD MEDICAL GROUP	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18615
2	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FDD77	JOHN WARE LIMITED	BS20 6LT	5309.0	2018-01-01	False	...	PORTISHEAD MEDICAL GROUP	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18615
3	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FD125	BOOTS UK LIMITED	BS34 5UP	26.0	2018-01-01	False	...	PORTISHEAD MEDICAL GROUP	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18615
4	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FEA59	LLOYDS PHARMACY LTD	BS16 7AE	1.0	2018-01-01	False	...	PORTISHEAD MEDICAL GROUP	PORTISHEAD MEDICAL GROUP,PORTISHEAD MEDICAL GR...	A	B	4	-2.767318	51.482805	L81004	BS20 6AQ	18615

5 rows × 76 columns

Look ate the time series of the prescription flow

monthly = fl18.set_index('Date')
monthly = monthly.groupby(monthly.index).mean()
cols = ["N_EPSItems"]
monthly['N_EPSItems'] = monthly['N_EPSItems'].replace({0:np.nan})

ax = plt.axes()
sns.set(rc={'figure.figsize':(20, 6)})
monthly['N_Items'].plot()
monthly['N_EPSItems'].plot(color = 'red')
#set legend
plt.legend(labels=['Paper Prescriptions', 'Electronic Prescriptions'])
# Add axis and chart labels.
ax.set_xlabel('Month', labelpad=15)
ax.set_ylabel('Count', labelpad=15)
ax.set_title('Average prescription flow in Avon, 2018', pad=10, size = 15)
# sort out thex axes
#plt.xticks([ 8, 20, 32, 44, 56, 68, 80 ], ('December 2012', 'December 2013', 'December 2014', 'December 2015', 'December 2016', 'December 2017', 'December 2018'),rotation=20);
#plt.xticks([ 0, 12, 24, 36, 48, 60, 72], ('January 2013', 'January 2014', 'January 2015', 'January 2016', 'January 2017', 'January 2018', 'January 2019'),rotation=20);

Text(0.5, 1.0, 'Average prescription flow in Avon, 2018')

monthly = fl17.set_index('Date')
monthly = monthly.groupby(monthly.index).mean()
cols = ["N_EPSItems"]
monthly['N_EPSItems'] = monthly['N_EPSItems'].replace({0:np.nan})

ax = plt.axes()
sns.set(rc={'figure.figsize':(20, 6)})
monthly['N_Items'].plot()
monthly['N_EPSItems'].plot(color = 'red')
#set legend
plt.legend(labels=['Paper Prescriptions', 'Electronic Prescriptions'])
# Add axis and chart labels.
ax.set_xlabel('Month', labelpad=15)
ax.set_ylabel('Count', labelpad=15)
ax.set_title('Average prescription flow in Avon, 2017', pad=10, size = 15)
# sort out thex axes
#plt.xticks([ 8, 20, 32, 44, 56, 68, 80 ], ('December 2012', 'December 2013', 'December 2014', 'December 2015', 'December 2016', 'December 2017', 'December 2018'),rotation=20);
#plt.xticks([ 0, 12, 24, 36, 48, 60, 72], ('January 2013', 'January 2014', 'January 2015', 'January 2016', 'January 2017', 'January 2018', 'January 2019'),rotation=20);

Text(0.5, 1.0, 'Average prescription flow in Avon, 2017')

monthly = fl16.set_index('Date')
monthly = monthly.groupby(monthly.index).mean()
cols = ["N_EPSItems"]
monthly['N_EPSItems'] = monthly['N_EPSItems'].replace({0:np.nan})

ax = plt.axes()
sns.set(rc={'figure.figsize':(20, 6)})
monthly['N_Items'].plot()
monthly['N_EPSItems'].plot(color = 'red')
#set legend
plt.legend(labels=['Paper Prescriptions', 'Electronic Prescriptions'])
# Add axis and chart labels.
ax.set_xlabel('Month', labelpad=15)
ax.set_ylabel('Count', labelpad=15)
ax.set_title('Average prescription flow in Avon, 2016', pad=10, size = 15)
# sort out thex axes
#plt.xticks([ 8, 20, 32, 44, 56, 68, 80 ], ('December 2012', 'December 2013', 'December 2014', 'December 2015', 'December 2016', 'December 2017', 'December 2018'),rotation=20);
#plt.xticks([ 0, 12, 24, 36, 48, 60, 72], ('January 2013', 'January 2014', 'January 2015', 'January 2016', 'January 2017', 'January 2018', 'January 2019'),rotation=20);

Text(0.5, 1.0, 'Average prescription flow in Avon, 2016')

monthly = fl15.set_index('Date')
monthly = monthly.groupby(monthly.index).mean()
cols = ["N_EPSItems"]
monthly['N_EPSItems'] = monthly['N_EPSItems'].replace({0:np.nan})

ax = plt.axes()
sns.set(rc={'figure.figsize':(20, 6)})
monthly['N_Items'].plot()
monthly['N_EPSItems'].plot(color = 'red')
#set legend
plt.legend(labels=['Paper Prescriptions', 'Electronic Prescriptions'])
# Add axis and chart labels.
ax.set_xlabel('Month', labelpad=15)
ax.set_ylabel('Count', labelpad=15)
ax.set_title('Average prescription flow in Avon, 2015', pad=10, size = 15)
# sort out thex axes
#plt.xticks([ 8, 20, 32, 44, 56, 68, 80 ], ('December 2012', 'December 2013', 'December 2014', 'December 2015', 'December 2016', 'December 2017', 'December 2018'),rotation=20);
#plt.xticks([ 0, 12, 24, 36, 48, 60, 72], ('January 2013', 'January 2014', 'January 2015', 'January 2016', 'January 2017', 'January 2018', 'January 2019'),rotation=20);

Text(0.5, 1.0, 'Average prescription flow in Avon, 2015')

From this we can see the the average number of prescriptions is slowly increasing over time. However, we can also see that the years 2017 and 2018 are fairly stable years in compare with the 2016 and 2015 which has some significant fluctations in the average prescribing.

Look at the variables in 2017 and 2018

plt.figure(figsize = [15,10])
plt.suptitle("Number of prescriptions on a flow", fontsize=25)
plt.hist(fl15['N_Items'], histtype='step', linewidth=6, color = '#310F78')
plt.hist(fl16['N_Items'], histtype='step', linewidth=5, color = '#581E9B')
plt.hist(fl17['N_Items'], histtype='step', linewidth=4, color = '#894ABD')
plt.hist(fl18['N_Items'], histtype='step',linewidth=3, color = '#B581D6')
plt.legend(labels=['2015', '2106', '2017', '2018']);


plt.figure(figsize = [15,10])
plt.suptitle("Distance of flow", fontsize=25)
plt.hist(fl15['great_circle_dist'], histtype='step', linewidth=6 ,  color = '#310F78')
plt.hist(fl16['great_circle_dist'], histtype='step', linewidth=5 ,color = '#581E9B')
plt.hist(fl17['great_circle_dist'], histtype='step', linewidth=4,color = '#894ABD' )
plt.hist(fl18['great_circle_dist'], histtype='step',linewidth=3 ,color = '#B581D6')
plt.legend(labels=['2015', '2106', '2017', '2018']);


plt.figure(figsize = [15,10])
plt.suptitle("Number of patients registered at each GP", fontsize=25)
plt.hist(fl15['NUMBER_OF_PATIENTS'], histtype='step', linewidth=6 , color = '#310F78')
plt.hist(fl16['NUMBER_OF_PATIENTS'], histtype='step', linewidth=5 ,color = '#581E9B')
plt.hist(fl17['NUMBER_OF_PATIENTS'], histtype='step', linewidth=4,color = '#894ABD' )
plt.hist(fl18['NUMBER_OF_PATIENTS'], histtype='step',linewidth=3 ,color = '#B581D6')
plt.legend(labels=['2015', '2106', '2017', '2018']);


plt.figure(figsize = [15,10])
plt.suptitle("Daytime population density around the pharmacies", fontsize=25)
plt.hist(fl15['Density'], histtype='step', linewidth=6 ,  color = '#310F78')
plt.hist(fl16['Density'], histtype='step', linewidth=5,color = '#581E9B' )
plt.hist(fl17['Density'], histtype='step', linewidth=4,color = '#894ABD' )
plt.hist(fl18['Density'], histtype='step',linewidth=3,color = '#B581D6' )
plt.legend(labels=['2015', '2106', '2017', '2018']);

fl15.iloc[:,[6, 48, 76, 65]].describe()

	N_Items	great_circle_dist	NUMBER_OF_PATIENTS	Density
count	79975.000000	79975.000000	79975.000000	79975.000000
mean	257.286565	6.904360	9345.992135	59.599734
std	1112.177455	6.579699	4298.057099	43.362001
min	1.000000	0.000000	530.000000	0.833333
25%	2.000000	2.425812	6615.000000	33.543636
50%	6.000000	4.768408	9454.000000	44.194643
75%	53.000000	9.289434	12104.000000	72.205556
max	22311.000000	45.367788	21238.000000	216.139024

fl16.iloc[:,[6, 48, 76, 65]].describe()

	N_Items	great_circle_dist	NUMBER_OF_PATIENTS	Density
count	77099.000000	77099.000000	77099.000000	77099.000000
mean	270.637596	6.881413	9919.814628	59.580914
std	1162.624077	6.576676	4604.358012	43.440339
min	1.000000	0.000000	16.000000	0.833333
25%	2.000000	2.434314	6752.000000	33.543636
50%	6.000000	4.752313	9669.000000	43.840426
75%	56.000000	9.188920	12805.000000	72.205556
max	25077.000000	45.334545	21588.000000	216.139024

fl17.iloc[:,[6, 48, 75, 65]].describe()

	N_Items	great_circle_dist	NUMBER_OF_PATIENTS	Density
count	74668.000000	74668.000000	74668.000000	74668.000000
mean	280.026316	6.920288	10663.137542	59.422628
std	1181.619314	6.493172	5293.086633	43.452671
min	1.000000	0.000000	8.000000	0.833333
25%	2.000000	2.465416	7117.000000	33.077419
50%	7.000000	4.828210	10191.000000	43.243548
75%	62.000000	9.427593	13344.000000	72.205556
max	25161.000000	45.334545	32033.000000	216.139024

fl18.iloc[:,[6, 48, 75, 65]].describe()

	N_Items	great_circle_dist	NUMBER_OF_PATIENTS	Density
count	71734.000000	71734.000000	71734.000000	71734.000000
mean	289.512031	6.958728	11863.355564	59.193355
std	1195.221113	6.490907	6718.805457	43.402032
min	1.000000	0.000000	5.000000	0.833333
25%	2.000000	2.513705	7367.000000	32.875000
50%	7.000000	4.851604	10520.000000	43.243548
75%	58.000000	9.491935	15562.000000	72.205556
max	22604.000000	46.139331	36737.000000	216.139024

fl18['great_circle_dist'].value_counts(normalize = True)

0.000000     0.026055
2.125845     0.000934
3.936654     0.000822
23.579141    0.000767
1.719747     0.000711
               ...   
24.894594    0.000014
30.107116    0.000014
5.912814     0.000014
22.904229    0.000014
6.988423     0.000014
Name: great_circle_dist, Length: 11299, dtype: float64

The only variable we might have an issue is the distance, which has a lowest value zero. The zero belong to 2.6% o the data which is significant but should not be a big issue to change.

Also we can notice that number of patients registered at GP is very differnt for the 2017 than for the other years, why is it so?

Let's have a look into it

Examining the ‘Number of patients Registered at GP’

The reson for the difference in the distributions is complex.

In 2016, NHS started to joining surgeries into a groups(health group, health centres with more branches,…). In 2016, each branch was recorded as row and each branch had its own number of patients. For example, those are records for all the branches of Fireclay Health Group (fictional). Each surgery has recorded of registered patients that is unique for each surgery and is more or less the same as the number of patients in 2015. None of the data contain any record of if the surgeries belong to group and which group could it be. This information is not availiable in the data, or anywhere else to my knowledge.

ID	Name	Number of Patients
1	Churchroad surgery	2000
2	Barewell surgery	3000
3	Longwell surgery	4000
4	Bond street surgery	6000

In 2017 they records show the number of patients for each the branch as well as for the whole group, but this is actually recorded as one of the branches. This essentially means that the patients are doubled or multiplied in the data, in other words, some patients are counted twice in the data.

ID	Name	Number of Patients
1	Churchroad surgery	2000
2	Barewell surgery	3000
3	Longwell surgery	4000
4	Bond street surgery	15000

This continue to 2018 January, but by the July the NHS changed the way they shared published the numbers and corrceted themselves. They now only post the grouped number of patients for all the branches, so the patients should not appear multiple times in the data. This has one issue though, although the little branches exists, the record will show not the group name but the biggest branch with all the patients and the little branches (although still existing) are not present in the data.

ID	Name	Number of Patients



4	Bond street surgery	15000

Question is, do we need to do something with that??

Adding low value to all flows

for i in np.linspace(0.001, 1 , num = 10):
    plt.hist(x= (np.log(fl18['great_circle_dist'] + i)), color='g')
    plt.legend(labels=[i])
    plt.figure()
    plt.show();

Impact on the model

We need to find out what is going to be less impactfull

Zero substitution only

Add small amount to all variables

I will be handling many variables now, those datasets with 1. will have subscript _sub and those with 2 will have subscript _add

For the Power law models

what	how	value	2015	2016	2017	2018
Model Power	substituted	0.001	mps531	mps631	mps731	mps831
Model Power	substituted	0.01	mps521	mps621	mps721	mps821
Model Power	substituted	0.1	mps511	mps611	mps711	mps811
Model Power	substituted	1	mps501	mps601	mps701	mps801
Model Power	added	0.001	mpa531	mpa631	mpa731	mpa831
Model Power	added	0.01	mpa521	mpa621	mpa721	mpa821
Model Power	added	0.1	mpa511	mpa611	mpa711	mpa811
Model Power	added	1	mpa501	mpa601	mpa701	mpa801

For the Exponential law models

what	how	value	2015	2016	2017	2018
Model Exponential	substituted	0.001	mes531	mes631	mes731	mes831
Model Exponential	substituted	0.01	mes521	mes621	mes721	mes821
Model Exponential	substituted	0.1	mes511	mes611	mes711	mes811
Model Exponential	substituted	1	mes501	mes601	mes701	mes801
Model Exponential	added	0.001	mea531	mea631	mea731	mea831
Model Exponential	added	0.01	mea521	mea621	mea721	mea821
Model Exponential	added	0.1	mea511	mea611	mea711	mea811
Model Exponential	added	1	mea501	mea601	mea701	mea801

Build the data for the substitute

fl180 = fl18.iloc[:,[6,8, 48, 75, 65]]
fl170 = fl17.iloc[:,[6,8, 48, 75, 65]]
fl160 = fl16.iloc[:,[6,8, 48, 76, 65]]
fl150 = fl15.iloc[:,[6,8, 48, 76, 65]]

# 0.001
fl180['dist_adjusted_0001']= fl180['great_circle_dist'].mask(fl180['great_circle_dist'] == 0, 0.001)
fl170['dist_adjusted_0001'] = fl170['great_circle_dist'].mask(fl170['great_circle_dist'] == 0, 0.001)
fl160['dist_adjusted_0001'] = fl160['great_circle_dist'].mask(fl160['great_circle_dist'] == 0, 0.001)
fl150['dist_adjusted_0001'] = fl150['great_circle_dist'].mask(fl150['great_circle_dist'] == 0, 0.001)
# 0.01
fl180['dist_adjusted_001']= fl180['great_circle_dist'].mask(fl180['great_circle_dist'] == 0, 0.01)
fl170['dist_adjusted_001'] = fl170['great_circle_dist'].mask(fl170['great_circle_dist'] == 0, 0.01)
fl160['dist_adjusted_001'] = fl160['great_circle_dist'].mask(fl160['great_circle_dist'] == 0, 0.01)
fl150['dist_adjusted_001'] = fl150['great_circle_dist'].mask(fl150['great_circle_dist'] == 0, 0.01)
# 0.1
fl180['dist_adjusted_01']= fl180['great_circle_dist'].mask(fl180['great_circle_dist'] == 0, 0.1)
fl170['dist_adjusted_01'] = fl170['great_circle_dist'].mask(fl170['great_circle_dist'] == 0, 0.1)
fl160['dist_adjusted_01'] = fl160['great_circle_dist'].mask(fl160['great_circle_dist'] == 0, 0.1)
fl150['dist_adjusted_01'] = fl150['great_circle_dist'].mask(fl150['great_circle_dist'] == 0, 0.1)
# 1
fl180['dist_adjusted_1']= fl180['great_circle_dist'].mask(fl180['great_circle_dist'] == 0, 1)
fl170['dist_adjusted_1'] = fl170['great_circle_dist'].mask(fl170['great_circle_dist'] == 0, 1)
fl160['dist_adjusted_1'] = fl160['great_circle_dist'].mask(fl160['great_circle_dist'] == 0, 1)
fl150['dist_adjusted_1'] = fl150['great_circle_dist'].mask(fl150['great_circle_dist'] == 0, 1)

plt.figure(figsize = [15,10])
plt.suptitle("2018, Distance of flow, zero replaced with", fontsize=25)
plt.hist(fl180['great_circle_dist'], histtype='step', linewidth=6 ,  color = '#310F78', bins = 50)
plt.hist(fl180['dist_adjusted_0001'], histtype='step', linewidth=5 ,color = '#581E9B', bins = 50)
plt.hist(fl180['dist_adjusted_001'], histtype='step', linewidth=4,color = '#894ABD' , bins = 50)
plt.hist(fl180['dist_adjusted_01'], histtype='step',linewidth=3 ,color = '#B581D6', bins = 50)
plt.hist(fl180['dist_adjusted_1'], histtype='step',linewidth=2 ,color = 'pink', bins = 50)
plt.legend(labels=['original','0.001', '0.01', '0.1', '1']);


plt.figure(figsize = [15,10])
plt.suptitle("2017, Distance of flow, zero replaced with", fontsize=25)
plt.hist(fl170['great_circle_dist'], histtype='step', linewidth=6 ,  color = '#310F78', bins = 50)
plt.hist(fl170['dist_adjusted_0001'], histtype='step', linewidth=5 ,color = '#581E9B', bins = 50)
plt.hist(fl170['dist_adjusted_001'], histtype='step', linewidth=4,color = '#894ABD' , bins = 50)
plt.hist(fl170['dist_adjusted_01'], histtype='step',linewidth=3 ,color = '#B581D6', bins = 50)
plt.hist(fl170['dist_adjusted_1'], histtype='step',linewidth=2 ,color = 'pink', bins = 50)
plt.legend(labels=['original','0.001', '0.01', '0.1', '1']);

plt.figure(figsize = [15,10])
plt.suptitle("2015, Distance of flow, zero replaced with", fontsize=25)
plt.hist(fl150['great_circle_dist'], histtype='step', linewidth=6 ,  color = '#310F78', bins = 50)
plt.hist(fl150['dist_adjusted_0001'], histtype='step', linewidth=5 ,color = '#581E9B', bins = 50)
plt.hist(fl150['dist_adjusted_001'], histtype='step', linewidth=4,color = '#894ABD' , bins = 50)
plt.hist(fl150['dist_adjusted_01'], histtype='step',linewidth=3 ,color = '#B581D6', bins = 50)
plt.hist(fl150['dist_adjusted_1'], histtype='step',linewidth=2 ,color = 'pink', bins = 50)
plt.legend(labels=['original','0.001', '0.01', '0.1', '1']);

fl15_sub = fl150
fl16_sub = fl160
fl17_sub = fl170
fl18_sub = fl180

Build the data for the add up

# 0.001
fl15_add = fl15
fl15_add['great_circle_dist_0001'] = fl15_add['great_circle_dist'] + 0.001
fl15_add['NUMBER_OF_PATIENTS_0001'] = fl15_add['NUMBER_OF_PATIENTS'] + 0.001
fl15_add['Density_0001'] = fl15_add['Density'] + 0.001

fl16_add = fl16
fl16_add['great_circle_dist_0001'] = fl16_add['great_circle_dist'] + 0.001
fl16_add['NUMBER_OF_PATIENTS_0001'] = fl16_add['NUMBER_OF_PATIENTS'] + 0.001
fl16_add['Density_0001'] = fl16_add['Density'] + 0.001

fl17_add = fl17
fl17_add['great_circle_dist_0001'] = fl17_add['great_circle_dist'] + 0.001
fl17_add['NUMBER_OF_PATIENTS_0001'] = fl17_add['NUMBER_OF_PATIENTS'] + 0.001
fl17_add['Density_0001'] = fl17_add['Density'] + 0.001

fl18_add = fl18
fl18_add['great_circle_dist_0001'] = fl18_add['great_circle_dist'] + 0.001
fl18_add['NUMBER_OF_PATIENTS_0001'] = fl18_add['NUMBER_OF_PATIENTS'] + 0.001
fl18_add['Density_0001'] = fl18_add['Density'] + 0.001

# 0.01
fl15_add['great_circle_dist_001'] = fl15_add['great_circle_dist'] + 0.01
fl15_add['NUMBER_OF_PATIENTS_001'] = fl15_add['NUMBER_OF_PATIENTS'] + 0.01
fl15_add['Density_001'] = fl15_add['Density'] + 0.01

fl16_add['great_circle_dist_001'] = fl16_add['great_circle_dist'] + 0.01
fl16_add['NUMBER_OF_PATIENTS_001'] = fl16_add['NUMBER_OF_PATIENTS'] + 0.01
fl16_add['Density_001'] = fl16_add['Density'] + 0.01

fl17_add['great_circle_dist_001'] = fl17_add['great_circle_dist'] + 0.01
fl17_add['NUMBER_OF_PATIENTS_001'] = fl17_add['NUMBER_OF_PATIENTS'] + 0.01
fl17_add['Density_001'] = fl17_add['Density'] + 0.01

fl18_add['great_circle_dist_001'] = fl18_add['great_circle_dist'] + 0.01
fl18_add['NUMBER_OF_PATIENTS_001'] = fl18_add['NUMBER_OF_PATIENTS'] + 0.01
fl18_add['Density_001'] = fl18_add['Density'] + 0.01

# 0.1
fl15_add['great_circle_dist_01'] = fl15_add['great_circle_dist'] + 0.1
fl15_add['NUMBER_OF_PATIENTS_01'] = fl15_add['NUMBER_OF_PATIENTS'] + 0.1
fl15_add['Density_01'] = fl15_add['Density'] + 0.1

fl16_add['great_circle_dist_01'] = fl16_add['great_circle_dist'] + 0.1
fl16_add['NUMBER_OF_PATIENTS_01'] = fl16_add['NUMBER_OF_PATIENTS'] + 0.1
fl16_add['Density_01'] = fl16_add['Density'] + 0.1

fl17_add['great_circle_dist_01'] = fl17_add['great_circle_dist'] + 0.1
fl17_add['NUMBER_OF_PATIENTS_01'] = fl17_add['NUMBER_OF_PATIENTS'] + 0.1
fl17_add['Density_01'] = fl17_add['Density'] + 0.1

fl18_add['great_circle_dist_01'] = fl18_add['great_circle_dist'] + 0.1
fl18_add['NUMBER_OF_PATIENTS_01'] = fl18_add['NUMBER_OF_PATIENTS'] + 0.1
fl18_add['Density_01'] = fl18_add['Density'] + 0.1

# 1
fl15_add['great_circle_dist_1'] = fl15_add['great_circle_dist'] + 1
fl15_add['NUMBER_OF_PATIENTS_1'] = fl15_add['NUMBER_OF_PATIENTS'] + 1
fl15_add['Density_1'] = fl15_add['Density'] + 1

fl16_add['great_circle_dist_1'] = fl16_add['great_circle_dist'] + 1
fl16_add['NUMBER_OF_PATIENTS_1'] = fl16_add['NUMBER_OF_PATIENTS'] + 1
fl16_add['Density_1'] = fl16_add['Density'] + 1

fl17_add['great_circle_dist_1'] = fl17_add['great_circle_dist'] + 1
fl17_add['NUMBER_OF_PATIENTS_1'] = fl17_add['NUMBER_OF_PATIENTS'] + 1
fl17_add['Density_1'] = fl17_add['Density'] + 1

fl18_add['great_circle_dist_1'] = fl18_add['great_circle_dist'] + 1
fl18_add['NUMBER_OF_PATIENTS_1'] = fl18_add['NUMBER_OF_PATIENTS'] + 1
fl18_add['Density_1'] = fl18_add['Density'] + 1

plt.figure(figsize = [15,10])
plt.suptitle("2018, Distance of flow, zero replaced with", fontsize=25)
plt.hist(fl18_add['great_circle_dist'], histtype='step', linewidth=6 ,  color = '#310F78', bins = 50)
plt.hist(fl18_add['great_circle_dist_0001'], histtype='step', linewidth=5 ,color = '#581E9B', bins = 50)
plt.hist(fl18_add['great_circle_dist_001'], histtype='step', linewidth=4,color = '#894ABD' , bins = 50)
plt.hist(fl18_add['great_circle_dist_01'], histtype='step',linewidth=3 ,color = '#B581D6', bins = 50)
plt.hist(fl18_add['great_circle_dist_1'], histtype='step',linewidth=2 ,color = 'pink', bins = 50)
plt.legend(labels=['original','0.001', '0.01', '0.1', '1']);

Build the models

# power modes
# 2015
mps531 = Gravity(flows = fl15_sub['N_Items'].astype(int).values, o_vars = fl15_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl15_sub['Density'].values, cost = fl15_sub['dist_adjusted_0001'].values, cost_func = 'pow',   constant=True)
mps521 = Gravity(flows = fl15_sub['N_Items'].astype(int).values, o_vars = fl15_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl15_sub['Density'].values, cost = fl15_sub['dist_adjusted_001'].values, cost_func = 'pow',   constant=True)
mps511 = Gravity(flows = fl15_sub['N_Items'].astype(int).values, o_vars = fl15_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl15_sub['Density'].values, cost = fl15_sub['dist_adjusted_01'].values, cost_func = 'pow',   constant=True)
mps501 = Gravity(flows = fl15_sub['N_Items'].astype(int).values, o_vars = fl15_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl15_sub['Density'].values, cost = fl15_sub['dist_adjusted_1'].values, cost_func = 'pow',   constant=True)
mpa531 = Gravity(flows = fl15_add['N_Items'].astype(int).values, o_vars = fl15_add['NUMBER_OF_PATIENTS_0001'].values, d_vars = fl15_add['Density_0001'].values, cost = fl15_add['great_circle_dist_0001'].values, cost_func = 'pow',   constant=True)
mpa521 = Gravity(flows = fl15_add['N_Items'].astype(int).values, o_vars = fl15_add['NUMBER_OF_PATIENTS_001'].values, d_vars = fl15_add['Density_001'].values, cost = fl15_add['great_circle_dist_001'].values, cost_func = 'pow',   constant=True)
mpa511 = Gravity(flows = fl15_add['N_Items'].astype(int).values, o_vars = fl15_add['NUMBER_OF_PATIENTS_01'].values, d_vars = fl15_add['Density_01'].values, cost = fl15_add['great_circle_dist_01'].values, cost_func = 'pow',   constant=True)
mpa501 = Gravity(flows = fl15_add['N_Items'].astype(int).values, o_vars = fl15_add['NUMBER_OF_PATIENTS_1'].values, d_vars = fl15_add['Density_1'].values, cost = fl15_add['great_circle_dist_1'].values, cost_func = 'pow',   constant=True)
# 2016
mps631 = Gravity(flows = fl16_sub['N_Items'].astype(int).values, o_vars = fl16_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl16_sub['Density'].values, cost = fl16_sub['dist_adjusted_0001'].values, cost_func = 'pow',   constant=True)
mps621 = Gravity(flows = fl16_sub['N_Items'].astype(int).values, o_vars = fl16_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl16_sub['Density'].values, cost = fl16_sub['dist_adjusted_001'].values, cost_func = 'pow',   constant=True)
mps611 = Gravity(flows = fl16_sub['N_Items'].astype(int).values, o_vars = fl16_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl16_sub['Density'].values, cost = fl16_sub['dist_adjusted_01'].values, cost_func = 'pow',   constant=True)
mps601 = Gravity(flows = fl16_sub['N_Items'].astype(int).values, o_vars = fl16_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl16_sub['Density'].values, cost = fl16_sub['dist_adjusted_1'].values, cost_func = 'pow',   constant=True)
mpa631 = Gravity(flows = fl16_add['N_Items'].astype(int).values, o_vars = fl16_add['NUMBER_OF_PATIENTS_0001'].values, d_vars = fl16_add['Density_0001'].values, cost = fl16_add['great_circle_dist_0001'].values, cost_func = 'pow',   constant=True)
mpa621 = Gravity(flows = fl16_add['N_Items'].astype(int).values, o_vars = fl16_add['NUMBER_OF_PATIENTS_001'].values, d_vars = fl16_add['Density_001'].values, cost = fl16_add['great_circle_dist_001'].values, cost_func = 'pow',   constant=True)
mpa611 = Gravity(flows = fl16_add['N_Items'].astype(int).values, o_vars = fl16_add['NUMBER_OF_PATIENTS_01'].values, d_vars = fl16_add['Density_01'].values, cost = fl16_add['great_circle_dist_01'].values, cost_func = 'pow',   constant=True)
mpa601 = Gravity(flows = fl16_add['N_Items'].astype(int).values, o_vars = fl16_add['NUMBER_OF_PATIENTS_1'].values, d_vars = fl16_add['Density_1'].values, cost = fl16_add['great_circle_dist_1'].values, cost_func = 'pow',   constant=True)
# 2017
mps731 = Gravity(flows = fl17_sub['N_Items'].astype(int).values, o_vars = fl17_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl17_sub['Density'].values, cost = fl17_sub['dist_adjusted_0001'].values, cost_func = 'pow',   constant=True)
mps721 = Gravity(flows = fl17_sub['N_Items'].astype(int).values, o_vars = fl17_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl17_sub['Density'].values, cost = fl17_sub['dist_adjusted_001'].values, cost_func = 'pow',   constant=True)
mps711 = Gravity(flows = fl17_sub['N_Items'].astype(int).values, o_vars = fl17_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl17_sub['Density'].values, cost = fl17_sub['dist_adjusted_01'].values, cost_func = 'pow',   constant=True)
mps701 = Gravity(flows = fl17_sub['N_Items'].astype(int).values, o_vars = fl17_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl17_sub['Density'].values, cost = fl17_sub['dist_adjusted_1'].values, cost_func = 'pow',   constant=True)
mpa731 = Gravity(flows = fl17_add['N_Items'].astype(int).values, o_vars = fl17_add['NUMBER_OF_PATIENTS_0001'].values, d_vars = fl17_add['Density_0001'].values, cost = fl17_add['great_circle_dist_0001'].values, cost_func = 'pow',   constant=True)
mpa721 = Gravity(flows = fl17_add['N_Items'].astype(int).values, o_vars = fl17_add['NUMBER_OF_PATIENTS_001'].values, d_vars = fl17_add['Density_001'].values, cost = fl17_add['great_circle_dist_001'].values, cost_func = 'pow',   constant=True)
mpa711 = Gravity(flows = fl17_add['N_Items'].astype(int).values, o_vars = fl17_add['NUMBER_OF_PATIENTS_01'].values, d_vars = fl17_add['Density_01'].values, cost = fl17_add['great_circle_dist_01'].values, cost_func = 'pow',   constant=True)
mpa701 = Gravity(flows = fl17_add['N_Items'].astype(int).values, o_vars = fl17_add['NUMBER_OF_PATIENTS_1'].values, d_vars = fl17_add['Density_1'].values, cost = fl17_add['great_circle_dist_1'].values, cost_func = 'pow',   constant=True)
# 2018
mps831 = Gravity(flows = fl18_sub['N_Items'].astype(int).values, o_vars = fl18_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl18_sub['Density'].values, cost = fl18_sub['dist_adjusted_0001'].values, cost_func = 'pow',   constant=True)
mps821 = Gravity(flows = fl18_sub['N_Items'].astype(int).values, o_vars = fl18_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl18_sub['Density'].values, cost = fl18_sub['dist_adjusted_001'].values, cost_func = 'pow',   constant=True)
mps811 = Gravity(flows = fl18_sub['N_Items'].astype(int).values, o_vars = fl18_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl18_sub['Density'].values, cost = fl18_sub['dist_adjusted_01'].values, cost_func = 'pow',   constant=True)
mps801 = Gravity(flows = fl18_sub['N_Items'].astype(int).values, o_vars = fl18_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl18_sub['Density'].values, cost = fl18_sub['dist_adjusted_1'].values, cost_func = 'pow',   constant=True)
mpa831 = Gravity(flows = fl18_add['N_Items'].astype(int).values, o_vars = fl18_add['NUMBER_OF_PATIENTS_0001'].values, d_vars = fl18_add['Density_0001'].values, cost = fl18_add['great_circle_dist_0001'].values, cost_func = 'pow',   constant=True)
mpa821 = Gravity(flows = fl18_add['N_Items'].astype(int).values, o_vars = fl18_add['NUMBER_OF_PATIENTS_001'].values, d_vars = fl18_add['Density_001'].values, cost = fl18_add['great_circle_dist_001'].values, cost_func = 'pow',   constant=True)
mpa811 = Gravity(flows = fl18_add['N_Items'].astype(int).values, o_vars = fl18_add['NUMBER_OF_PATIENTS_01'].values, d_vars = fl18_add['Density_01'].values, cost = fl18_add['great_circle_dist_01'].values, cost_func = 'pow',   constant=True)
mpa801 = Gravity(flows = fl18_add['N_Items'].astype(int).values, o_vars = fl18_add['NUMBER_OF_PATIENTS_1'].values, d_vars = fl18_add['Density_1'].values, cost = fl18_add['great_circle_dist_1'].values, cost_func = 'pow',   constant=True)

# exponential models
# base, no change
base5 = Gravity(flows = fl15_sub['N_Items'].astype(int).values, o_vars = fl15_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl15_sub['Density'].values, cost = fl15_sub['great_circle_dist'].values, cost_func = 'exp',   constant=True)
base6 = Gravity(flows = fl16_sub['N_Items'].astype(int).values, o_vars = fl16_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl16_sub['Density'].values, cost = fl16_sub['great_circle_dist'].values, cost_func = 'exp',   constant=True)
base7 = Gravity(flows = fl17_sub['N_Items'].astype(int).values, o_vars = fl17_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl17_sub['Density'].values, cost = fl17_sub['great_circle_dist'].values, cost_func = 'exp',   constant=True)
base8 = Gravity(flows = fl18_sub['N_Items'].astype(int).values, o_vars = fl18_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl18_sub['Density'].values, cost = fl18_sub['great_circle_dist'].values, cost_func = 'exp',   constant=True)
# 2015
mes531 = Gravity(flows = fl15_sub['N_Items'].astype(int).values, o_vars = fl15_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl15_sub['Density'].values, cost = fl15_sub['dist_adjusted_0001'].values, cost_func = 'exp',   constant=True)
mes521 = Gravity(flows = fl15_sub['N_Items'].astype(int).values, o_vars = fl15_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl15_sub['Density'].values, cost = fl15_sub['dist_adjusted_001'].values, cost_func = 'exp',   constant=True)
mes511 = Gravity(flows = fl15_sub['N_Items'].astype(int).values, o_vars = fl15_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl15_sub['Density'].values, cost = fl15_sub['dist_adjusted_01'].values, cost_func = 'exp',   constant=True)
mes501 = Gravity(flows = fl15_sub['N_Items'].astype(int).values, o_vars = fl15_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl15_sub['Density'].values, cost = fl15_sub['dist_adjusted_1'].values, cost_func = 'exp',   constant=True)
mea531 = Gravity(flows = fl15_add['N_Items'].astype(int).values, o_vars = fl15_add['NUMBER_OF_PATIENTS_0001'].values, d_vars = fl15_add['Density_0001'].values, cost = fl15_add['great_circle_dist_0001'].values, cost_func = 'exp',   constant=True)
mea521 = Gravity(flows = fl15_add['N_Items'].astype(int).values, o_vars = fl15_add['NUMBER_OF_PATIENTS_001'].values, d_vars = fl15_add['Density_001'].values, cost = fl15_add['great_circle_dist_001'].values, cost_func = 'exp',   constant=True)
mea511 = Gravity(flows = fl15_add['N_Items'].astype(int).values, o_vars = fl15_add['NUMBER_OF_PATIENTS_01'].values, d_vars = fl15_add['Density_01'].values, cost = fl15_add['great_circle_dist_01'].values, cost_func = 'exp',   constant=True)
mea501 = Gravity(flows = fl15_add['N_Items'].astype(int).values, o_vars = fl15_add['NUMBER_OF_PATIENTS_1'].values, d_vars = fl15_add['Density_1'].values, cost = fl15_add['great_circle_dist_1'].values, cost_func = 'exp',   constant=True)
# 2016
mes631 = Gravity(flows = fl16_sub['N_Items'].astype(int).values, o_vars = fl16_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl16_sub['Density'].values, cost = fl16_sub['dist_adjusted_0001'].values, cost_func = 'exp',   constant=True)
mes621 = Gravity(flows = fl16_sub['N_Items'].astype(int).values, o_vars = fl16_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl16_sub['Density'].values, cost = fl16_sub['dist_adjusted_001'].values, cost_func = 'exp',   constant=True)
mes611 = Gravity(flows = fl16_sub['N_Items'].astype(int).values, o_vars = fl16_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl16_sub['Density'].values, cost = fl16_sub['dist_adjusted_01'].values, cost_func = 'exp',   constant=True)
mes601 = Gravity(flows = fl16_sub['N_Items'].astype(int).values, o_vars = fl16_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl16_sub['Density'].values, cost = fl16_sub['dist_adjusted_1'].values, cost_func = 'exp',   constant=True)
mea631 = Gravity(flows = fl16_add['N_Items'].astype(int).values, o_vars = fl16_add['NUMBER_OF_PATIENTS_0001'].values, d_vars = fl16_add['Density_0001'].values, cost = fl16_add['great_circle_dist_0001'].values, cost_func = 'exp',   constant=True)
mea621 = Gravity(flows = fl16_add['N_Items'].astype(int).values, o_vars = fl16_add['NUMBER_OF_PATIENTS_001'].values, d_vars = fl16_add['Density_001'].values, cost = fl16_add['great_circle_dist_001'].values, cost_func = 'exp',   constant=True)
mea611 = Gravity(flows = fl16_add['N_Items'].astype(int).values, o_vars = fl16_add['NUMBER_OF_PATIENTS_01'].values, d_vars = fl16_add['Density_01'].values, cost = fl16_add['great_circle_dist_01'].values, cost_func = 'exp',   constant=True)
mea601 = Gravity(flows = fl16_add['N_Items'].astype(int).values, o_vars = fl16_add['NUMBER_OF_PATIENTS_1'].values, d_vars = fl16_add['Density_1'].values, cost = fl16_add['great_circle_dist_1'].values, cost_func = 'exp',   constant=True)
# 2017
mes731 = Gravity(flows = fl17_sub['N_Items'].astype(int).values, o_vars = fl17_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl17_sub['Density'].values, cost = fl17_sub['dist_adjusted_0001'].values, cost_func = 'exp',   constant=True)
mes721 = Gravity(flows = fl17_sub['N_Items'].astype(int).values, o_vars = fl17_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl17_sub['Density'].values, cost = fl17_sub['dist_adjusted_001'].values, cost_func = 'exp',   constant=True)
mes711 = Gravity(flows = fl17_sub['N_Items'].astype(int).values, o_vars = fl17_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl17_sub['Density'].values, cost = fl17_sub['dist_adjusted_01'].values, cost_func = 'exp',   constant=True)
mes701 = Gravity(flows = fl17_sub['N_Items'].astype(int).values, o_vars = fl17_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl17_sub['Density'].values, cost = fl17_sub['dist_adjusted_1'].values, cost_func = 'exp',   constant=True)
mea731 = Gravity(flows = fl17_add['N_Items'].astype(int).values, o_vars = fl17_add['NUMBER_OF_PATIENTS_0001'].values, d_vars = fl17_add['Density_0001'].values, cost = fl17_add['great_circle_dist_0001'].values, cost_func = 'exp',   constant=True)
mea721 = Gravity(flows = fl17_add['N_Items'].astype(int).values, o_vars = fl17_add['NUMBER_OF_PATIENTS_001'].values, d_vars = fl17_add['Density_001'].values, cost = fl17_add['great_circle_dist_001'].values, cost_func = 'exp',   constant=True)
mea711 = Gravity(flows = fl17_add['N_Items'].astype(int).values, o_vars = fl17_add['NUMBER_OF_PATIENTS_01'].values, d_vars = fl17_add['Density_01'].values, cost = fl17_add['great_circle_dist_01'].values, cost_func = 'exp',   constant=True)
mea701 = Gravity(flows = fl17_add['N_Items'].astype(int).values, o_vars = fl17_add['NUMBER_OF_PATIENTS_1'].values, d_vars = fl17_add['Density_1'].values, cost = fl17_add['great_circle_dist_1'].values, cost_func = 'exp',   constant=True)
# 2018
mes831 = Gravity(flows = fl18_sub['N_Items'].astype(int).values, o_vars = fl18_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl18_sub['Density'].values, cost = fl18_sub['dist_adjusted_0001'].values, cost_func = 'exp',   constant=True)
mes821 = Gravity(flows = fl18_sub['N_Items'].astype(int).values, o_vars = fl18_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl18_sub['Density'].values, cost = fl18_sub['dist_adjusted_001'].values, cost_func = 'exp',   constant=True)
mes811 = Gravity(flows = fl18_sub['N_Items'].astype(int).values, o_vars = fl18_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl18_sub['Density'].values, cost = fl18_sub['dist_adjusted_01'].values, cost_func = 'exp',   constant=True)
mes801 = Gravity(flows = fl18_sub['N_Items'].astype(int).values, o_vars = fl18_sub['NUMBER_OF_PATIENTS'].values, d_vars = fl18_sub['Density'].values, cost = fl18_sub['dist_adjusted_1'].values, cost_func = 'exp',   constant=True)
mea831 = Gravity(flows = fl18_add['N_Items'].astype(int).values, o_vars = fl18_add['NUMBER_OF_PATIENTS_0001'].values, d_vars = fl18_add['Density_0001'].values, cost = fl18_add['great_circle_dist_0001'].values, cost_func = 'exp',   constant=True)
mea821 = Gravity(flows = fl18_add['N_Items'].astype(int).values, o_vars = fl18_add['NUMBER_OF_PATIENTS_001'].values, d_vars = fl18_add['Density_001'].values, cost = fl18_add['great_circle_dist_001'].values, cost_func = 'exp',   constant=True)
mea811 = Gravity(flows = fl18_add['N_Items'].astype(int).values, o_vars = fl18_add['NUMBER_OF_PATIENTS_01'].values, d_vars = fl18_add['Density_01'].values, cost = fl18_add['great_circle_dist_01'].values, cost_func = 'exp',   constant=True)
mea801 = Gravity(flows = fl18_add['N_Items'].astype(int).values, o_vars = fl18_add['NUMBER_OF_PATIENTS_1'].values, d_vars = fl18_add['Density_1'].values, cost = fl18_add['great_circle_dist_1'].values, cost_func = 'exp',   constant=True)

R215, R216, R217, R218 = [], [], [], []
model_nameP = ['gravity_power_sub_0.001', 'gravity_power_sub_0.01',  'gravity_power_sub_0.1','gravity_power_sub_1', 'gravity_power_add_0.001', 'gravity_power_add_0.01', 'gravity_power_add_0.1', 'gravity_power_add_1']
col_namesP = ['R2_2015', 'R2_2016', 'R2_2017', 'R2_2018']
models15 = [mps531, mps521, mps511, mps501, mpa531,mpa521,mpa511,mpa501]
models16 = [mps631, mps621, mps611, mps601, mpa631,mpa621,mpa611,mpa601]
models17 = [mps731, mps721, mps711, mps701, mpa731,mpa721,mpa711,mpa701]
models18 = [mps831, mps821, mps811, mps801, mpa831,mpa821,mpa811,mpa801]

for model in models15:
    R215.append(model.pseudoR2)
for model in models16:
    R216.append(model.pseudoR2)
for model in models17:
    R217.append(model.pseudoR2)
for model in models18:
    R218.append(model.pseudoR2)

colsP = {'model_name': model_nameP,'R2_2015': R215, 'R2_2016': R216,'R2_2017': R217,'R2_2018': R218}

dataP = pd.DataFrame(colsP).set_index('model_name')

R215, R216, R217, R218 = [], [], [], []
model_nameE = ['base_exponential','gravity_exponent_sub_0.001', 'gravity_exponent_sub_0.01',  'gravity_exponent_sub_0.1','gravity_exponent_sub_1', 'gravity_exponent_add_0.001', 'gravity_exponent_add_0.01', 'gravity_exponent_add_0.1', 'gravity_exponent_add_1']
col_namesE = ['R2_2015', 'R2_2016', 'R2_2017', 'R2_2018']
models15 = [base5, mes531, mes521, mes511, mes501, mea531,mea521,mea511,mea501]
models16 = [base6, mes631, mes621, mes611, mes601, mea631,mea621,mea611,mea601]
models17 = [base7, mes731, mes721, mes711, mes701, mea731,mea721,mea711,mea701]
models18 = [base8, mes831, mes821, mes811, mes801, mea831,mea821,mea811,mea801]

for model in models15:
    R215.append(model.pseudoR2)
for model in models16:
    R216.append(model.pseudoR2)
for model in models17:
    R217.append(model.pseudoR2)
for model in models18:
    R218.append(model.pseudoR2)

colsE = {'model_name': model_nameE,'R2_2015': R215, 'R2_2016': R216,'R2_2017': R217,'R2_2018': R218}

dataE = pd.DataFrame(colsE).set_index('model_name')
dataE[col_namesE]

	R2_2015	R2_2016	R2_2017	R2_2018
model_name
base_exponential	0.577145	0.572300	0.559345	0.557001
gravity_exponent_sub_0.001	0.577135	0.572288	0.559332	0.556991
gravity_exponent_sub_0.01	0.577047	0.572181	0.559213	0.556892
gravity_exponent_sub_0.1	0.575993	0.570945	0.557879	0.555773
gravity_exponent_sub_1	0.546718	0.540835	0.529192	0.529886
gravity_exponent_add_0.001	0.577149	0.572303	0.559348	0.557006
gravity_exponent_add_0.01	0.577181	0.572333	0.559381	0.557043
gravity_exponent_add_0.1	0.577483	0.572611	0.559688	0.557389
gravity_exponent_add_1	0.579162	0.574184	0.561462	0.559391

dataP[col_namesP]

	R2_2015	R2_2016	R2_2017	R2_2018
model_name
gravity_power_sub_0.001	0.439593	0.437777	0.433232	0.419891
gravity_power_sub_0.01	0.501261	0.497098	0.492077	0.478877
gravity_power_sub_0.1	0.566077	0.556465	0.550752	0.558417
gravity_power_sub_1	0.456260	0.443518	0.441651	0.466390
gravity_power_add_0.001	0.439037	0.437308	0.432843	0.419657
gravity_power_add_0.01	0.498087	0.494550	0.489984	0.476431
gravity_power_add_0.1	0.578871	0.572455	0.567675	0.555453
gravity_power_add_1	0.632234	0.623348	0.617584	0.611254

Conlusion

dataE.iloc[0:5,].plot()
plt.suptitle("Performances of exponential law gravity models with zero substituted only", fontsize=18);
dataP.iloc[0:4,].plot()
plt.suptitle("Performances of power law gravity models with zero substituted only", fontsize=18);

dataE.iloc[5:9,].plot()
plt.suptitle("Performances of exponential law gravity models with added values to independent vars", fontsize=18);
dataP.iloc[4:8,].plot()
plt.suptitle("Performances of power law gravity models with added values to independent vars", fontsize=18);

Firstly, we know that the higher value we add up or substitute, the more drastic the change is and the less trust we have into the model accuracy. It is interesting though That using power law models, adding 1 to all variables actually results in the best model of all, over 60%.

Secondly, In all case, power law models show bigger accuracy differences every step we use higher value. On the other hand exponential law models seems to be more stable for the values belw the one, and only show visible difference when we use value of 1.

Thirdly, the power law models perform similarly for all years, but the exponential law models perform much better in teh 2015 and 2016 then in 217 and 2018. However, the difference is less truth when we only substitute the zero and more drastic when we are adding up values.

Why is it so?

Could it be the Number of patient's variable that is dodgy for the last two years?
Something else?

Which model to choose

From this, I am Inclining to exponential law model, as they seem to be more stable across little changes. It also can be more advantegous im the future work;

The advantages of the exponential gravity model are as below: first, it is independent of dimension; second, its underlying rationale is clear because it is derivable from the principle of entropy maximization (Chen, 2015)

Then, the base exponential model is actually working without any changes and results in same accuracy as the little changes. So Is there any reason for not to use the base exponential model?

cost function for the power law model in SpInt

$$np.log$$

cost function for the exponential law model in SpInt

$$\lambda x: x * 1.0$$

## If best model is the base exponential (base8,7,6,5) the we can test what the same model does with th ecomplete data

file = './../../data/NHS/working/year_data_flow/complete_flow_2017.csv'
comp_fl17 = pd.read_csv(file, index_col=None)

plt.figure(figsize = [15,10])
plt.suptitle("Number of prescriptions on a flow in 2017", fontsize=25)
plt.hist(comp_fl17['N_Items'], histtype='step', linewidth=6, color = '#310F78')
plt.hist(fl17['N_Items'], histtype='step', linewidth=4, color = '#B581D6')
plt.legend(labels=['complete data with 0', 'incomplete as published']);

comp_fl17['N_Items'].value_counts()

0.0       469076
1.0        15262
2.0         8029
3.0         5120
4.0         3601
           ...  
3911.0         1
6502.0         1
3905.0         1
3901.0         1
2216.0         1
Name: N_Items, Length: 3921, dtype: int64

print( str(round((469076/len(comp_fl17))*100,2)) + '% of the complete data is 0')

86.27% of the complete data is 0

comp_fl17['dist_adjusted_0001'] = comp_fl17['great_circle_dist'].mask(comp_fl17['great_circle_dist'] == 0, 0.001)

comp_base7 = Gravity(flows = comp_fl17['N_Items'].astype(int).values, o_vars = comp_fl17['NUMBER_OF_PATIENTS'].values, d_vars = comp_fl17['Density'].values, cost = comp_fl17['great_circle_dist'].values, cost_func = 'exp',   constant=True)
comp_base7_eucd = Gravity(flows = comp_fl17['N_Items'].astype(int).values, o_vars = comp_fl17['NUMBER_OF_PATIENTS'].values, d_vars = comp_fl17['Density'].values, cost = comp_fl17['Euc_distance'].values, cost_func = 'exp',   constant=True)
comp_base7_pow = Gravity(flows = comp_fl17['N_Items'].astype(int).values, o_vars = comp_fl17['NUMBER_OF_PATIENTS'].values, d_vars = comp_fl17['Density'].values, cost = comp_fl17['dist_adjusted_0001'].values, cost_func = 'pow',   constant=True)

print('complete data, great circle distance, pseudo R2 ' + str(round(comp_base7.pseudoR2,2)))
print('complete data, euclidian distance, pseudo R2 ' + str(round(comp_base7_eucd.pseudoR2,2)))
print('complete data, power law, adjusted 0.001, pseudo R2 ' + str(round(comp_base7_pow.pseudoR2,2)))

complete data, great circle distance, pseudo R2 0.67
complete data, euclidian distance, pseudo R2 0.65
complete data, power law, adjusted 0.001, pseudo R2 0.48

print('published data, great circle distance, pseudo R2 ' + str(round(base7.pseudoR2,2)))

published data, great circle distance, pseudo R2 0.56

from scipy import sparse as sp
import numpy as np
from collections import defaultdict
from functools import partial
from itertools import count

def CPC(model):
    """
    Common part of commuters based on Sorensen index
    Lenormand et al. 2012
    """
    y = model.y
    try:
        yhat = model.yhat.resahpe((-1, 1))
    except BaseException:
        yhat = model.mu((-1, 1))
    N = model.n
    YYhat = np.hstack([y, yhat])
    NCC = np.sum(np.min(YYhat, axis=1))
    NCY = np.sum(Y)
    NCYhat = np.sum(yhat)
    return (N * NCC) / (NCY + NCYhat)


def sorensen(model):
    """
    Sorensen similarity index
    For use on spatial interaction models; N = sample size
    rather than N = number of locations and normalized by N instead of N**2
    """
    try:
        y = model.y.reshape((-1, 1))
    except BaseException:
        y = model.f.reshape((-1, 1))
    try:
        yhat = model.yhat.reshape((-1, 1))
    except BaseException:
        yhat = model.mu.reshape((-1, 1))
    N = model.n
    YYhat = np.hstack([y, yhat])
    num = 2.0 * np.min(YYhat, axis=1)
    den = yhat + y
    return (1.0 / N) * (np.sum(num.reshape((-1, 1)) / den.reshape((-1, 1))))

CPC(base7)

# but the SpInt has already sorensen index in it
base7.SSI

0.3182449860984687

comp_base7.SSI

0.04548680791330862

base7.n

Comparing the graph structures of the patients flow

Thu, 26 Mar 2020 00:00:00 +0000

# we will first import all necessary libraries
import pandas as pd
import os
import numpy as np
import glob
from datetime import datetime
import geopandas as gpd
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px
from shapely.geometry import Point, LineString
#from mpl_toolkits.basemap import Basemap
import osmnx as ox
import networkx as nx
import matplotlib as mpl
from networkx.algorithms import bipartite as bi
import statsmodels.api as sm
from scipy import stats
from statsmodels.formula.api import glm

wgs84 = {'init':'EPSG:4326'}
bng = {'init':'EPSG:27700'}

Prescription flow model validation

Using graphs to detect changes in spatial structure of the prescription flow

Using a graphs to look at the properties of the flow network, we need to first think about what kind of graph the network actually is and what kind of connections it has.

Based on the number of edges between two points we recognize

Simple graph - Undirected graph with no loops and one edge between two points
Multiple graph - Undirected graph with no loops
Pseudograph - Graph with loops

Based on the type of the link we recognise;

Undirected graph - the flow does not have a direction
Directed graph - the flow have a direction, flow from point A to point B

We also recognize;

Incomplete graph - graph in which at least one set of nodes is missing connection
Complete graph - graph in which all the nodes are connected to all the other nodes

Lastly, we can also recognize very special case of a graph called Bipartite. What is Bipartiv graph? Recently, “bipartivity” has been proposed as an important topological characteristic of complex networks. A network (graph) G = (V,E) is called bipartite if its vertex set V can be partitioned into two subsets V1 and V2 such that all edges have one endpoint in V1 and the other in V2.

The GP and pharmacy flow are in nature Directed Incomplete pseudograph. that is because the flow goes from GP to pharmacy, always one direction, it does not contain the flows between all the pairs and also contains some loops; Gp can also dispense small amount odf prescriptions. Nevertheles, as the loops are very small proportoin of the data and we can benefit from the graph as a bipartite, I have attached dummy GPs into a pharmacy points so we can treat the loops as normal flows.

NOTE: what is the implication of this? We are loosing the information about connection of dome flows to origin, however, we are gaining an information on the process which reminds zero-inflated or hurdle process;

Step 1 - patient come to GP but leaves without a prescriprion
Step 2 - patient come to GP get the prescription virtually and recieve the drug from GP directly
Step 3 - patient come to GP, get prescription, go to nearest pharmacy to get the drug
Step 4 - patient come to GP, get the prescription and go to pharmacy close to shopping opportunity to get the drug
Step 5 - patient come to GP, get the prescription and go to other pharmacy to get the drug

How do we represent the graph in Networkx?

Here are types of graphs in the networkx package

Networkx Class	Type	Self-loops allowed	Parallel edges allowed
Graph	undirected	Yes	No
DiGraph	directed	Yes	No
MultiGraph	undirected	Yes	Yes
MultiDiGraph	directed	Yes	Yes

We know our graph is directed so Digraph should serve the purpose. However, the data consist from monthly prescription flow so for most of the nodes we can find 12 edges throughout the year, so we need to work with MultiDiGraph to make sure we can account for all of those.

1. Load the data

# load the output of the gravity models, the flow data
# set the path
file = r'./../../data/NHS/working/predicted_flow_17.csv'
# read the file
flow17 = pd.read_csv(file, index_col=None)
# plot the top of the table
flow17.head()

	P_Code	P_Name_x	P_Postcode	D_Code_x	D_Name_x	D_Postcode	N_Items	Date	loop	...	D_Left_y	D_X_y	D_Y_y	D_E_y	D_N_y	D_Setting_y	residents	Density	predict_items	predict_items_doubly
0	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FA251	HORIZON PHARMACY LTD	BS34 6AS	130.0	2017-01-01	False	...	20180115.0	-2.562191	51.539195	361105.872423	182404.906640	50	456.517857	36.592857	29.316590	10.581379
1	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FA613	DOWNHAM FB LTD	BS7 9JT	39.0	2017-01-01	False	...	NaN	-2.582219	51.480445	359664.895464	175881.865127	50	249.144737	54.493421	29.473430	2.853510
2	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FCL91	MYGLOBE LTD	BS15 4ND	12.0	2017-01-01	False	...	NaN	-2.478870	51.459944	366826.940029	173549.785593	50	249.128205	30.569231	24.155556	2.061345
3	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FCT35	LLOYDS PHARMACY LTD	BS20 7QA	661.0	2017-01-01	False	...	20180131.0	-2.758272	51.484474	347443.775309	176441.826480	50	369.081633	34.918367	1648.893267	1078.144522
4	L81004	PORTISHEAD MEDICAL GROUP	BS20 6AQ	FDD77	JOHN WARE LIMITED	BS20 6LT	4058.0	2017-01-01	False	...	NaN	-2.781349	51.481409	345837.744901	176117.839266	50	391.076923	29.092308	1194.123265	3570.014826

5 rows × 72 columns

# create a lists of the unique codes so we can subset the GP and PH point data
list_flow_GP = list(flow17['P_Code'].unique())
list_flow_PH = list(flow17['D_Code_new'].unique())

# get the pharmacies in
fl = './../../data/NHS/edispensary/PH_joined_daytimepop.geojson'
PH = gpd.read_file(fl)
PH = PH.to_crs(wgs84)
PH = PH[PH['D_Code_new'].isin(list_flow_PH)]
PH.plot();

C:\Users\tk18583\AppData\Local\Continuum\anaconda3\envs\geonet\lib\site-packages\pyproj\crs\crs.py:53: FutureWarning:

'+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6

# get the GP's points in
GP = gpd.read_file("./../../data/NHS/epraccur/GP_AVON.geojson")
GP = GP[['P_Code','P_Name', 'full_address', 'P_Postcode', 'Open_date', 
                                         'Close_date', 'Status_Code', 'Sub_type', 'Join', 'left', 'Setting', 'X', 'Y', 'geometry']]
GP = GP.rename(columns={'P_Postcode':'P_Postcode2','Open_date': 'P_Open_date', 'Close_date':'P_Close_date','full_address':'P_full_address' ,'Status_Code':'P_Status_Code', 'Sub_type':'P_Sub_type', 'Join':'P_Join', 'left':'P_Left', 'Setting':'P_Setting', 'X':'P_X', 'Y':'P_Y'}, errors="raise")
GP = GP.to_crs(wgs84)
GP = GP[GP['P_Code'].isin(list_flow_GP)]
GP.plot();

C:\Users\tk18583\AppData\Local\Continuum\anaconda3\envs\geonet\lib\site-packages\pyproj\crs\crs.py:53: FutureWarning:

'+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6

2. Create a graph

We now know that the graph is bipartite, has multiple edges and has a directions, which means that we need to create MultiDiGraph. We also have weight on edge, that is the number of prescriptions on flow and predicted items on flow for both of the used models.

G1 = nx.from_pandas_edgelist(flow17, source='P_Code', target='D_Code_new', edge_attr=['N_Items'], create_using=nx.MultiDiGraph())
print('Is the graph bipirtite?   ' + str(nx.is_bipartite(G1)))
print('Is the graph directed??   ' + str(nx.is_directed(G1)))

Is the graph bipirtite?   True
Is the graph directed??   True

G2 = nx.from_pandas_edgelist(flow17, source='P_Code', target='D_Code_new', edge_attr=['predict_items'], create_using=nx.MultiDiGraph())
print('Is the graph bipirtite?   ' + str(nx.is_bipartite(G2)))
print('Is the graph directed??   ' + str(nx.is_directed(G2)))

Is the graph bipirtite?   True
Is the graph directed??   True

G3 = nx.from_pandas_edgelist(flow17, source='P_Code', target='D_Code_new', edge_attr=['predict_items_doubly'], create_using=nx.MultiDiGraph())
print('Is the graph bipirtite?   ' + str(nx.is_bipartite(G3)))
print('Is the graph directed??   ' + str(nx.is_directed(G3)))

Is the graph bipirtite?   True
Is the graph directed??   True

3. Look at the properties of the graph

# Look at number on graph properties
nb_nodes = len(G1.nodes())
nb_arr = len(G1.edges())
print("Number of nodes : " + str(nb_nodes))
print("Number of edges : " + str(nb_arr))

Number of nodes : 485
Number of edges : 74668

degree_freq = np.array(nx.degree_histogram(G1)).astype('float')
plt.figure(figsize=(12, 8))
plt.stem(degree_freq)
plt.ylabel("Frequence")
plt.xlabel("Degre")
plt.show();

C:\Users\tk18583\AppData\Local\Continuum\anaconda3\envs\geonet\lib\site-packages\ipykernel_launcher.py:3: UserWarning:

In Matplotlib 3.3 individual lines on a stem plot will be added as a LineCollection instead of individual lines. This significantly improves the performance of a stem plot. To remove this warning and switch to the new behaviour, set the "use_line_collection" keyword argument to True.

#degree_freq[degree_freq > 0]
x = degree_freq[degree_freq < 80]

plt.figure(figsize=(12, 8))
plt.stem(x)
plt.ylabel("Frequence")
plt.xlabel("Degre")
plt.show();

C:\Users\tk18583\AppData\Local\Continuum\anaconda3\envs\geonet\lib\site-packages\ipykernel_launcher.py:2: UserWarning:

In Matplotlib 3.3 individual lines on a stem plot will be added as a LineCollection instead of individual lines. This significantly improves the performance of a stem plot. To remove this warning and switch to the new behaviour, set the "use_line_collection" keyword argument to True.


plt.figure(figsize=(14,14))
nx.draw_networkx(G1, with_labels=True, width=0.02)

4. Define the functions to calculate centrality and snap the resluts to the point data

The Bipirtate graph is very specific cas of graph and has its own dedicated centraliy functions that can be calculated with networkx package.

This is the closeness and degree centrality, however none of those contain any weight, they are only based on the number of flows coming into the nodes.

No centrality can be measured in weighted bipirtite graph

Other measures we can look at are the Link Analysis measures. Those are Page Rank and HIIT (hubs and authorities). However, only Page Rank include a weight on edges,

Page Rank is the only measure we can apply on weighted directed bipartite graph!

def close_bi(graph, nodes, name, joinsA):
    # calculate the closenest
    close = bi.closeness_centrality(G = graph, nodes = nodes)
    # put it into a dataframe 
    x = pd.DataFrame.from_dict(close,  orient='index')
    x = x.reset_index()
    x = x.rename({0: name},axis = 1)

    df = joinsA.merge(x, left_on = 'P_Code', right_on = 'index', how = 'left')
    return df

def degree_bi(graph, nodes, name, joinsA, joinsB):
    # calculate the closenest
    close = bi.degree_centrality(G = graph, nodes = nodes)
    # put it into a dataframe 
    x = pd.DataFrame.from_dict(close,  orient='index')
    x = x.reset_index()
    x = x.rename({0: name},axis = 1)

    df = joinsA.merge(x, left_on = 'P_Code', right_on = 'index', how = 'left')
    df2 = joinsB.merge(x, left_on = 'D_Code_new', right_on = 'index', how = 'left')
    return df, df2

def pgr(graph, name, joinsA, joinsB, weight):
    # calculate the closenest
    close = nx.pagerank_scipy(G = graph, weight = weight, max_iter = 50)
    # put it into a dataframe 
    x = pd.DataFrame.from_dict(close,  orient='index')
    x = x.reset_index()
    x = x.rename({0: name},axis = 1)

    df = joinsA.merge(x, left_on = 'P_Code', right_on = 'index', how = 'left')
    df2 = joinsB.merge(x, left_on = 'D_Code_new', right_on = 'index', how = 'left')
    return df, df2

def pgr(graph, name, joinsA, joinsB, weight):
    # calculate the closenest
    close = nx.pagerank_scipy(G = graph, weight = weight, max_iter = 50)
    # put it into a dataframe 
    x = pd.DataFrame.from_dict(close,  orient='index')
    x = x.reset_index()
    x = x.rename({0: name},axis = 1)

    df = joinsA.merge(x, left_on = 'P_Code', right_on = 'index', how = 'left')
    df2 = joinsB.merge(x, left_on = 'D_Code_new', right_on = 'index', how = 'left')
    return df, df2

# eigenvector centrality does not work for multidigraphs
def eigen_ce(graph, name, joinsA, joinsB, weight):
    # calculate the closenest
    close = nx.eigenvector_centrality(G = graph, max_iter = 100, weight = weight)
    # put it into a dataframe 
    x = pd.DataFrame.from_dict(close,  orient='index')
    x = x.reset_index()
    x = x.rename({0: name},axis = 1)

    df = joinsA.merge(x, left_on = 'P_Code', right_on = 'index', how = 'left')
    df2 = joinsB.merge(x, left_on = 'D_Code_new', right_on = 'index', how = 'left')
    return df, df2

5. Apply the functions for Bipartate graph

GP2 = close_bi(graph = G1, nodes = list_flow_GP, name = 'close_G1', joinsA = GP)
GP2 = close_bi(graph = G2, nodes = list_flow_GP, name = 'close_G2', joinsA = GP2)
GP2 = close_bi(graph = G3, nodes = list_flow_GP, name = 'close_G3', joinsA = GP2)

GP2, PH2 = degree_bi(graph = G1, nodes = list_flow_GP, name = 'degree_G1', joinsA = GP2, joinsB = PH)
GP2, PH2 = degree_bi(graph = G2, nodes = list_flow_GP, name = 'degree_G2', joinsA = GP2, joinsB = PH2)
GP2, PH2 = degree_bi(graph = G3, nodes = list_flow_GP, name = 'degree_G3', joinsA = GP2, joinsB = PH2)

GP2, PH2 =  pgr(graph = G1, name = 'pgr_G1', weight = 'N_Items', joinsA = GP2, joinsB = PH2)
GP2, PH2 =  pgr(graph = G2, name = 'pgr_G2', weight = 'predict_items', joinsA = GP2, joinsB = PH2)
GP2, PH2 =  pgr(graph = G3, name = 'pgr_G3', weight = 'predict_items_doubly', joinsA = GP2, joinsB = PH2)

# eigenvector centrality does not work for multidigraphs
#GP2, PH2 =  eigen_ce(graph = G1, name = 'eig_G1', weight = 'N_Items', joinsA = GP2, joinsB = PH2)
#GP2, PH2 =  eigen_ce(graph = G2, name = 'eig_G2', weight = 'predict_items', joinsA = GP2, joinsB = PH2)
#GP2, PH2 =  eigen_ce(graph = G3, name = 'eig_G3', weight = 'predict_items_doubly', joinsA = GP2, joinsB = PH2)

GP2.drop(['index_x','index_y' ,'index'], axis=1, inplace=True)
PH2.drop(['index_x','index_y'], axis=1, inplace=True)

PH2.info()

<class 'geopandas.geodataframe.GeoDataFrame'>
Int64Index: 357 entries, 0 to 356
Data columns (total 25 columns):
 #   Column          Non-Null Count  Dtype   
---  ------          --------------  -----   
 0   D_Code          357 non-null    object  
 1   D_Name          357 non-null    object  
 2   D_full_address  357 non-null    object  
 3   D_Postcode2     357 non-null    object  
 4   D_Open_date     357 non-null    int64   
 5   D_Close_date    27 non-null     float64 
 6   D_Status_Code   357 non-null    object  
 7   D_Sub_type      357 non-null    object  
 8   D_Join          356 non-null    float64 
 9   D_Left          27 non-null     float64 
 10  D_X             357 non-null    float64 
 11  D_Y             357 non-null    float64 
 12  D_E             357 non-null    float64 
 13  D_N             357 non-null    float64 
 14  D_Setting       357 non-null    int64   
 15  D_Code_new      357 non-null    object  
 16  residents       357 non-null    float64 
 17  Density         357 non-null    float64 
 18  geometry        357 non-null    geometry
 19  degree_G1       357 non-null    float64 
 20  degree_G2       357 non-null    float64 
 21  degree_G3       357 non-null    float64 
 22  pgr_G1          357 non-null    float64 
 23  pgr_G2          357 non-null    float64 
 24  pgr_G3          357 non-null    float64 
dtypes: float64(15), geometry(1), int64(2), object(7)
memory usage: 72.5+ KB

# look at the values
GP2.iloc[:,14:23].head()

	close_G1	close_G2	close_G3	degree_G1	degree_G2	degree_G3	pgr_G1	pgr_G2	pgr_G3
0	1.262397	1.262397	1.262397	1.456583	1.456583	1.456583	0.001684	0.001684	0.001684
1	1.262397	1.262397	1.262397	2.207283	2.207283	2.207283	0.001684	0.001684	0.001684
2	1.262397	1.262397	1.262397	1.834734	1.834734	1.834734	0.001684	0.001684	0.001684
3	1.262397	1.262397	1.262397	1.733894	1.733894	1.733894	0.001684	0.001684	0.001684
4	1.262397	1.262397	1.262397	0.806723	0.806723	0.806723	0.001684	0.001684	0.001684

PH2.iloc[:,19:25].head()

	degree_G1	degree_G2	degree_G3	pgr_G1	pgr_G2	pgr_G3
0	2.140625	2.140625	2.140625	0.002018	0.001812	0.001996
1	1.687500	1.687500	1.687500	0.002194	0.001957	0.002179
2	0.648438	0.648438	0.648438	0.001689	0.001774	0.001689
3	2.937500	2.937500	2.937500	0.002471	0.002330	0.002497
4	3.773438	3.773438	3.773438	0.001700	0.001867	0.001718

6. Changing graph structure

We can see that the degree centrality and closeness centraility is the same for all the graphs, because the wight is not included.

The Pagerank for the GP's is not changing either as the flows are leaving them and page rank focuses on the incoming flows only.

The Page Rank algorithm computes a ranking of the nodes in the graph G based on the structure of the incoming links, so we can only observe the importance of pharmacies.

plt.figure(figsize=(30, 6))
# Degree Centrality
f, axarr = plt.subplots(1,3, num=1)

plt.sca(axarr[0])
PH2['pgr_G1'].plot.hist(grid=True, bins=20, rwidth=0.9, color='#607c8e')
axarr[0].set_title('Page Rank Graph 1', size=16)

plt.sca(axarr[1])
PH2['pgr_G2'].plot.hist(grid=True, bins=20, rwidth=0.9, color='#607c8e')
axarr[1].set_title('Page Rank Graph 2', size=16)

plt.sca(axarr[2])
PH2['pgr_G3'].plot.hist(grid=True, bins=20, rwidth=0.9, color='#607c8e')
axarr[2].set_title('Page Rank Graph 3', size=16);

# which one is the most important? the one that sticks out
PH2[PH2['pgr_G1'] > 0.0044]

	D_Code	D_Name	D_full_address	D_Postcode2	D_Open_date	D_Close_date	D_Status_Code	D_Sub_type	D_Join	D_Left	...	D_Code_new	residents	Density	geometry	degree_G1	degree_G2	degree_G3	pgr_G1	pgr_G2	pgr_G3
112	FLQ56	BOOTS UK LIMITED	BOOTS UK LIMITED,59 BROADMEAD,BRISTOL,nan,BS1 3ED	BS1 3ED	19480101	NaN	A	1	20040401.0	NaN	...	FLQ56	1021.5	202.028571	POINT (-2.58972 51.45774)	8.898438	8.898438	8.898438	0.004406	0.002691	0.003961

1 rows × 25 columns

ax = plt.figure(figsize=(10, 10))
plt.axis([PH2['pgr_G1'].min(), PH2['pgr_G1'].max(), PH2['pgr_G1'].min(), PH2['pgr_G1'].max()])
# plots
plt.scatter(PH2['pgr_G1'],PH2['pgr_G1'])
plt.scatter(PH2['pgr_G1'],PH2['pgr_G2'])
plt.plot(PH2['pgr_G1'].mean(),PH2['pgr_G1'].mean(), c = 'r', marker = 'X', markersize = 30)
# regression plots
sns.regplot(PH2['pgr_G1'],PH2['pgr_G1'])
sns.regplot(PH2['pgr_G1'],PH2['pgr_G2'])

plt.legend(['Mean','Page Rank from data', 'Page Rank from base gravity']);

PH2['pgr_G1'].mean()

0.002197291434997281

ax = plt.figure(figsize=(10, 10))
plt.axis([PH2['pgr_G1'].min(), PH2['pgr_G1'].max(), PH2['pgr_G1'].min(), PH2['pgr_G1'].max()])
# plots
plt.scatter(PH2['pgr_G1'],PH2['pgr_G1'])
plt.scatter(PH2['pgr_G1'],PH2['pgr_G3'])
plt.plot(PH2['pgr_G1'].mean(),PH2['pgr_G1'].mean(), c = 'r', marker = 'X', markersize = 30)
# regression plots
sns.regplot(PH2['pgr_G1'],PH2['pgr_G1'])
sns.regplot(PH2['pgr_G1'],PH2['pgr_G3'])

plt.legend(['Mean','Page Rank from data', 'Page Rank from doubly gravity model' ]);

Results

What we can see on the figure above is how the node centrality in the graph conctructed from prediction relates to a node centrality in the graph constructed from raw data. In ideal situation, the line of best fit and the observations would follow the blue line, however, this is not the case. From the first figure, which looks at the importance of the nodes based on base gravity model (very simple one), you can observe that where the actual importannce of the node is high, the importance based on predicted flows drop down drastically, and where the actual importance of the node is low, the importance based on predicted flows is higher. You can also notice line of zeros trying to hide behind the y axes. So although the model predicts flow with certain accuracy, th quality of the flow is not the same as from the data. More specifically the model is not really able to see the hierarchical structure of the nodes and creates predictions that are close to mean for all nodes.

Not suprisingly this is not the case for more complex gravity model such as doubly constrained gravity model. Here the hierarchy of the nodes is sustained by the prediction, however, that is expected as each node has it's own parameter in the model. Seems like a simple and smart solution, however, you need to remember the with doubly constrained model, where each node is considered as unique, predictions can be made only to existing nodes. So question like ‘What will happen with the flows if one of the GP's closes’ are hardly answerable. It's like a cheeky shortcut; if I am not able to meodel complicated flow structure, let's model each node separately.

Drop me a message on twitter if you have any comments :)

Inter-National Migration; Part 2 - Calculating Distances

Thu, 09 Jan 2020 00:00:00 +0000

This is a second part of Intra-National Migration data exploration. This post will also construt distance variables for the data and crack on with some Spatial Interction Modeling. This is a contribution to my PhD research that examines Origin-Destination and its spatial aspects.

Introduction

UK intra-migration data are open source data that can be downloaded from ONS. ONS uses the NHS Patient Register Data Service (PRDS) to find out the changes in patients adresses. Since most people change their address with their doctor soon after moving, these data are considered to provide a good proxy indicator of migration.

The migration data is published every year and consist of Origin Local Authority ID, Destination Local Authority ID, sex, age and movement factor field. This movement factor is based on resscaled number of people on the flow.

Please follow ONS to find more information about the methodology.

Follow Part 1 to see some basic graphs of the data.

Recap

I'm just going to quickly recap what we know alrady about the data.

The origins and destinations are the same, UK Local Authorities.
The flow volume is represented by movement factor wich is already scaled variable and is extremely skewed to the left.
The year 2011 has very different nature from all the other years.
Because we are not able to judstify the difference in nature of the 2011, we are going to work with the later years only.

In this post…

I want to calculate distances between the Originn and Destination.

Now, it might sound easy, but the theory behind this is more colmpicated than you would think. Read through my other post that explains most of the theory behind this.

I need to;

Calculate the Eclidian Distance between the Origins and Destinations
Calculate the Great Circle Distance between the Origins and Destinations
Calculate the Network Distance (walking path along the streets)

1. Calculate the Euclidian Distance between the Origins and Destinations

To do that, we first need to load the libraries and the data we mungedin the post 1

# we will first import all necessary libraries
import pandas as pd
import os
import numpy as np
import glob
from datetime import datetime
import geopandas as gpd
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px
from shapely.geometry import Point, LineString

# set the path
file = r'./../data/migration/full_mig.csv'
# read the file
df = pd.read_csv(file, index_col=None, low_memory=False)
# plot the top of the table
df.head()

	OutLA	InLA	Age	Sex	Moves	Date	sex_bin
0	E09000001	E09000002	27	2	0.0002	2011-01-01	2
1	E09000001	E09000002	36	1	0.0004	2011-01-01	1
2	E09000001	E09000002	26	2	0.0004	2011-01-01	2
3	E09000001	E09000002	25	2	0.0005	2011-01-01	2
4	E09000001	E09000002	34	1	0.0014	2011-01-01	1

# set the path
file = r'./../data/migration/munged/LAcentrooids.shp'
# read the file in
la = gpd.read_file(file)
# show the map
la.plot();

Now I can create lat and long columns in the centrooid data which I'll merge to the flow data twice. First based on the Out-flow LA ID and second based on In-flow LA ID.

# create new callumn x and y wit lat and long coordinates
la['x'] = la.geometry.x
la['y'] = la.geometry.y
la.head()

	objectid	lad17cd	lad17nm	geometry	x	y
0	1	E06000001	Hartlepool	POINT (-1.259479894328456 54.66947264221945)	-1.259480	54.669473
1	2	E06000002	Middlesbrough	POINT (-1.222265950359884 54.54199624640567)	-1.222266	54.541996
2	3	E06000003	Redcar and Cleveland	POINT (-1.020534867661266 54.55159000727666)	-1.020535	54.551590
3	4	E06000004	Stockton-on-Tees	POINT (-1.332307567288941 54.56157879321239)	-1.332308	54.561579
4	5	E06000005	Darlington	POINT (-1.552585765196361 54.54869927572448)	-1.552586	54.548699

# lets get rid of the rows we dont have spatial information for
# list of Unique ID's
d = list(la['lad17cd'].unique())
# new dataset with just those that are in a list
df2 = df.loc[df['OutLA'].isin(d)]
df3 = df2.loc[df2['InLA'].isin(d)]
# see how many rows we lost
len(df) - len(df3)

# create two datasets from the la's base on Outflow and Inflow
# drop th geometries and create dataframes
outla = pd.DataFrame(la)
inla = pd.DataFrame(la)
# select just what we need
outla = outla.loc[:,['lad17cd','x','y']]
inla = inla.loc[:,['lad17cd','x','y']]
# rename the columns
outla.rename(columns = {'x':'out_x','y':'out_y'}, inplace = True)
inla.rename(columns = {'x':'in_x','y':'in_y'}, inplace = True)
# look at what we have
outla.head()

	lad17cd	out_x	out_y
0	E06000001	-1.259480	54.669473
1	E06000002	-1.222266	54.541996
2	E06000003	-1.020535	54.551590
3	E06000004	-1.332308	54.561579
4	E06000005	-1.552586	54.548699

# join the spatial points to the data, first for the OutLA and secondly for the InLA
df4 = df3.merge(outla, how = 'left', left_on = 'OutLA', right_on = 'lad17cd')
df4 = df4.merge(inla, how = 'left', left_on = 'InLA', right_on = 'lad17cd')
df4.tail()

	OutLA	InLA	Age	Sex	Moves	Date	sex_bin	lad17cd_x	out_x	out_y	lad17cd_y	in_x	in_y
10701098	W06000024	W06000023	67	F	1.1416	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737
10701099	W06000024	W06000023	73	F	0.9519	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737
10701100	W06000024	W06000023	74	M	1.0020	2018-01-01	1	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737
10701101	W06000024	W06000023	74	F	0.9866	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737
10701102	W06000024	W06000023	77	F	3.0000	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737

# make sure there is no missing values
x = df4.isna()
x.loc[x['out_x'] == True]

	OutLA	InLA	Age	Sex	Moves	Date	sex_bin	lad17cd_x	out_x	out_y	lad17cd_y	in_x	in_y

That looks like exactly what we need!

Next step will be calculating the Euclidian distance between the Drigin and the Destination.

There are actually quite few way to do this.

1. You can just use pythagoras function to do that

test = df4
test['Euc_distance'] = (((test.out_x - test.in_x) ** 2) + (test.out_y - test.in_y) ** 2) ** .5
test.head()

	OutLA	InLA	Age	Sex	Moves	Date	sex_bin	lad17cd_x	out_x	out_y	lad17cd_y	in_x	in_y	Euc_distance
0	E09000001	E09000002	27	2	0.0002	2011-01-01	2	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735
1	E09000001	E09000002	36	1	0.0004	2011-01-01	1	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735
2	E09000001	E09000002	26	2	0.0004	2011-01-01	2	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735
3	E09000001	E09000002	25	2	0.0005	2011-01-01	2	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735
4	E09000001	E09000002	34	1	0.0014	2011-01-01	1	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735

Very simple, we just need to pay attention to the units of the distance which is in degrees (coordinates of the points in degrees).

2. You can use scipy.spatial.distance.cdist() function that will basically do the same job, however needs more data manipulation

# import the library
# this could be done different ways too so feel free to do what suits you
import scipy.spatial.distance as ssd
# importing the library you use the the function as x=ssd.cdist()
from scipy.spatial import distance
# then use the function as x=distance.cdist()
import scipy
# then use the function as x=scipy.spatial.distance.cdist()

# Make selection
# I'll test this on selection of points as I did not find out why it does not like my vector of size 11575714...to big for this I guess
selection = test.iloc[0:5,:]

# to use this function, we neet to create numpy arrays from the coordinates, as the cdist() requires
coords_out = np.array(selection[['out_x','out_y']])
coords_in = np.array(selection[['in_x','in_y']])
# note that this function return a matrix
mat = distance.cdist(coords_out, coords_in, 'euclidean')
mat

array([[0.22773512, 0.22773512, 0.22773512, 0.22773512, 0.22773512],
       [0.22773512, 0.22773512, 0.22773512, 0.22773512, 0.22773512],
       [0.22773512, 0.22773512, 0.22773512, 0.22773512, 0.22773512],
       [0.22773512, 0.22773512, 0.22773512, 0.22773512, 0.22773512],
       [0.22773512, 0.22773512, 0.22773512, 0.22773512, 0.22773512]])

Then you need to convert the matrix into the collumn of the dataframe… I did not worked that one out, so let me know if you know how O:-)

However, BIG ADVANTAGE of this function is that it has predefined 24 different calculation of distances. Check out the documentation, and because it's based on C++ its quite fast.

2. Calculate the Great Circle Distance between the Origins and Destinations

This could be also done several ways.

The easiest way is using the GeoPy library, but if you find a faster way, let me know!

To do this we need to get the coordinates in one column as it was in original point dataset just stripped out of the geometry column.

from geopy.distance import geodesic, great_circle
import itertools

# create stripped column of coordinates 
la['xy'] = la.geometry.apply(lambda x: [x.y, x.x])

# turn it into dataframe
points = pd.DataFrame(la)
# select just what we need
points1 = points.loc[:,['lad17cd','xy']]
points2 = points.loc[:,['lad17cd','xy']]
# rename the columns
points1.rename(columns = {'xy':'xy_1'}, inplace = True)
points2.rename(columns = {'xy':'xy_2'}, inplace = True)
# join the spatial points to the data, first for the OutLA and secondly for the InLA
test2 = test.merge(points1, how = 'left', left_on = 'OutLA', right_on = 'lad17cd')
test2 = test2.merge(points2, how = 'left', left_on = 'InLA', right_on = 'lad17cd')
test2.tail()

	OutLA	InLA	Age	Sex	Moves	Date	sex_bin	lad17cd_x	out_x	out_y	lad17cd_y	in_x	in_y	Euc_distance	lad17cd_x	xy_1	lad17cd_y	xy_2
10701098	W06000024	W06000023	67	F	1.1416	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737	0.584839	W06000024	[51.74094175880734, -3.361056696671829]	W06000023	[52.323736681870706, -3.4099121042932894]
10701099	W06000024	W06000023	73	F	0.9519	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737	0.584839	W06000024	[51.74094175880734, -3.361056696671829]	W06000023	[52.323736681870706, -3.4099121042932894]
10701100	W06000024	W06000023	74	M	1.0020	2018-01-01	1	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737	0.584839	W06000024	[51.74094175880734, -3.361056696671829]	W06000023	[52.323736681870706, -3.4099121042932894]
10701101	W06000024	W06000023	74	F	0.9866	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737	0.584839	W06000024	[51.74094175880734, -3.361056696671829]	W06000023	[52.323736681870706, -3.4099121042932894]
10701102	W06000024	W06000023	77	F	3.0000	2018-01-01	2	W06000024	-3.361057	51.740942	W06000023	-3.409912	52.323737	0.584839	W06000024	[51.74094175880734, -3.361056696671829]	W06000023	[52.323736681870706, -3.4099121042932894]

# now calculate the great circle distance
# this takes good 15 min
test2['great_circle_dist'] = test2.apply(lambda x: great_circle(x.xy_1, x.xy_2).km, axis=1)
test2.head()

	OutLA	InLA	Age	Sex	Moves	Date	sex_bin	lad17cd_x	out_x	out_y	lad17cd_y	in_x	in_y	Euc_distance	lad17cd_x	xy_1	lad17cd_y	xy_2	great_circle_dist
0	E09000001	E09000002	27	2	0.0002	2011-01-01	2	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735	E09000001	[51.51484485100944, -0.09217115162643398]	E09000002	[51.54527247765296, 0.13352209887032826]	15.97471
1	E09000001	E09000002	36	1	0.0004	2011-01-01	1	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735	E09000001	[51.51484485100944, -0.09217115162643398]	E09000002	[51.54527247765296, 0.13352209887032826]	15.97471
2	E09000001	E09000002	26	2	0.0004	2011-01-01	2	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735	E09000001	[51.51484485100944, -0.09217115162643398]	E09000002	[51.54527247765296, 0.13352209887032826]	15.97471
3	E09000001	E09000002	25	2	0.0005	2011-01-01	2	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735	E09000001	[51.51484485100944, -0.09217115162643398]	E09000002	[51.54527247765296, 0.13352209887032826]	15.97471
4	E09000001	E09000002	34	1	0.0014	2011-01-01	1	E09000001	-0.092171	51.514845	E09000002	0.133522	51.545272	0.227735	E09000001	[51.51484485100944, -0.09217115162643398]	E09000002	[51.54527247765296, 0.13352209887032826]	15.97471

YAAAAAAAAY

We have got a distance in kilometres!

Ok it's a mess, let's clean the dataset a little bit as there is several extra columns.

# create back up first
test3 = test2
test3 = test3[['OutLA','InLA','Age','Moves','Date','sex_bin', 'Euc_distance','great_circle_dist','out_x', 'out_y','in_x','in_y', 'xy_1','xy_2']]
test3.head()

	OutLA	InLA	Age	Moves	Date	sex_bin	Euc_distance	great_circle_dist	out_x	out_y	in_x	in_y	xy_1	xy_2
0	E09000001	E09000002	27	0.0002	2011-01-01	2	0.227735	15.97471	-0.092171	51.514845	0.133522	51.545272	[51.51484485100944, -0.09217115162643398]	[51.54527247765296, 0.13352209887032826]
1	E09000001	E09000002	36	0.0004	2011-01-01	1	0.227735	15.97471	-0.092171	51.514845	0.133522	51.545272	[51.51484485100944, -0.09217115162643398]	[51.54527247765296, 0.13352209887032826]
2	E09000001	E09000002	26	0.0004	2011-01-01	2	0.227735	15.97471	-0.092171	51.514845	0.133522	51.545272	[51.51484485100944, -0.09217115162643398]	[51.54527247765296, 0.13352209887032826]
3	E09000001	E09000002	25	0.0005	2011-01-01	2	0.227735	15.97471	-0.092171	51.514845	0.133522	51.545272	[51.51484485100944, -0.09217115162643398]	[51.54527247765296, 0.13352209887032826]
4	E09000001	E09000002	34	0.0014	2011-01-01	1	0.227735	15.97471	-0.092171	51.514845	0.133522	51.545272	[51.51484485100944, -0.09217115162643398]	[51.54527247765296, 0.13352209887032826]

3. Calculating the network distances

I will demonstrate this on random points on Bristol as our migration data are not that relevant to a network distance. We just need to upload two libraries osmnx and networkx

There is nothing easier than get your network from osmnx, no need for looking for data and extra faf

import osmnx as ox
import networkx as nx

# define the place
place_name = "Bristol, United Kingdom"
# define the graph
graph = ox.graph_from_place(place_name, network_type='drive')

Now get some random points from Bristol

# 51.453700, -2.595778 # Dominos pizza in Bristol city centre
# 51.456294, -2.605236 # Bristol Gallery and Museum
# 51.468233, -2.588812 # Montpelier rail station
# 51.456569, -2.626453 # Clifton observatory

# create the list of coordinates
lat_list = [51.453700, 51.456294, 51.468233, 51.456569]
lon_list = [-2.595778, -2.605236, -2.588812, -2.626453]

#put it together
random_points = zip(lon_list, lat_list)

# create dataframe
random_points = pd.DataFrame(random_points)
# change the names of columns
random_points.rename(columns = {0:'x',1:'y'}, inplace = True)

# create spatial dataframe from the dataframe
random_points_gpd = gpd.GeoDataFrame(
    random_points, geometry=gpd.points_from_xy(random_points.x, random_points.y))
# assign coordinates
random_points_gpd.crs = la.crs

Now we need to convert everything to UTM as we want to get the lenght in metres not degrees

# define the crs
utm = "+proj=utm +zone=33 +ellps=WGS84 +datum=WGS84 +units=m +no_defs"
# points to utm
random_points_gpd = random_points_gpd.to_crs(utm)
# convert the graph into utm...projct_graph function does it all
graph = ox.project_graph(graph, to_crs = utm)

# lets just see how it looks like
fig, ax = ox.plot_graph(graph_proj)
plt.tight_layout()

<Figure size 432x288 with 0 Axes>

# separate the nodes and edges from the graph
nodes_proj, edges_proj = ox.graph_to_gdfs(graph_proj, nodes=True, edges=True)
# look at the crs just to check
print("Coordinate system:", edges_proj.crs)

Coordinate system: +proj=utm +zone=33 +ellps=WGS84 +datum=WGS84 +units=m +no_defs

# create nearest nodes for each point

#get the coordinates only into dataframe
random_points_gpd['a'] = random_points_gpd.geometry.y
random_points_gpd['b'] = random_points_gpd.geometry.x
target_xy = pd.DataFrame(random_points_gpd[['a','b']])

# get the nearest node id for each row
def nearest_node(Lat,Lon):
    nearest_node,dist=ox.get_nearest_node(graph, (Lat,Lon), return_dist=True)  
    return nearest_node 

# apply the function
target_xy['node'] = np.vectorize(nearest_node)(target_xy['a'],target_xy['b'])

Now we create the Origin-Destination data this is going to be little fidly, indeed, but at least we have just a sample dataset

# separate just the node id
target2 = target_xy.iloc[:,[2]]

# double this for destinations
target3 = target_xy.iloc[:,[2]]
# reset the index and reverse the order
target3 = target3.sort_index(ascending=False).reset_index().reset_index()
target2 = target2.reset_index()

# merge those two node datasets into OD data
OD = target2.merge(target3, left_on='index', right_on= 'level_0')

# get rid of unnecessary columns
OD = OD.loc[:,['node_x','node_y']]
OD.head()

	node_x	node_y
0	6502760749	26127008
1	5432376184	358536159
2	358536159	5432376184
3	26127008	6502760749

# define a function that calculates shortest path between the nodes on graph
def short_path_length(row):
    return nx.shortest_path_length(graph, row['node_x'], row['node_y'], weight='length')

# apply the function to our OD data.....Note: as we are working with UTM projection, the units should be metres
OD['short_path_length'] = OD.apply(short_path_length, axis=1)
OD.head()

	node_x	node_y	short_path_length
0	6502760749	26127008	4537.728
1	5432376184	358536159	4031.132
2	358536159	5432376184	4039.488
3	26127008	6502760749	4611.492

Bare in mind I calculated the network distances on sample points in small area. That is because graph of whole UK is just huuuge and the computation would be time-expensive. In my opinion, we do not need the network distance for the migration data, however it could be usefull to know how to do that, so let me know if you found any quick and efficient way to do that.

Intra-National Migration, Part 1 - EDA

Mon, 30 Dec 2019 00:00:00 +0000

Introduction

What do I do here?

This post is an exploration of the UK intra-migration data. Those are open source data that can be downloaded from ONS. ONS uses the NHS Patient Register Data Service (PRDS) to find out the changes in patients adresses. Since most people change their address with their doctor soon after moving, these data are considered to provide a good proxy indicator of migration.

Please follow ONS to find more information about the methodology.

Why do I do this?

My PhD research is looking at origin-destination flow data and explores Machine Learning modelling possibilities, considering the spatial structure of the flows, including the scale. The ONS migration data is one of the datasets I'm going to use for modelling. Thus, Exploration Data Analysis is a vital process of the reserach.

How?

I'll be looking at

The nature of the variables
The structure of the migration in time
The structure of the migration in space

To do that I need to

Load the data in and join them all into one dataset
Make sure we retain the date to look at the timestamp
Join a spatial information to ensure the validity of all origins and destinations

1. Load the data in and join them all into one dataset + 2. Make sure we retain the date to look at the timestamp

# we will first import all necessary libraries
import pandas as pd
import os
import numpy as np
import glob
import datetime 
import datetime as dt
from datetime import datetime
import geopandas as gpd
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px

# set the path
path = r'./../../data/migration'

os.chdir(path) # navigate to path

all_files = glob.glob(path + "/*.csv") # access all the csv files in the folder and create list

li = [] # create empty list

# function which reads each csv, set up same header for each, create a new column with name of each file and store them in a list
for filename in all_files:
    df = pd.read_csv(filename, index_col=None, header=0, names =['OutLA', 'InLA', 'Age', 'Sex', 'Moves'], low_memory=False)
    df['filename'] = filename
    li.append(df)

# rowbind (concatonate) all the csv's in a list together
frame = pd.concat(li, axis=0, ignore_index=True, sort=False)

# seperate just the year from the filename variable
frame['filename'] = frame['filename'].str[23:] # this number depends on how long is your path, this row deletes 23 characters from left
frame['filename'] = frame['filename'].str[:-9] # and this row deletes 9 characters from right

# convert the year to date
frame['filename'] = frame['filename'].apply(lambda x: 
                                    dt.datetime.strptime(x,'%Y'))
# rename the date collumn
frame = frame.rename({'filename':'Date'},axis = 1)

# fix the sex column - convert all to binary
frame.loc[frame['Sex']==1,'sex_bin'] = 1
frame.loc[frame['Sex']==2,'sex_bin'] = 2
frame.loc[frame['Sex']=='M','sex_bin'] = 1
frame.loc[frame['Sex']=='F','sex_bin'] = 2
frame['sex_bin'].fillna('Other', inplace=True)
frame['sex_bin'] = frame['sex_bin'].astype(int)

# set the path
path = r'./../../../post/migration'
# fix the path
os.chdir(path) # navigate to path

# Write the cleaned dataset to back it up if we need
# frame.to_csv(index = False , path_or_buf = './../../data/migration/full_migration.csv')

# now I just create a backup in the environment so I don't need to go through all the steps above in case I do something wrong
df = frame
df = df.set_index('Date') # set the date as an index
df.head(3) # look at the first 3 rows

	OutLA	InLA	Age	Sex	Moves	sex_bin
Date
2011-01-01	E09000001	E09000002	27	2	0.0002	2
2011-01-01	E09000001	E09000002	36	1	0.0004	1
2011-01-01	E09000001	E09000002	26	2	0.0004	2

You can see how the data looks like here. There is 7 collumn, from which 1st is the date, 2nd and 3rd retain the ID of the Origin and Destination Local Authority. There are two demographic variables, age and sex. The sex_bin is the sex variable in binary format, as the original variable was combination of Binary and characters. Most important variables is the ‘Moves’ that represents the amount of moves in one flux. By flux, I mean unique flow defined by the combination of Origin, Destination, Date, Age and Sex.

From the cell below you can see that the dataset has 11 milion rows combining the migration data from 2011 till 2018.

df.info()

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 11575714 entries, 2011-01-01 to 2018-01-01
Data columns (total 6 columns):
OutLA      object
InLA       object
Age        int64
Sex        object
Moves      float64
sex_bin    int32
dtypes: float64(1), int32(1), int64(1), object(3)
memory usage: 574.1+ MB

# Use seaborn style defaults and set the default figure size
sns.set(rc={'figure.figsize':(10, 4)})
# Plot the moves as a time series
df['Moves'].plot();

Well that is not very pretty. As the date of the flow is always the 1st day of the 1st month, the time series expect that the rest of th month is zero and show that on the graph. To avoid that, let's try group the values by year.

# create new variable with grouped values
yearly = df.groupby(df.index).mean()
# use the same plot as before
sns.set(rc={'figure.figsize':(10, 4)})
yearly['Moves'].plot();

This looks like the first recorded year 2011 had less then half of the migration flows as the years after. This could be very much true, but lets have a look at the plot witout the 2011.

# create new variable with grouped values
yearly = df.groupby(df.index).mean()
yearly = yearly[1:8] # remember the first row has index 0
# use the same plot as before
sns.set(rc={'figure.figsize':(8, 4)})
yearly['Moves'].plot();

Other than very low migration rate in 2011, there is a certainly an overall increase of the migration between 2016 and 2018.

We can have a look at the distribution of genders in the migration flows.

# get percentages of each gender throughout the dataset
sb = df['sex_bin'].value_counts(normalize = True)
# create dataframe
sb = pd.DataFrame(sb)
# rename the value collumn
sb = sb.rename({'sex_bin':'percentage'},axis = 1)
# create new value for sex column
sex = ['Female', 'Male']
# add the new column to dataframe
sb['sex'] = sex
# change index for the new column
sb = sb.set_index('sex')
# see how the table looks like
sb

	percentage
sex
Female	0.506348
Male	0.493652

ax = sb.plot(kind='bar', title ="Gender structure", figsize=(5, 3), legend=False, fontsize=12)
ax.set_xlabel("Sex", fontsize=12)
ax.set_ylabel("Percentage from all flows", fontsize=12)
plt.show()

Looks like there is slightly more women than men moving within the UK. Is it true for all of the years?

# group by the date and sex
sb_year = frame.groupby(['Date', 'sex_bin'])
# create variable from the grouping
sb_year = sb_year.size()
# create dataframe
sb_year = pd.DataFrame(sb_year)
# rename the value collumn
sb_year = sb_year.rename({0:'frequency'},axis = 1)
# create new value for sex column
sex = ['Male', 'Female','Male', 'Female','Male', 'Female','Male', 'Female','Male', 'Female','Male', 'Female','Male', 'Female','Male', 'Female']
# add the new column to dataframe
sb_year['sex'] = sex
# reset index
sb_year = sb_year.reset_index()
sb_year

	Date	sex_bin	frequency	sex
0	2011-01-01	1	1233198	Male
1	2011-01-01	2	1241611	Female
2	2012-01-01	1	611320	Male
3	2012-01-01	2	625818	Female
4	2013-01-01	1	616615	Male
5	2013-01-01	2	634451	Female
6	2014-01-01	1	635949	Male
7	2014-01-01	2	656613	Female
8	2015-01-01	1	632116	Male
9	2015-01-01	2	651020	Female
10	2016-01-01	1	632000	Male
11	2016-01-01	2	650336	Female
12	2017-01-01	1	673271	Male
13	2017-01-01	2	698670	Female
14	2018-01-01	1	679906	Male
15	2018-01-01	2	702820	Female

fig, ax = plt.subplots(figsize=(12, 8))
x = np.arange(len(sb_year.Date.unique()))

# Define bar width. We'll use this to offset the second bar.
bar_width = 0.4

# Note we add the `width` parameter now which sets the width of each bar.
b1 = ax.bar(x, sb_year.loc[sb_year['sex'] == 'Male', 'frequency'],
            width=bar_width, label = 'Man')
# Same thing, but offset the x by the width of the bar.
b2 = ax.bar(x + bar_width, sb_year.loc[sb_year['sex'] == 'Female', 'frequency'],
            width=bar_width, label = 'Woman')

# Fix the x-axes.
ax.set_xticks(x + bar_width / 2)
ax.set_xticklabels(sb_year.Date.unique(), rotation=45)

# Add legend.
ax.legend()

# Axis styling.
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['left'].set_visible(False)
ax.spines['bottom'].set_color('#DDDDDD')
ax.tick_params(bottom=False, left=False)
ax.set_axisbelow(True)
ax.yaxis.grid(True, color='#EEEEEE')
ax.xaxis.grid(False)

# Add axis and chart labels.
ax.set_xlabel('Year', labelpad=15)
ax.set_ylabel('Count', labelpad=15)
ax.set_title('UK Intra-Migrants by the gender', pad=15)

fig.tight_layout()

It is definitely true that more woman move within UK than man, even for each year.

Bellow you can see how the difference between the genders differ in each year.

# separate the Males and Females into seperate datasets
M = sb_year.loc[sb_year['sex'] == 'Male']
F = sb_year.loc[sb_year['sex'] == 'Female']
# merge those two datasets
X = M.merge(F, left_on = 'Date', right_on = 'Date')
# calculate difference between males and females
X['Diff'] = X['frequency_y'] - X['frequency_x']
# choose just desired columns
XX = X.loc[:,['Date','Diff']]
# plot the time-series of the gender difference
sns.set(rc={'figure.figsize':(8, 4)})
XX['Diff'].plot();

Wait a second …

It looks like in 2011, the frequency of the flows is much higher than all the other years, however, from the graphs above showing the ‘Moves’ factor, we know that the 2011 actually has visibly lower movement factor. In other words, although the 2011 has the most amount of unique fluxes, the amount of people moving is much lower than in recent years.

This is an interesting discovery, but what could be a cause?

Any change in methodology between 2011 and 2012.
Differences in the structure of the data before and after 2011
Change in the nature of the data. I could be that people changed their behaviour pattern.
Could it be any mistake from my side?

Let's investigate all options.

Changes in the methodology

The only change in methodology found in the methods guide is this;

“For the mid-2012 to mid-2016 internal migration estimates we improved our methods by linking the health registration data with data from the Higher Education Statistics Agency (HESA). The HESA data showed where students were registered by their university as living, and this allowed us to make more accurate estimates of people moving to study in each area."

Difference in the data structure

From the download site we can see that the 2011 dataset is the only one that has 3 parts, it has the highest number of rows (unique fluxes). It contains 2.47 milion records, which is almost half as much as 2018 dataset with its 1.38 milion records.

Is it that the 2011 dataset contains more areas than the later one?

3. Join a spatial information to the data

Let's join in a spatial data to see it on a map. There are several datasets that could be used for this, for example ONS LA lookup.

# set the path
file = r'./../data/Local_Authority_Districts_December_2017_Full_Clipped_Boundaries_in_Great_Britain/Local_Authority_Districts_December_2017_Full_Clipped_Boundaries_in_Great_Britain.shp'
# read the file in
la = gpd.read_file(file)
# show the map
la.plot()

# look at the data
la.head()

	objectid	lad17cd	lad17nm	lad17nmw	bng_e	bng_n	long	lat	st_areasha	st_lengths	geometry
0	1	E06000001	Hartlepool	None	447157	531476	-1.27023	54.676159	9.355951e+07	71707.407523	(POLYGON ((447213.8995000003 537036.1042999998...
1	2	E06000002	Middlesbrough	None	451141	516887	-1.21099	54.544670	5.388858e+07	43840.866561	(POLYGON ((448958.9007000001 521835.6952999998...
2	3	E06000003	Redcar and Cleveland	None	464359	519597	-1.00611	54.567520	2.448203e+08	97993.391012	(POLYGON ((455752.6002000002 528195.7048000004...
3	4	E06000004	Stockton-on-Tees	None	444937	518183	-1.30669	54.556911	2.049622e+08	119581.595543	(POLYGON ((444157.0018999996 527956.3033000007...
4	5	E06000005	Darlington	None	428029	515649	-1.56835	54.535351	1.974757e+08	107206.401694	POLYGON ((423496.602 524724.2984999996, 423497...

# create point dataset of the LSOA centrooids
# copy polygons to new GeoDataFrame
points = la.copy()
# change the geometry
points.geometry = points['geometry'].centroid
# same crs
points.crs = la.crs
# lets reproject the data too
points = points.to_crs({'init': 'epsg:4326'})
# change projection
la = la.to_crs({'init': 'epsg:4326'})
# cut out unnecessary columns
points = points.loc[:,['objectid','lad17cd', 'lad17nm', 'geometry']]
points.head()

	objectid	lad17cd	lad17nm	geometry
0	1	E06000001	Hartlepool	POINT (-1.259479894328456 54.66947264221945)
1	2	E06000002	Middlesbrough	POINT (-1.222265950359884 54.54199624640567)
2	3	E06000003	Redcar and Cleveland	POINT (-1.020534867661266 54.55159000727666)
3	4	E06000004	Stockton-on-Tees	POINT (-1.332307567288941 54.56157879321239)
4	5	E06000005	Darlington	POINT (-1.552585765196361 54.54869927572448)

Now we need to create lists of the values in the LA ID columns to

# create list of unique LA id's from each dataset
x = list(df['OutLA'].unique())
d = list(points['lad17cd'].unique())
# chck if the number of LA codes matches 
len(x) - len(d)

There is indeed some LA's that we don't have a spatial information for. Let's filter them out.

# lets get rid of the rows we dont have spatial information for, based on origins
df2 = df.loc[df['OutLA'].isin(d)]
len(df) - len(df2)

# Do the same thing for the destination
df3 = df2.loc[df2['InLA'].isin(d)]
len(df) - len(df3)

According to this we have filtered out all together 874 thousand record that could not be found in the spatial dataset.

Let's write both datasets into a folder for further use.

Explore rest of the variables

The main variable in the migration flows is the ‘Moves’ which describes the intensity of the flow or the amount of people on the flow.

From the historam below, we can see that most of the flows has a volume between 0 and circa 70. Thise are the less intensive flows, although the most frequent. However, little peak can be seen on number 800; high intensity flows that are less frequent.

plt.figure(figsize=(9, 8))
sns.distplot(df3['Moves'], color='g', bins=1);

The last variable is the age of the migrants. From the histogram below, it is clear that most of the migrating population is aged between 20 and 40. A little peak is also visible for the early age of 0 to 2 years old.

df3['Age'].hist(figsize=(8, 5), bins=50, xlabelsize=8, ylabelsize=8);

Can we do histograms of age by each year?

The order of the years is as follows;

2011	2012
2013	2014
2015	2016
2017	2018

df4 = df3.reset_index()
# plot all the histograms alligned, with same y axe limit, to ensure we can compare them
f, axes = plt.subplots(4, 2, figsize=(15, 20), sharex=True)

sns.distplot( df4['Age'].loc[df4['Date'] == '2011-01-01'] , color="skyblue", label="2011", ax=axes[0, 0]).set_ylim(0,0.06)
sns.distplot( df4['Age'].loc[df4['Date'] == '2012-01-01'] , color="blue", label="2012", ax=axes[0, 1]).set_ylim(0,0.06)
sns.distplot( df4['Age'].loc[df4['Date'] == '2013-01-01'] , color="purple", label="2013", ax=axes[1, 0]).set_ylim(0,0.06)
sns.distplot( df4['Age'].loc[df4['Date'] == '2014-01-01'] , color="darkred", label="2014", ax=axes[1, 1]).set_ylim(0,0.06)
sns.distplot( df4['Age'].loc[df4['Date'] == '2015-01-01'] , color="red", label="2015", ax=axes[2, 0]).set_ylim(0,0.06)
sns.distplot( df4['Age'].loc[df4['Date'] == '2016-01-01'] , color="chocolate", label="2016", ax=axes[2, 1]).set_ylim(0,0.06)
sns.distplot( df4['Age'].loc[df4['Date'] == '2017-01-01'] , color="darkorange", label="2017", ax=axes[3, 0]).set_ylim(0,0.06)
sns.distplot( df4['Age'].loc[df4['Date'] == '2018-01-01'] , color="gold", label="2018", ax=axes[3, 1]).set_ylim(0,0.06)

(0, 0.06)

Even here we can see that the 2011 have significantly different pattern from all the other years.

To sum up this issue, we know that:

the 2011 has more flows
the 2011 has les volume of people on flows
has very different pattern in both Age and gender
there might have been changes in the methodology, but we cannot find exactt description of the changes

This leads tho a decision to exclude the 2011 data in future analysis. Data with significantly different pattern without justified reasons would majorly affect the modelling and the results.

## Maps of Origins and destinations

# extract the long and lat from geometry into separate columns
points['lon'] = points.geometry.x
points['lat'] = points.geometry.y

# join the spatial points to the data, first for the OutLA and secondly for the InLA
outla = points.merge(df3, how = 'right', left_on = 'lad17cd', right_on = 'OutLA')
inla = points.merge(df3, how = 'right', left_on = 'lad17cd', right_on = 'InLA')

# Setup figure and axis
f, ax = plt.subplots(1, figsize=(9, 9))
# Add hexagon layer that displays count of points in each polygon
hb = ax.hexbin(outla.geometry.x, outla.geometry.y, gridsize=50, alpha=0.8, cmap='OrRd')
# Add title of the map
ax.set_title("Hex-bin heatmap of UK internal migration - Leaving the Local Authority")
# Add a colorbar (optional)
plt.colorbar(hb);

# Setup figure and axis
f, ax = plt.subplots(1, figsize=(9, 9))
# Add hexagon layer that displays count of points in each polygon
hb = ax.hexbin(inla.geometry.x, inla.geometry.y, gridsize=50, alpha=0.8, cmap='OrRd')
# Add title of the map
ax.set_title("Hex-bin heatmap of UK internal migration - Moving to Local Authority")
# Add a colorbar (optional)
plt.colorbar(hb);

<matplotlib.colorbar.Colorbar at 0x24c9186ccc8>

Although the heatmaps are quite easy maps to show, it is one of the worst choices one can make. You can read more about the heatmap issue here. In short, the issue is in the way our brains percieve the shading of the colors, this is also called the checker shadow illusion.

In Part 2 I'll look more closely at the spatial structure using R and start some modelling.

# for that I just need to export the data I munged here, to use them later.
# df3.to_csv(index = False , path_or_buf = './../data/migration/munged/full_migration.csv')

# points.to_file(driver = 'ESRI Shapefile', filename = './../data/migration/munged/LAcentrooids.shp')