Case Studies

00

Introduction

Proposing a framework to evaluate IoT deployments against the UN Sustainable Development Goals

Objectives

The World Economic Forum’s IoT for Sustainable Development project aims to encourage the use of the Internet of Things (IoT) to accelerate progress on the 17 United Nations Sustainable Development Goals (SDGs). To achieve this goal, the project explores scalable and replicable models of business, investment and collaboration across industries and with public authorities to support the design of commercially viable IoT deployments that can maximize their social value potential.

The first phase of the project focuses on the following questions:

  1. Is there a way to compare the impact of IoT projects across different sustainable development areas?
  2. Do IoT projects support or have the potential to support the SDGs? If yes, how do they map against the 17 SDGs?
  3. What are some of the inspiring or most illustrative IoT projects addressing the SDGs, and how can their impact be compared against each other?

In an attempt to start answering these questions, the project proposes:

  • A framework for a preliminary evaluation of the impact of IoT projects across sustainable development areas.
  • A map of IoT projects that support or have the potential to support the SDGs according to this framework
  • The application of the framework to case studies to help visualize how they address the SDGs and how they could compare with each other

A second part of the initiative will build on these results and produce a set of guidelines on scalable and replicable models for business, investment and collaboration for sustainable IoT deployment strategies. These guidelines will be published at a later stage.

The Framework

The challenge

The main challenge while developing the framework to determine the impact of IoT projects on the SDGs was to find a way to compare projects on a “like-for-like” basis. This is difficult to achieve given that projects vary in purpose, geography, size, focus area, etc.

Disclaimer: The framework is but one way of looking at IoT projects and their impact; it does not claim to be the most comprehensive or the most illustrative and it focuses only on the most prominent SDG being addressed by a given project. There might be better ways to assess Internet of Things for Development (IoT4D) impact developed by other organizations; however, they do not appear to be easily accessible in the public domain.

SDGs as a reference for sustainable development

The UN SDGs are used as reference to classify the full spectrum of sustainable development areas for the planet. There are other ways to classify and think about sustainable development; however, the SDGs appear to be the closest to a common classification system widely adopted across the public and private sectors.

In total, there are 17 SDGs with 169 targets. Twenty of the targets are quantitative in nature (e.g. a call to reduce the global maternal mortality ratio to less than 70 per 100k live births). The rest are more qualitative (e.g. how to increase substantially the share of renewable energy in the global energy mix).

KPIs based framework

To solve the above challenge, a five-point KPI-based framework was developed. Each of the KPIs is scored on a scale of 1 to 5 using criteria detailed below.

KPIs to evaluate a project
KPI Applied on Display
1Scale of projects
Determine today’s project reach from micro to macro level by scoring the number of individuals impacted, geography reach, and usage reach
640+ projects from IoT-analytics database Map of IoT deployments with potential to support SDGs
2Targets penetration
Determine the percentage of SDG targets that the project benefit touches
640+ projects from IoT-analytics database Map of IoT deployments with potential to support SDGs
3Influence on targets
Determine how much the project outcome can influence an individual target within an SDG
5 illustrative examples KPI framework applied on Illustrative IoT4D examples
4Scalability and replicability potential
Determine to what extent a project is structurally scalable (same target) or replicable (across targets, SDGs)
5 illustrative examples KPI framework applied on Illustrative IoT4D examples
5Focus on vulnerable groups
Determine whether a project has a significant focus on vulnerable, under developed or underserved groups
5 illustrative examples KPI framework applied on Illustrative IoT4D examples

Acknowledgements

This analysis has been conducted by the World Economic Forum in collaboration with IoT-Analytics GmbH under the guidance of a working group. We would like to thank Knud Lasse Lueth and Abhay Rastogi at IoT-Analytics GmbH for their efforts as well as the members of the working group formed by:

Danielle Osler, Google; Edmund DiSanto, American Tower; Ivan Huang, Huawei; Julian Ashworth, BT; Jeffery Torrance, Qualcomm; Karl Bream, Nokia; Mani Manimohan, GSMA; Michael Zeto, AT&T; Mikael Back, Ericsson and Stephen Collins, VEON with additional support from Ann Ewasechko, HPE; Ariel Brassil, AT&T; Becca Gould and Sanjay Aggarwal, American Tower; Manuel Kohnstamm, Liberty Global; Matilda Gustafsson, Ericsson; Roland Klemann, Cisco; and Sam Clitus, Infosys.

We would also like to thank the different individuals and organizations involved in the case studies complementing this framework (please see full list of acknowledgements for each case study under the “Case Studies” section).

We also acknowledge the contribution of subject matter experts Daniel Obodovski, The Silent Intelligence; Gustaf Landahl, City of Stockholm; Karl-Filip Coenegracht, City of Ghent and Michael Mulquin, TM Forum.

This analysis is part of the World Economic Forum’s IoT for Sustainable Development project led by Rodrigo Arias, Content Lead, Technology, Media and Digital Industries, and a Global Leadership Fellow at the World Economic Forum.

01

Scale of projects

Determine today’s project reach from micro to macro level by scoring the number of individuals impacted, geography reach, and usage reach

This KPI aims to determine the current scale of project impact. This is done by assessing the current reach on three dimensions: number of individuals impacted; the geographical area of operation; and usage across sectors or industries. Each of these dimensions is scored on a five-point scale: micro (1), small (2), medium (3), large (4) and macro (5). The KPI score is the highest score among the three dimensions

Scale of Impact Individuals Usage Geography
Macro > 10M Across multiple sectors Worldwide
Large 1M – 10M Across industry, value chain or sector Pan continent
Medium 100K – 1M Multiple generic products or projects Country or Countries
Small 10K – 100K Multiple specific products or projects City or region
Micro < 10K Single product or project Community
Uses best of 3 dimensions
02

Target penetration

Determine the percentage of SDG targets that the project benefit touches

This KPI measures how many of the SDG targets are currently “touched” by the project impact, i.e. how many of the targets under a given SDG are likely to be directly affected by the project impact. The number of targets touched are then converted to a percentage (% = targets touched/total number of targets under a SDG) and ranked on a five-point scale: 0-20% (1), 20%-40% (2), 40%-60% (3), 60%-80% (4) and >80%(5).

% of SDG targets project touches converted to a 5 point scale
x Criteria Explanation
5 >80% Project touches more than 80% of targets within SDG
4 60%–80% Project touches between 60%-80% of targets within SDG
3 40%–60% Project touches between 40%-60% of targets within SDG
2 20%–40% Project touches between 20%-40% of targets within SDG
1 <20% Project touches less than 20% of targets within SDG
03

Influence on targets

Determine how much the project outcome can influence an individual target within SDG

This KPI aims to determine the potential effectiveness of a project towards achieving a particular target (i.e. whether the project is likely to achieve x% or y% of the target) under a SDG. For quantitative targets, the score is determined by answering the following question: For a given scale and scope, what % of target can the project achieve? The score is on a scale from one to five: <1% (1), 1%-10% (2), 10%-33% (3), 33-66% (4) or >66% (5).

The score is not linearly distributed. This is to reflect the fact that targets under the SDGs are very ambitious and hence the scale needs to be realistic. For example, even a 10%+ influence on a target is a significant achievement and deserves a score of 3; a 2/3 influence (>66%) deserves a score of 5.

For qualitative targets, the score is determined by experts responding to the following question: For a given scale and scope, what is the likely project impact on the target? The score is on a scale from one to five: nominal (1), limited (2), material (3), significant (4), or very significant (5).

Quantitative targets

e.g. By 2030, reduce the global maternal mortality ratio (MMR) to less than 70 per 100,000 live births.

Score Criteria Explanation
For a given scale and scope what % of target can the project achieve?
5 >66% Greater than 66%
e.g. initial MMR 100; target MMR < 80.2
4 33%–66% Between 33% and 66%
e.g. initial MMR 100; 80.2 < target MMR < 90.1
3 10%–33% Between 10% and 33%
e.g. initial MMR 100; 90.1< target MMR < 97
2 1%–10% Between 1% and 10%
e.g. initial MMR 100; 97< target MMR < 99
1 <1% Less than 1%
e.g. initial MMR 100; 99< target MMR < 100

Qualitative targets

e.g. By 2030, enhance international cooperation to facilitate access to clean energy research and technology

Score Criteria Explanation
For a given scale and scope what is the likely project impact on target?
5 Very significant For a given scale and scope what is the likely project impact on target?
4 Significant Very significant impact on targets; project is probably designed to achieve the targets
3 Material Material impact on targets; probably amongst top 3 focus areas for the project
2 Limited Limited impact on targets; project touches but probably not designed for the target
1 Nominal Nominal impact, project touches but does not influence the targets
04

Scalability and replicability

Determine to what extent a project is structurally scalable (same target) or replicable (across targets, SDGs)

This KPI aims to answer the following question: Can more benefits be structurally reaped from a project? More benefits can be reaped by either scaling up the project or replicating the setup for a different environment (targets or SDGs).

A score is provided by determining to what extent a project is structurally scalable (same target/same SDG) or replicable (different targets/same SDG, different targets/different SDGs).

Scalability: How scalable is the project in terms of increasing the number of individuals impacted, the geographical reach and usage reach for a given target and SDG? Score: none (1), low (2), medium (3), high (4), or very high (5).

Replicability: Estimate the ease with which the project can be replicated to achieve other targets in same or different SDGs. Score: none (1), low (2), medium (3), high (4), or very high (5).

As both scalability and replicability measure in some form whether more benefits can be structurally reaped from an existing project, the final KPI4 score is a higher of the two scores.

Scalability

How scalable (number of individuals impacted, geographical and usage reach) is the project for achieving individual SDG target?

Score Label Explanation
Estimates ease of scaling in terms of i. # individuals ii. geography iii. usage reach
5 Very high Project in its current form can easily be scaled up in all 3 dimensions
4 High Project in its current form can easily be scaled up in 2 dimensions
3 Medium Project in its current form can be scaled up in at least 1 dimension
2 Low Project would require significant structural changes and / or resources to scale up
1 None Unique project, cannot be scaled up in current form

Replicability

How replicable is the project for achieving targets in same or other SDGs?

Score Label Explanation
Estimate ease of replication to achieve other targets in same or different SDGs
5 Very high Project in its current form can easily be replicated to achieve 3 or more targets
4 High Project in its current form can easily be replicated to achieve 2 targets
3 Medium Project in its current form can easily be replicated to achieve at least 1 other target
2 Low Project would require significant structural changes / resources to be replicated
1 None Specific project, cannot be replicated in current form
05

Focus on vulnerable groups

Determine whether a project has a significant focus on vulnerable, under developed or underserved groups

This KPI determines whether a project has a significant focus on vulnerable, underdeveloped or underserved groups. Projects with extensive focus on vulnerable groups would score high on this KPI.

Certain public and private entities may be especially interested in these high-scoring projects.

There are many socioeconomic indicators to identify vulnerable groups. In this framework, any of 12 national performance indicators could be used to ascertain the score. The indicators are split into three categories: growth and development (GDP per capita, labour productivity, employment %, healthy life expectancy); inclusion (median household income, wealth Gini, income Gini, poverty rate/% below minimum); and intergenerational equity and sustainability (adjusted net saving, dependency ratio, public debt, carbon intensity of GDP).

The score is calculated using a three-step process:

  1. Identify which of the 12 indicators is the most relevant to use for the project under consideration
  2. Create a normalized scale for the chosen indicator (minimum 0, maximum 1)
  3. Plot where on the scale the project beneficiaries are likely to fall and read the “vulnerable group focus score” from the left of the chart

The score is ranked on a scale from one to five: nominal (1), limited (2), material (3), significant (4), or very significant (5).

Framework

Application of the Framework

Application of framework on 640+ projects in the IoT database

The framework was applied to an IoT analytics database of 640+ IoT projects to answer the question: Do IoT projects support or have potential to support the SDGs? If yes, how do they map against the 17 UN SDGs.

Choice of database and KPIs

The IoT-Analytics database provided a handy compilation of over 640 worldwide IoT deployments, with enough information to apply KPI1 (scale of project) and KPI 2 (target penetration). An alternative approach would have been to search for a list of IoT projects from the internet or extended network; the approach was not considered viable in the study timeframe. The database did not have information to apply KPIs 3-5 with reasonable confidence, so it was decided to illustrate the application of the framework using four inspiring examples.

Results

The results from the exercise above were plotted against the 17 UN Sustainable Development Goals to create a map of IoT4D projects which support or have the potential to support the SDGs.

  • 84% of the 643 analysed IoT deployments can address the SDGs
  • 75% of these focus on five SDGs; (25%) SDG 9 (industry, innovation and infrastructure); (19%) SDG 11 (sustainable cities and communities); (19%) SDG 7 (affordable and clean energy); (7%) SDG 3 (good health and well-being); (5%) SDG 12 (responsible production and consumption)
  • 95% of projects are small/medium sized; 70% are led by private sector and 80% originate in the Americas and Europe.
  • Enhanced focus is seen on SDG 11 and SDG 16 when mapping projects by a secondary SDG (i.e. an additional SDG on top of the main focus area)

KPI 1: Scale of projects

Regional Analysis of SDG IoT Projects

Corporation Models of SDG IoT Projects

Enhanced focus on SDG 11 and 16 when mapping projects by secondary SDG

KPI 2: Target penetration