Time for Sensemaking 3.0? The potential of AI-powered portfolio analytics to drive impact
By Enrico Gaveglia and Shumin Liu
The Regional Innovation Centre of UNDP Bangkok Regional Hub and the UNDP Philippines together with data science company Dataverz have explored the use of artificial intelligence (AI) to improve the UNDP sensemaking process. This is part of the Regional Innovation Centre's work to continuously evolve sensemaking to meet the needs of the UNDP Country Offices so as to fine-tuning its process to accommodate a post-pandemic digital imperative as well as to complement it with foresight disciplines.
Where is the idea coming from?
Two years ago, the Regional Innovation Centre introduced a portfolio sensemaking process which has now been adopted by the UNDP across the Asia Pacific region. The Philippines Country office (CO) as one of the first adopters of this work has now institutionalized sensemaking as a whole-of-office effort. The sensemaking process involves the Country Office reflecting on the portfolio of projects as a whole and helping project teams to surface common patterns (recurring features from one project to another), identify connections and gaps, and facilitating collective decision-making on how those gaps can be addressed. This process enables the office to get a better grasp of the strategic direction they are heading in as a team and understand the changing external environment to ensure they can adapt and pivot their programmes.
We asked UNDP Philippines Country office senior management what made them believe sensemaking would have been a viable option in support to their office: “the main work of any lead is to uplift the collective consciousness of our staff to a sense of shared purpose. Sensemaking has opened a gate of meaning stolen from the many practices that dumb us towards our work. We made sensemaking a regular practice and a CO self-discipline twice a year. The ambition is to mature it into a constant and distinct trait of character that can make us simply better development professionals” said Enrico Gaveglia, DRR at UNDP Philippines.
Making sense of rich information is not an easy task. It requires an ability to actively listen, deal with the complex data coming from multiple sources, and abstract this information into meaningful patterns and insights. To enable this, we use structured project sharing, questioning, and mapping as well as active “balcony” listening to provide a higher-level overview and pattern mapping.
This process brings significant value in terms of collective reflection, as well as, bringing diverse perspectives by leveraging cross-program and cross-functional expertise. However, from the experience of two years running in offices across the region, we have noted some constraints of this process. This was particularly visible when there are a significant number of projects that cannot all be embraced in the run of the protocol because offices have limited time to do this work. The latter has become particularly relevant if you dedicate many hours to Zoom it can lead to significant fatigue. These constraints have pushed us to explore the possibility of visualizing the patterns and connections within the rich but unstructured data that is already captured in our project documents — this has become an aha moment for us! Just like what Marcel Proust once said, “The real voyage of discovery consists not in seeking new landscapes, but in having new eyes.”, we are enabling the new eye of AI as a complementary angle to delve deeper into our portfolio.
The reaction of the Philippine CO was positive yet cautious: “As you know I am not exactly a fan of an artificial synapse snooping around our programme — (Ref. Give me AI break, blog) but if you manage to optimize our sensemaking craft by elevating the process into an experience that concentrates on quality and less on the basic ask of self-recognition, we are all in!” Enrico Gaveglia, DRR at UNDP Philippines.
What did we do?
We were happy to find an ambitious Country Office who wanted to develop the use of artificial intelligence (Natural Language Processing, Machine Learning, and Network Analysis) to help make sense of our project portfolios in order to make the sensemaking process more meaningful and more impactful. Our primary hypothesis is that different types of data from various sources such as UNDP Project Documents, Annual Progress Reports, Annual Work Plans, and open.undp.org can be extracted and interlinked for the analysis on pattern recognition and connectivity.
Our experiment runs as an adaptive and iterative process with several sprints. With the help of AI applications, we have extracted semi-structured data (from open.undp.org) for ten selected projects that were used in the recent sensemaking workshop to prototype the patterns and connections based on the thematic areas of development challenges. Here is what we have learned so far:
- Connecting silos: Each UNDP project is associated with the self-reported SDG (Sustainable Development Goals) markers. The data visualization on the left-hand side shows projects connected uniquely based on their reported SDGs. This naturally creates silos because from that perspective you are either relevant to the SDGs or you are not.
In contrast, if we visualise the projects using connections that are based on the terms and topics extracted from rich unstructured data (right-hand side), we see connections emerging between the silos. This shows the highly interlinking nature of the language used between the projects — hinting at the hidden connections underneath the surface and provides a different understanding of SDG relevance. For example, Adaptable Digitally-Enabled Post-crisis Transformation is linking with SDG 1 no poverty, SDG 13 climate action, and SDG 17 partnerships for the goals, while the network map also draws a new connection with e-governance. This enables us to find new opportunities within our current assets for team and project configuration, for fundraising, and for delivering impact in a systematic way by developing new and better programmes.
- Asking different questions: We are aware of the bias that might be introduced by this data that cannot fully represent reality. So, the assumption here is not about presenting the “truth” based on the evidence, rather, using it as an engagement tool to spur interaction between people and machine where we collectively reflect on it, validate it, or challenge it — Are these connections happening and how do they play out? If no, why and is it possible and valuable to foster that? How do we spark off the connections for better coherence, coordination, and collaboration between our teams?
- Providing actionable intelligence: We are experimenting with a way to embed portfolio analytics in the sensemaking process that yields actionable intelligence for decision making. For instance, one of the propositions from the recent sensemaking workshop is about integrating the data offering of all projects in the Country Office. Two projects — DELVELIVE+ and Pintig Lab which have data expertise and portfolios appear to be the obvious ones to engage with. Our next question is with which project teams or which thematic areas to start. Using the Pintig Lab as an example, we can see areas like recovery and resilience building, democratic governance programme, and localizing e-government could be new territories to explore (if they are not yet) as shown by the specific node of the network map. It gives a sense of possible directions to delve deeper into the relationship between projects and identify entry points for further collaboration and learning.
While there is a technical side of analytics we are experimenting with, there is also a deeper level of thinking we are keen to explore:
- Will the insights from data-driven portfolio analytics be able to complement the human sensemaking process with new ways to observe new types of dynamics in the portfolio?
- Will different modes of analysis (qualitative and quantitative) reveal similar or different patterns? Can this approach bring new intelligence about our country program?
- Will the data-driven insights be able to support the generation of actionable intelligence and facilitate the decision-making by the office?
- Will this added intelligence lead to better strategy and more impact for UNDP and for the people we seek to serve?
Philippine Country Office management first thoughts are teasing us towards a series of next steps: “What we discovered in the Philippines with the use of Ai in our protocol is that by colliding verbal data between UNDP Project Documents, Annual Progress Reports, Annual Work Plans, and open.undp.org we have uncovered some limits of our programme offer if measured in the realm of semantic. This is unchartered territory and a long-awaited and much welcome challenge. We are used to mastering data science and its use in an ocean of possible computations, but quite frankly we have yet to start anything of the sort in the logical aspects of our lexicon or how words and meanings relate to each other in different contexts. Ultimately our goal is to elevate our cognitive experience in development as close as possible to the reality out there. If AI can facilitate it by enhancing sophistication in an iterative process between what we told others we would do and what we really did then let's push the sensemaking to a 3.0 protocol” Enrico Gaveglia, DRR at UNDP Philippines.
Next steps
We are now building our second prototype, which will use data extracted from the documents of ongoing projects from 2019 to 2021 and expanding the dimensions to include development challenges, interventions, approaches, partnerships, outcome measurement, learning, and gender lens. We are also making plans to integrate this into the UNDP Sensemaking process.
As Deborah Ancona said, “Sensemaking is not about finding the “correct” answer; it is about creating an emerging picture that becomes more comprehensive through data collection, action, experience, and conversation.” So, we don’t mean to prove something with data, rather, we aim to create a space to build the collective intelligence of the portfolio together, improve the effectiveness of the portfolio by reducing its opacity, capitalize on discovered synergies between projects, and induce new cultural dynamics of how our offices are leveraging their operational or knowledge assets.
If you are interested in knowing more about our approach to portfolio sensemaking, please kindly find more here and here.
This blog is written by Enrico Gaveglia, Deputy Resident Representative at UNDP Philippines, and Shumin Liu, Data and Policy Consultant, Regional Innovation Centre of UNDP Bangkok Regional Hub. We will keep you posted on our findings and in the meantime, should you be interested you can share your thoughts with us on Twitter @enrico_gaveglia and @SL_ShuminLiu.
With enormous thanks to Alex Oprunenco, Giulio Quaggiotto, Marian Theresia Valera Co, Kate Sutton, Prateeksha Singh, and Pedro Parraguez Ruiz for being part of the journey and sharing their valuable inputs for this work. We also appreciate the participation and support from the colleagues of the Technical Working Group in the UNDP Philippines.
