Decarbonising Existing Homes in Wales: A Participatory Behavioural Systems Mapping Approach

Reviewer's comment

Authors' response

The abstract could hint more at the connection of behavioural and systems worldviews – it reads as if these are natural companions, but to me it is more intuitive to think of them as alternatives (e.g. many systems thinkers are critical of the narrow focus of behavioural approaches). This comes out in the introduction a bit, but is missing in the abstract.

We consider both approaches natural companions, so we had neglected this important view. We have made additions to the abstract to highlight that these are often seen as alternatives and that our approach allows for considering behavioural determinants in a systemic way (p. 2).

Introduction: I would ideally like more discussion of the tension between behaviour and systems approaches – the critiques they make of each other, and whether they can fit neatly together. I think they can, but it warrants more consideration.

We have briefly highlighted the perceived tension between these approaches in the introduction (p. 3), and we have commented on their fit in the discussion (p. 37). As our paper focuses on a way of addressing these perceived tensions using a particular systems approach (CLDs) and a particular behavioural framework (the Behaviour Change Wheel) it feels beyond the scope to give a comprehensive discussion of the critiques that systems and behavioural literature make of each other, which we think may sometimes reflect narrow views of the fields. We therefore point to others’ discussions in the introduction, particularly: Chater, N., & Loewenstein, G. (2022). The i-Frame and the s-Frame: How Focusing on Individual-Level Solutions Has Led Behavioral Public Policy Astray (SSRN Scholarly Paper No. 4046264). Social Science Research Network.https://doi.org/10.2139/ssrn.4046264

It would be helpful to acknowledge the distinction between complexity approaches and systems thinking here I think. ABM in particular, and arguably network analysis too, come more from a complexity science/theory angle than systems thinking. I think it is fine to lump them together in general, but I would acknowledge this distinction here.

We agree it is helpful to make the distinctions and have added this (p. 6).

There is some unfortunate duplication of the use of the term ‘Participatory Systems Mapping’ in the literature – it would be helpful to explain the difference between Artunes et al and Barbrook-Johnson and Penn more clearly. My understanding is that Artunes et al is causal loop diagrams, whereas Barbrook-Johnson and Penn is more like fuzzy cognitive mapping but using network analysis for analysis, not turning the map into a simulation. The paper skims over these nuances too quickly.

We agree this could be clearer and have pointed out this difference (pp. 7-8). We have also clarified that we use the term ‘participatory systems mapping’ to mean involving stakeholders in the process of building and using any kind of systems map; we would not restrict behavioural systems mapping to only causal loop diagrams.

It would be helpful to make clearer the real difference between (i) the method used/developed here and (ii) annotated versions of causal loop diagrams or the PSM proposed by Barbrook-Johnson and Penn. In the early text of the paper, it seems like there is a big difference, but it is a little vague, then when you see the maps in results, it looks like an annotated causal loop diagram. More commentary on how this is helpful, but also the issues it introduces would be beneficial too.

We also consider the maps produced here as similar to annotated causal loop diagrams. We have clarified that the main difference is in guiding and specifying what type of information should be usefully represented in the systems map, rather than how the information is represented (p. 8)

Was any analysis of the maps done – how were the maps interpreted with and without stakeholders?

We used visual inspection of the maps to identify feedback loops and variables which might influence relatively many others, but we did not carry out quantitative analyses on the maps. We have added some details about how the maps were interpreted with and without stakeholders (p. 17).

The researchers take a lot of control of the process, refining and merging maps, and asking stakeholders to build simple versions. Could you comment on any bias or shifts in focus this brought, if any?

The main focus that the researchers (JH & PC) brought to the mapping process was to direct participants’ attention to actors, behaviours and influences on behaviour. Apart from this, our role was to represent the ideas of the advisory group as accurately as possible. Since  we had no prior knowledge of the retrofit system, we felt that any bias that would be introduced would we diffuse and distributed throughout the map, rather than driven by a particular viewpoint on retrofit. We have added this consideration to the discussion section (p. 35) and added a clarification to the methods section (p. 16).

Comment

Response

Expanding on the level of validity of the study: I would be interested in more analysis and critique of the composition of the Advisory Group, whose knowledge was core to the maps produced, and therefore to the policy recommendations.  Does the advisory group represent all stakeholders?  Or rather all domains of expertise thought relevant?

The Advisory Group was established by the Minister for Housing and Regeneration. It represented 33 organisations which covered a broad range of knowledge areas, but did not include all possible stakeholders or relevant domains of expertise. We were not able to influence the composition of the group. We have mentioned in the discussion section that a different set of stakeholders might have arrived at different maps and recommendations (pp. 36-37), and we have added better signposting to the Methods section to help the reader find the details of the organisations represented (p. 11).

Expanding on the level of completeness of the study: I believe that the most significant contribution of the research is methodological, but I think that the case for retrofit as a vital area which demands such methodological innovation would have been strengthened by more reference  to the technical requirements of retrofit (e,g, CCC 2019 “UK Housing: fit for the future”, CLC 2021 “National Retrofit Strategy v2.0) and a recognition that this can be measure by measure, whole house, or over time (e.g. FAWCETT, T. 2013. Exploring the time dimension of low carbon retrofit: owner-occupied housing. Building Research & Information, 42, 477-488.)

Thank you for these helpful suggestions. Although our focus is on the methodological contributions, we agree retrofit is a vital area for innovation. We have referred to these reports in the introduction to strengthen the framing of the challenge (pp. 4-5). We have also highlighted in the discussion section that a valuable future direction would be to explore in more detail the technical roll-out of retrofit measures in different temporal scenarios (p. 37).

While I fully accept that owner occupiers, private landlords and Welsh policy makers are the major actors whose behaviour is being analysed, I think that putting the whole of the supply chain into a single box in the prototype systems diagram, labelled ‘suppliers’ is open to some challenge and could be further refined in future work.  Materials and equipment manufacturers have different behaviours to merchants (e.g. Killip G, Owen A, Topouzi M. 2020. Exploring the practices and roles of UK construction manufacturers and merchants in relation to housing energy retrofit. Journal of Cleaner Production. 251), who in turn have different behaviours to the construction firms working in RMI, which are the main focus of the research cited at the moment. 

We recognise that some areas of the systems maps would benefit from greater unpacking and elaboration, and have highlighted as an example these actors within the supply chain (p. 37). This may have been partly influenced by the composition of the Advisory Group, which had relatively little representation of the supply chain compared to other areas.

The understanding the sociotechnical nature of the retrofit delivery system is set out well.  Might the problem statement be strengthened further by returning to some of the core frameworks in STS – such as the multi-level perspective or the co-evolutionary systems?

As the purpose of the research was not to test or compare these frameworks and they did not specifically guide our development of behavioural systems mapping, we have chosen not to introduce these in what is already a long and framework-heavy introduction.

I strongly support the need for methodological innovation in this area of research/practice, and we therefore need to explore how transferable are new methodologies.  In this case, I am keen to know how are the results obtained using this method validated?  On page 18 a short statement says that the workshop results were validated by two of the authors.  Is this rigorous enough to allow the research findings to be generalizable?  How was this validation undertaken? The advisory group are an expert group; their perspectives and their depth of knowledge are very important.  And yet, to achieve changes in behaviour, other perspectives need to be represented.  This challenge is acknowledged on p37 : “Several rounds of review and refinement helped us to be confident in the validity of the maps from [the advisory group’s] perspective as stakeholders, although it is possible that the map would look different had different stakeholders been involved in each workshop.” I think this is worth further consideration and testing in future rounds of mapping.

The method presented here was one of our first exploratory attempts at behavioural systems mapping and subject to practical constraints. As such a more detailed validation against other perspectives was not possible within the timeframe. We fully agree that this part of the process is deserving of greater attention as behavioural systems mapping is developed and refined. We have extended our comment in the discussion section (p. 37) and highlighted this area for development in the future directions (p. 38).

I also wondered if we needed a simple definition of the behaviours that would lead to decarbonising homes – and how these behaviours might differ for different actors.  The team has clearly mapped out these behaviours in some detail in Appendix B but they are not described in the body of the paper.  The list in Appendix B1 covers a huge variety of behaviours, with different loci and impacts.  Some categorisation of these, perhaps in terms of how they relate to the prototype system map, might be helpful?  By describing the technical tasks of retrofit, but not the range of behaviours needed, the balance between C, O and M in policy measures risk becoming something of a pick’n’mix menu for policymakers, rather than a coherent and consistent package …  p28 mentions the “desired behaviour change of owner occupiers and private landlords” but this could cover a variety of behaviours including funding retrofit, commissioning retrofit, using a building renovation passport etc.  On p38 then it seems that other organisational behaviours are also within scope.

We listed the behaviours in Appendix B for ease of reference, and in the behavioural systems map diagrams these are organised more clearly in terms of sectors through colour-coding. The primary purpose of our paper is to share the behavioural systems mapping methodology, rather than point to specific behaviours to be targeted in retrofit policy. Therefore, defining the behaviours was not part of our process (since the meanings were known by the participants) and it would not be feasible to obtain the Advisory Group’s consensus around definitions at this stage.

We would hesitate to pick out and define a particular sub-set of behaviours that would lead to decarbonising homes, because we believe the value of mapping these is to help policy-makers consider their interrelatedness and select suitable policy measures on the basis of potential causal impacts, so as to discourage a less targeted pick’n’mix approach.

10 headline recommendations are referred to on page 19, but 26 policy recommendations are identified in the results.  I understand that the 26 were generated by workshops which used the initial 10 headlines as stimulus.  Some critical discussion of the role of these headline recommendations would be valuable.  If a different mix of headline recommendations were used as the stimulus, how might this have changed the longer list generated through workshops?  I think that this might lead to some reflections on the completeness of the set of recommendations developed.

In this paper we focus on the role of behavioural systems mapping in the process of developing recommendations. It was not our intention or expectation that the mapping process should lead to a fully complete list of recommendations from a behavioural perspective, and we did not seek to test this. Other types of input besides the mapping went into the development of recommendations and they were finalised after the involvement of the researchers. In the methods section (pp. 18-19) we have adjusted the wording about ‘headline recommendations’, which had been used in two different instances (to describe the stimulus and also the ideas generated in the workshop) to aid clarity.

I’m also intrigued to see that in both the headline recommendations and the complete list, some recommendations do not connect to C, O or M.  These are lobbying recommendations.  One what basis are they justified if there is not a connection to the desired behaviours?  Or is there a secondary connection that lobbying would create WG opportunities, that would then translate into owner occupier or landlord opportunities?

We have revisited the COM-B mapping and made some updates. In the headline recommendations we have updated that number 7 (make loans conditional and information available) would alter motivation and capability, and that number 10 (lobbying) would alter social opportunity (p. 30). In the final recommendations we have updated that 1.3 and 2.1 (which both refer to improved standards) would alter opportunity, since this would involve new rules (p. 31).

The Advisory Group’s process of developing the final recommendations did not require that each recommendation had to be justified on the basis of connecting to COM-B. As researchers we also do not see this as a necessary requirement, since valuable recommendations could be directed at a behaviour or outcome rather than at modifying influences on behaviour. The purposes of our COM-B analysis were to prompt and challenge participants during workshop 2 and to consider the behavioural relevance of the final recommendations, rather than to judge the value or justification of any recommendations.