Enabling interdisciplinary research capacity for sustainable development: Self- evaluation of the Blue Communities project in the UK and Southeast Asia

Reviewer Comment

Response

Review #1 Washbourne (2023)

Title: I am not convinced that the title fully does justice to the scope and nature of the article. I would be tempted to switch out ‘Growing’ for a different term that sounds more purposeful / intentional like ‘Enabling’ or ‘Supporting’. While I realise it makes it a lot longer, the sub-title could be rephrased to better capture 1) the nature of the evaluation 2) the link with the project 3) maybe the counties involved e.g. : self-evaluation of the ‘Blue Communities’ project [in the UK and Southeast Asia]

The title has been changed to reflect the reviewer’s comments. The new title better reflects the content of the paper

ODA funding might need further explanation, or perhaps don’t include in the abstract and only in the body text

This term was removed from the abstract and is explained further in the body of the text in Footnote #1.

Line 50: ‘Results were mainly positive’ is too vague here. What do you mean by ‘positive’?

This sentence was expanded to make the positive relationship more explicit. The text now reads: “Greatest improvement was seen at the self level where results indicated a positive relationship between an individual’s current success or skill and their improvement over the course of the research project across 18 out of 22 aspects of research capacity for Southeast Asian, and 2 for UK respondents.” Line:18-21

Is it possible to include any of the more detailed insights / findings?

As above, further details on the findings have been included in Lines 18-21.

Line 63: Why was the UoL 2026 strategy highlighted here? It is a good example, but feels a little arbitrary as so many institutions have similar strategies. Was this one particularly ground-breaking?

This has been edited to make more general to institutions, such as universities (now Line 33)

Line 71 / 72: Consistency with the terms ‘capacity strengthening’ and ‘capacity building’. Make it clear if there is an intended difference between the two uses here and throughout the paper.

This has been edited to show that these terms are used interchangeably throughout although sometimes with an emphasis on building from zero capability and sometimes strengthening what is there already (Line 36-38)

Line 94 / 95: It would be good to have a footnote with a bit more detail about the nature of GCRF. Especially if the reference to ODA funding is retained in the abstract.

A footnote has been added with more explanation on GCRF; Footnote #1 reads: “The Global Challenges Research Fund (GCRF) is a UK fund that promotes achievement of the UN Sustainable Development Goals in developing countries, through supporting international research. It is part of the UK’s Official Development Assistance (ODA) programme that aims to promote sustainable growth of OECD (Organisation for Economic Cooperation and Development) selected developing countries. https://royalsociety.org/~/media/grants/schemes/ODA-GCRF.pdf?la=en-GB&hash=B51F1E2140346184856E2F87D6F4B32A”

Line 106: As above re: GCRF. It would be good to be a bit more explicit about the way in which the nature of this funding influence the scope and approach of the project

As above, see Footnote #1

Line 114: Were these funding calls based on the redistribution of funds already won for the project, or was this something in addition?

Now Line 87-88 This text has been edited to make clear that these funds were set aside from the main project grant.

Line 140: More detail about the survey is needed here. Who developed the survey (Author FC is stated in the author contributions, were their other contributors)? Who distributed the survey and how (via email to the project members, during meetings, through newsletters, via social media etc.)? How long was the survey open for / when did you decide how to close the survey?

More detail has been added to the methods to describe additional details on all aspects of the survey process highlighted by the reviewer. See Section 2.1, Lines 113-134 (not copied here)

Line 143: Explain more here how you defined the different career stages (shown in Section 1 of the survey). You do come back to this later in the paper, but it would be helpful to have a brief sense here of what the categories were and how you chose to define these stages.

More detail has been added to the methods section to describe additional details about how the career stage categories were chosen. See Lines 124-126 which read “Researchers within the project ranged from those with little research experience to those with long careers in research, and categories in the survey were chosen to capture all of these career stages.”

Line 184: What kinds of ‘other’ institutions were represented?

‘Other’ institutions consisted only of government agencies from this study’s respondents. This has now been specified. See Table 2

Table 2: Check final formatting of table, as the separation between the definitions in the final row could be made clearer

The table has been edited to distinguish these rows more clearly

Results: Did you use any more detailed statistical approaches to explore the correlations and differences between responses? This strikes me as being especially useful in the case of the data currently presented in Figures 6-9. These are really nicely illustrative, but a further exploration of the correlation would be very instructive.

Further analyses have been carried out, including Fishers exact tests to look for associations between variables, and Spearman rank correlations to explore the relationship between the current success or skill and the degree of improvement. These analyses make the results much clearer and more informative. See Section 2.2 of Methods and Results sections

Figure 1, 2, 3a, 4a, 5a: Reconsider the colour scheme used here. The ‘UK and other European’ category would benefit from being recoloured in something more contrasting  

The colour scheme has now been changed to a gradient as suggested in the point below.

Figure 3b, 4b, 5b: Reconsider the colour scheme used here. It would be better to use a scheme more clearly distinct from the accompanying charts to the left.  Perhaps it would be better onscreen to use a gradient based on one colour for the categories in (a) and a different colour for (b).

The colour scheme has now been changed to a gradient across all of these figures

Figures 6-9: Why are the categories in reverse order top to bottom? (i.e. ending with A rather than starting)

These have been edited for Figure 6 to read in the right order. Figures 7-9 have been replaced with a different figure format.

Line 251: See also comments below re: creating a separate section for ‘Limitations of the study’

A new section, Section 4.3 on limitations has been added and this text has been moved there

Figures 7-9: For quick visual communication of the results, I would advise keep the x axis consistent on all charts even where there is no data (e.g. some of the (b) charts showing the difference in response)

Figures 7-9 have now been replaced with a different figure format.

Line 429-430: By ‘this study’ you are referring to the project overall, rather that this manuscript?

This sentence has been clarified in the text to read: “lead authorship in BC the project” Line 443

Line 491-492: I think it is fine to be more definite here. There is plenty of evidence that the value placed on different aspects of the University differs from place to place.

The text has been edited to make more definite, it now reads: “the values in UK universities differ from those that may be found in other cultures” Line 506

Line 503-504: Not absolutely necessary to include, but notable that there are some attempt to change this through the links between REF and Knowledge Exchange Framework (KEF)  

This is a relevant point, and we have added the following statement to reflect this: “The increasing importance of impact in the UK’s evaluation of Higher Education providers through evaluations by funding bodies such as the UK Research and Innovation’s (UKRI) Research Excellence Framework and Knowledge Excellence Framework may go some way towards valuing and  incentivising researchers who participate in capacity building research”. Line 521-525

Line 564-565: As above re: explaining a little more about GCRF and ODA link

As above, Footnote #1

Line 595-603: I would personally relocate this from the conclusion to a separate section on limitations of the study. This could do right at the end of the methods, or discussion, depending on the preferences of the author. Add in the point about the survey only being available in English.

A new limitations section 4.3 has been created at the end of the discussion. The point about the language of the survey has been moved here.

Review #2 Okamura (2023)

However, it appears that the authors may not be fully utilising the obtained data. Each respondent answered multiple questions, encompassing individual-level, team-level, and organisation-level aspects, as well as questions regarding attributes and demographics like gender, country of affiliation, and research career stage. As a result, numerous cross-tabulations or regression analyses could be conducted among these question responses.

For instance, attributing a specific perception exhibited by early career researchers in response to a particular question solely to the brevity of their career can be misleading. In reality, it could be influenced by the research environment they are situated in or even their predisposition. 

The revised version of the manuscript contains much more detailed analyses based on the comments of the reviewers. We are grateful for their guidance and advice.

Additionally, exploring the correlation between responses to individual-level questions and those related to team-level or organisation-level inquiries would be beneficial.

We did not correlate responses between individual, team and institutional level. There could be some interesting questions to explore here but we feel it is beyond the scope of this initial paper.

Regression analysis is a useful tool that can provide valuable insights beyond simple cross-tabulation. However, it is essential to ensure that the regression model satisfies various conditions necessary for causal inference. While acknowledging the limitations, conducting regression analysis can yield helpful insights. Using the CSV data publicly available in Ref. [2], I present below an example of regression analysis that I have attempted and found suggestive. I hope you find it beneficial.

As indicated by the reviewer, the conditions of causal inference are questionable in this case. The level of success and skill and the degree of improvement were only measured at one point in time, and the samples sizes in different groups were small. We felt that associations measured using Fishers exact test and Spearman rank correlations were appropriate and sufficiently instructive. We hope the reviewer agrees.

If conducting regression analysis poses challenges, I suggest conducting more detailed descriptive analyses as a minimum. To gain further insights, it would be beneficial to create histograms for different attributes such as gender and country of affiliation, and overlay them for comparison, for both the x and y variables defined earlier.

More detailed analyses have been carried out, including Fishers exact tests to look for associations between variables, and Spearman rank correlations to explore the relationship between the current success or skill and the degree of improvement. These analyses make the results much clearer and more informative. See Section 2.2 of Methods and Results sections. We hope the reviewer finds these to be sufficient.

Additionally, creating scatterplots with corresponding x and y pairs as axes would provide valuable information. To account for the discrete nature of the responses, consider introducing jittering. Varying the marker styles in the scatterplot based on attributes like gender and country of affiliation and overlaying them for comparison is also recommended. These visual representations will enhance the understanding of the data and facilitate comparisons across different attributes

We have created scatterplots with jittered points, with the region information overlaid, as suggested by the reviewer (Figures 7-9). The Fishers exact test showed that region was the overriding factor that explained most of the differences, and therefore we feel it is justified to carry out the correlations with only this variable overlaid.

Overall, I get the impression that the authors’ manuscript resembles more of a report on the organisational activities or activity records specifically of the BC project, rather than a research paper that contributes to the academic knowledge base or provides lessons for other (future) cases.

While we do report on organisational activities and records, we believe that this is relevant to (1) demonstrate the types of approaches and initiatives used in the project that could be adapted to implement elsewhere, and (2) the approaches that we discuss are linked to the results. For example, we report on the work that was done by project members on evidence synthesis and systematic reviews. This was reflected in the data which showed high level of improvement amongst SE Asian respondents and it was also evidenced by publications being led by SE Asian partners.

In the Discussion and Conclusion sections, there are instances where the authors attempt to extrapolate their analysis and interpretations to general theories related to career development, the academic environment, or interdisciplinary collaboration, in an effort to draw significant implications. However, there seem to be logical leaps in many parts of the manuscript. For example, generalising specific comments from certain individuals in the open-ended responses and using them to justify the overall evaluation of the BC project or attempting to derive universal conclusions lacks sufficient credibility.

We used comments from individuals to provide more insight than using the data alone. These comments were used to illustrate points that we already made based on the results from the survey. The new data analyses carried out based on the suggestion of the reviewers makes these results more explicit and our hope is that it is now clear how the illustrative comments link to the evidence. Therefore, we believe we have made logical interpretations that are based on evidence and hope that this is clear in the revised manuscript. If the reviewer still believes this not to be the case, please indicate the specific points that are not well supported by data or the wider literature.

It is important to ensure that the conclusions drawn in the manuscript are supported by robust evidence and rigorous analysis. Additionally, generalisations should be made cautiously, considering the limitations of the study and the specific context of the BC project. Providing a clear rationale and using appropriate references or theoretical frameworks can strengthen the credibility and reliability of the manuscript’s conclusions.

We hope the reviewer agrees that the additional analyses carried out supports the conclusions made in the manuscript.

it is desirable to uncover valuable insights beyond the specific BC project. By doing so, this paper will become even more valuable as a publication in the journal, as it will offer insights and lessons that can be applied to a wider range of contexts and contribute to the broader academic knowledge base.

We think that several of the insights discussed in this paper are applicable to many research contexts, including the emergence of the conflict between research aims (e.g. advancing knowledge and publishing papers), influencing policy and building capacity; and the importance of interactive dialogue and not just one-way training, for mutual capacity building.

It is recommended to include the actual number of respondents alongside the response rate in Tables 1 and 2. This will provide readers with a better understanding of the sample size and the proportion of participants who responded to the survey.

The number of respondents has been added to these tables

In my view, the numbers indicated with ‘%’ next to the bar graphs in Figures 1–5 should be removed. These numbers can be misleading and confusing since they do not correspond to the length of the bars.

The % values have been removed in these figures

When stating phrases like ‘Most respondents felt...’, it is advisable to quantify the extent of ‘most’ using the format like ‘X out of Y respondents (Z%)’. This will provide a quantitative representation and enhance the clarity of the statement.

The percentage number of respondents has been added throughout the results section.

In Section 2.2 (Questionnaire), it is necessary to provide more specific and detailed explanations about the methods of questionnaire development, distribution, and collection, including the survey duration.

More detail has been added to the methods to describe additional details of the survey process. See Section 2.1, Lines 113-134 (not copied here)

In the Supplementary Material and the file named ‘Survey_Questions.pdf’, each question item should be labelled with a number or symbol for individual identification, and it is strongly recommended to ensure a one-to-one correspondence between each response in the data file.

Corresponding numbers have been added to the survey questions in the supplementary material that match the numbers in the data file.

In the response data (CSV file), it seems that responses related to age groups and sectors of affiliation have been removed. If there is a deliberate reason for omitting these responses, it should be mentioned in the document to avoid any confusion or misinterpretation.

The metadata file for the csv data explains that these groups were removed to protect confidentiality where there were small groups and individuals that could be identified due to combinations of demographic factors

It would be advantageous to provide a more compelling rationale for the relative superiority of the ‘learning-by-doing’ approach compared to other approaches. While it is generally expected that any approach implementing in a project could yield positive outcomes, the key lies in demonstrating how the benefits derived from the ‘learning-by-doing’ approach outweigh those that would have been obtained through alternative approaches. This will strengthen the argument and provide a clearer understanding of why the ‘learning-by-doing’ approach is recommended.

We did not test other approaches within the project, and we only have data from one project, therefore it is difficult to describe more than what we observed in this project, including evaluations of alternative approaches carried out in other projects with different groups of people.

This manuscript has the potential to significantly enhance its academic and societal value by conducting a more comprehensive analysis considering both internal and external validity. To achieve this, it would be beneficial to provide meticulous descriptions of the approach employed to draw conclusions, ensuring transparency and clarity. Additionally, improving the methods of data visualisation, presentation, and delivery will contribute to a more effective communication of the research findings. By implementing these enhancements, the overall quality and impact of the paper can be greatly improved, leading to a more valuable contribution to the academic and societal discourse.

We hope that the reviewer agrees that we have improved the manuscript sufficiently in all areas based on valuable comments by both reviewers.