Approach
Image: Stefanie JohnWe rely on participatory processes with potential user groups and apply proven agile working methods from software development.
We adapt established concepts from science and research for the development of the monitoring tool and the assessment process for projects. Central to this is the use of impact pathways and the focus on predictive indicators.
We develop the current version of the monitoring tool and the assessment approach for the specific requirements in the agricultural, food and environmental sectors. We are using the open source current research information system DSpace-CRIS as a technical starting point. In this way, we create the optimal conditions to facilitate the establishment and dissemination of the tool among potential user groups and to enable future context-specific adaptations without legal barriers.
The development of the monitoring tool and the assessment process is based on an agile software development approach. The development takes place step by step and intermediate versions are presented in iterative processes to potential end users from research, research funding, knowledge transfer and the evaluation community.
Concrete drafts of user interfaces, functions or applied concepts, for example, are reflected upon. In this way, we stay close to the needs of the various user groups and can discover and solve striking problems even before the actual pilot operation.
Impact Pathways
The monitoring tool will be available with a user interface for the creation of so-called impact pathways.
With an impact pathway, the impact logic of a project can be developed taking into account restrictions. Concretely, the following impact pathway for research and development projects is used in the monitoring tool:
An impact pathway is a well-proven instrument for impact planning in international development cooperation. Accordingly, the Impact Pathway offers a productive approach to sharpen the desired impact orientation of research projects and to present the project's contributions to increasing the impact potential in a structured and clear manner.
In adapting this instrument for the impact orientation of research projects, particular attention was paid to the limits of impact measurement of research.
Limitations of impact measurement of research are essentially due to two factors: First, the open nature of research results means that impacts can be diverse or less specific. Secondly, the societal impact of research results is difficult to measure, since effects usually only occur with a long time lag and, at the same time, societal changes can hardly be traced back to specific research activities.
Predictive indicators
A fair rewarding of research performance is realised by focusing on factors that can also be actively shaped by research and at the same time are subject to a low influence of external framework conditions. These factors should also be able to provide information on whether a societal impact of research performances is likely. We use predictive indicators for this purpose.
Central to this is the concept of so-called productive interactions. This concept is based on the empirically proven knowledge that an application and impact of research must be preceded by interaction processes with participants from the field of action.
We also derive further indicators from innovation research. Innovation potentials are described, among other things, on the basis of application possibilities or the degree of readiness of a developed solution.
Productive interactions include all interactions between researchers and stakeholders that are likely to have a societal impact. Interactions are direct (e.g. workshops), indirect (e.g. a brochure) or financial (e.g. co-financing) processes of exchange between researchers and stakeholders from practice and society. They are considered productive when stakeholders are eager to apply results, outputs or experiences from the research (Spaapen, Jack und van Drooge, Leonie (2011): Introducing 'productive interactions' in social impact assessment, in: Research Evaluation, 20 (3), S. 211–218).
Research funding increasingly aims to ensure that research and development provide answers to societal challenges. Research results should lead to innovations and thus contribute to sustainable development. For this, it is crucial how research works.
We develop a project assessment based on these different dimensions of societal benefits. Our multi-criteria set brings together nine dimensions:
In the conception of project assessment, we adapt approaches from transdisciplinary research and innovation research. We take up the discourse of Responsible Research and Innovation, which is considered decisive for European research funding. Approaches to sustainability assessment and classic technology transfer are also incorporated.
Assessment criteria for the variety of research and development
The assessment criteria we have developed should be applicable to the variety of different project and innovation types as well as funding programmes.
Since the aim is to assess potential societal benefits at an early stage, we rely on predictive indicators. Central to our concept are productive interactions between research and non-scientific stakeholders as a suitable "measure" of transfer and exchange processes. Equally important is information on application and innovations and their potential societal impacts.