Pilot 1
Pilot 1 comprises evaluations at the level where the research is conducted. The department is often a natural body to evaluate, but another delimitation may sometimes be more appropriate. In the pilot project, four departments and two subjects as a part of a larger department are evaluated. The evaluations comprise a review of the environment, which includes research quality and effect as well as conditions for research and the unit's strategies and quality assurance activities.
The evaluations aim to be quality-driving for each department or subject based on their specific conditions, challenges and opportunities.
How can this benefit me?
Departments that take part invest their time and energy, but in return they receive:
- substantial analyse material;
- a review and recommendations by qualified external experts;
- opportunity to expand their network with prominent researchers within the field through the external experts;
- opportunity to use the results to enhance their research;
- opportunity to influence and enhance future peer reviews at Umeå University;
Time plan
2024
February
Draft instructions for self-evaluation.
March
Departments receive quantitative documentation.
April to October
Departments perform self-evaluation according to their own time plan.
2025
March
Site visits in the organisation and presentation of report.
April onwards
Drafting a plan for measures, implementation and follow-up of measures, evaluation of pilot studies and drafting of decision guidance documentation.
Frequently asked questions about the pilot project
What benefits do we as a department have from being one of the pilots?
The evaluations are meant to be an investment in higher research quality. The departments involved have to commit a certain amount of time and energy, but in return receive a thorough set of metrics as a basis for analysis and reflection; the assessment and recommendations of prominent researchers within the field, and the opportunity to make long-term connections with these; and hopefully concrete measures for development to work with in dialogue with the Faculty. Being part of the pilot also provides the opportunity to help shape the future of research evaluations at Umeå University!
What will the evaluations be based on?
They will be based on self-evaluations, a set of metrics and a site visit where the unit under evaluation and the external reviewers can discuss and clarify different issues. The self-evaluations will also contain examples of societal impact.
How is the self-evaluation supposed to be quality-enhancing, and what will it contain?
The self-evaluation work can be quality-enhancing by gathering the faculty around a reflection and strategic discussion on strengths, weaknesses, opportunities and challenges. To form a good basis for relevant assessments and recommendations from the panels, it needs to be as honest and reflective as possible. The self-evaluations will not be published, and need not be expansive (10-15 pages should be enough). Involved departments and Faculties will be able to provide input on a draft self-evaluation template.
What is the purpose of descriptions of societal impact?
The societal impact of research is important, but much harder to follow and assess than scientific impact which can be measured in publications and citations. Describing cases of societal impact is something that is increasingly used in evaluations, internationally and in Sweden. Writing and receiving feedback on such case descriptions can be quality-enhancing through increasing insight in ways that impact can happen, generating new research questions, and stimulating active work to achieve positive societal impact. It can also contribute to building competencies and experience that may be useful when outside actors require such descriptions, e.g. for project applications.
What is the role of the external reviewers?
Even if the self-evaluation work is quality-enhancing in its own right, external reviewers with expertise and integrity are essential for the evaluations. Having such external peers to write the self-evaluation for makes that work more meaningful, and their assessment and recommendations are important for calibrating the evaluated unit's own quality assessments and gaining new insights and perspectives. Establishing and deepening contacts with prominent researchers within the field is also valuable in itself.
How are the panels of external reviewers selected?
Based on nominations from each department/subject, the vice-chancellor or a deputy will appoint panels consisting of three reviewers external to Umeå University. They should be professors with expertise in areas relevant to the unit under evaluation, and have experience from academic leadership and research strategy. The panel Chairs should have thorough knowledge of the Swedish higher education and research system, but the panels should also preferably contain reviewers from abroad. Each panel must consist of both women and men, and gender balance will be strived for in the total number of reviewers.
What material will be provided for the self evaluation? Can we influence it?
A dataset will be provided, fulfilling two purposes. One is serving as a good basis for reflection and analysis in the self-evaluation. The other is providing the panels with necessary context and a picture of the unit's profile, conditions, and results which the self-evaluation can be read in relation to. Where possible, relevant national or international benchmarks will be used. Important components will be financial data, staff and recruitment, external research funding, and scientific impact in the form of publications and citations.
The departments involved can propose content and will have the opportunity to see and comment on a preliminary version.
Is the research graded on a scale? Are there financial consequences of the evaluations?
No, the evaluations are not about grading, comparing within Umeå University, or financial consequences as reward or punishment. The purpose is quality enhancement. Measures as a result of the evaluations may of course include strategic spending or changes in priorities at different levels, within the regular planning and budget cycles.
What happens when the evaluation is done? How is it to be used for higher research quality?
The whole point of the evaluation is to be used as a basis for quality-enhancing measures and strategic development. They are primarily a tool for quality enhancement in the evaluated unit itself, and the first question is therefore what conclusions they draw and what measures they see as necessary. But Faculty- and University-level management also need to take the panel observations and recommendations to heart, for a dialogue about what measures may be needed at what levels, and in order to spread insights that may be applicable in other environments. Designing, implementing, and following up quality-enhancing measures is a work with a longer time horizon then the pilot project itself.
Who are included in the pilots?
Pilots on department or subject level are Philosophy (Department of Historical, Philosophical and Religious Studies), the Department of Medical Biochemistry and Biophysics, Neuroscience (Department of Clinical Sciences), the Department of Psychology, the Department of Geography, and the Department of Ecology and Environmental Science. The Faculty of Medicine will be pilot for the faculty-level evaluation. Theme or themes for the transversal evaluation is still do be determined.
Our department is not one of the pilots, when will we be evaluated?
There is no schedule for that; plans for future evaluations will be decided after the pilot project. They will start no earlier than 2026.
Pilot 2
The faculty-level reviews focus on the faculty's research strategy for quality assurance and quality enhancement of research, as well as conditions for research. The setup will resemble the faculty audits that are used in the quality system for education. The review group will consist of people from other faculties, with an external chair. More information will be posted in due course.
Pilot 3
The third pilot evaluates how Umeå University in its entirety works with one or a few specific aspects of conditions for research. The theme will be chosen this spring semester.