Mysterious assessment process

When I was studying for my basic degree and completing further studies, the assessment of applications seemed to be a very mysterious practice. The assessors remained nameless and reasons were rarely given for decisions. However, the academic world has changed a lot since then. Nowadays training is provided in research ethics, university pedagogy and also on compiling job and grant applications. Students are also required to do peer reviews, in other words, make critical assessments of texts written by other students. However, the assessment of applications is still not taught and the assessment process is not otherwise widely discussed. In the following, I will try to shed a little light on the process from my perspective as an assessor and explain the problems that I often come up against.

I have developed my assessment skills through practice and by discussing assessment principles with my colleagues. As my experience has increased my confidence in my assessment skills has grown. The assessment process is more or less a routine now.

I start off by reading the funding body’s instructions. What kinds of principles do they want me to observe in the assessment? Kone Foundation provided a clear guideline here: they want bold and unprejudiced initiatives, and interdisciplinary projects.

After this I like to read the application headings and summaries to gain an overall picture. I want to know whether I am potentially biased and what subjects the applications deal with. What kind of spectrum can I expect? The words the funding body emphasizes in its call for grant applications are frequently repeated in the summaries. In the case of Kone Foundation these were “boldness”, “innovation” and “multidisciplinary”. However, as an assessor, I am not influenced by certain individual words – this method does not guarantee a thing. Instead I consider whether the principles will be achieved in the project as a whole. Even though I am already starting to get a mental picture of the quality of the application by this stage I don’t make any final decisions yet.

During my second read-through of the applications I focus on the research plans. I ask the same set of questions to each application that I read: What is being researched? Why is this being researched? How is this being researched? Does the applicant have the prerequisites to achieve the targets? What are the expected results of the project? I make notes on all of the project descriptions and list the good and bad characteristics of each.

The research question, i.e. the definition of what is being researched, must be crystal clear. It must be summarised into one idea, possibly even into one sentence. Far too often an applicant will provide a half-hearted definition of the research question and steam ahead onto the subsequent questions. However, these minor subsequent questions need to follow on from and support the main question.

“What is being researched?” is closely connected to “Why is this being researched?”. The application should explain what the previous research lacks and what gap it is filling. The applicant needs to know the history of the research area and the earlier research related to it. My suspicions are raised whenever I read an application that proudly asserts that this or that subject has never ever been researched before. Such a categorical statement is hardly ever true. Particularly in the world of humanities research there is hardly likely to be a subject that has never been touched on to some extent before by an earlier researcher. This makes me wonder whether the applicant is sufficiently informed of the research history or the current situation in the field. It is possible that there may be little previous research, the research may be of the wrong type and this project might have a fresh perspective, but the applicant has to convince me of this.

The question, “How is this being researched?” is linked to the methods. A good application might be founded on established methods, but in my field, archaeology, many scientific breakthroughs are often based on application. Natural sciences methods in particular are important in contemporary archaeology. It is no longer important whether the project has some kind of new natural sciences method but how the results are linked with a humanities interpretation. In my assessments I favour plans that can successfully cross boundaries between fields of research in such a way that the humanities part is as strong as the natural sciences part. I also believe that projects that focus on the development of high-level natural sciences research methods so that the results have a clear international significance are important.

In the research plans I also consider whether the applicant has the prerequisites to implement the project. I verify this based on the research location and researcher network, and also with the CV and list of publications. These can be long and impressive or then show that the researcher is just at the start of his/her career. However, the main thing is whether the applicant has the type of skills required by the plan. Will the applicant be able to apply the methods? Does the applicant know the subject matter and research material? Why is this person or group, in particular, the right one to carry out the project?

Postdoc funding applicants sometimes focus too much on presenting the results of their doctoral thesis. However, I am not primarily interested in what the applicant has done before, but what he/she plans to do after completing the thesis. This is a new project that needs to stand on its own two feet. The doctoral thesis is usually just the foundation for the project and does not have a starring role; the aim of the new project can’t just be to tie up the loose ends of the doctoral thesis.

In the final stage I remind myself of the principles defined by the funding body. I go through my notes and think about how each project fits into the funding body’s targets.

It is very easy to separate the applications into three groups: the strongest, the weakest and the ones that fall between these two categories. The hardest thing is to place the strongest projects in order of which is best. To do this you need to weigh up the funding body’s principles, the clarity of the research question, the national and international impact of the projects and the conditions for implementing the project. Then, last of all, I give each project a numerical grade.

As a result I have a well-founded list which places the projects in order of superiority. I then sent this list electronically to Kone Foundation and transferred the responsibility for the actual funding decisions to someone else.