What is research? What is evaluation?
Research is a process for generating new general knowledge. What you do education research, you discover new things about how humans learn, grow, interact, and develop. You share your discoveries with the scientific community as well as with your participant populations so that your discoveries can become new knowledge and your new knowledge can help shape future research projects as well as policies, programs, or projects. Your research may be case-based or pattern-based; your discoveries can have small scales or large ones; and the mechanisms for your sharing can vary (but often include peer-reviewed papers).
In contrast, evaluation is a process for generating new local knowledge, tied tightly to a local context or activity, and for the purposes of figuring out how (and in what ways) the activity meets its goals and meets the needs of the participants. When you grade your students, you’re evaluating their work in light of the work you asked them to perform and how well they performed it. You’re not conducting research on student understanding; you’re giving specific feedback to individual students about their performance.
| aspect | Research | Evaluation |
|---|---|---|
| Central goal | Generate new general knowledge | Generate new local knowledge |
| Impetus | Answering research questions | How does this program meet its goals? |
| Shared among | a research community | stakeholders for the program |
In education projects, it can be hard to navigate the roles of research and evaluation. If your project is about developing new curricula to improve students’ learning of partial differential equations, you will need to evaluate how they interact with your new curriculum in order to improve the curriculum. This evaluation is not, itself, research: the goal is to produce local knowledge for the purposes of making changes to your curriculum. That’s ok: not all projects in education are research projects.
You might also do research to produce new generalizable knowledge around students’ ideas about partial differential equations; your research would treat the curriculum as a context in which students learn. However, you will probably use overlapping data streams and analyses for the evaluation of your curriculum and your research on student learning. This overlap in data and analysis can make it difficult to distinguish research and evaluation.
Some people think that the intent to publish is what distinguishes research and evaluation, but that’s not accurate. The actual distinction is not about whether you want to publish your work; it’s about what kind of knowledge you intend to generate.
If you are only generating local knowledge, then you are not doing research and you do not need IRB approval. If your project is about developing a curriculum, the curriculum development is not research, even if you intend to publish your curriculum later.
The difference between evaluation and research is important, if subtle. Generally, education research projects need the oversight of an ethics review board, but evaluation does not. If your project does not include an intent to produce generalizable knowledge, then you are not doing research and you do not need IRB approval.
Relatedly, it’s difficult to publish evaluation results. Why? Because evaluation produces new local knowledge. If you want other people in the world – people who don’t have a stake in the success of your project – to care about your work, you need to connect it to a larger context and show that it has general implications. Your results need to, in some way, generate new generalizable knowledge. At that point, you’re doing research.
What needs evaluation? Everything.
All work is better if you enter into it mindfully, reflect on how it proceeds, and debrief on its accomplishments afterwards. Part of entering mindfully is articulating goals and developing a timeline of activities and milestones; these activities feed into evaluation. Reflecting on how work proceeds and debriefing on accomplishments are both evaluative activities.
What projects need evaluation? All of them.
In a research project, you should evaluate your interim results to decide which avenues to pursue, whether you need more data (or different data), and how your research questions are changing. You should evaluate the human processes of doing research: are we (collectively) proceeding apace? are some people overwhelmed and others frustrated? how are we balancing the load of this research with our individual capacities for it?
This need for evaluation is independent of the kind of research you’re doing: bench science needs it, education research needs it, and the humanities need it.
Not all projects in education are research projects. If your project is about developing new curricula, improving a program to promote student retention, or evaluating whether your undergraduates are adequately prepared for graduate school, then you’re (probably) not doing research. You still need to evaluate whether (and how) your project is achieving its goals. You still need evaluation.
What kinds of evaluation are possible?
There are lots of ways to evaluate whether (and how) your work is meeting its goals or how its goals are changing in response to new developments. Across all of these kinds of evaluation, there are some big points to keep in mind:
- Evaluation needs to consider both the processes of doing research and the results. It’s tempting to only check whether you have achieved your milestones, but that’s only part of the story. How you make your milestones is just as important as whether.
- Evaluation is driven by goals. It’s hard to know if you have been successful if you don’t know what success means.
- All projects, even classic bench science projects, need evaluation. However, different fields conceptualize evaluation differently, and different projects will use different emphases on how or why to do it.
Let’s look at some of the major ways to conceptualize evaluation.
Formative and Summative evaluation
| aspect | Formative |
Summative |
|---|---|---|
| when | throughout the project | at the end of the project |
| intent | learn about processes and interim results | reflect on processes and results |
| so that | you can make good choices of how to proceed | you can assess success and generalize for next time |
| academic example | grading your students’ rough drafts | final grades in a course |
Formative and summative evaluationFormative evaluation occurs throughout a project to help you reflect on what has happened and make decisions about how to proceed, while summative evaluation happens at the end of a project to help you assess how and in what ways you have met your goals.
I’ve presented formative and summative evaluation as if they are exclusive categories, but in practice the lines can be a bit blurry. For example, your external evaluation might include presenting your preliminary results at a conference and soliciting feedback from other researchers. On the one hand, this is formative: you’re using their feedback to make changes in the next stage of your project. On the other hand, this could be summative: perhaps the conference presentation marks the capstone of your student’s research experience, and other students will be taking up the project later. This kind of blurry line between summative and formative evaluation is pretty common, especially for large projects which span multiple people and multiple years.
Internal and external evaluation
| aspect | Internal |
External |
|---|---|---|
| who | you and members of your team | someone external to your team |
| intent | reflect and record | get feedback |
| so that | you can be mindful of what’s happening | you can connect with stakeholders and community norms |
| academic example | post-lecture notes to yourself for next time | meeting with your chair about your mid-tenure review |
Internal and external evaluationInternal evaluation is work that you (together with the members of your team) perform to assess the processes and results of your project, while external evaluation is work that someone outside of your team performs to give you feedback.
Comparing kinds of evaluation
These two axes – formative and summative assessment, and internal and external assessment – are both useful and important for assessing the processes and successes of your projects. You can think of them as making a 2x2 grid, and mindfully select kinds of assessment to fill in each box of the grid. In Table 4, I’ve put together some common (and minimal) evaluation choices for a research project.
| type | Formative |
Summative |
|---|---|---|
Internal |
Lab meetings about research progress | Project debrief |
External |
Quarterly meetings with advisory board | Publication of results in peer-reviewed journals |
Formal and informal
A third major axis of evaluation is about how formal the evaluation is. Formal mechanisms for evaluation, such as peer review, tend to be easier to identify. They’re often linked to specific milestones or events: peer review of submitted papers hinges on writing papers and submitting them for peer review. They often have explicit criteria by which you can judge your progress: your tenure review, for example, is conducted following specific policies and procedures.
In contrast, informal evaluations may not follow explicit rules, and they may be more relationship-driven. Going out for coffee with a mentor might yield great informal evaluation of your progress with guidance for the future. Asking questions with a presenter after their talk is another informal route, as is checking in with your research students about their progress.
:::{callout-tip title=“Do everything”} Generally speaking, you will want to build opportunities for all of these kinds of evaluation: formative, summative, internal, external, formal, and informal.
:::
Measurement: what will you measure? How will you know?
Central to the practice of evaluation is the question of measurement: what is happening, and how do you know? Because evaluation needs to reflect on processes as well as outcomes, your measurements need to include both.
Some measures are quantitative: how many students enrolled or graduated? how many papers did we publish, and where? Other measures are qualitative: what do our participants cite as being impactful, and why? Common data types in education research are often applied to evaluation as well.
In general, a robust evaluation will include both qualitative and quantitative measures, though it’s ok to only have qualitative measures. It’s never enough to have only quantitative measures; you must contextualize and augment quantitative information with qualitative information about what the numbers mean and why.
Setting up for success
As you plan a research project, you can set yourself up for success by including regular mechanisms for evaluation which span all axes of evaluation: formative, summative, internal, external, formal, and informal.
There are two common avenues here: identifying milestones, and setting procedures.
Identify milestones
Identifying milestones in your project is a calendar-based activity. It’s generally used for formal evaluation, both internal and external
- Starting with a rough project timeline, identify milestones in your project. What will you accomplish and when?
- For each iterative cycle of your project, identify what new features you will develop or activities you will undertake.
- If there are external deadlines (e.g. for semester breaks or grant deadlines), connect your planned milestones with their dates.
- If publication is a goal, where do you want to publish? when do you want to submit?
Set procedures
Setting procedures is about building the practice of evaluation into your project.
- If you are working with undergraduates or other team members, schedule regular meetings as a group to check in on progress.
- Develop lab practices to support good documentation. What are you doing? why?
- If you are working with an advisory board or external evaluator, meet with them regularly.
- Set aside specific meetings to check in with each of your project members and major stakeholders about how the project is going for them and any excitements or concerns they may have. Plan to do these meetings regularly.
Research is a human endeavor.
As you think about how to evaluate your project, it’s important to attend to the human aspects of doing research. Are your team members working well together? Do they feel valued? Do they feel like they have the opportunity to grow? When conflicts arise, do they feel safe in articulating what’s happening and trust that the resolution will be fair and appropriate?
These human aspects of doing research are important to evaluate, even if they don’t tend to align well to specific milestones like papers published or students graduated. In an informal sense, you can check in on your team as humans regularly in group meetings and individual ones. In a formal sense, it’s a good idea to have a regular, formalized review process. If you engage in a formal process – and I think you should – then your process needs to include mechanisms for your team to learn the results of the evaluation as well as mechanisms for you (and your team) to make decisions about how to operate together in the future.
Working with an external evaluator is a great choice for getting good formative and summative feedback for how your team works together. They might meet with you about your goals and questions, then meet with or survey each of your team members. Afterwards, they will distill what they’ve learned in a report to you, either verbal or written, so that you and your team can make good choices about how to proceed.