This article follows directly from Planning Research Projects; start there if you’re getting started in your research design process. If you are looking to design an education research project, the exercises on research design will help you. You might also look at research process models to help you think about how research projects progress. If you’re thinking about research philosophy, check out Generation, Iteration, Reflection for other core ideas in doing research.
Why use iterative design?
As you engage in research, you will learn new things. You can’t plan exactly in advance what new ideas and opportunities will emerge in the course of your research project. However, you can anticipate the character of what they might be, and build space in your research plan to foster opportunities for emergence and mitigate risks. The best way is to use an iterative design.
How does iteration foster opportunities for emergence?
Iterative design is an important tool to plan for emergence . By dividing your project into a series of progressively more complicated and refined iterations, you build in opportunities to foster emergence during your early iterations that could reshape the project in unforeseeable and delightful ways. There are a few key ways to build opportunities for emergence into your project planning.
Collect rich data
During each iteration, plan to collect rich, explanatory data of what happens in each iteration so that you can make good choices about how to adjust for the next time. You want to be able to connect process information (how do people interact or learn?), contextual information (what are they working on? what happened immediately before and after?), and metadata (when was this collected? who are these people?). For many projects, this might involve collecting additional data streams to help you triangulate your conclusions. For example, if you are interviewing students about their experiences in a mechanics class, you might want to coordinate with the instructor to see what they covered, when, and how, and you might also want to collect information from the students about how their backgrounds affected their experiences. If you are developing curricula to improve their understanding, you will want to collect information about how they interact with your materials, not just summative information about what they learned.
Rich data helps with emergence because it enables you to notice interesting features and have enough information to follow up on them.
Document and reflect
At the end of each iteration, document and reflect on what you’re learning and what’s surprising, and use that information to make a new plan about the next few iterations. Spend some time to deliberately collect your thoughts and reflect on the generative writing you made along the way. Pay attention to both the substance of what you’re learning and the processes by which you learn it. Were there missed opportunities to learn something that would be useful next time? Information that was costly to acquire but ultimately not useful? How did you allocate your time (and your team’s time) in this iteration, and how might you rebalance for the next one?
Reflection fosters emergence because it gives you the intellectual space to connect ideas; documentation helps you generate those connections and learn from them for the next iteration.
Talk to people
In every iteration, include ways to share your work with people outside your project team and deliberately seek feedback about your project and its progress. This might include conversations with stakeholders about their goals and constraints, presentations at research group meetings or conferences, meetings with advisors or research friends, hallway conversations or walk-and-talks with your dean, etc. The more you talk about your project, the better you will become at encapsulating it; the more people you talk to, the more cool ideas you can discover together.
Talking to people fosters emergence in two ways: first, because the act of communicating will help you crystallize your ideas or generate new ones; second, because they have cool ideas too and those ideas may emerge from your conversations. To be clear: their ideas might not help you immediately, but they might help you conceptualize new projects or collaborations later on, which is also delightful.
Building in iteration: an activity
As you put together your iterative design, here are some ways to think about building in iteration to your design.
- What’s the major knowledge milestone for each piece?
- Think about preliminary analyses which lead to more robust ones of the same data, or exploratory analyses of a data subset to confirm in the full corpus.
- Think about pilot phases which lead to larger phases.
- For time series, what analyses can be done on only the first parts?
- If you’re building something, what’s the minimum viable product (MVP) at each stage?
- If you’re presenting something, what are the preliminary presentations (locally, at conferences, etc) that build to a larger paper?
- How is each major knowledge milestone / iteration moving forward on your big question / problem?
This activity blends between building a research design and project planning. While smaller projects might not need a lot of iteration, generally speaking it’s a wise choice to plan for iteration in many different time scales.
How does iteration mitigate risk?
What are common risks in education research projects?
In research projects, especially those that involve multiple people and multiple years, the biggest sources of risk are about staffing, timing, and access.
- Common staffing risks include project members leaving the project early (or having less available time than anticipated), as well as the need to develop the expertise to do the work.
- Common timing risks center around how well you can estimate how much time it will take to do this work, or how well you can estimate the availability of your research team; and how quickly the offices you depend on perform their work (e.g. your IRB).
- Common access risks are about whether you can actually get the data you want to have, make the instructional changes you plan for, or find enough participants.
Ahead of time, you can (and should) build the best plan possible for mitigating all of these sources of risk, but as your project develops you will inevitably need to make adjustments to your original plan.
How does iteration mitigate risk?
By separating your project into successive iterations, you can mitigate the effects of these risks. For each iteration, you can define milestones to meet and a minimum viable product to develop. This reduces the risk that your project could fail altogether, because you can “back up” to the previous product.
For research projects, iteration usually looks like planning for multiple cycles of data collection with interim analysis after each cycle. Pilot interviews lead to interviews with a broader population, for example, or unstructured observations in class lead you to select formalized observation protocols or develop interview questions later. A key feature of these iterative designs is that your pilot data collection and analysis are designed to serve two purposes.
Pilot work guides the development of future data collection and analysis plans. At the end of each cycle, your reflections will help you decide what data you should collect next (and how much data), in light of the data you have currently collected. For example, your upcoming plans might include collecting very similar data (but more): more interviews with a slightly different protocol, more survey respondents from a broader population, etc. Starting small lets you develop expertise iteratively, which is a smaller lift than trying to develop it all at once. Alternately, you might have planned to collect more data, but your preliminary analysis suggests that isn’t necessary, or that you aren’t ready yet to get more data because your existing data need more analysis time. An iterative design means less wasted effort and a gentler onramp to developing expertise and access.
Pilot work generates preliminary conclusions which are, themselves, useful. Perhaps your pilot data are already useful to help you design an intervention, or perhaps they could be used as the nucleus of a conference paper submission. Perhaps your dean needs an answer more quickly than a 3-year study can handle, or you’d like to use these conclusions to support a proposal for more resources.
Both of these features help mitigate the major risks inherent in doing research projects.
What are common iterative designs in education research?
There are two major models for iterative design in education.
Getting bigger
In iterative “getting bigger” designs, early phases of the research project often start with noticing: what’s something interesting, concerning, or surprising in your class or environment? After you notice something, you formalize it: how frequently does it occur? under what circumstances or for which students? how can you measure it reliably? Finally, you extend it: does it work this way in other settings? Notice-formalize-extend is a very common way for research projects to grow, especially if they are about generating new knowledge but not about developing new interventions.
Another way that projects “get bigger” is to start with baseline data, then a small pilot intervention, then revise to include more classes or topics. Baseline-pilot-revise is a very common way for development projects to grow. It is central to design-based research.
Arthur’s department teaches an introductory class with 8 different instructors and 10 different GTAs. Many students are struggling with the mathematical formalism, so the department is going to revise the curriculum to support students’ math skills and problem solving. Arthur plans to collect some baseline data so that they will know if the changes are working.
Arthur plans to collect some baseline data so that they will know if the changes are working. As he develops new curricula, he’s going to pilot the new units in one or two sections to see how they work. He plans to make revisions before he implements them in all sections.
This is a classic baseline-pilot-revise project which uses a design-based research framework.
Going deeper
Another common model for iterative design in education research is to “go deeper”. This model also starts with noticing, but instead of moving to formalize, it moves to complicate: what nuances are here? what other issues are at play? Finally, it moves to connecting: what causal story or mechanism drives these details? how do they bring together different theories? Notice-complicate-connect is the most common choice for case studies and other qualitative research studies.
This model often spends a lot of time engaging back and forth with complicated data streams and theoretical frameworks to develop case studies and new theories. It’s the primary driver behind my video-supported research.
Julian is curious about how students in general chemistry connect diagram representations of molecules with their chemical formulas. His department has 15 sections of general chemistry which enroll 30 students each. He plans to collect students’ homework from 3 sections to see how they use diagrams or formulas to understand chemical properties.
Julian’s project could be a great opportunity to go deeper. He plans to start with noticing what students do in their lab notebooks. From there, he could develop an interview protocol to ask students about what they drew (and why). After his interviews, he might want to return to the lab notebooks to focus on a particular kind of problem or diagram.
Other benefits to iterative design.
There are some other benefits of engaging in iterative design:
If your research program involves mentoring undergraduates in research, you can design each iteration so that your research students complete a meaningful project in the time that they have, and also that their projects build meaningfully to a research program that lasts longer than any particular student.
Each iteration can, itself, look like a miniature research project using the parallel processes model for research. This helps you build expertise in doing this kind of research generally, and solve logistical issues in doing this research in specific.
Engaging in iterative design helps you put together a longer research project from a series of smaller pieces that are tuned to the the rhythms of your professional life. For many academics, identifying what milestones or minimum viable product you can produce each semester feels more approachable than building an entire 3-year project at once.
Additional topics to consider
History
This article was first written on May 2, 2023, and last modified on February 14, 2024.