What is design-based research (DBR)?
Design-based research (DBR) begins with a design problem in education, such as a teaching a difficult concept, improving a dismal undergrad success rate, or similar. DBR turns to learning theories (or other theories in education) to explain why that problem might be difficult to learn or undergrads might not succeed, and to suggest what the shape of solution might be. These ideas from theory are used to inform the design of the intervention: new curricula, new learning environment, new experiences.
The intervention is tested on real learners in an ecologically-valid context, like a classroom (though early cycles may occur in more controlled environments). Data are collected on how learners interact with the intervention, preferably process data as well as outcome data. Data are analyzed in light of the theory to understand the ways in which the design and implementation of the intervention are successful and to suggest what elements of the design or implementation could be altered to improve student learning (or whatever the key outcomes of the design problem are). Researchers engage in multiple cycles of design construction, implementation, evaluation, and revision in order to (a) solve the design problem and (b) generate new knowledge & learning theory about student learning and development.
The outcome of a DBR project is twofold: the solution to the design problem as well as the new knowledge about learning theories. Both aspects of the outcome are necessary for a successful project; you can’t just do stuff and not generate new knowledge about it, and you can’t just generate new knowledge without finding a solution to the design problem.
While there is an aspect of emergence in subsequent cycles for intervention design and learning theories, theory and intervention are necessary for all cycles. Because theory guides which kinds of data are meaningful, the general family of theories usually does not change radically from cycle to cycle. Similarly, while the design problem may grow or change in subsequent cycles, because the intervention is intentionally chosen and modified, the design problem generally does not change radically from cycle to cycle.
A “design problem” stands at the same level as “research question”: it’s a guiding idea that shapes your project. Design problems and research questions are living. They will grow and change with time.
I’m using “students” to mean “participants” because this is often used in classroom settings, but “participants” is also ok.
I’m using “intervention” to mean a wide variety of things: new curriculum, new course policies, new learning environments, new learning experiences, etc. The key parts are: there’s something new about it, and it is enacted in a natural environment (e.g. a classroom, not just a psych lab).
Designing a design-based research project
Articulate a design problem
Physics graduates need computational skills, but Maria’s departmental curriculum doesn’t cover computation. She would like to add training in computational skills, but her department is unwilling to require a new course for everyone.
Articulating a good design problem entails figuring out what problem you want to solve, why it is a problem, and what the shape of a feasible solution might look like. As you work on this,
- Be specific, and ground it in a specific context.
- Who has this problem? How do you know?
- What’s the scope of this problem? What can you actually change or improve?
- What is the current situation, and why is that a problem?
- Talk to stakeholders: do they think this is a problem too? why?
- What are other solutions (perhaps partial) that exist?
As you work on articulating your design problem, you may find that you need to conduct a literature review to understand elements of the problem or look for solutions to related problems. As you talk to more people about it, you may find that your ideas shift about what the problem is and why it is a problem. This is great.
A design problem is a problem in your students’ environment that you could, in principle, solve. You can build new curricula or improve programs to solve your design problem and help your students, but you don’t get to design your students.
For example, you might notice that the student failure rate (“DFW rate”) in calculus is 30% (yikes!). The design problem is not “my students are too poorly prepared” (a problem with the students); the design problem might be “our placement test doesn’t sort students well” (a problem with the environment before calculus enrollment) or “calculus class doesn’t support student learning well” (a problem with the environment of the calculus class).
Use Theory to refine your problem and suggest a solution
Maria decides to use two theories to support her work: the Resources Framework helps her identify how students bring ideas together to solve problems, and Computational Thinking helps her identify three particular ideas (abstraction, automation, and analysis) for her intervention to improve.
Design-based research relies on theory to shape why these problems are problems and to suggest the shape of a solution. If you want to fix a problem, you need to understand why it is a problem so that you can tackle the cause. The job of theory is to suggest possible causes for the problem, and to help shape potential solutions.
For example, you might notice that some students struggle to solve mathematical problems in their physics classes. Different theories conceptualize this struggle differently:
A cognitive theory of learning might suggest that students have a hard time seeing how to apply math ideas in physics contexts. We need to support their transfer of ideas by explicitly eliciting math class procedures and drawing direct parallels.
A different cognitive theory of learning might suggest that their problem solving skills need support, especially for complicated problems that coordinate math and physics ideas. We need to support their growing problem solving ability by scaffolding the steps of problem solving.
A socio-cultural theory of learning might suggest that different students struggle at different points. We need to support them working together to connect and rediscover math ideas in this new context.
A critical theory of learning might suggest that students struggle to solve these problems because they seem irrelevant to their lives in the face of gross structural inequities. We should reimagine the topics our physics courses cover to better support a just and educated populace.
There are tons of different theories available to you. Thinking about why your problem is a problem and what kind of solution might solve it can suggest to you what family of theory you should use. Through your conversations with stakeholders and literature, you will discover potential theories. Try them on: how does each one suggest the shape of a solution? What feels satisfying to you? You may discover that you want to use different theories to articulate and refine different aspects of your design problem. That’s normal.
Design an intervention
Through conversations with stakeholders and literature, you will start to see the shape of a possible solution to your design problem. The shape of this solution should include some kind of intervention that changes the environment in which your participants learn. For example, you might decide to develop new curricula, as guided by the theory you chose.
Scope: How “big” is your intervention?
Maria decides to focus on one course, Electromagnetism, that she teaches every year. This course is required of all physics majors, so she will be able to affect everyone without adding another course to their workload. Because the course occurs once every year, Maria will have enough time to revise her materials when she isn’t teaching it.
- How much time do you expect your participants to interact with it? Very brief interventions are easier, but generally have less potential for impact.
- How many topics will you cover? Which aspects of the class?
- How many participants will you have? Are they enrolled together, or are these multiple classes?
- How many times can you iterate your intervention to make adjustments?
DBR projects rely on iterative cycles of implementation and development. As you think about your intervention, think about how many times you can iterate through it. Two cycles are a minimum; 3-4 are more common. It’s also common to iterate through each piece several times, and to use lessons learned in each piece to suggest developments in other pieces. For example, if you want to develop seven new labs, you might plan to develop two in the first year, then refine them and develop five more in the second year, and refine all seven for coherence in the third year. For each iteration, identify: what’s the minimum viable product for each cycle? What additional features would be nice to have, but not necessary in the core?
Map your project
Maria decides to develop homeworks and in-class activities to improve her students’ computational thinking. She replaces homework problems that rely on manual problem solving with problems that use computational methods, writing about 20% new homework problems spread across the whole semester.
Build a plan for your intervention that explicits maps between the theory you choose, the elements of your design problem, and the elements of your intervention. For each piece of the map, ask yourself:
- How does this piece of the intervention help solve my design problem? How does it use theory?
- How does this theory help articulate my design problem? How does this theory suggest the shape of the intervention?
- Why does this design problem require this piece of the intervention? Why do I need this theory?
- How does each iteration build new features or topical coverage in the intervention?
Building this map is a lot like engaging in conjecture mapping, a more robust tool from the Learning Sciences that explicitly draws together your theory and your observations to make meaning and conclusions.
Gather evidence
To assess how her students are learning the three elements of computational thinking, Maria will look at their homework solutions. For progress in abstraction, she will look at how they represent physical quantities in their code. For progress in automation, she will look at how they structure their code. For progress in analysis, she will look at how they interpret their results for physical reasonableness. Because she thinks her students will be grappling with these ideas during class, Maria also plans to record their work during class.
The last piece of a DBR design is data. How will you know if your intervention is successful? In what ways does it succeed, and how should you make changes for the next iteration?
Your theory and design will help guide you about what kinds of evidence can show success. If you want students to learn more, for example, you will need to show what they knew coming in and what they know coming out; if you want them to have stronger problem solving skills, you need evidence of their problem solving skills; if you want them to feel like they belong in STEM, you need measures of belongingness. All of these are outcome measures of success: what is true at the end of the intervention?
Because DBR relies on iterations to make changes and improve the intervention, it is crucial that you also collect process measures of how your participants interact with your materials. It’s not enough to know that your materials are helpful (or not); you need to know how they use them so that you can improve them. Common process measures include observations or interviews; it’s also possible to look at artifacts like whiteboards or homeworks. Because you’re going to use this information to help you make decisions about what or how to change, it’s important that your data are as rich as possible: they should show processes of interaction among students and how they arrive at their answers, and include information like the problems students are working on as well as their solutions. You can’t just compare answers on the homework to answers on the final exam: that information doesn’t tell you about how they got to each answer.
As you think about the design of each iteration, plan to collect evidence. Ask yourself:
- What data will you collect? How will it be analyzed? Make sure this data is well-aligned with your theory.
- Will some data streams be consistent in each iteration? Will some be unique to specific iterations?
- How much data do you need to make good development decisions? To show success?
Dangers in data collection plans
For some people, it’s really tempting to collect all possible data because they don’t know what they’ll need, or because they think it might be interesting or useful later. A major danger is that they’ll spend so much time on data collection that they won’t have time for meaningful analysis.
Analyzing all of her classroom video & in-class assignments would take too much time before Maria needs teach this course again. Maria decides to scale back: she will develop half as many problems at at time so that she can focus her analysis and development efforts each year.
If this is you, ask yourself: How much capacity do you have to collect and analyze data between iterations, so that you can make good choices? Which data streams are necessary, and which are “nice to have”? Judiciously pare out data streams that do not directly help you solve your design problem and speak to your theory.
For some people, their access to these learning environments constrains their data-taking abilities. Perhaps they cannot do classroom observations because of university policies, or they don’t have enough research staff to perform interviews.
If this is you, ask yourself: What information do I need to make evidence-informed choices about how to iterate? Can I get this information in another way? For example, instead of classroom observations you might ask faculty for their reflections on what went well, or instead of 20 interviews in person, you might have 2-3 shorter zoom interviews with a selection of stakeholders.
Classroom video won’t help Maria know what’s on her students’ screens. Maria decides to change her data stream: instead of classroom video, she asks a senior (who already took this class) to take field notes on select days.
Return to your map
Return to your map between design problem, theory, and intervention. For each element, identify which data you will collect and how you will analyze it in order to (a) show success; and (b) suggest iterative changes.
Doing your DBR project
Once you’ve designed your DBR project, it’s time to do it!
The second part of your DBR project is to iteratively try your intervention, conduct research on its effectiveness, and redevelop it. You’ll engage with these three activities cyclically several times until you’ve solved your design problem and developed new theory.
Hassan developed some simulations to help undergraduate biology students visualize ion transport through cell membranes. Students can control physical characteristics of the cell membrane as well as the environments inside and outside the cell. He would like to investigate how these simulations help students think about biochemistry concepts, and improve the simulations. Eventually, he would like to have other faculty use his simulations in their classes.
Hassan’s project really centers the development – and redevelopment – of his simulations. Even though he already has some simulations completed, design-based research is still a great choice for him as he considers how to make them better or develop new ones.
Rory has been teaching upper-level quantum mechanics for several years and always scans written homework, quizzes, and exams for their records before grading them. They have decided that they would like to use this written work to explore what resources students bring to quantum mechanics in order to design activities to build on these resources. They hope to publish their findings in The Physics Teacher so other instructors can also benefit from their work.
Rory’s project might be a good candidate for design-based research, if they want to do a research project. But it’s also possible that Rory isn’t super interested in doing research, and they just want to develop some great materials, then tell the world about them. If that’s true, they still need to engage in evaluation to make sure their materials are helpful for their students
Additional topics to consider
History
This article was first written on May 3, 2023, and last modified on February 8, 2024.