Design Projects

Abstract

Research is about generating new knowledge for the scientific community, but sometimes you see a problem and you just want to fix it. Do you need to do research? Probably not. Consider a design project instead. In a design project, you start with noticing a problem that you want to solve, possibly in your course or department, gather information to better understand and articulate the problem, and then build and test an intervention to solve the problem.

Oh hey!

I’ve been writing this Handbook since 2014. It’s time to finish. In the next few months, you can expect:

• formatting changes as I prepare to package the Handbook into book format.
• content updates I as clean up chapters to make them more coherent

Do you want to suggest edits or new chapters for this Handbook? You have until June 1, 2025.

Design projects solve real problems

While research projects are often motivated by curiosity – why does this happen? – design projects are often motivated by problem solving. A design project often starts with noticing a problem: enrollments are falling, the curriculum doesn’t cover a topic well, faculty are exhausted, etc. These problems arise from real educational situations, and they demand functional, real-world solutions.

In education research, the most common kinds of design problems arise when researchers or practitioners want to * develop new curriculum or improve an existing one around a particular content topic, instructional setting, or in support of specific skills * improve a program at their institution to support a specific population of students, like first year advising or ESL courses for grad students * develop a new survey, concept inventory, or research instrument * develop resources for other faculty, such as workshops, instructors’ guides, or online resources

Sometimes (often!) the (re)development of these curricula, activities, or resources involves doing research to understand what’s going on and suggest changes. That’s awesome! However, sometimes you don’t want or need to engage in a research project in order to solve your problem: you just want to solve it.

If your research questions sound like “which teaching method is better, the new way or the old way?” or “can we develop a program to support students?”, it’s likely that you don’t have a research project in front of you. Ask yourself: where is my passion? If your passion is generating new knowledge, revise your research questions to support that goal. If your passion for this project is fixing the problem, then your project is more likely to be a design project. If you want both (both is good), check out Design-Based Research (DBR).

Case study: Maria and Computational Thinking

Physics graduates need computational skills, but Maria’s departmental curriculum doesn’t cover computation. She would like to add training in computational skills, but her department is unwilling to require a new course for everyone.

Maria wants to do research and development, so she decides that a Design-Based Research (DBR) project fits her passion.

Projects grow and change with time

Many projects start with research as they figure out what is going on, and then move to design as they figure out good ways to improve the situation. This is a great model for making changes, because it help ensure that the changes you make are likely to have the effects you intend. However, it can take a lot of time and resources, and sometimes it’s overkill.

Check with the people around you: what’s important here? Check with your passion: what do you want to do?

Articulate a design problem

At the heart of every design project is a design problem which articulates what problem the project hopes to solve. The design problem is analogous to a research question, in that it sets up the intent and scope of your project. Just like research questions are living questions, design problems are living problems: they grow and change with time and experience.

Articulating a good design problem entails figuring out what problem you want to solve, why it is a problem, and what the shape of a feasible solution might look like. As you work on this,

• Be specific, and ground it in a specific context.
• Who has this problem? How do you know?
• What’s the scope of this problem? What can you actually change or improve?
• What is the current situation, and why is that a problem?
• Talk to stakeholders: do they think this is a problem too? why?
• What are other solutions (perhaps partial) that exist?

As you work on articulating your design problem, you may find that you need to conduct a literature review to understand elements of the problem or look for solutions to related problems. As you talk to more people about it, you may find that your ideas shift about what the problem is and why it is a problem. This is great.

Design problems are not problems with students.

A design problem is a problem in your students’ environment that you could, in principle, solve. You can build new curricula or improve programs to solve your design problem and help your students, but you don’t get to design your students.

For example, you might notice that the student failure rate (“DFW rate”) in calculus is 30% (yikes!). The design problem is not “my students are too poorly prepared” (a problem with the students); the design problem might be “our placement test doesn’t sort students well” (a problem with the environment before calculus enrollment) or “calculus class doesn’t support student learning well” (a problem with the environment of the calculus class). These different design problems imply different interventions to solve them.

Case study: Blaise and Bridging

Blaise teaches a course on differential equations, and he noticed that his students aren’t learning as much as he would like. Blaise’s goal is to help his students learn more, and he starts thinking more carefully about the best ways to do that.

Blaise thinks about where Differential Equations sits in the curriculum. Many students take it in their fourth semester, and it’s a gateway to the upper division courses in the department. Most of the students in his course are math minors and are taking it in support of their other major: physics, electrical engineering, education, or data science. As a mathematician, Blaise believes that understanding differential equations is foundational for his students’ future work.

Blaise starts with a design problem: “Course materials need to support and engage a wide variety of student interests by embedding differential equations across many applications.”

Learn from the field

Because design projects are grounded in real problems, it is crucial to make sure that your solution is solving the right problem. If your solution is poorly aligned with the needs of your participants or other stakeholders, it won’t be effective. If your proposed solution is impractical, you won’t be able to implement it well. If it reinvents the wheel, you’re not using your time effectively, you aren’t growing from what is known, and you aren’t participating in a scholarly conversation.

In a design project, you must check

• whether your design problem is the right problem to solve (a needs analysis)
• how your project fits into the landscape of other projects (often, a literature review) by seeking related work.

Needs analysis

As you articulate your design problem, you need to talk to stakeholders: other people who work in the context where this problem occurs and who should be interested in solving it. Because many problems in education are thorny (otherwise we would’ve solved them already!), there are probably a lot of different groups of stakeholders involved in your problem. You probably don’t need to talk to everybody, but you should talk to people from all of the groups.

Start with a listening tour: what do they see as the problem? what systemic barriers do they see that perpetuate it? what are they willing to do to create or implement a solution? what resources or knowledge can they bring to the table?

This listening tour might blend informal elements (e.g. hallways conversations) and formal elements (e.g. interviews). Depending on the scope of your project, your listening tour might lead to forming an advisory board.

Case study: Blaise and Bridging

Blaise thinks about the scope of his problem. He really just wants to improve his course for his students. His primary stakeholders are his students, current and future. Other stakeholders include the rest of his department (who teach follow-on courses as well as decide on Blaise’s tenure and promotion), and faculty in the other departments his students major in.

Blaise chats with a few colleagues in his department about the problems he sees. He invites some physicists and engineers for coffee, where he learns that two of his former students are doing research in their labs. Excited, he plans to ask them informally about their experiences as well. From the education department, he learns about state standards for mathematics and how they connect to his course topics.

As your listening tour develops, you will find that some stakeholders challenge your view of what the problem “really” is. This is important information for you to know, so don’t scoff or ignore it. Depending on their role, they could have valuable insight into the context, represent future barriers or support, or simply help you reshape the scope or aims of your project. As you talk to more people, record and refine your developing thoughts on this problem.

In a formal needs analysis, you will refine the information from your listening tour into a research project in its own right. You might follow-up with more structured inquiry using surveys or interviews, incorporate theory to strengthen your conclusions, and develop presentations like posters or papers on it. In an informal needs analysis, your generative writing is enough for you to proceed.

Case study: Hassan and Simulations

Hassan developed some simulations to help undergraduate biology students visualize ion transport through cell membranes. Students can control physical characteristics of the cell membrane as well as the environments inside and outside the cell. He would like to investigate how these simulations help students think about biochemistry concepts, and improve the simulations. Eventually, he would like to have other faculty use his simulations in their classes.

Hassan has three major groups of stakeholders:

1. For improving his simulations, he needs to engage with students.
2. For conceptualizing what improvement might look like, he needs to engage with other faculty in his department.
3. To have other faculty use his simulations, he needs to engage with other people who teach similar courses to see what their needs are, and how their needs might overlap with his.

If Hassan eventually intends to see outside funding for his design project, he will need to show that his simulations have a ready and willing market at other universities. It’s not enough for Hassan to assert that lots of people teach biochemistry and therefore lots of people will use his simulations; he needs to show that his simulations meet the perceived needs of the faculty who might choose to use them. Hassan should probably do a more formal needs analysis in support of this goal.

Related work

No matter what your design problem is, surely there is someone else in the world who has encountered a similar problem and developed a solution. It is exceptionally likely that many people have encountered related problems and put together related solutions. You need to learn from them, and ideally participate in their communities so that they can learn from you.

You need to ask: how is your work similar or different to other existing curricula, instruments, or approaches?

You have two major avenues for discovering related work. As part of your needs analysis, you’ve probably already started with the first avenue: talking to people to learn what they know. If you’re improving a program on your campus, your stakeholders probably include people who developed the current program. Ask them where they drew their inspiration from. Maybe your friends at other institutions know of similar projects. Ask. Post on social media. As you ask more people, you will find some people with great expertise, pro-social demeanor, and the capacity for help. You might consider asking them to sit on your advisory board if your project is big enough to warrant one.

Research is a collaborative process with other humans. It is normal practice to ask people for advice.

Your second major avenue for discovering related work is the literature review. Oftentimes, the results of development studies show up in conference proceedings, though sometimes they appear in journals. If you can’t get enough information from a conference proceedings (possibly because only an abstract is published), it is normal practice to write to the authors for more information.

Professional societies and curriculum repositories are also great sources of programmatic information, curricula, and pedagogical methods. No matter what you’re trying to do, there is (at least!) one professional society where the people who do that work congregate and share materials.

Does the world need another concept inventory?

Classic DBER research studies from the last century often focused on a single topic in the introductory curriculum (“Topic X”), developed a concept inventory to test students’ understanding of Topic X, and built curricula to improve students’ understanding of Topic X. The evidence for success of the curriculum was improved scores on the concept inventory. As a result of literally hundreds of studies using this design, there are literally hundreds of concept inventories of varying quality and intent.

The hope for these studies was that other instructors would be convinced by the improved scores, adopt the curricula, and use the concept inventory as well. For the overwhelming majority of these studies, this hope has not been realized. There is a glut of concept inventories and other survey instruments, and the vast majority of instructors do not use the vast majority of them. If you want to make a new survey instrument, you should be aware that developing a good one is a lot of work and nobody is likely to use it after your study is concluded.

Does the world need another concept inventory? Probably not.

Depending on the scope of your project, your lit review might be large and formal (e.g. if you are writing a dissertation) or small and informal (e.g. if you’re just working on your class). No matter the scope of your project, it pays to think broadly about topical coverage, accessibility, student populations served, instructional settings, faculty effort, and/or pedagogical principles. As you grow as a developer, knowing more about the landscape will help you design better projects.

Case study: Blaise and Bridging

Blaise learns that Research on Undergraduate Mathematics Education (RUME) community has developed a ton of materials for courses like his.

He looks through some curriculum repositories and finds some materials to use as inspiration for his project. He writes to the developers to ask if they have any more leads, and to thank them for their great stuff.

Researchers are humans and they like it when people like their stuff.

Case study: Hassan and Simulations

Hassan knows that there are a ton of simulations already available, so he divides his literature review into two pieces: first, he looks for simulations on other topics to learn about the state of the art in user design and back end development; second, he looks for simulations on similar topics to learn about how his simulations fit into the landscape of other products.

Because he wants other people to use his sims later on, Hassan also looks at where these other sims are published: broad curriculum repositories? professional society websites? github? He takes notes on how he finds those sims to improve his eventual dissemination project.

Pilot work

Oftentimes, your design project is about improving an existing program or curriculum, not inventing one anew. You might have already engaged in some pilot work, making small-scale changes or trying out new ideas. Alternately, maybe your colleagues developed something and you’re inheriting it.

The parallel processes model for research also applies to design projects: no matter what state your project is in right now, it’s ok to refine and revise your design problem, look to the literature for suggestions, and iterate through cycles of testing and improvement.

Scope and design

No really, what do you want to do?

Project teams spend a lot of time focusing on the problem they want to solve and the expected benefits to the world (and/or their students) should they solve it. But actual project activities can get short shrift in the discussion. Don’t shortcut this part.

What do you want to accomplish?

1. An iterative design will help you articulate what you want to do now, and how what you’re doing now can map to a longer-term future.
2. Be realistic about how much time it will take you to do this work, and think carefully about what will happen if you only get part of it done on time. Articulating a minimum viable product for each iteration of development and testing will help insulate you if your project encounters difficulties, because you’ll have something to fall back on.
3. Work through planning research projects.
4. Develop clear goals: what does success look like? partial success? abject failure?

Case study: Blaise and Bridging

Blaise’s design problem is about connecting course materials with student interests, but he hasn’t yet figured out what that looks like. As he learns about materials from the RUME community, he realizes that he wants to focus on developing student-centered projects for them to engage their interests, not updating problem sets to have a variety of contexts for word problems.

What is your capacity (and your team’s capacity, if relevant) to do this work?

1. Does your project team have appropriate expertise and capacity you will need to do this work?
2. Who else might you add to the project team or advisory board?
   
3. What opportunities for professional growth does this project include? (postdoc mentoring, undergraduate student researchers, new faculty, etc)
4. If you are working with undergraduate developers (e.g. for software, data collection, or analysis) and their work goes poorly, will your project be able to recover? How?

Case study: Maria and Computational Thinking

Maria’s DBR project centers on her class. Much as she would like her whole department to engage in their students in computational work, organizing that effort is more work than Maria has capacity for. She makes sure that her project scope doesn’t depend on her colleagues’ topic coverage of computational topics.

Evaluation

Every project needs evaluation, including design projects.

Evaluation will tell you how (and in what ways) your project is successful. It guides you into making improvements for the next cycle, and helps you document your successes (and growth opportunities) for your stakeholders.

For design projects, evaluation will help you answer these key questions:

• How will you know if your curricula are successful at meeting your learning goals?
• How will you know if your instrument is valid and reliable?
• How will you know if your materials are meeting users’ needs?
• How will your project show early results and engage in an iterative development and testing plan?
  • Is the plan likely to generate actionable, formative insights?
    
  • Does each stage articulate a minimum viable product?

There are lots of ways to do evaluation well depending on your project.

Usability

If your design project is generating a product, such as curriculum, simulations, apps, or a survey, you need to conduct usability testing. Usability testing looks at how people – often students, but sometimes faculty – actually use the materials you develop. This is somewhat independent of learning gains, especially in coursework: if your students learn the material poorly, how do you know what aspects of your materials need to be refined? Alternately, if your students learn the material well, but it takes them so much time they don’t have time for other topics, other faculty are unlikely to use your materials.

There are three core questions in usability testing:

1. How will usability testing data will be collected and analyzed? What data, from whom?
2. How will the ideas from this usability testing formatively feed back into interface changes?
   
3. How will you engage with likely users throughout your development process to ensure that your developing product is meeting their needs?

If your scope is small – just your class, just your department – then you might skip the third question for now. However, as your project grows, you might decide later on that you would like other people to use your materials as well. The best time to start thinking about the third question is when you start thinking about whether other people might want to use your materials, and the second best time is now.

Case study: Hassan and Simulations

Hassan decides that he’s going to collect two kinds of usability testing data. He’s going to look at the software logs of real students interacting with his sims to see where they click and how, and he’s also going to sit down with a few students and watch them interact with his sims while he asks them what they’re thinking about.

Separately from this information, Hassan also plans to check students’ understanding of the key concepts his sims are targeting.

Hassan feels somewhat daunted by the idea of engaging with a community of likely users at this point, but he knows this is really important if anyone else is ever going to use his stuff. He resolves to corral a few friends at a conference and walk them through his sims to see how they react.

Dissemination, propagation, and publishing

Your project grew and learned from similar projects in your community. To be a good community member, you need to share your work with them.

• Which repositories and existing resources will you contribute to?
  
• What materials will you develop to make it easier for likely users to actually use your stuff?
  
• What formats will you make your materials available in?
• Students do not make curriculum decisions; faculty do. If relevant, how will you make sure faculty have easy access and students do not?

Additional considerations: software, apps, sims, AR/VR, and chatbots

Does your project involve developing software, apps, sims, AR/VR materials, and/or chatbots? You should consider these questions.

Software development

1. How will usability testing data will be collected and analyzed? How will the ideas from this usability testing formatively feed back into interface changes?
2. If students are in charge of developing and testing the software interface, show how your development plan can be insulated from the effects of unstable or inconsistent student employment. If this is research for them, show what they are learning; if it’s a campus job, show that you’re paying them competitively.
3. Is this proprietary software for educational purposes? Open source? Freely available (to whom)? What (if anything) is the monetization plan? What license are you releasing it under?

Smart phone or app development

1. Smartphones are ubiquitous, but not universal. How will you account for students who do not have a smartphone? If students have a limited data plan, does that restrict their ability to perform in this setting?
2. Smartphones are diverse in their capabilities. How does this plan account for phones which are too old to install new software, have broken sensors, or limited functionality? Are development plans / software available for both Apple and Android? How will instructors handle differences in Apple or Android apps?
3. Well-designed software can support diverse students with disabilities; poorly designed software will inhibit their full participation. How will the project handle these access issues?
4. Third party software may collect additional data about users and usage for other or unknown purposes; students taking data at home with their cameras may inadvertently include private information about their families or roommates. How will you address data privacy issues for students?
   

Virtual or augmented reality (VR/AR)

1. If your app is dependent on specific hardware, what’s the plan for updates when the hardware changes? How expensive is the hardware, and how ubiquitous?
2. What’s the comparison group or baseline? AR/VR should be compared to simulations where appropriate, not lecture.
   
3. How do your materials handle users with limited visual acuity in one or both eyes? color-blindness?
4. Third party software may collect additional data about users and usage for other or unknown purposes. How will you address data privacy issues for students?
   

AI-driven chatbot/feedback/agents

1. How is your development plan insulated from the fast pace of AI development? What are the costs of switching back end tools?
2. Where do student interactions & data go, who will own that data, how are you protecting privacy?
3. How will you ensure that AI does not give harmful advice or very wrong feedback?
4. If this is a local implementation, that’s good for privacy but bad for dissemination; if an external implementation, that’s bad for privacy and good for dissemination. How are you navigating this tension?
5. Generative AI can take huge amounts of processing power, which corresponds to huge amounts of electrical power and cooling needs. That’s an ecological disaster in the making. How are you handling that?

Additional topics to consider

History

This article was first written on March 6, 2025, and last modified on April 4, 2025.