If you are looking to design an education research project, the exercises on this page will help you. If you’re looking for advice on how to plan research projects is a good choice. You might also look at research process models to help you think about how research projects progress, or Iterative Design to think about to structure them for maximum likelihood of success.
If you’re doing video-based observational research, here’s a good companion piece to consider. If you’re thinking about Design-based research, check out this article.
More broadly, check out all articles tagged with “doing research”.
What does my project need?
Every project in education research needs to address four areas. While the details of these areas can be (should be) emergent, well-formed and successful research projects identify as much as possible ahead of time.
Every project needs:
| Area | Details |
|---|---|
| Research question | What do you want to study? |
| Access | What populations can you study, and how much time / which modalities are available? |
| Theory | What theoretical frameworks guide your work? |
| Methods | How will you generate observations and interpret them to become data? How many observations? |
Additionally, when you present your work for publication or funding, you will need to consider two more areas:
| Area | Details |
|---|---|
| Relevance | What intellectual merit or broader impacts will this project have? Why is this important or interesting work? |
| Audience | Where are you planning to publish your work? What counts as novel and important to them? |
We’re going to leave these two aside for now because they reference a broader sense of where the research community is, what societal needs are, and how your project fits into a much larger narrative. Those considerations are outside the scope of this guide, though you might consider reading ahead to other guides on writing. Let’s work on the four primary areas.
In a good research project, the four areas are all tightly related and supportive of each other. You should develop them in concert with each other. The exercises on this page will help you design a research study, and they will also help you develop your design skills in general.
Details of the four areas
Research Questions
Your research question(s) tie together your theoretical frameworks, methods, and access. They give purpose to your data collection and analysis. Answering them generates new knowledge about human behavior. In the ordinary process of doing research or thinking about the world, you will ask lots of questions. As you pursue some of them, you’ll develop follow-up questions and related lines of inquiry.
Research Question templates
If you’re in the very beginning stages of thinking about your project, you might need help brainstorming some possible research questions. Here are some templates to get you started. It’s not an exhaustive list.
- Theory X says A, but theory Y says B. How can they be made commensurate?
- This paper used population A, but I have population B. How can I apply their findings to my population?
- Surveys shows that students can do X. What is the actual process of learning to do X?
- What are the moderating factors which control success at task X?
- Our previous work shows X happens sometimes. Why does X occur?
- What’s better at teaching X, curriculum A or curriculum B?
- How do teachers make sense of X in light of Y?
Making your research question better
When you have an idea about what you’d like to investigate, you need to refine your ideas into a research question that suggests how you will answer it and how you will know when it is answered.
A good research question has the following properties:
- It is phrased a question, not a statement of problem
- Specific enough to be answerable
- Open to complicated, robust answers
- Interesting to investigate
You will want to have two versions of your research question: one that uses regular language, and a longer one (possibly with subquestions!) that uses specific, technical language.
This exercise helps you refine your ideas into a research question.
Write your question in the form of a question. Use regular language.
Make it specific. Your research question needs to be answerable in principle, and your research design needs to have a high likelihood of answering it.
- If your research question uses comparison language, what are you comparing? For example, if your research question is about whether a new curriculum is “better” or if students are learning “more”, what will you be comparing it to? Do you need to collect baseline data? Will you be able to run a treatment group and a control group at the same time?
- If your research question uses development language (e.g. “learning”), over what time are your subjects changing? An hour? Four years? Their lives? How will you know if change is durable? how will you know if it occurs at all?
Open it up to complicated, robust answers.
- If your research question has a binary answer (“does X happen?”), revise it to permit a more subtle answer (“to what extent does X happen?”; “how much does Y mediate X happening?”; “under what conditions can X be optimized to happen?”)
- If your research question is too specific (“what is the correlation coefficient of X with Y?”), you are too specific. Revise your question to have more robust answers (“how do X and Y relate?”; “what factors affect X and by how much?”)
Check: does answering this question sound like fun to you?
- If you refined your question so much that finding the answer sounds boring, trivial, or insurmountably hard, try new ways to refine it so that it really captures your interest in this topic.
In the process of refining your research question, you might realize that there are a bunch of interesting sub-questions to pursue. Go ahead and list them out, and follow this same process to refine them. Your refinements probably also include technical language and reserved words that mean something specific to the research project. Define each reserved word and link it to specific theoretical frameworks, methods, or data streams.
A good research question is a living question. As you interact with theories and data, it will necessarily change. The more specific you can make it in the beginning, the better you will be able to see it change and adjust your future work in an intentional way. You may find it useful to read Engle et al’s “Progressive Refinement of Hypotheses in Video-Supported Research” to understand how research questions can change and in response to repeated engagement with data, and Iterative Design to think about how to design for this feature.
Access
The Access area is about practical constraints on your project: what populations do you have access to, and in which modalities? how much time do you have, and which analysis resources can you marshal? Of all the areas, Access is the one which is usually fixed earliest in the project, because the kinds and amounts of data you have access to are usually determined before you can collect any data at all, and the scope of your project is usually outside your control.
Questions that detail your access to data:
- What kinds of people will you measure? Some examples: introductory students, pre-service teachers, graduate teaching assistants, third graders in a specific elementary school.
- In what modalities can you collect data from them? Some examples: I can talk to them individually in interviews once per person, I can video them in class every day, I can put a problem on their final exam, I have three years of archival data but cannot collect new data, etc.
- How many people / how much data? One or two significant figures are ok here: about 10 students, about 300 students, about 20 hours of video, about 100 matched pre-post tests, etc
You probably can’t answer all of these questions alone. Get specific guidance from your collaborators, advisor, and people who control your access to research subjects (their instructors, their principals, the registrar, the data librarian, etc) – the members of your Advisory Board. At early design stages, you don’t need to seek IRB approval yet, and you don’t need written permission from every stakeholder. When your study is more fully designed, you will talk to these people again to firm up the details of your access and adjust your research questions and methods.
Questions that detail your access to resources:
- How long can you spend collecting data? How long analyzing it?
- How many researchers will be involved in data collection and analysis? What are their skill levels?
- How much data (and what kinds) can you reasonably expect to collect / analyze in the amount of time and effort that are available to you?
It is entirely possible that you have access to more data and analysis resources than you will need or use in your project. That’s great! You don’t need to collect (or use) everything. Alternately, you might not have enough access (or the right kind of access) to do the study you really want to do. That’s disappointing. You will need to adjust your research questions and methods in light of how much (and what kinds) of data you can collect or analyze with your resources.
On rare occasions, you can use your research questions to argue for access to more resources or different modalities. For example, suppose your research question is about student epistemology and persistence, and you already have access to students’ attitude survey scores. You might be able to ask the registrar for students’ demographics and final grades to enhance your analysis.
Theoretical frameworks
The role of your theoretical frameworks is to tell you why your observations are meaningful and in what ways your analyses generate new knowledge. Without a theoretical framework, your observations are meaningless and your work is unpublishable.
The primer on theory covers what you need theory for, an organizing framework for deciding which theory or theories to use, some theory options in education research, and some other common considerations.
The best theoretical frameworks are a) explicit; b) well-matched to your research question and methods; and c) intentionally chosen. There isn’t a “best framework” for everyone, or even every research question, and there are a lot of options available.
I’m using “Theoretical framework” in a loose sense to include things like knowledge-in-pieces, communities of practice, speech genres, models of institutional change, error-based learning, etc. (I’ve used all of these, and there are a lot more out there.) Some people use the phrase “theoretical-methodological framework” to acknowledge that good frameworks must tie theory, methods, and data together.
In this article, I’m not going to explore those subtleties.
Methodology
The role of your methodology is to tell you how to generate observations to answer your research question, how to convert those observations into data, and how to analyze that data. While theoretical frameworks are mostly concerned with why those observations and analyses are meaningful or interesting, methodologies are mostly concerned with the practicality of converting observations into analyses and the reasons for those analyses.
It is becoming a lot more common in discipline-based education research to be explicit about the methods that you choose and why. While it used to be sufficient in papers to outline what you did, now you also need to discuss why you did it and how it fits into a broader research tradition.
Many projects – especially large projects – coordinate multiple kinds of data and multiple kinds of analyses in order to make robust conclusions. This is (broadly) called “mixed-methods” or “multi-methods” design. There are lots of ways to mix methods well (and some ways to do it badly). If your research questions demand multi- or mixed-methods, you will need to write sub-research questions and choose theoretical frameworks for each method, and you will need to think about how the analyses from each method will interact to generate new knowledge. Before you jump into a mixed-methods design, ask yourself carefully if your research questions really warrant it, and if your access really allows it.
Sources for theoretical frameworks and methodologies:
There are books and papers written on this subject. Some of them are textbook-style for students; others are monograph style for researchers. To find them, you will have to step outside your particular discipline and look at the broader educational research literature, the learning sciences, or psychology (depending on your research questions).
- The Journal of the Learning Sciences has an excellent series on methodology and many beautiful papers on theory.
- Reviews in PER has a few papers with brief overviews of some kinds of methods and theories.
- Probably the most highly-cited book on methods is Creswell’s book on research design. It is not comprehensive, but it is extensive.
- There’s a quick overview of coding qualitative data (aimed at UX researchers) on Delve
- Shayan Doroudi wrote an excellent primer on learning theories.
- When you read papers, make note of their frameworks and methods (and their citations!).
You can also talk to other humans!
- Talk to your advisor or collaborators about what they would use (or require you to use).
- Write a one-page prospectus that outlines what you want to do and why you think it’s interesting or important, and send it to someone who does similar work. Ask them (nicely) for suggestions.
Develop the four areas in concert for a specific research project
In this exercise, you’re going to iteratively refine each of the four areas so that they are tightly integrated with each other.
On a whiteboard, write down a preliminary research question. If you don’t have a preliminary research question, start with one of the research question templates or do the exercise on making better research questions.
Write down what kind of access you have. Be specific about what populations, what kind of resources you have to undertake this research and how long it will take, and what kind of data modalities are available to you.
If you’re structurally constrained (by your funder, or your advisor, or your equipment) to use particular methods or theories, write them down as well.
Return to your research question, and update it so that it is constrained to the populations you have access to (as well as other structural constraints). It will get longer and more detailed. That’s great.
Which theories support your research question? Write them down. Amend your research question to explicitly reference at least one theoretical framework. If your question is about how individuals develop, you might look at the Resource Framework. If it’s about how communities form, try Communities of Practice. If you don’t know any theories, what have you read that makes you think this would be an interesting research question? You might need to use two or three frameworks in concert with each other to fully answer your research question.
What kind of data will you collect? Here’s a quick overview of the common kinds of data.
- Make sure that your access permits this kind of data, that your theoretical framework will be able to use the data from it, and that it will be able to answer your research question.
- Amend your research question and theoretical framework(s) to reflect the kind of observations you will collect. You might triangulate across several different data streams: preliminary surveys will identify participants for in depth interviews, and you ask them for their homework, for example.
How much data will you need to collect or analyze to show the effects you are looking for? Part of the answer to this question is about where you plan to publish your results at the end of your study: if you want to exhaustively prove your solution, you need a lot of evidence, but if you are only looking to prove its existence, you don’t need as much. Even a thoroughly theory-driven, theory-generating project needs something data-like (reinterpretation of old data, for example).
- If your project is based on finding patterns of human behavior, there are formalized methods for estimating effect sizes (generally known as a “power analysis” or “power estimate”). A quick-n-dirty estimate is that your error bars will go like 1/sqrt(N). If you can estimate differences in your treatments based on the literature, you can guess about how many subjects you will need. If your estimates suggest you will need many more subjects than you have access to, you need to revise your research question.
- If your project is based on finding cases of human behavior, you will need to think carefully about episode selection. How many episodes will you need to prove your point substantially? A good estimate is 3-5, most of which should be similar and one of which should be contrasting. More or fewer are possible.
Adjust your research question and methods in light of how much data you will be able to generate.
Write down a preliminary data collection and analysis plan.
- You may find that drawing a logic model or conjecture map is helpful. You may find that a narrative of what you’re planning to do and how is helpful.
- Compare your plan with your chosen theories and research question. Does your plan make use of your theories? Is it likely to answer your research question? Is it possible with the time and resources you have allotted?
Imagine that everything in data collection goes swimmingly and all of your data are fantastic. What does the answer to your research question look like? To what extent can you answer it with your methods and access? If course, you won’t know exactly what the answer will be – if you already knew, it wouldn’t be research – but you should be able to guess at an approximate shape to the answer.
- If you think you’ll need additional kinds of data to better triangulate an answer your question, amend your access and methods.
- If you think you’ll need a lot more data than you can get, amend your research question.
Another process which can help with intentional research design is conjecture mapping; you might also consider the emergent processes outlined in “Progressive Refinement…”. If your research project is larger than you can complete in one semester, you are strongly encouraged to think about an iterative design using the principles in Planning Research Projects. Alternately, if your research project has a substantial curriculum development aspect, you should consider Design-Based Research (DBR). Lastly, you might consider whether your project is research at all: maybe you’re doing evaluation, not research.
Develop your skill in designing research projects
Exercises
These exercises will develop your skill in designing research projects. If you do them a lot, then designing research studies will become a habit for you.
When you read papers, imagine using their theory and methods with a different population, or using their access with different theory and methods, or their research question with different access and methods. Make notes about your choices, so that later you can cite these papers in your own work. This exercise also makes you a better reader of papers.
Read through the abstracts of NSF’s recent awards for either IUSE or EDU:CORE. For every project, imagine that you have been given a supplement to do some research related to that project. What would be interesting? What would be possible, but not personally interesting? What would be exciting, but you don’t know very much about? You should be able to find something personally interesting or exciting in almost all of the projects. Design a study for each. This exercise also makes you a better citizen of the broader education research community, because you will know a lot more about the shape of current work in the community.
Read through the NSF’s upcoming deadlines for programs sponsored by Directorate for STEM Education (EDU), particularly the DUE and DRL divisions. For each one, sketch out a research study: what would you investigate? who might you partner with? This exercise also makes you a better researcher, because you will become more knowledgeable about how to frame your work to get funding.
Generative writing is the biggest tool in your researcher toolbox. Go back to your old notes about research designs, and enrich them with your new thoughts as you learn more.
Readings
Check out all articles in this Handbook tagged with “doing research”.
Read this delightful piece by the former editor of Sociology of Education.
Read this paper on quality in qualitative research design: Tracy, S.J., 2010. Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative inquiry, 16(10), pp.837-851.
Read this paper on elements of research project designs.
Read this overview on designing projects for the scholarship of teaching and learning.
Additional topics to consider
History
This article was first written on January 1, 2015, and last modified on May 30, 2024.