2.5.2 Research Methodology

Table 1 Methodology for Research

Pose significant empirical question(s). Identify gaps in current knowledge. Describe newly observed phenomena. Link research to relevant literature. Locate a comprehensive and up-to-date set of sources. Outline the current status of theory, conceptual models, and research methods. Synthesize findings from previous research. Refine or reformulate a research question or hypothesis. Pose a question or a hypothesis. Phrase it in a way that is appropriate to the research approach used, whether qualitative or quantitative. Set boundaries limiting the focus of the investigation. Employ methods for the direct investigation of the question or hypothesis. Align the research method with the question or hypothesis. Adopt accepted methods or devise new methods. Design experiments that avoid subjectivity and lend themselves to being replicated by others. Provide an explicit chain of reasoning that leads to a conclusion. Organize the data for analysis and interpretation. Describe limitations that are due to context, method, or errors in measurement. Replicate it and generalize it across studies. Conduct quantitative studies under similar conditions. Conduct qualitative studies in different contexts. Disclose research to peers to invite their professional scrutiny and critique. A small team of peers checks the work for its clarity, defensibility, and significance. Present the research to the larger community to invite criticism or replication, and to add to the existing knowledge base.

Qualitative Versus Quantitative Research

Researchers can use qualitative or quantitative approaches to the research process. It is useful to recognize the differences between the two approaches. These are shown in Table 2. Qualitative research is inductive and seeks to develop new theory. Quantitative research is usually deductive and is designed to test existing theory.

Methodology for Research

The Research Methodology shown in Table 1 is advocated by the National Research Council as being applicable to all fields of scientific inquiry (Shavelson and Towne, 2002).

Discussion of the Methodology

Empirical Questions—Research is fundamentally empirical, which means that it is derived from observation, experiment, or experience. Research is undertaken to resolve a suspected inconsistency among the findings of others, to fill a gap in current knowledge, to describe a phenomenon, or to reframe an old problem to adapt it to changes brought by new technology.

Relevant Theory—It is necessary to thoroughly review the existing literature to develop a well-stated question or hypothesis, to employ appropriate research methods, and to interpret the results in light of existing theories and findings. The researcher should read seminal works that may be decades old as well as recent work. The researcher should also examine all relevant areas of practice; methods, instruments, equations, and conceptual models from diverse fields may provide powerful approaches. The final step in the literature review is to synthesize pertinent findings with the question or hypothesis.

Refinement of the Question or Hypothesis—As the research progresses, the research question or hypothesis can transform significantly. This can happen when a previously unidentified data gap becomes apparent. It can also happen when new methods are identified that show better promise in answering central questions. Such situations require researchers to reframe the question or hypothesis. In qualitative research, the research question is stated using exploratory verbs such as discover, explore, describe, report, while it avoids directional words such as affect, influence, cause, and relate. In quantitative research the hypothesis discriminates between different variables in relation to a response variable.

Methods—Research methods must be aligned to the research question or hypothesis. These methods should support the systematic recording and documenting of observations to prevent subjectivity. Examples of common qualitative methods are observation, open-ended surveys, interviews, and focus groups. Examples of common quantitative methods include randomized controlled trials (with “treatment” and “control” groups), matching, post-test-only designs, establishing baseline data, longitudinal designs, and variation of dependent variables (Olds, Moskal, & Miller, 2005). Research in education frequently employs a combination of quantitative and qualitative methods, referred to as “mixed methods.” Qualitative methods are usually performed first to establish the context and to identify important variables, and quantitative methods are used later to rank factors or to study interactions.

The complexity of the experiment design must be appropriate to the resources available. Delimitations can be used to narrow the scope of a study, allowing only certain variables to vary (Castetter & Heisler, 1977; Creswell, 2003). When reporting the research methods, it is important to recognize known limitations in the study so that others are aware of issues they may encounter in generalizing findings. Any research method that involves human subjects, such as education research, must be reviewed by the host institution’s human assurances committee.

Much education research is performed by multi-disciplinary teams. Different disciplines have different jargon, different ways of conceptualizing parameters used to measure behavior, and different methods of behavior, and different methods of inquiry. In multidisciplinary research, various tasks in the research plan are parsed out to each expert who employs the defensible methods from his or her own discipline to accomplish those tasks. This is in contrast to interdisciplinary research, in which the methods are a hybrid of tools and techniques found in contributing disciplines (NAS/NAE/IM, 2005).

Reasoning/Analysis/Conclusions—Data collected to answer the research question or hypothesis must be analyzed and interpreted in the context of pre-existing findings. It is useful to recheck alignment in the research question, the methods, and the findings. If a misalignment is detected, it may be possible to rephrase the research question to reflect what hypothesis was really tested. Otherwise, the misalignment must be reported as a limitation. Another limitation that should be considered is experimental error and the degree to which such errors could bias the findings. All findings should also be reviewed within the context of alternative theories. The endpoint of the reasoning process is a well-stated conclusion, or set of conclusions, that summarizes the findings, context, and limitations. The conclusion(s) should provide a qualified answer to the research question or hypothesis.

Replication/Generalization—Replication refers to ability of the investigation to be repeated in more than one setting to reach a similar set of conclusions (Shavelson & Towne, 2002). Qualitative studies must include rich discussion of the context in which observations were made and the limitations in data collection so that others can test conclusions in related settings. In quantitative studies, convincing proof of reliability in results should be given and summarized statistically.

Peer Review—The first step in peer review involves a small team of reviewers who will check the work for its clarity, defensibility, and significance. The researcher(s) address the problems or suggestions raised by the reviewers to the satisfaction of an editor. Then the work is disseminated to the research community. The goal of research reporting is “to communicate the findings of the investigation, to open the study to examination, criticism, review and replication by peer investigators, and to integrate the new knowledge into the field” (Shavelson & Towne, 2002).

Scenario 1—Quantitative Research

Dr. Gwey, a groundwater engineering professor, is investigating the transport of pathogens in groundwater. Historically it was thought that porous media, such as sand aquifers, filtered out microbes. Recently, however, it has been determined that microbes can be transported through porous materials such as medium to large-grained sand particles.

Empirical Question—Is Bacillus cereus more mobile in groundwater in its germinated (live) form or in its sporulated (dormant) form?

Relevant Theory—It is believed that colloid (e.g., clay particle) transport theory adequately describes the transport of micron-sized particles in a porous material such as soil or sand. This theory includes transport-aiding mechanisms such as dispersion (electrostatic repulsion) and transport-preventing mechanisms such as attachment (electrostatic or chemical attraction) and straining (physical trapping).

Refinement of Hypothesis—It is difficult to distinguish the effects of straining and attachment in the same experiment, so Dr. Gwey imposes the delimitation that only attachment will be investigated. To ensure that little to no straining occurs in the experiments, he plans to use only large-grained sand as the porous medium. The following refined hypothesis emerges: Is attachment between microbe and silica sand particles stronger for the germinated (live) form or the sporulated (dormant) form?

Methods—Dr. Gwey is familiar with the established methods for performing column transport studies; however, he has limited experience with microbes. Therefore, he seeks the collaboration of a disciplinary expert, microbiologist Dr. Jones. Together, they plan methods that will meet the multidisciplinary standard of being defensible for either an engineer doing column tests or for a microbiologist inducing sporulation and assessing microbial surface charge. The two collaborators conclude that it is not necessary for them to develop new interdisciplinary methods.

Reasoning/Analysis/Conclusions—Dr. Gwey and Dr. Jones conduct the experiments and find that the germinated form of the microbe is more mobile than the sporulated form. However, during the course of the investigation, Dr. Jones observes that the charge of the sporulated microbes varies significantly over their lifetime. Although variability of charge among spores from the same day (24 hours) age-group is found to be insignificant, spores more than 14 days old have a significantly different charge than those that are 1-5 days old. Because charge can affect attachment, and the attachment of spores to sand in the column experiment was explored with spores varying in age from 2 days to 28 days, this potential limitation of the finding (a confounding variable) is reported in the conclusions section of the final manuscript.

Peer Review—The two investigators decide to report their findings in a multidisciplinary environmental-quality journal rather than a groundwater journal or an applied microbiology journal. When the paper is submitted for peer review, the editor selects one reviewer who is an expert on groundwater transport of microbes and another who is an expert on modeling porous media flow. The expert in the transport of microbes identifies two additional limitations of the completed work. Dr. Gwey, a junior faculty member, is distressed about the review and feels that all the contributions of their findings have been overshadowed by the new limitations. Dr. Jones, a more senior faculty member, coaches Dr. Gwey to rewrite the final conclusions in view of the posed limitations. The two of them soon become excited about the value of the newly-identified limitations for making their paper stronger and for their future research plans. The second reviewer provides comments and questions that demonstrate that he or she lacks knowledge of microbes. This prompts the authors to improve their definitions and explanations to reach an interdisciplinary audience that has less knowledge about biology.

Scenario 2—Mixed-Methods Research

Dr. Morales and Dr. Wong are committed to improving undergraduate performance in teamwork. They are also committed to having a well-aligned transition from the freshman class, taught by Morales, to the sophomore class, taught by Wong. As they discuss the strategies they employ to help students develop teaming skills, they identify a structural difference between their two classes. In the freshman class, student teams regularly analyze the productive and unproductive dynamics that occur among team members. They do this as part of their team meetings, but this does not include peer evaluation. In the sophomore class, students are required to submit written peer evaluations at the end of the semester, and these peer evaluations are used in grading the students. The two faculty members realize that they have an opportunity to poll the students and ask them how peer evaluation impacted their performance. Their population of target students will be students who have taken both classes.

Empirical Question—Does student commitment to teams improve if students know that they will be evaluated by their peers and that the professor will use those evaluations in their course grades?

Relevant Theory—Morales and Wong search the literature on team member accountability and peer evaluation. They also speak to a sociologist colleague about preparing a survey. They would like to prepare survey questions with scaled answers (e.g., strongly agree, agree, neutral, disagree, and strongly disagree) so that they can quantify the responses. However, because they are asking questions relating to personal commitment and other affective issues, they don’t want to bias the answers by the wording of their questions. The sociologist affirms that their concern is valid, and suggests that they address this concern by preparing a survey with both types of questions. He also urges them to ask a peer group of students to review the survey confidentially for unclear or threatening questions.

Refinement of Question—After reviewing literature on team accountability, Morales and Wong recognize that they are not asking whether team performance improves because of the pressure of peer evaluation, neither are they seeking to directly measure changes in student commitment. They refine their research question to ask, “Do students perceive that their commitment is higher if they know that they will be evaluated in writing by their peers as part of the course grade?”

Methods—On the basis of their consultation with the sociologist, Morales and Wong prepare a two-part survey that includes both scaled questions and open-ended response questions. They plan to query all the students who have taken both courses, so they ask students in another major to review the survey to identify problems with questions. They are impressed with the insights these students bring to this task. When they finalize the questionnaire, they ask the university’s human assurance committee to review it.

Reasoning/Analysis/Conclusions—Students articulated logical explanations for the value of peer evaluation, but they did not perceive that it strongly influenced their behavior . In all cases, the scaled responses to questions such as, “When we evaluated each other I worked harder,” were neutral. Although the research question yielded a non-spectacular finding, a valuable insight did emerge. A majority of the students identified mid-project peer evaluation as valuable because it allowed them to let the professor know, before the project was done, if someone wasn’t pulling his or her weight. Although this finding was not aligned with the original research question (rather, it expressed a concern about enforcement), it provided evidence that students wanted a checkpoint when they could trigger an intervention from the professor.

Peer Review—The paper was submitted to an engineering education journal, and, after several limitations were identified by social science and education researchers, the revised document was published.

Concluding Thoughts

Deliberately navigating the steps in the Research Methodology facilitates collaboration within and between disciplines. Furthermore, writing proposals with the Research Methodology in mind increases the likelihood that the research will be ranked highly by peers on funding panels. Finally, by explicitly referencing the research process while they work, researchers help graduate students and other junior investigators conceptualize and internalize successful behaviors that are modeled by their advisors and mentors.

References

Castetter, W. B., & Heisler, R. S. (1977). Developing and defending a dissertation proposal. Philadelphia, PA: Graduate School of Education.

Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2^nd ed.). Thousand Oaks, CA: Sage.

National Academy of Sciences, National Academy of Engineering, & Institute of Medicine (NAS/NAE/IM). (2005). Facilitating interdisciplinary research. Washington, DC: National Academies Press.

Olds, B. M., Moskal, B. M., & Miller, R. L. (2005). Assessment in engineering education: Evolution, approaches and future collaborations. Journal of Engineering Education, 94, 13-26.

Shavelson, R. J., & Towne, L. (Eds.). (2002). Scientific research in education. Washington, DC: National Academy Press.