Reliability, Validity, Generalizability of Research & IRB

Reliability, validity and generalizability are perhaps the most important aspects of a research methodology. In the following part, I will firstly compare these concepts based on my own experience and understanding, and then I will touch upon some ethical issues.

In LeCompte’s textbooks, it is mentioned that reliability and validity are the principal criteria for judging the quality of research and “conventional criteria for reliability and validity derive from positivism and the requirement for experimental research.” From this point of view, and as a quantitative researcher, I am not optimistic about the quality of the studies many researchers have done and I also question the quality of most of the papers that have been published in the realm of quantitative research. It is not uncommon to see that in the limitations part of papers, the authors always say “Thus our sample comprised a subpopulation of potential population….”, “It would be worthwhile to replicate the study using a general population of …” Frankly speaking, I think these are just the excuses of these authors for their not being able to do really high quality research. What is more frustrating is the fact that authors even say that their results could be generalized to the whole population based on the “subpopulation of potential population”. Sometimes they even tout that that their result can have implications for policy makers, practitioners and other researchers.

Another thing I am not satisfied with the current quantitative research is that I find most of the researchers shy away from creating their own instrument even if they know that their research context needs a specific context dependent instrument. For example, in the field of privacy research, people’s “privacy concern” is an important independent viable in determining people’s behavior intentions. In the whole literature, there are only 2 instruments that could be used to measure people’s privacy concerns. One is Jeff Smith’s “Concern for Information Privacy” (CFIP), which is used to measure people’s general privacy concerns. Another is the “Internet Users’ Information Privacy Concerns” (IUIPC), and it is used to measure users’ privacy concerns towards online behaviors. However, many researchers from medical research, location based service studies, or marketing researches, who know that their research context needs a new instrument to measure their participants’ privacy concerns, are reluctant to create and test new instrument to measure their participants’ “privacy concern”. The reason for their reluctance is obvious. It is really a very complex procedure to test the reliability, the validity of a new instrument. I have heard many times from different researchers saying that “if you can find an existing instrument, never create your own”. While I admit that by doing like this, researches could be much easier, I really doubt the result of these studies.  On the flip side, although many times qualitative research are criticized by quantitative researchers that “qualitative research is hard to be generalizable”, I think the nature that qualitative research could provide rich, deep understanding of a specific context, could ensure that the result of qualitative research to be more reliable and valid. From this perspective, I like qualitative research better than I like quantitative research because qualitative research is revealing the truth.

As to research ethics, I think IRB is a good thing, but it is too rigid and not so flexible. Although IRB officers always want to convey to researchers that IRB is not intend to stop the research, but to facilitate the research with the researchers, I would say that it do sometimes prohibit researchers conducting researches. This is analogous to a search warrant. In most of the police movies, because the process of applying for a search warrant is too complex and may waste a lot of time, the brave, handsome FBI detective usually acts on himself at the risk of violating the law because he usually has no search warrant with him. At the end, usually the result is that the police with a search warrant appear right after the detective has successfully caught the criminals. It is the same as IRB. When doing researches, there is always something that is unexpected and the chance that the opportunity would come again is very low.