The meaningfulness of lab-based aggression research
I consider myself to be an aggression researcher. And I often use lab-based research methods. So I guess I consider myself a lab-based aggression researcher. One of the things I often worry about is whether I am producing meaningful, informative, credible research that helps us understand "real-world" aggression. Or am I merely playing a game where I can follow the right methods and say the right things and I can publish the results as a study on "aggression" regardless of the relevance of that research for "real-world" aggression. Am I adding lines to my CV, but not making an inch of theoretical progress?
Adding to my constant angst about this issue, I recently read a blog post by the statistician Andrew Gelman. The post was not a direct critique of lab-based aggression research per se, but the message was clear: Aspects of many lab-based aggression studies are so ludicrous that, from the outside looking in, some real studies appear as if they are an April Fool's Day joke. Aggression researchers use Voodoo Dolls, games that involve noise blasts, etc. in our lab-based research. Yet we claim to be studying when and for whom "real" aggression--like punching, intimate partner violence, etc.--are likely to occur. While I disagree with some of the specific criticisms raised in the blog post, I cannot disagree there is a perception that some aspects of lab-based aggression research appear foolish.
My initial reaction to this post was "you just don't get it." If only this author knew why we do lab-based research or how hard it is to study aggression in a lab or the arguments we make for the validity of our methods, surely the author would not be so harsh. Merely brushing off these criticisms is lazy. Blaming the audience for not understanding is lazy. The real reason this blog post hit me like a gut punch was that some of these criticisms might just be accurate. Or worse, there is a possibility that our entire enterprise is complete bullshit. What if we have been studying how undergraduates play games in our labs and communicating this to the public as aggression research? What if we have deluded ourselves with arguments that serve to mollify our doubts about our research and continue with the status quo more than actually address shortcomings with our research. That would confirm my worst fears about lab-based aggression research.
I am not saying that I believe lab-based aggression research is bunk, I am saying that many people view lab-based aggression research as bunk.
I want to be proud of my area of research. I want the public to believe that my fellow lab-based aggression researchers are good stewards of the public's trust and that we are diligently and objectively working towards tackling difficult and socially-important topics. In short, lab-based aggression researchers need to be MORE critical of our own work than outsiders. We shouldn't have to wait for other people to hold a mirror up to our research to force us to be critical. We shouldn't merely expect people to accept our arguments because we are "scientists." We need to be faster and harsher critics of our own research than outsiders are. We must always be probing our methods and theories for weaknesses. We must be more skeptical of the validity of our methods than other researchers or the general public. We must always keep active the possibility that we are collectively deluding ourselves. We need to earn our seat at the table. This doubt should motivate us and inspire us to produce better research.
In that vein, here are 3 areas that I would like lab-based aggression researchers to focus on.
#1 Prioritize replicability
Taking inspiration from the Data Colada post by Joe Simmons, I want lab-based aggression researchers to prioritize replicability. Essentially, above all, if somebody publishes a finding, I would like to know that if I replicate the methods that I have a reasonable chance to observe the same empirical effect. To me, this takes priority over the validity of our methods and even some methodological shortcomings with the particular study.
Prioritizing replicability is extra important for research with undergraduates. If we opt for convenience in our samples over representativeness, then there is no reason not to make our lab-based research replicable.
Let's use a hypothetical example. Suppose a researcher reports a study that randomly assigns participants to play Tetris or Mortal Kombat. Then the researcher has the participants play a game where they get to send annoying sound blasts back-and-forth to each other. Suppose this hypothetical study finds that participants who played Mortal Kombat sent louder sound blasts than participants who played Tetris, and this was taken as evidence that playing violent video games cause aggression.
There are many criticisms that can be leveled at this hypothetical study: Tetris and Mortal Kombat differ in a lot of ways other than their violent content, participants may intuit the study's hypotheses and behave in a way that confirms the hypotheses (i.e., demand effects), the outcome variable may not actually be an aggressive behavior, etc. All of these are fine criticisms, but I believe they are all moot unless we can establish the basic empirical effect. Namely, if you and I both follow the same methods, do we both get similar results? If all competent researchers can reliably produce the effect, the other problems seem tractable.
Is this a problem in the extant literature? I would argue "yes." We have now documented there is a huge amount of data analysis flexibility in our most commonly-used measure of aggression, there is asymmetry in the funnel plots of one of our most commonly-studied hypotheses, and pre-registered research has failed to detect theoretically-consistent effects. These are essentially arguments about our observed empirical effects, which pre-empts discussions about how to properly interpret these effects.
#2 Focus on validity
Within laboratory settings, we cannot allow participants to behave more than mildly aggressive. We cannot allow non-trivial amounts of aggression to occur. And we certainly cannot allow violence to occur. If we want to make claims about "real-world" aggression that is more impactful than what is allowable in the lab, we must (a) infer that the behaviors exhibited within our lab-based aggression paradigms are valid representations of behaviors that would occur in different settings outside of the lab and (b) that the amount of aggression displayed in our lab-based settings also "scales up" to more impactful levels of aggression. That is, the mild aggression inside of the lab "scales up" to non-trivial aggression outside of the lab.
Sadly, in my opinion, the validity for many of our lab-based aggression methods are weak. Yet, validity information is extremely important for demonstrating the meaningfulness of our lab-based research. I am glad to see people like David Chester starting to do the hard-yet-necessary work of trying to validate some of our existing lab-based aggression paradigms. I don't know how our methods will fare when we expose our existing methods to the scrutiny they deserve. It could turn out to be great, it could turn out to be crap. But I know that process will be super important for demonstrating the relevance of our research in the future.
#3 Be more modest
Until the replicability and validity of our methods are sound, we have to remain humble about our claims. I don't mean that we merely have to make a nod to replicability in our Discussion sections and only have enough validity research to satisfy the editor and peer reviewers, I mean let's strive to have replicable evidence that any competent researcher can produce and validity information that satisfies the most skeptical nay-sayer. I don't know if we will ever achieve these goals, but that is what I believe we should strive for. Then, and only then, should we dip our toes into the waters of making claims about "real-world" phenomena.
I know it is interesting to study extreme aggression or intimate partner violence. I know those are the topics that I want to study. And those topics are probably more fundable than claiming to study "mildly aggressive" behaviors. But let's be real, our current lab-based methods probably just aren't there yet. This is my personal opinion, but I have serious reservations about whether, for example, choosing how long somebody has to hold their hand in ice water is a very good proxy for whether somebody would, for example, get into a fight. I am very willing to be proven wrong, but I must be proven wrong with replicable and valid evidence.
I know the arguments in favor of lab-based research: Within the well-controlled confines of a lab, we may have control over factors that we do not have outside of the lab, it may be easier to use some research designs (e.g., experiments) that allow for certain inferences (e.g., causal inferences), we can test our theories of aggression while placing participants at the minimal risk of harm, we can artificially create theoretically-relevant conditions that may not exist in the "real-world," this research is an invaluable piece to the overall puzzle, etc. But knowing these arguments is not the same as really, really believing these arguments. Doing lab-based aggression research is severely limiting in a lot of ways too. Most prominently, we cannot actually allow more than trivial amounts of aggression to occur in a lab. Let's not pretend like these very real limitations do not exist.
Currently, I see a lot of lab-based aggression researchers who want it both ways. We want to study the "real-world" aggression that is most interesting and most socially-relevant in a lab and we want sidestep all of the thorny issues that come from studying an extremely complex phenomena in an artificial laboratory environment. I don't want to sidestep these issues though. I would like our field to tackle them head-on. I personally and selfishly want better research to justify my belief in these arguments. I want to be able to say with a straight face that the scientific community has done our due diligence on these methods before claiming we have demonstrated a real-world phenomenon. Until then, I would like us to be much more humble than we currently are.
In summary...
Lab-based aggression research is my community. And I am seriously concerned that if we do not change our ways that we will continue to be seen as foolish by some people. We will be seen as a community that continues to fill our journals with each other's research, but ultimately not contributing credible and informative research about the phenomena that we care about. This is not the future I want. Rather than treating those as criticisms of uninformed outsiders, I want to take those criticisms to heart. If somebody believes our area of research is bullshit, I want to be able to say that we have sincerely considered that possibility. Let's roll up our sleeves and get to work.
Adding to my constant angst about this issue, I recently read a blog post by the statistician Andrew Gelman. The post was not a direct critique of lab-based aggression research per se, but the message was clear: Aspects of many lab-based aggression studies are so ludicrous that, from the outside looking in, some real studies appear as if they are an April Fool's Day joke. Aggression researchers use Voodoo Dolls, games that involve noise blasts, etc. in our lab-based research. Yet we claim to be studying when and for whom "real" aggression--like punching, intimate partner violence, etc.--are likely to occur. While I disagree with some of the specific criticisms raised in the blog post, I cannot disagree there is a perception that some aspects of lab-based aggression research appear foolish.
My initial reaction to this post was "you just don't get it." If only this author knew why we do lab-based research or how hard it is to study aggression in a lab or the arguments we make for the validity of our methods, surely the author would not be so harsh. Merely brushing off these criticisms is lazy. Blaming the audience for not understanding is lazy. The real reason this blog post hit me like a gut punch was that some of these criticisms might just be accurate. Or worse, there is a possibility that our entire enterprise is complete bullshit. What if we have been studying how undergraduates play games in our labs and communicating this to the public as aggression research? What if we have deluded ourselves with arguments that serve to mollify our doubts about our research and continue with the status quo more than actually address shortcomings with our research. That would confirm my worst fears about lab-based aggression research.
I am not saying that I believe lab-based aggression research is bunk, I am saying that many people view lab-based aggression research as bunk.
I want to be proud of my area of research. I want the public to believe that my fellow lab-based aggression researchers are good stewards of the public's trust and that we are diligently and objectively working towards tackling difficult and socially-important topics. In short, lab-based aggression researchers need to be MORE critical of our own work than outsiders. We shouldn't have to wait for other people to hold a mirror up to our research to force us to be critical. We shouldn't merely expect people to accept our arguments because we are "scientists." We need to be faster and harsher critics of our own research than outsiders are. We must always be probing our methods and theories for weaknesses. We must be more skeptical of the validity of our methods than other researchers or the general public. We must always keep active the possibility that we are collectively deluding ourselves. We need to earn our seat at the table. This doubt should motivate us and inspire us to produce better research.
In that vein, here are 3 areas that I would like lab-based aggression researchers to focus on.
#1 Prioritize replicability
Taking inspiration from the Data Colada post by Joe Simmons, I want lab-based aggression researchers to prioritize replicability. Essentially, above all, if somebody publishes a finding, I would like to know that if I replicate the methods that I have a reasonable chance to observe the same empirical effect. To me, this takes priority over the validity of our methods and even some methodological shortcomings with the particular study.
Prioritizing replicability is extra important for research with undergraduates. If we opt for convenience in our samples over representativeness, then there is no reason not to make our lab-based research replicable.
Let's use a hypothetical example. Suppose a researcher reports a study that randomly assigns participants to play Tetris or Mortal Kombat. Then the researcher has the participants play a game where they get to send annoying sound blasts back-and-forth to each other. Suppose this hypothetical study finds that participants who played Mortal Kombat sent louder sound blasts than participants who played Tetris, and this was taken as evidence that playing violent video games cause aggression.
There are many criticisms that can be leveled at this hypothetical study: Tetris and Mortal Kombat differ in a lot of ways other than their violent content, participants may intuit the study's hypotheses and behave in a way that confirms the hypotheses (i.e., demand effects), the outcome variable may not actually be an aggressive behavior, etc. All of these are fine criticisms, but I believe they are all moot unless we can establish the basic empirical effect. Namely, if you and I both follow the same methods, do we both get similar results? If all competent researchers can reliably produce the effect, the other problems seem tractable.
Is this a problem in the extant literature? I would argue "yes." We have now documented there is a huge amount of data analysis flexibility in our most commonly-used measure of aggression, there is asymmetry in the funnel plots of one of our most commonly-studied hypotheses, and pre-registered research has failed to detect theoretically-consistent effects. These are essentially arguments about our observed empirical effects, which pre-empts discussions about how to properly interpret these effects.
#2 Focus on validity
Within laboratory settings, we cannot allow participants to behave more than mildly aggressive. We cannot allow non-trivial amounts of aggression to occur. And we certainly cannot allow violence to occur. If we want to make claims about "real-world" aggression that is more impactful than what is allowable in the lab, we must (a) infer that the behaviors exhibited within our lab-based aggression paradigms are valid representations of behaviors that would occur in different settings outside of the lab and (b) that the amount of aggression displayed in our lab-based settings also "scales up" to more impactful levels of aggression. That is, the mild aggression inside of the lab "scales up" to non-trivial aggression outside of the lab.
Sadly, in my opinion, the validity for many of our lab-based aggression methods are weak. Yet, validity information is extremely important for demonstrating the meaningfulness of our lab-based research. I am glad to see people like David Chester starting to do the hard-yet-necessary work of trying to validate some of our existing lab-based aggression paradigms. I don't know how our methods will fare when we expose our existing methods to the scrutiny they deserve. It could turn out to be great, it could turn out to be crap. But I know that process will be super important for demonstrating the relevance of our research in the future.
#3 Be more modest
Until the replicability and validity of our methods are sound, we have to remain humble about our claims. I don't mean that we merely have to make a nod to replicability in our Discussion sections and only have enough validity research to satisfy the editor and peer reviewers, I mean let's strive to have replicable evidence that any competent researcher can produce and validity information that satisfies the most skeptical nay-sayer. I don't know if we will ever achieve these goals, but that is what I believe we should strive for. Then, and only then, should we dip our toes into the waters of making claims about "real-world" phenomena.
I know it is interesting to study extreme aggression or intimate partner violence. I know those are the topics that I want to study. And those topics are probably more fundable than claiming to study "mildly aggressive" behaviors. But let's be real, our current lab-based methods probably just aren't there yet. This is my personal opinion, but I have serious reservations about whether, for example, choosing how long somebody has to hold their hand in ice water is a very good proxy for whether somebody would, for example, get into a fight. I am very willing to be proven wrong, but I must be proven wrong with replicable and valid evidence.
I know the arguments in favor of lab-based research: Within the well-controlled confines of a lab, we may have control over factors that we do not have outside of the lab, it may be easier to use some research designs (e.g., experiments) that allow for certain inferences (e.g., causal inferences), we can test our theories of aggression while placing participants at the minimal risk of harm, we can artificially create theoretically-relevant conditions that may not exist in the "real-world," this research is an invaluable piece to the overall puzzle, etc. But knowing these arguments is not the same as really, really believing these arguments. Doing lab-based aggression research is severely limiting in a lot of ways too. Most prominently, we cannot actually allow more than trivial amounts of aggression to occur in a lab. Let's not pretend like these very real limitations do not exist.
Currently, I see a lot of lab-based aggression researchers who want it both ways. We want to study the "real-world" aggression that is most interesting and most socially-relevant in a lab and we want sidestep all of the thorny issues that come from studying an extremely complex phenomena in an artificial laboratory environment. I don't want to sidestep these issues though. I would like our field to tackle them head-on. I personally and selfishly want better research to justify my belief in these arguments. I want to be able to say with a straight face that the scientific community has done our due diligence on these methods before claiming we have demonstrated a real-world phenomenon. Until then, I would like us to be much more humble than we currently are.
In summary...
Lab-based aggression research is my community. And I am seriously concerned that if we do not change our ways that we will continue to be seen as foolish by some people. We will be seen as a community that continues to fill our journals with each other's research, but ultimately not contributing credible and informative research about the phenomena that we care about. This is not the future I want. Rather than treating those as criticisms of uninformed outsiders, I want to take those criticisms to heart. If somebody believes our area of research is bullshit, I want to be able to say that we have sincerely considered that possibility. Let's roll up our sleeves and get to work.
Great response to criticism.
ReplyDeleteThanks, Uli.
Delete