The issue of contradictory results of biosafety studies

Michelle Marvier from Santa Clara University (USA) illuminates the reasons for conflicting study results and new approaches on how to manage such uncertainties and to improve the significance of biosafety research results. She is promoting large open-access databases for biosafety studies and the application of meta-analyses. Such meta-analyses use the results of numerous studies to get a statistically more reliable picture of the environmental impact of GM crops.

GMO Compass: What is your personal motivation to conduct research on the environmental effects of GM crops?

Michelle Marvier: I have always loved growing vegetables in my home garden, and I'm interested in food production and the environmental impacts of agriculture, in general. As a graduate student, I worked on several agroecology projects, sampling beneficial insects in cotton, strawberries, and cabbage. I also really enjoy thinking about experimental design and statistical analysis. So, that combination of interests led me to look a bit more critically at the experiments being used to study potential risks of GM crops.

GMO Compass: All, or at least most, of the concerned studies were published in peer-reviewed scientific journals. Is the peer-review process not rigorous enough?

Michelle Marvier: I don't think that this indicates there is a problem with the peer-review process. It is important that results get communicated to the broader scientific community, even if the study is somewhat preliminary in the sense that it might involve high levels of exposure in a lab setting. These studies are important because they spur more detailed attempts to quantify exposure levels in the field.

GMO Compass: Do you think that the design of environmental impact studies should follow obligatory guidelines to prevent the publication of statistically insufficient studies?

Michelle Marvier: I do think there is one simple thing that can be done to improve the credibility of ecological risk assessment studies and that is to increase the numbers of animals exposed to the treatments. Larger studies, involving more subjects, are more believable than smaller studies. In lab studies, in particular, it would not cost very much to increase the numbers of insects exposed to the GM crops and the relevant control treatments. On the other hand, field studies are extremely labour intensive and it is far more understandable that the number of plots or fields used in such studies will continue to be on the small side.

GMO Compass: You conducted what are known as meta-analyses to improve the significance of single studies. How do they work?

Michelle Marvier: Meta-analysis has gained prominence in clinical trials and the medical arena, where one has to be very careful before deciding that a treatment is relatively risk-free. The crux of meta-analysis is the realisation that an absence of significant effects in a collection of individual studies is not necessarily as convincing as it might first seem. The problem is – as already mentioned - that the individual risk assessment or toxicity studies may be poorly replicated and thus have low statistical power. For example, a study might expose three groups of honey bees to a Cry protein incorporated into a standard diet and three groups of honey bees to a control diet, lacking the Cry protein. No matter how many honey bees are in each "group," the number of replications of the study is only 3. With such low replication, only a large and very consistent difference between the two treatments in the survival, development, or growth of the honey bees could be detected as statistically significant. The weak statistical power of these studies means that a finding of no significant effect is not very convincing.

Even a tally of the results – that means the number that found versus didn’t find significant effects - from a collection of weak studies is not much more convincing than the findings of each individual study on its own. Among statisticians, such tallies are called "vote counts" and if one thinks about it a bit, it is pretty obvious that even a dozen studies, all with poor replication, finding no effect would not constitute convincing evidence that no effect actually exists.

Fortunately, meta-analysis provides an alternative to vote counting. By statistically combining the observed differences between treatments and controls across a group of independent studies, and weighting the results of each experiment by the variance in the data, one comes up with an estimate of the general effect size across experiments. This resultant effect size is much richer than simply stating that 9 of 11 experiments or even 11 of 11 experiments found no significant impact on honey bee survival. It is possible that, by applying meta-analysis to a set of poorly replicated studies, a more reassuring picture may emerge. Of course, it is also possible that a meta-analysis will draw out small but potentially biologically important effects that went undetected by any individual study.

GMO Compass: What were the topics and outcomes of your meta-analyses?

Michelle Marvier: A meta-analysis of field studies found that Bt crops are generally more benign for non-target invertebrates than insecticides. A second meta-analysis of lab studies found no harmful effects of Cry proteins - the toxins produced by Bt crop - for honey bees.

GMO Compass: What is the reaction of the biosafety community to your work? Are meta-analyses the new "golden standard" for the safety assessment of transgenic organisms?

Michelle Marvier: Generally the reaction has been quite positive. Of course, the results of our meta-analyses suggest that Bt crops are relatively benign for non-target invertebrates. I'm not sure if the response would have been as warm if our analyses had revealed harmful effects. I would hope that the approach and the findings would be welcomed regardless of which way the findings turned out, but this is a contentious arena and people have some very strong opinions on the subject.

GMO Compass: You promote large open-access databases on risk assessment studies. What is the idea behind this?

Michelle Marvier: In order to facilitate the execution of meta-analyses the scientific community needs easy access to data from the numerous biosafety studies on GM crops. Yet there are so many scientists doing so many different experiments and risk assessments that the information all too easily overwhelms decision-makers or causes the debate to zig-zag around. Meta-analysis has the potential to move the debate about the safety of genetically modified crops beyond a situation in which competing sides argue that "study X shows this" only to be countered with "yes, but studies y and z show the opposite." Indeed, no single study should, by itself, be taken too seriously until other studies have confirmed the findings.

If meta-analyses and large databases of completed studies were to become a routine part of risk assessment, then there would not be the distraction of single experiments capturing media attention and inappropriately alarming or comforting the public and policy-makers. An investment in the creation and maintenance of risk assessment databases will have high payoff in terms of improved transparency, increased public confidence in the process, and more rapid advancement of scientific understanding.

GMO Compass: At the moment the database you developed together with your colleague Peter Kareiva only contains data on non-target effects of insecticide-resistant Bt crops. Do you intend to expand your database with data on other traits and GM crops?

Michelle Marvier: Actually, we set up the database as a model, in hopes that regulatory agencies - such as the US Environmental Protection Agency, which funded our research - would adopt a similar approach. It is a lot of work to extract data from published papers and unpublished reports and track down details by corresponding directly with authors. It would be way easier to keep a database up to date and complete if the researchers entered the data themselves, potentially as a condition for registration and deregulation.

GMO Compass: How do you imagine the set-up of a comprehensive database containing all relevant studies of GMO environmental risk assessment? Who will be responsible for that?

Michelle Marvier: I hope that that an internationally active organisation such as the International Society of Biosafety Research will be in charge of data collection, quality control and coordination. I would love to see a single publically-accessible database, detailing the methods and results for all of the completed studies. It would be an enormous service to science, to the public, and to industry.

GMO Compass: What is your personal view on the future challenges of biosafety research?

Michelle Marvier: Scientists and farmers were very familiar with Bt before Bt toxins were incorporated into transgenic crops. Other types of "plant-incorporated-protectants" that are currently under development and testing are less familiar, and we will likely find surprises along the way as these next generation transgenic insect-resistant plants are tested. I believe it was helpful that Bt was the first to go through this type of testing, so we could learn how to perform risk assessment and we can now apply what we have learned to crop traits that are less familiar.

GMO Compass: Thank you for talking to us.