How to Analyze Scientific Research
Science is at the heart of what TrainerRoad does, and that means evaluating the latest research. Whether it’s creating new training plans or preparing for deep dives on the Ask A Cycling Coach podcast, our goal is to empower you with the knowledge that will make you faster. With some practice, anyone can critically analyze scientific research, and Amber Pierce has some tips to help you get started.
Special thanks to TrainerRoad’s Amber Pierce for assistance in developing this blog post. For more on analyzing research, check out Ask A Cycling Coach Ep 299.
Science is for everyone! That’s why sharing the results from an experiment through published, peer-reviewed articles is an essential final step for scientists. It is a chance to share the newly gained information with the public and provides an opportunity for critical discussion and analysis. This article will cover how studies are formatted and some questions to ask as you read through a study.
Abstract
The Abstract is usually the most accessible part of a study, and you can quickly find it with a simple internet search. It serves as a very brief overview of the research to help you decide whether to read the study. Because they are usually less than five hundred words long, abstracts lack nuanced explanations about the methods, data, and conclusions.
Research Tip: Avoid drawing conclusions based on the Abstract. Instead, use the Abstract to determine whether the full article will contain the information you’re seeking, and wait to draw conclusions until after reading the entire study.
Adaptive Training
Get the right workout, every time with training that adapts to you.
Check Out TrainerRoadIntroduction
The Introduction serves as a place where the author can explain the goals and central question of the paper. Typically, the Introduction provides context by explaining the current state of knowledge on the topic, the need for the study, or gaps in current knowledge. In many studies, the Introduction includes key background information and terms that can give you a broad sense of the topic.
Research Tip: The Introduction is the best place to find the central question the authors aim to answer with their study. You can usually find the central question or aim stated in a single sentence at the very beginning or very end of this section, but the background information provided can be very valuable.
Methods
The Methods section explains how the researchers decided to answer the central question or address the central aim; in other words, this section explains what the author actually did in the experiment. It should include the steps and testing protocols, which provide a basis for interpreting the results later in the paper. In a well-designed study, researchers will design the methods to ensure valid, reliable, and objective results.
Typically, this section will include the characteristics of participants (for example, trained or untrained subjects), instrumentation, and specific testing protocols the researchers incorporated in their experiment. Additionally, this is where the authors will explain how their experiment tested factors relevant to their central question while controlling for factors that could confound their results.
In technical terms, these would be the treatments and controls for each group in the study. Honestly, the methods section is dry reading, but it provides you with vitally important information: how the researchers generated their results informs how they (and you) can reasonably interpret those results. For example, you would need to know if some of the equipment introduced error into measurements or if the authors simply could not control certain variables.
Research Tip: The Methods section is critical to read because this is where the researchers explain how they generated their results, which determines the best way to analyze and interpret the results.
Results
The Results section is straightforward and provides detailed documentation of what was measured when the authors executed the methods. If the authors applied any statistical analyses, this is where they will explain why they used certain statistical tests and what the results of those tests were. Often, statistical tests show whether or not the results were statistically significant.
Statistical Significance
Statistical significance does not mean significant in the sense of being relevant or important. Instead, statistical significance is often measured in terms of a p-value, which describes how likely an observed result or key difference is due simply to chance. The smaller the p-value, the less likely the observed effect or result is due to chance. The authors decide what p-value is small enough to determine a significant (not due to chance) result. In many cases, authors designate a p-value of less than or equal to 0.05 as indicating a significant result, meaning there is a less than five percent chance of seeing that result by pure randomness. There is nothing magical about 0.05. All you need to know is that the smaller the p-value, the less likely the result is due to chance.
Sample Size
During the podcast, you might hear the host talk about a large or small “N” when discussing studies on the podcast. N refers to the number of subjects tested in a study. In a study of sports performance, N probably refers to the number of people who participated as subjects in that study. The more subjects who participate, the more confidently the authors can rely on their statistical tests of significance.
Studies on human performance are tricky due to big individual differences (fitness, gender, physiology, training history, health, etc) and due to ethical constraints. A study with 100 subjects that shows a significant result can make more confident conclusions than a study with five subjects that shows a significant result, but it can be very difficult for researchers to find 100 willing participants who also qualify to take part in their study. Nevertheless, as a critical thinker, you’ll want to take into consideration the number of participants when interpreting the results of a study.
Effect Size
Effect size is a statistical technique for assessing the magnitude of difference between two groups and can help to overcome some of the limitations of a small N. When testing for statistical significance, researchers can calculate a p-value from a small number of subjects, but we know that individual differences might disproportionately affect the p-value in a study with N=5 subjects, in contrast to a study of N=100.
In a smaller study, calculating effect size can help to determine how strongly a particular intervention affected the outcome or result when measurements of statistical significance are limited by a small N. There are many ways to calculate effect size. One way is to divide the difference between the experimental and control groups, by the standard deviation of the control group.
Research Tip: Since this is where the authors report all of the results without interpretation, it’s a good place for you to start forming your own interpretations and then compare your thoughts to the researchers’ discussion in the next section. You don’t need to be an expert in statistics to see that an N=5 study might have different limitations than a study with N=100. Remember, researchers are experts in what they do, but they’re also human and make mistakes too! Mistakes or bad analyses should be caught in the peer-review process or covered in the discussion, but not always!
Discussion
The Discussion is all about interpreting the results within the context of current research on the topic and given the limitations and constraints of the study. Here, the authors will point out the limitations of methods and statistics and how those limitations affect the interpretation of the results.
Importantly this is the section where researchers offer probable explanations for the results of their study. This is also where authors compare their results to similar studies in the literature and discuss why their results might differ from other studies, or why their results confirm what other studies have observed. Most studies teach us something at the same time as raising new questions, so this is usually where the authors identify future research needs and outstanding questions.
Helpful Questions to Ask:
- Do the methods and results square with the author’s conclusions?
- Is there a way to explain/interpret the results that the authors have missed?
- Do the authors reasonably discuss the limitations of their methods and analyses?
Research Tip: A fun exercise is to write your own list of possible limitations and explanations of the study results, based on what you read in the Methods and Results sections, and to compare your list to points addressed in the Discussion. You’ll probably surprise yourself with how well you can do this, even as a non-expert! You may even detect mistakes or bias in the authors’ interpretations, or identify possible explanations the authors omitted. A well-written paper can give you all of the information and context you need to come up with your own, well-reasoned analysis. Researchers are very good at what they do, but even the best studies have limitations, especially when it comes to studies on human subjects. It’s important to apply your own logic, even when reading scientific studies.
Conclusions
This section is often combined with the Discussion. If separate, it should distill the most basic conclusions that one can fairly glean from the study in light of the limitations of methods, and analysis already covered in the Discussion. This is where the authors state what they see as the most important takeaways, without introducing new content.
Research Tip: Rarely, a study can definitively prove something, which is why the central question is usually incredibly specific. Most conclusions necessarily include at least some degree of uncertainty.
Figures and Tables
This is where you can find a visual summary of results or statistical analyses. Figures and tables are important, but not enough to analyze the study on its own. Typically, authors choose formats that make the data look more compelling, so it’s crucial always to view them critically, especially if they are presented without context.
Research Tip: Read the captions because looks can be deceiving. Ask yourself why the author chose to display the data in this particular format. Pay close attention to scale, units, and labels.
Other Considerations:
It’s always a good idea to check the citations or references given in a published study. This can be a time-consuming process, but it’s a best practice. Intentional misrepresentation is rare, but when authors borrow statements/citations from others without checking, it becomes a game of telephone. You may even find more studies that pique your interest!
Additionally, you can check to see if other articles have cited the study to see if the results have been confirmed or refuted. You can even find out if the author completed follow-up studies on the same topic. An easy way to do this is to search for the author’s name on Google Scholar. The author’s page will include a list of their publications as well as their h-index. The h-index is a quantitative metric based on the number of published peer-reviewed studies and citations an author has. For example, TrainerRoad’s own Amber Pierce has an h-index of 2.