Several systematic reviews have examined the incidence rates and risk factors of running related injuries [1-4]. Such injuries are common, following a systematic review of the literature, Tonoli et al [4] state that injury incidence varies between 0.1% and 2.6% (P<.05). The most common injuries sustained by long-distance runners were found to be Achilles tendinopathy, iIliotibial friction syndrome, and medial tibial stress syndrome [4]. The identification of risk factors for these injury types in runners remains controversial, with conflicting results published in the literature [1,2,5-7].

There are two types of injury that are commonly reported in association with running and exercise: bone stress injuries (including medial tibial stress syndrome and tibial stress fracture) and tendon injuries (including tendinopathy and tendon rupture) [1,6]. In the case of tendon injuries, genetic risk factors have been identified that correlate with increased risk of, or protection from, injury [8]. It has been acknowledged that further investigation is required to confirm and correctly interpret the association of identified polymorphisms with specific injuries [8].

Published genetic studies on exercise-induced injuries primarily focus on soft tissue injuries [9], and there is only one study demonstrating association between genetic polymorphisms and increased risk of bone stress injuries [10]. In an attempt to address this, we have designed a study to analyze Australian runners using genome-wide association techniques to identify gene variants that contribute to increased risk of, or protection from, Achilles tendinopathy and/or bone stress injuries. To achieve a large sample size, the study must provide a reliable and convenient method of data collection from prospective participants, allowing them to participate in the research from home and with their own device. We therefore developed an Internet activity survey to prospectively recruit participants relevant to our test groups. Retrospective surveys of sporting injuries have been shown to have some limitations for injury epidemiology research because of issues with injury recall [11,12]. In order to counteract these issues, participants were only asked to recall injuries that had occurred in the previous 2 years.

Surveys should be evaluated for reliability, a critical measurement property for health-related and physical activity variables. Reliability refers to the consistency of answers obtained by the same participant when a measurement is repeated on different occasions [13]. Test-retest reliability is measured by having the same participants complete a survey at two different points in time to see how stable the responses are. Consistency is then quantified with a kappa coefficient statistic for categorical data [14].

Internet survey instruments are increasingly being used as a preferred method of conveniently collecting data from a wide range of participants from various locations. The reliability of survey instruments delivered on the Internet has been previously demonstrated, showing that in the instance of retrospective tobacco exposure and risk, this method of data collection is reliable [15]. Furness et al [16] also demonstrated success using an Internet survey instrument for the study of injuries collected in a retrospective manner in surfing athletes, whereby they recruited 1348 surfers. This high level of response enabled the researchers to produce a comprehensive dataset that is the largest study of surfing injuries in Australia [16].

The purpose of the study reported here is to test the test-retest reliability of an Internet survey platform from a convenience population of first responders, and pilot test participants who accepted an invitation to describe their recreational running epidemiology. This will contribute to the broad aims of the wider study, which are to recruit thousands or even tens of thousands of individuals to investigate injury epidemiology across a wide group of recreational runners in order to gain insight into how a variety of demographic factors may influence injury epidemiology; and to perform genetic assessments for the injury risk factors in a small subsample of the overall injury epidemiology study.

The questionnaire administered was designed to enable recruitment as part of the wider study entitled, “The Genetics of Exercise-Induced Injuries in Tendon and Bone.” This study has very broad aims in the range of wanting to recruit very large numbers of participants to provide injury epidemiology among recreational runners, as well as performing genetic assessments for the injury risk factors in a smaller subsample of the overall injury epidemiology study.

The platform used to deliver the survey was the Internet and commercial survey software (SurveyGizmo). The Internet survey was advertised through appropriate forums relevant to the intended population to analyze sporting injuries and training practices. Participants provided us with data on their demographics and running injuries, but the survey provides a platform for determining whether a participant met the wider genetics study’s inclusion and exclusion criteria. Participation was entirely voluntary and required consent to participate in the research processes described by checking a required “I ACCEPT” box necessary to access the questionnaire. Further information about the project was made available via a check box diverting the participant to another screen with all relevant information and contact details for further clarification if required.

On accepting conditions of participation in this project, participants were taken to the questionnaire and instructed that it would take no more than 30 minutes to complete. The software used allowed participants to exit the survey at any time and complete at a later date, allowing participants to provide their data at the time most suitable to them. A questionnaire was deemed complete when the participant had answered all required questions (customized within the software) and participants had submitted their questionnaire by checking the “SUBMIT” tab inserted at the end of the questionnaire.

The questionnaire developed here combined new items of assessment with adapted versions of the SCOFF questionnaire for examining eating disorders [17]. The full survey is provided in Multimedia Appendix 1. Regular running habits were assessed across several categories including how many years participants had been running for, how many kilometers (km) per week they ran, what terrain running was performed on, and whether orthotics were worn during running. In addition, participants reported any other sports or intentional exercise they participated in regularly over the last 2 years and were specifically asked to report any injuries of the lower limbs within the last 2 years that resulted in discontinuation of running for a consecutive period of 2 weeks or more. This was further examined by asking the participants to report on the signs and symptoms of the injury, how the injury was diagnosed, and whether it was an Achilles tendon or bone stress injury. They were also asked to stipulate any previous hip, knee, or ankle surgery over their lifetime.

In addition, participants were asked to report on their general health, provide any knowledge of existing conditions or diseases, and any known antibiotic or prescribed drug administration. They were asked to describe their regular dietary practices including sport supplement use, as well as any body weight fluctuations. Participants were also asked to report their age, sex, height, weight, and ethnic background to collect demographic data for further epidemiology analysis.

Some of the key variables, important to clarify inclusion or exclusion from the wider genetics study previously mentioned, were then selected for test-retest reliability here (Table 1). Questions of interest included those that addressed diagnosis of the injuries, as these variables may result in less than optimal reliability due to the retrospective design of the study [11]. Several questions from the tail end of the survey were also selected to assess if respondent fatigue had an effect on reliability. To approach this experimentally, we used a test-retest reliability design. This study was approved by the Human Research Ethics Committee of Bond University and the Australian Institute of Sport (RO1688B).

Descriptive statistics were used to describe the cohort characteristics of participants for each group, and reported as a percentage (%) of total participant number within the group illustrated. Some of the key variables important to clarify inclusion or exclusion from the wider genetics study previously mentioned were then selected for test-retest reliability and scored using the kappa statistic, with asymptotic standard error [14]. Reliability was then rated using the scale by Landis and Koch for the purposes of comparing the reliability of key questions [14]. Reliability was rated as poor (below .00), slight (.00-.20), fair (.21-.40), moderate (.41-.60), substantial (.61-.80), or almost perfect (.81-1.00). Data were collected using SurveyGizmo and analyses were conducted using SPSS Statistics version 22.0.

This self-report questionnaire worked toward the development of a standardized instrument via which participants not only provide us with data on their demographics and running injuries, but also provide us with a platform for determining whether interested participants meet the wider genetics study’s inclusion criteria. This study analyzed the test-retest reliability of key inclusion/exclusion criteria for the wider genetics study, collected with the developed survey involving the 2 groups. The choice of different methods for test-retest analysis of each group is discussed in the “Methods” section. The results of both approaches scored using the kappa statistic, with asymptotic standard error, the valid number of responses, and number of response options, are highlighted here.

There were 20 participants that fulfilled group 1 criteria by completing the survey on the Internet then again on the Internet within 21 days after the first completion. Participants were between 18-68 years of age, with an equal distribution of males (50%, n=10) and females (50%, n=10). As much as 45% (9/20) of participants recorded that they had been running for 6 years or less and the remaining 55% (11/20), 8 years or more. Their running habits were variable, with 50% (10/20) predominantly running 15-30 km/week, 25% (5/20) running 30-40 km/week, and 25% (5/20) running 40 km or more per week (Table 2).

There were 30 participants that fulfilled group 2 criteria by completing the survey on the Internet then again by phone interview within 14 days after Internet completion. Participants were between 18 and 67 years of age, with an almost equal distribution of males (53%, n=16) and females (47%, n=14). There were 40% (12/30) of participants that had been running for 6 years or less and the remaining 60% (18/30) 10 years or more. Their running habits were similar to test group 1 with 43% (13/30) predominantly running 15-30 km/week, 30% (9/30) 30-40 km/week, and 27% (8/30) 40 km or more per week (Table 2).

Reliability was almost perfect for the type of terrain run on (kappa=.863), injuries of the lower limbs reported in the last 2 years (kappa>.99), and the number of these lower limb injuries diagnosed in the same time frame (kappa>.99). Reliability was also almost perfect for the injury being diagnosed by a doctor (kappa=.820), the injury being a bone stress injury (kappa>.99), reporting any hip/knee/ankle surgery (kappa=.857), and the use of orthotics while running (kappa>.99). Reliability was substantial for the number of years running on a regular basis (kappa=.791), average kilometer run per week (kappa=.684), other sports or intentional exercise participated in regularly in the last 2 years (kappa=.773), and reporting the injury as an Achilles tendon injury (kappa=.625). Reliability was moderate for being diagnosed by a physiotherapist (kappa=.467; Table 3).

Reliability was almost perfect for the majority of all general health questions reported including having ever smoked cigarettes (kappa>.99), any knowledge of being treated with calcium tablets (kappa=.886), any knowledge of being treated with vitamin D supplementation (kappa>.99), reporting on whether participants followed a gluten-free diet (kappa>.99), or stating any food allergies or avoidances (kappa>.99). Reliability for any knowledge of being treated with corticosteroid medication (kappa=.492) or quinolone (kappa=.468) was moderate (Table 3).

The reliability of running habit and running injury key variables in test group 1 was more variable than that of test group 1 (Table 4). Reliability was almost perfect for injuries of the lower limbs in the last 2 years, which forced discontinuation of running for a period of 2 weeks or more (kappa=.930), reporting the injury as an Achilles tendon injury (kappa=.814), reporting any hip/knee/ankle surgery (kappa>.99), and the use of orthotics while running (kappa>.99). Reliability was substantial for the number of years running on a regular basis (kappa =.639), the number of lower limb injuries diagnosed within the last 2 years (kappa=.697), injury diagnosed by a doctor (kappa=.727), and injury diagnosed by a physiotherapist (kappa=.615). Reliability was moderate for kilometer run per week (kappa=.540), the type of terrain run on (kappa=.469), other sports or intentional exercise participated in regularly in the last 2 years (kappa=.473), and reporting the injury as bone stress injury (kappa=.571; Table 4).

Overall, reliability was lower for general health questions reported in test group 2 in comparison to test group 1 (Table 4). Reliability was almost perfect for having ever smoked cigarettes (kappa=.839), and stating any food allergies or avoidances (kappa=.922). Reliability for any knowledge of being treated with calcium tablets (kappa=.720), any knowledge of being treated with vitamin D supplementation (kappa=.672), and reporting on whether participants followed a gluten-free diet (kappa=.783) was substantial. Reliability was moderate for any knowledge of being treated with corticosteroid medication, however, any knowledge of being treated with quinolone had slight reliability (kappa=.150; Table 4).

The wider study, for which this survey will provide epidemiology information and a participant cohort to recruit from, will be the first, to our knowledge, to examine the genetic predisposition of tendon and bone injury in adult recreational runners. Recruitment for the larger study will involve the use of this survey, tested here initially for test-retest reliability. The results indicate that the self-reported questionnaire, delivered using the Internet commercial software, provided stable and reliable answers for many of the most important measures required for recruiting to the larger study on tendon and bone injury.

The Internet delivery of a retrospective injury study is a convenient method for remote collection of data from participants and has been used successfully in the analysis of sports-induced injuries in surfing [16] and strongman athletes [18]. As mentioned, the survey developed here combined new items of assessment with adapted versions of the SCOFF questionnaire for examining eating disorders [17]. In most aspects, its reliability was comparable to that of other similar survey delivery methods used for examining health and injury-related factors, including the Military Pre-training Questionnaire, which assesses risk factors for injury among military trainees across 5 domains (physical activity, injury history, diet, alcohol, and smoking) [19]. Test-retest reliability, for the study reported here, was shown to be acceptable for all variables for all participants, supporting the stability of the questionnaire. The only variable questionable for reliability was “treated using quinolone” (kappa=.150) for test group 2. On first completion of the Internet survey, there were 10 participants (n=30; 27 valid; 3 excluded) that were “unsure” if they had been treated in the past with quinolone. In the phone interview, these same 10 participants reported that they had not been treated previously with quinolone, and therefore this shift in reporting is responsible for the low reliability.

Poor reliability of the question addressing treatment with quinolone may suggest poor understanding of or unfamiliarity with “quinolone” upon first exposure to the question. It is known that test-retest reliability can be influenced by many factors. Interpretation of a question, such as familiarity of content or ambiguity, as well as memory can cause random answers [20]. Questions which involve unfamiliar knowledge or are ambiguous, such as asking about treatment with various prescribed medications in this survey, may lead to considerable variability in test-retest responses [21]. The nature of the question can also shift test-retest reliability scores. As we report here, it is not unusual that constant behaviors, such as smoking, result in higher reliability than more variable behavior such as obtaining an injury [11,12], or being diagnosed or treated with different medications over time [22]. Responses to questions inquiring about variable behaviors, including those addressing all aspects of running habits, would probably be more reliable if a training diary was referred to when obtaining data rather than relying on retrospective recall. However, the data presented in our study indicate moderate to almost perfect reliability for questions asking about variable behaviors without requiring participants to refer to such a tool.

Respondent fatigue is a well-known phenomenon affecting survey participants [23]. We developed this survey to minimize such fatigue by making the questions as easy to understand and answer as possible, in addition to being mindful of the time required to complete the survey. Questions regarding a gluten-free diet and known food allergies were among some of the last questions addressed in survey. We show high reliability for these variables, which supports our belief of minimal user fatigue across the questionnaire in both test groups (Tables 3 and 4). Increasing the number of possible responses did not markedly affect the reliability of the questions. Responses for questions containing only 2 variables (yes/no answers) had perfect or high reliability, while questions with 7 or 11 possible responses had moderate to high variability. This is consistent with previous research demonstrating that increasing the number of response categories only negligibly affects reliability [24,25].

Results when addressing the 2 different test groups should be considered in the context of several limitations. Our study contained a different number of participants in each group with a variable distribution of age. This limitation resulted from the sequential manner of recruitment of the two groups, inviting participation from within our networks for test group 1, then soft launching to recruit a larger group from the running club for test group 2.

There was a slightly higher test-retest reliability for test group 1, in which the survey was repeated using the same Internet platform. It is possible that this was due to learning effects, or that the retest was administered in exactly the same format.

The slightly lower test-retest reliability for group 2, in which the survey was administered by the Internet platform and then followed up by a telephone call, may be related to the wording of the question over the telephone by the interviewer compared to the participants reading the computer screen. During the telephone assessment, it is also possible that participants felt more uncomfortable answering sensitive questions or got confused in regards to what constitutes a response for each question asked. Our findings indicate that using 2 test-retest protocols resulted in participants providing the same information with high reliability for the survey administered in this study. Maintaining consistency of the protocol, however, can improve test-retest reliability.

We successfully developed an Internet survey to meet the recruitment needs of the larger genetic injury study. We tested its reliability using 2 different test-retest methods and used a participant sample targeted to our needs (adult recreational runners) in aid of focusing our findings to support our larger study. We had equal gender participation and a broad range of running habits and participants with reportable Achilles and bone stress injuries. Despite the aforementioned limitations, the results of this reliability study demonstrate that the Internet survey developed does provide a relatively quick, easy to complete, and cost-effective method to collect epidemiological data from recreational runners and evaluate these participants for inclusion into a genetic study.