Preprint
Article
Change over Five Years in Important Measures of Methodological Quality and Reporting in Cardiovascular Clinical Research Trials
This version is not peer-reviewed.
Submitted:
30 October 2023
Posted:
01 November 2023
You are already at the latest version
A peer-reviewed article of this preprint also exists.
Abstract
Objectives: The aim of our current study was to analyze whether the use of important measures of methodological quality and reporting of randomized clinical trials published in the field of cardiovascular disease research have changed over time. Further aim was to investigate whether there was an improvement over time in the ability of these trials to provide a good estimate of the true intervention effect. Methods We conducted two searches in the Cochrane Central Register of Controlled Trials (CENTAL) database to identify cardiovascular clinical research trials published in either 2012 or 2017. Randomized clinical trials (RCTs) trials in cardiovascular disease research with adult participants were eligible to be included. We randomly selected 250 RCTs for both publication year 2012 and 2017. Trial characteristics, data on measures of methodological quality and reporting were extracted and risk of bias for each trial was assessed. Results As compared to 2012 in 2017 there were significant improvements in the reporting of the presence of a data monitoring committee (42.0% in 2017 compared to 34.4% in 2012), and a positive tendency of registering cardiovascular disease research RCTs in clinical trial registries (83.6% in 2017 compared to 72.0% in 2012). On the other hand, we also observed that significantly fewer RCTs reported sample size calculation (60.4% in 2017 compared to 98.4% in 2012) in 2017 as compared to 2012. RCTs in 2017 were more likely to have low overall risk of bias ( RoB ) than in 2012 (29.2% in 2017compared to 21.2% in 2012). Conclusion: As compared to 2012 in 2017 there were significant improvement in some, but not all the important measures of methodological quality. Although more trials in the field of cardiovascular disease research had a lower overall RoB in 2017, the improvement over time was not consistently perceived in all RoB domains.
Keywords:
Subject:
Public Health and Healthcare - OtherIntroduction
Randomized clinical trials (RCTs) constitute the foundational background of modern medical practice (1). In the last three decades the cardiovascular randomized clinical trial has emerged as the principal method by which new therapies are evaluated (2). Moreover, evidence generated from randomized clinical trials has greatly influenced the diagnosis and treatment of many heart diseases including arterial hypertension, arrhythmias, acute myocardial infarction, heart failure and coronary revascularization (3-5).
The increasing prevalence of cardiovascular disease around the world requires high quality of clinical research and translation of its findings into new therapeutic and diagnostic strategies (6).
Unfortunately, although there was a significant increase in the quantity of scientific literature concerning cardiovascular disease published in recent years, it was indicated that this has not resulted in guideline recommendations with more certainty and supporting evidence. The American College of Cardiology and the American Heart Association (ACC/AHA) clinical practice guidelines are still based on lower quality of evidence and expert opinions, indicating the lack of high-quality studies with relevant data (7).
Several tools exist, which support researchers to plan and conduct high-quality research and make trial results completely and transparently available. Guidelines for clinical trial protocols (e.g. SPIRIT) facilitate trial planning in all important details. Reporting guidelines (e.g. CONSORT for RCTs) have the aim to decrease the risk of non-reporting bias, i.e. facilitating that clinical trial methods are described as they were conducted and trial results are fully published (8). The requirement of clinical trial registration supports transparency in research. Although in the USA it is a requirement from the Food and Drug Administration (FDA) that all clinical trials are registered before the first patient is enrolled (11), and European Medicines Agency and WHO also support clinical trial registration (12,13), in the field of cardiology insufficient registration tendencies were reported (14). Cardiac and cardiovascular system journals infrequently require, recommend and enforce use of obligatory clinical trial registration (15).
Methodological flaws in the design, conduct, analysis and reporting of randomized clinical trials can cause the true intervention effect to be under- or overestimated. This is why these systemic errors (defined as risk of bias) are assessed when systematic reviews are conducted or evidence-based guidelines are developed (16). Concerns arising due to high risk of bias in trials included in evidence syntheses lead to the downgrading of evidence level and consequently will decrease our certainty in the pooled results.
Our previous study compared risk of bias in industry-funded and non-industry funded cardiovascular disease research trials published in 2017 (17). In the present study we would like to investigate tendencies over time and answer whether there was an improvement in measures of methodological quality and reporting in cardiovascular randomized clinical trials between 2012 and 2017. Further, we would like to assess how well these trials were able to estimate the true intervention effect in 2017 as compared to 2012.
Methods
We conducted two searches to identify cardiovascular clinical research trials published in either 2012 or 2017. We searched the Cochrane Central Register of Controlled Trials (CENTAL) database, as this is the most comprehensive resource available containing randomized clinical trials.
We used the same search strategy for both years, containing subject headings and key words related to adults (aged >18 years) and cardiovascular diseases, restricted to the years 2012 or 2017. The first author (OB) searched CENTRAL and screening trials for eligibility. We included studies which were published in 2012 in English language journals, where the investigated intervention was related to cardiovascular practice and where participants 18 years or older were included. Our search resulted in 2566 trials. All identified records were exported to Excel, where they were randomly ordered with the following method: we used the RAND function to assign a number between 0 and 1 to each record. In a subsequent step we were reordering trials from the smallest to the highest number.
We included the first 250 (about 10%) eligible randomized clinical trials for both year 2012 and 2017.
We used a data extraction tool which was developed for assessing methodological quality of RCTs in child health research (17). Two authors (OB, OF) independently extracted data for each study included. We discussed all unclear decisions until consensus was established. For each cardiovascular disease study, we identified information about journal type (e.g., general or specialty medical journal, general or specialty cardiovascular journal), corresponding author’s country , study type , study design, intervention type, type of control, number of study centers, study sample, primary diagnostic category in the study using the ICD-10 classification system, presence of data monitoring committee, type of primary outcome, outcome results, and trial registration. Trial registration number, author names and keywords related to the specific cardiovascular intervention were used to find the published protocol via Google and Google Scholar. We completed data extraction after precise analysis of full text article, trial registration and published protocols. To retrieve information on primary outcome of trial, we defined primary outcome as 1) the outcome defined under objective of the study; 2) the outcome used to calculate sample size; or 3) the first outcome reported in the randomized clinical trial.
We used Cochrane RoB tool (18) to assess methodological quality of randomized clinical trials. This tool is evaluating 7 domains of bias and thereby determining the extent to which the RCT’s design, conduct, analysis, and presentation was appropriate to answer the trials research question. These 7 domains are: 1) Sequence generation (whether the allocation sequence was adequately generated); 2) Allocation concealment (whether the allocation of group assignment could not been foreseen prior to randomization); 3) Blinding of participants ad personnel (Whether the knowledge of the allocated intervention was adequately prevented during study); 4) Blinding of outcome assessors; 5) Incomplete outcome data (whether the incomplete outcome data were adequately addressed); 6) Selective outcome reporting (whether the study was free of apparent selective outcome reporting); 7) Other sources of bias (whether the study was free of other problems that could introduce bias).
Statistical analysis
Statistical analyses were conducted by the statistical software R version 4.1.2 (R Development Core Team 2021) (19). To analyze for 5-year changes in main study characteristics, we compared the 2017 sample with 250 RCTs published in 2012 (17). All collected binomial variables were analyzed using logistic regression analyzes with generalized linear models. All collected categorical variables with more than two categories were analyzed with multinomial regression models. Variables of methodological quality and report were analyzed with separate univariable logistic regressions.
A multivariable logistic regression analysis was conducted to investigate the association between pre-specified study characteristics and the odds of high/unclear against low RoB. The independent variable in the multivariable logistic regression analyzes were the study centers (single or multicenter); trial registration; type of the intervention (drug vs non-drug); sample size; availability of Data Monitoring Committee (DMC) and statistical significance of the primary outcome on each RoB domains. Explanatory variables for statistical analyzes was the Overall Risk of Bias assessment results (low vs. high/unclear). Model assumptions on residuals were checked using “model-checking plots”. Statistical significance tests in the models were carried out with Chi-square tests. The value of p < 0.05 was considered as a significant result
Results
Descriptive analysis
The main characteristics of included cardiovascular RCTs are shown in Table 1. Data from 2017 have been previously partly reported (17). Values from 2012, some data on additional measures of methodological quality and reporting for both years and statistical comparisons are novel.
We saw significant differences in the country of origin defined based on the first author’s affiliation between 2012 and 2017. In our 2017 sample more publications were published in specialty medical journals (19.6% compared to 10.4%; p<.001). In 2017 we included more RCTs with parallel design (92.4% compared to 80.4%; p<0.01); and among the interventions there were more drug trials (55.6% compared to 46.8%) and surgical interventions (1.2% compared to 0.4%), (p<0.001). In the 2017 sample we had a larger number of multinational trials (27.6% compared to 18%), (p<0.05), and developing (8.4% compared to 1.2%) and transitional economy countries (5.2% compared to 3.2%) were more often concerned (p<0.001). In 2017 included trials were more often funded by pharmaceutical company or industry (p<0.001).
Table 2 shows changes in important measures of methodological quality and reporting.
As compared to 2012 we saw an improvement in 2017 in the reporting of the presence of a data monitoring committee (42.0% compared to 34.4%; p<0.001). As compared to 2012 there was a positive tendency of registering trials in trial registries in 2017; and among clinical trial registries the clinicaltrials.gov database had increased popularity (registration rate in clinicaltrials.gov was: 78.4% compared to 68.9%; p=0.03). On the other hand, significantly fewer RCTs reported sample size calculation (60.4% compared to 90.4%; p<0.001) in 2017 as compared to 2012. Although, fewer RCTs specified plan to collect adverse effects (48.4% compared to 74%) (p<0.001) in 2017, they reported harms more often (68% compared to. 52%; p<0.001) in 2017. When we investigated reporting of results, we observed that the number of RCTs with statistically significant results of the primary outcome was lower in the 2017 sample (69.2% compared to 78.8%; p<0.01). Further, there were more publications with neutral conclusions in 2017 (18.4% compared to 7.2%; p<0.01). There were no statistically significant differences between 2012 and 2017 in the number of intentions to treat analyses, in the type of outcomes (as most outcomes were objective), or specific types of primary outcomes.
Rısk of bias assessment.
We provided risk of bias assessment by each domain for trials published in 2012 and 2017 year (Table 3). Compared with 2012, more 2017 RCTs were rated low (70.4% compared to 38.8%) and less were rated unclear (20.4% compared to 50%; p<0.001) risk for allocation concealment. Fewer 2017 RCTs were rated low (50.8% compared to 65.6%; p<0.001) risk for blinding of participants and personnel and for blinding of outcome assessors (82.4% compared to 90.8%; p<0.001). A similar proportion of 2017 RCTs were rated low risk for random sequence generation (59.6% compared to 56.0%), for incomplete outcome data (74% compared to 3.6 %;) and selective outcome reporting (62.8% compared to 80.0%) compared to 2012. In 2017 more RCTs were rated low (42.8% compared to 33.6%) risk for other risk of bias (p<0.01). More trials were rated low (29.2% compared to 21.2%) for overall risk of bias in 2017 compared to 2012 (p<0.01).
In 2017 multicenter trials (OR 0.39, 95% CI 0.18 to 0.80), drug trials (OR 0.53, 95% CI 0.29 to 0.97) and registered trials (OR 0.06, 95% CI 0.003 to 0.31) were also more likely to have a low overall RoB. In 2012 there was not yet a significance difference between multicenter or single center trials (OR 0.52, 95% CI 0.24 to 1.22), drug trials and non-drug trials (OR 0.82, 95% CI 0.44 to 1.56). Trial registration was not yet shown to have positive effects on RoB in 2012 either (OR 0.85, 95% CI 0.38 to 1.84).
Discussion
Summary of main findings
We found that cardiovascular disease clinical trials changed significantly from 2012 to 2017 years in several characteristics related to their study design and reporting. Respectively RCTs in the 2017 sample were published mostly in specialty cardiovascular journal with higher number of authors from Asia and Europe compared to trials published in 2012. The 2017 sample icluded more parallel trials, evaluating drug interventions. The number of industry-funded clinical trials was increased from 2012 to 2017, while the number of trials funded by the academy decreased. In the 2017 sample more trials were from developing and transitional economy coutries. Multinational trials had significantly higher proportion in the 2017 sample compared with 2012.
As compared to 2012 in 2017 there were significant changes in important measures of methodological quality and reporting, including an improvement in the reporting of the presence of a data monitoring committee, and a positive tendency of registering trials in trial registries. On the other hand, we also observed that significantly fewer RCTs reported sample size calculation in 2017 as compared to 2012.
We also observed notable changes over five years in the ability of cardiovascular disease research trials to properly estimate the true intervention effect. 2017 trials were more likely to have low RoB than 2012 for overall RoB. However, the 5-year-change was not clearly in the direction of improvement, as we observed lower number of RCTs with low RoB for blinding of participants and personnel and blinding of outcome assessors in 2017 as compared to 2012. In 2017 multicenter trials, drug trials and registered trials were also more likely to have a low overall RoB, than single center, non-drug on non-registered trials. In 2012 these RoB differences were not yet present between RCTs with specific characteristics.
Strengths and Weaknesses of the Study
For both investigated publication years we selected our sample randomly from Cochrane CENTRAL as the most comprehensive resource of RCTs. The samples covered areas of the prevention, diagnosis and treatment of cardiovascular diseases, including acute myocardial infarction, heart failure, arrhythmia, coronary revascularization, and chronic coronary artery disease. Most of our trials were registered in clinical trial registries, so essential trial details were double checked in both the full text article and the registry. We used the most accurate tool for risk of bias (methodological quality) assessment of includes RCTs. Two independent reviewers performed data extraction and RoB assessment, discrepancies always resolved by discussion.
This study has also some limitations. Our sample included about 10% from all eligible cardiovascular disease trials published in the years 2012 and 2017 only in English language. This study was not pre-registered with detailed statystical analysis plan. We have chosen cardiovascular trials with participants aged 18 years or older, therefore our results are not applicable to pediatric trials in cardiovascular medicine.
It also has to be emphasized, that the 2017 sample differed from the 2012 sample in many study design and reporting features, which may have impacted our RoB results.
Discussion of Findings Considering Other Studies
The risk of bias in CVD RCTs has generally decreased over the 5 years. This is consistent with the conclusions of Vinkers et al (22), who reported significant improvement in the level of risk of bias of RCTs over the past years in connection with increased knowledge about mandatory trial registration and journal requirements.
Our study revealed that trial registration influenced positively RoB.This finding is in line with prior researches investigating clinical trial registration and risk of bias. A study among clinical trials included in Cochrane systematic reviews of interventions published between 2014 and 2019 found that clinical trial registration was associated with low risk of bias for all bias domains examined except for attrition bias, and for overall risk of bias (23). Registered trials were at lower risk of overall bias than non-registered trials in Latin America and the Caribbean’s (24). Prospectively registered trials had a significantly lower risk of bias compared to unregistered trials across all domains in health research (25). We found that multi-center trials were more likely at low risk of bias than single center trials. This is could be associated with that multi-center studies allow for better control of study quality than single-center studies (26). Our findings is consistent with Tamborska et al., who found that RCTs at lower risk of bias were more likely to use multicenter recruitment in neurolgy trials (27).
Our investigation has shown that drug trials had favourable impact on RoB than non drug trials, which might be related with the strict regulations these pharmaceuticals trials must follow. Similarly, Cho Y et al., found that most of drug trials were at low risk of bias for blinding of participants and personnel, while almost two third of non drug trials were at high risk of bias for blinding of participants and personnel in cardiopulmonary resuscitation and emergency cardiovascular care (28).
Existing differences and the positive beneficial impact of regulations can be observed already in the planning phase of trials, when regulated clinical trials protocols were described to follow reporting guidelines to a greater extent than non-regulated trials (29).
Implication for practice and future research
We observed some improvements with respect to some important study design features and some specific sRoB domains over 5 years. However, there were also some methodological features and RoB domains which changed in an unfavourable direction or remained unchanged. This points to the need to continue to pay close attention to the planning and conduct of RCTs in the field of cardiovascular clinical research.
This study has identified several features of clinical trial planning and conducting that need further improvement in the field of cardiovascular research. Improvments in study design, conduct and reporting will decrease research waste and support the realization of evidence-based decisions in the field of cardiology. Journals adoption of existing reporting guidelines may lead to potential mechanisms to ensure improvements over all clinical trials quality. We would emphasize that a paper that adheres to reporting guidelines better places a clinical decision maker to assess the quality of the trial design and conduct and to interpret its findings accurately, improving the potential of the research to be impactful and meaningful to patients and clinical practice.
Conclusion
We call cardiovascular disease researchers to evaluate possible risks of bias for each cardiovascular RCT to ensure validity of trial results and their effective translation to evidence-based cardiovascular patient care.
Author Contributions
SL conceptualized the research, supervised the research, and reviewed and edited the manuscript. OB and OF performed the research. OB led data extraction and assessment, responsible for data curation, and wrote the first draft of the manuscript. DN conducted statistical analyses. All authors critically revised and approved the final version of the manuscript.
Funding
SL was supported by a research grant from the National Research, Development and Innovation Office of Hungary (NKFIH_FK_143633).
Ethics Approval and Consent to Participate
Not applicable.
Consent for Publication
Not applicable.
Availability of Data and Materials
The raw data supporting the conclusions of this article will be made available from corresponding author (Szimonetta Lohner MD, PhD ; email:lohner.szimonetta@pte.hu) without undue reservation.
Acknowledgments
Not applicable.
Competing of Interests
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
- Chow JT, Lam K, Naeem A, Akanda ZZ, Si FF, Hodge W. The pathway to RCTs: How many roads are there? Examining the homogeneity of RCT justification. Trials. 2017 Feb 2;18(1):51. [CrossRef]
- Scott, D. Solomon, Marc A. Pfeffer. The Future of Clinical Trials in Cardiovascular Medicine. Circulation, 2016; 2662-2670;133;25. [CrossRef]
- Jones WS, Roe MT, Antman EM, Pletcher MJ, Harrington RA, Rothman RL, Oetgen WJ, Rao SV, Krucoff MW, Curtis LH, Hernandez AF, Masoudi FA. The Changing Landscape of Randomized Clinical Trials in Cardiovascular Disease. J Am Coll Cardiol. 2016 Oct 25;68(17):1898-1907. [CrossRef]
- Heart Protection Study Collaborative Group et al. MRC/BHF Heart Protection Study of cholesterol lowering with simvastatin in 20 536 high-risk individuals: A randomised placebocontrolled trial //The Lancet. – 2002. – Т. 360. – №. 9326. – С. 7-22. [CrossRef]
- SOLVD Investigators. Effect of enalapril on mortality and the development of heart failure in asymptomatic patients with reduced left ventricular ejection fractions //New England Journal of Medicine. – 1992. – Т. 327. – №. 10. – С. 685-691. [CrossRef]
- Jackson N, Atar D, Borentain M, Breithardt G, van Eickels M, Endres M, Fraass U, Friede T, Hannachi H, Janmohamed S, Kreuzer J, Landray M, Lautsch D, Le Floch C, Mol P, Naci H, Samani NJ, Svensson A, Thorstensen C, Tijssen J, Vandzhura V, Zalewski A, Kirchhof P. Improving clinical trials for cardiovascular diseases: A position paper from the Cardiovascular Round Table of the European Society of Cardiology. Eur Heart J. 2016 Mar 1;37(9):747-54. [CrossRef]
- Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC. Scientific Evidence Underlying the ACC/AHA Clinical Practice Guidelines. JAMA. 2009;301(8):831–841. [CrossRef]
- Carpenter CR, Meisel ZF. Overcoming the Tower of Babel in Medical Science by Finding the "EQUATOR": Research Reporting Guidelines. Acad Emerg Med. 2017 Aug;24(8):1030-1033. [CrossRef]
- Bai Y, Gao J, Zou DW, Li ZS. Methodological reporting of randomized clinical trials in major gastroenterology and hepatology journals in 2006. Hepatology. 2009 Jun;49(6):2108-12. [CrossRef]
- Lai TY, Wong VW, Lam RF, Cheng AC, Lam DS, Leung GM. Quality of reporting of key methodological items of randomized controlled trials in clinical ophthalmic journals. Ophthalmic Epidemiol 2007;14(6):390-398. [CrossRef]
- Wood SF, Perosino KL. Increasing transparency at the FDA: The impact of the FDA Amendments Act of 2007. Public Health Rep. 2008 Jul-Aug;123(4):527-30. [CrossRef]
- European Medicines Agency.Clinical trials.http://euclinicaltrials.eu/.
- WHO. Trial Registration. 2016 http://www.who.int/ictrp/trial_reg/en/.
- Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009 Sep 2;302(9):977-84. [CrossRef]
- Sims MT, Bowers AM, Fernan JM; et al.Trial registration and adherence to reporting guidelines in cardiovascular journals.Heart 2018;104:753-759. [CrossRef]
- Higgins J P T, Altman D G, Gøtzsche P C, Jüni P, Moher D, Oxman A D et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials BMJ 2011; 343 :d592. [CrossRef]
- Baasan O, Freihat O, Nagy DU, Lohner S. Methodological Quality and Risk of Bias Assessment of Cardiovascular Disease Research: Analysis of Randomized Controlled Trials Published in 2017. Front Cardiovasc Med. 2022 Mar 17;9:830070. [CrossRef]
- Higgins JPT, Savoviæ J, Page MJ, Elbers RG, Sterne JAC. Assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ. editors. Cochrane Handbook for Systematic Reviews of Interventions. Hoboken, NJ: Wiley-Blackwell; (2019). p. 205–28.
- R Core Team. R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing; (2021).
- WooL, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: Meta-epidemiological study. BMJ. 2008 Mar 15;336(7644):601-5. [CrossRef]
- Panagiotou OA, Contopoulos-Ioannidis DG, Ioannidis JP. Comparative effect sizes in randomised trials from less developed and more developed countries: Meta-epidemiological assessment. BMJ. 2013 Feb 12;346:f707. [CrossRef]
- Vinkers CH, Lamberink HJ, Tijdink JK, Heus P, Bouter L, Glasziou P, Moher D, Damen JA, Hooft L, Otte WM. The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. PLoS Biol. 2021 Apr 19;19(4):e3001162. [CrossRef]
- Lindsley K, Fusco N, Li T, Scholten R, Hooft L. Clinical trial registration was associated with lower risk of bias compared with non-registered trials among trials included in systematic reviews. J Clin Epidemiol. 2022 May;145:164-173. [CrossRef]
- Reveiz L, Sangalang S, Glujovsky D, Pinzon CE, Asenjo Lobos C, Cortes M, Cañón M, Bardach A, Bonfill X. Characteristics of randomized trials published in Latin America and the Caribbean according to funding source. PLoS One. 2013;8(2):e56410. [CrossRef]
- Tan AC, Jiang I, Askie L, Hunter K, Simes RJ, Seidler AL. Prevalence of trial registration varies by study characteristics and risk of bias. J Clin Epidemiol. 2019 Sep;113:64-74. [CrossRef]
- Jo, D. The interpretation bias and trap of multicenter clinical research. Korean J Pain. 2020 Jul 1;33(3):199-200. [CrossRef]
- Arina Tamborska, Michael T.C. Poon and Rustam Al-Shahi Salman. Characteristics of Randomized Trials Focusing on Stroke due to Intracerebral Hemorrhage. 2018;49:594-600. [CrossRef]
- Cho Y, Kim C, Kang B. Risk of bias assessment of randomised controlled trials referenced in the 2015 American Heart Association guidelines update for cardiopulmonary resuscitation and emergency cardiovascular care: a cross-sectional review. BMJ Open. 2019 May 5;9(5):e023725. [CrossRef]
- Lohner S, Gryaznov D, von Niederhäusern B, Speich B, Kasenda B, Ojeda-Ruiz E, Schandelmaier S, Mertz D, Odutayo A, Tomonaga Y, Amstutz A, Pauli-Magnus C, Gloy V, Bischoff K, Wollmann K, Rehner L, Meerpohl JJ, Nordmann A, Klatte K, Ghosh N, Heravi AT, Wong J, Chow N, Hong PJ, McCord K, Sricharoenchai S, Busse JW, Agarwal A, Saccilotto R, Schwenkglenks M, Moffa G, Hemkens LG, Hopewell S, von Elm E, Blümle A, Briel M. Reporting quality of trial protocols improved for non-regulated interventions but not regulated interventions: A repeated cross-sectional study. J Clin Epidemiol. 2021 Nov;139:340-349. [CrossRef]
Table 1.
Characteristics of cardiovascular trials from 2012(n=250) and 2017 (n=250).
| Characteristics | 2012, n(%) | 2017, n (%) | P value |
|---|---|---|---|
| Type of Journal | <0.001 | ||
| Specialty cardiovascular journal | 96(38.4%) | 100(40.0%) | |
| General cardiovascular journal | 41(16.4%) | 46(18.4%) | |
| Specialty medical journal | 26(10.4%) | 49(19.6%) | |
| General medical journal | 50(20.0%) | 41(16.4%) | |
| Other | 37(14.8%) | 14(5.6%) | |
| Continent of corresponding author | <0.05 | ||
| Africa | 3(1.2%) | 0(0.0%) | |
| Asia | 57(22.8%) | 65(26.0%) | |
| Australia | 10(4.0%) | 2(0.8%) | |
| Europe (excluding UK) | 70(28.0%) | 93(37.2%) | |
| North America | 89(35.6%) | 69(27.6%) | |
| South America | 8(3.2%) | 13(5.2%) | |
| United Kingdom | 13(5.2%) | 8(3.2%) | |
| Total | 250(100%) | 250(100%) | |
| Study type | 0.093 | ||
| Efficacy/Superiority | 244(97.6%) | 237(94.8%) | |
| Equivalence | 2(0.8%) | 3(1.2%) | |
| Non-inferiority | 4(1.6%) | 4(1.6%) | |
| None of the above | 0(0.0%) | 6(2.4%) | |
| Study design | <0.01 | ||
| Cluster | 7(2.8%) | 0(0.0%) | |
| Parallel | 201(80.4%) | 231(92.4%) | |
| Crossover | 34(13.6%) | 15(6.0%) | |
| Factorial | 5(2.0%) | 4(1.6%) | |
| Other | 3(1.2%) | 0(0.0%) | |
| Intervention type | <0.001 | ||
| Alternative therapeutic | 24(9.6%) | 32(12.8%) | |
| Behavioral | 0(0.0%) | 2(0.8%) | |
| Cell therapy | 0(0.0%) | 1(0.4%) | |
| Communication, organizational, or educational | 4(1.6%) | 13(5.2%) | |
| Device | 17(6.8%) | 23(9.2%) | |
| Diet, nutrition | 26(10.4%) | 10(4.0%) | |
| Drug | 117(46.8%) | 139(55.6%) | |
| Prevention or screening | 43(17.2%) | 20(8.0%) | |
| Rehabilitation or psychosocial | 18(7.2%) | 6(2.4%) | |
| Surgery or radiotherapy | 1(0.4%) | 3(1.2%) | |
| Other | 0(0.0%) | 1(0.4%) | |
| Type of control | 0.628 | ||
| Active intervention | 153(61.2%) | 160(64.0%) | |
| No intervention | 10(4.0%) | 21(8.4%) | |
| Placebo | 86(34.4%) | 68(27.2%) | |
| Other | 1(0.4%) | 1(0.4%) | |
| Was the study multicenter? | 0.063 | ||
| Yes | 117(46.8%) | 157(62.8%) | |
| No | 131(52.4%) | 93(37.2%) | |
| Unclear | 2(0.8%) | 0(0.0%) | |
| Was the study multinational? | <0.05 | ||
| Yes | 45(18.0%) | 69(27.6%) | |
| No | 205(82.0%) | 181(72.4%) | |
| Where were participants recruited from? | <0.001 | ||
| Developing country | 3(1.2%) | 21(8.4%) | |
| Transitional country | 8(3.2%) | 13(5.2%) | |
| Established market economy | 239(95.6%) | 216(86.4%) | |
| 250(100%) | 250(100%) | ||
| Who funded the study? | <0.001 | ||
| Academic or Research institute | 113(45.2%) | 94(37.6%) | |
| Government | 44(17.6%) | 24(9.6%) | |
| Industry for device | 4(1.6%) | 10(4.0%) | |
| No external funding | 3(1.2%) | 4(1.6%) | |
| Pharmaceutical | 36(14.4%) | 48(19.2%) | |
| Private | 13(5.2%) | 50(20.0%) | |
| Unclear | 37(14.8%) | 21(8.4%) | |
| Total | 250(100%) | 250(100%) | |
| How was the study population selected? | 0.775 | ||
| Inpatients | 144(57.6%) | 133(53.2%) | |
| Outpatients | 98(39.2%) | 116(46.4%) | |
| Unclear | 7(2.8%) | 1(0.4%) | |
| Primary diagnostic category in the study | 0.971 | ||
| Circulatory system | 250(100%) | 244(97.6%) | |
| Congenital malformations | 0(0.0%) | 1(0.4%) | |
| Factors influencing health status | 0(0.0%) | 2(0.8%) | |
| Metabolic disease | 0(0.0%) | 2(0.8%) | |
| Unclear | 0(0.0%) | 1(0.4%) | |
| Footnote: Intervention categories were defined based on Wood et al, 2008 (20) Multicenter trials were defined as trials with two or more administratively distinct study centers. Multinational applied to the countries from which patients were enrolled. Economic status of the country was defined based on Panagiotou et al, 2013( 21 ) | |||
Table 2.
Changes in important measures of methodological quality and reporting.
| Study characteristics | 2012, n (%) | 2017, n (%) | p |
|---|---|---|---|
| Funding source | 0.002 | ||
| Specified | 243(97.2%) | 229 (91.6%) | |
| Not specified | 7(2.8%) | 21 (8.4%) | |
| Consent obtained | 0.895 | ||
| Reported | 250(100%) | 248(99.2%) | |
| Not reported | 0(0.0%) | 2(0.8%) | |
| Number of patients approached to participate in the study | 0.854 | ||
| Reported | 2 (0.2 %) | 12 (4.8%) | |
| Not reported | 248(99.8%) | 238(95.2 %) | |
| Number of patients consented to participate in the study | 0.534 | ||
| Reported | 2 (0.2 %) | 12 (4.8%) | |
| Not reported | 248(99.8%) | 238(95.2 %) | |
| Number of participants randomized | 0.972 | ||
| Reported | 2 (0.2 %) | 2 (99.8 %) | |
| Not reported | 248(99.8%) | 248(2.0%) | |
| Number of participants analysed | 0.887 | ||
| Reported | 2 (0.2 %) | 1(0.4%) | |
| Not reported | 248(99.8%) | 249(99.6 %) | |
| Sample size calculation | <0.001 | ||
| Reported | 246 (98.4%) | 151 (60.4%) | |
| Not reported | 4 (1.6%) | 99 (39.6%) | |
| Data Monitoring Committee | <0.001 | ||
| Yes | 86 (34.4%) | 105 (42.0%) | |
| No | 39 (15.6%) | 94 (37.6%) | |
| Unclear | 125 (50.0%) | 51 (20.4%) | |
| Analysis described as intention to treat | 0.120 | ||
| Yes | 232(92.8%) | 222(88.8%) | |
| No | 18 (7.2%) | 28 (11.2%) | |
| Primary outcome specified in trial registry | 0.823 | ||
| Yes | 135(54.0 %) | 157 (62.8%) | |
| No | 115(46.0%) | 93 (37.2%) | |
| Primary outcome was objective | 0.652 | ||
| Objective | 247(98.8%) | 248 (99.2%) | |
| Subjective | 3 (1.2%) | 2 (0.8%) | |
| Type of primary outcome | 0.124 | ||
| Behavioural | 20 (8.0%) | 6 (2.4%) | |
| Biomarker | 40(16.0%) | 21 (8.4%) | |
| Physiological | 172(68.8%) | 206(82.4%) | |
| Psychological | 5 (2.0%) | 5 (2.0%) | |
| Techniques/Training | 8(3.2%) | 6 (2.4%) | |
| Quality of life | 3 (1.2%) | 1 (0.4%) | |
| Other | 2(0.8%) | 3 (1.2%) | |
| At least one statistically significant outcome | 0.899 | ||
| Yes | 213(85.2%) | 215(86.0%) | |
| No | 37 (14.8%) | 35(14.0%) | |
| Significant statistical primary outcome | <0.01 | ||
| Yes | 197(78.8%) | 173(69.2%) | |
| No | 53(21.2%) | 77(30.8%) | |
| The authors overall conclusion | <0.01 | ||
| Negative | 32(12.8%) | 34(13.6%) | |
| Neutral | 18(7.2%) | 46(18.4%) | |
| Positive | 193(77.2%) | 170(68.0%) | |
| Insufficient evidence (intermediate) | 7(2.8%) | (0.0 %) | |
| Planning to collect adverse effects/ events or side effects | <0.001 | ||
| Reported | 185(74.0%) | 121(48.4 %) | |
| Not reported | 65(26.0 %) | 129(51.6%) | |
| Harms reported | <0.001 | ||
| Yes | 130(52.0%) | 170(68.0 %) | |
| No | 120(48.0%) | 80(32.0%) | |
| Blinding performed | 0.087 | ||
| Yes | 126(50.4%) | 145(58.0%) | |
| No | 124(49.6%) | 105(42.0%) | |
| Trial registered | 0.238 | ||
| Yes | 135 (54.0%) | 192(76.8%) | |
| No | 115 (46.0%) | 58 (23.24%) | |
| Primary register | 0.031 | ||
| clinicaltrials.gov | 124(68.9%) | 164(78.4%) | |
| Other | 56(31.1%) | 45(21.6%) | |
| Primary outcome stated the same in trial registry and in the publication | <0.001 | ||
| Yes | 132(52.8%) | 183(73.2%) | |
| No | 76(30.4%) | 26(10.4%) | |
| N/A | 42(16.8%) | 41(16.4%) |
Footnote: Behavioural outcome included attitudes and specific (e.g. eating) behaviours; biomarkers were defined as markers measured as an indicator of biologic or pathogenic processes or pharmacologic responses to an intervention; physiological outcomes reflected how a patient feels, functions or survives; psychological and quality of life outcomes included different scales measuring these variables.
Table 3.
Risk of bais assessments by domain in 2012 (n=250) and in 2017 (n=250).
| RoB domains | N(%) in 2012 |
N(%) in 2017 |
P |
|---|---|---|---|
| Random sequence generation | |||
| Low | 140(56.0%) | 149(59.6%) | 0.381 |
| Unclear | 95(38.0%) | 68(27.2%) | |
| High | 15(6.0%) | 33(13.2%) | |
| Allocation concealment | |||
| Low | 97(38.8%) | 175(70.0%) | <0.001 |
| Unclear | 125(50.0%) | 51(20.4%) | |
| High | 28(11.2%) | 24(9.6%) | |
| Blinding participants and personnel | |||
| Low | 164(65.6%) | 127(50.8%) | <0.001 |
| Unclear | 73(29.2%) | 112(44.8%) | |
| High | 13(5.2%) | 11(4.4%) | |
| Blinding outcome assessors | |||
| Low | 227(90.8%) | 206(82.4%) | <0.001 |
| Unclear | 19(7.6%) | 33(13.2%) | |
| High | 4(1.6%) | 11(4.4%) | |
| Incomplete outcome data | |||
| Low | 184(73.6%) | 185(74.0%) | 0.469 |
| Unclear | 60(24.0%) | 57(22.8%) | |
| High | 6(2.4%) | 8(3.2%) | |
| Selective outcome reporting | |||
| Low | 200(80.0%) | 157(62.8%) | <0.001 |
| Unclear | 48(19.2.0%) | 67(26.8%) | |
| High | 2(0.8%) | 26(10.4%) | |
| Other bias | |||
| Low | 84(33.6%) | 108(42.8%) | <0.01 |
| Unclear | 131(52.4%) | 106(42.4%) | |
| High | 35(14.0%) | 36(14.4%) | |
| Overall bias | |||
| Low | 53(21.2%) | 73(29.2%) | <0.01 |
| Unclear | 142(56.8%) | 99(39.6%) | |
| High | 55(22.0%) | 78(31.2%) |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Alerts
MDPI Initiatives
Important Links
© 2025 MDPI (Basel, Switzerland) unless otherwise stated