Development and validation of a biostatistical framework for consistent evaluation of food supplement efficacy and safety using real-world data

Introduction

The global market for food supplements has expanded rapidly over the past decade, reflecting growing public interest in preventive healthcare and long-term wellness management. Vitamins, minerals, probiotics, botanical extracts, and nutraceutical formulations have become part of routine health practices for millions of consumers worldwide. Despite the scale of this industry, regulatory supervision remains inconsistent across many countries, creating difficulties in assessing product quality, clinical efficacy, and long-term safety [1]. Differences between commercial availability and scientific validation continue to generate concerns regarding reliability of health claims and post-market monitoring standards. Randomized Controlled Trials (RCTs) traditionally remain the primary method for evaluating medical interventions. However, their application to food supplements is associated with several practical and methodological limitations. Such trials require considerable financial resources, lengthy implementation periods, and strict participant selection criteria [2]. In many cases, individuals with comorbidities, irregular lifestyles, or concurrent medication use are excluded from participation. As a result, highly controlled trial conditions often fail to reflect the variability of real consumers and everyday clinical practice. This reduces external validity and complicates translation of trial outcomes into broader population settings.

Against this background, Real-World Data (RWD) has emerged as an important source of evidence for healthcare analytics and regulatory evaluation [3]. RWD includes information collected outside conventional clinical trials, including electronic health records, observational registries, insurance databases, wearable device outputs, and patient-reported outcomes. After appropriate statistical processing, these datasets can generate Real-World Evidence (RWE) capable of supporting practical decision-making and post-market surveillance activities [4]. Regulatory organizations such as the Food and Drug Administration (FDA) and the European Food Safety Authority (EFSA) have increasingly incorporated RWE into modern evaluation frameworks [5].

At the same time, analysis of RWD presents substantial methodological challenges. Unlike controlled experimental datasets, real-world information is frequently incomplete, heterogeneous, and structurally inconsistent. Missing observations, coding variability, measurement inaccuracies, and differences in reporting standards complicate statistical interpretation. In addition, observational data is highly susceptible to confounding by indication and selection bias, particularly when treatment allocation is influenced by patient condition or physician preference [6]. Conventional regression approaches often fail to capture these complexities adequately, especially in high-dimensional environments with multiple interacting variables. Another significant issue involves the persistence of "clinical rigidity" within observational supplement studies. Even non-interventional investigations frequently rely on restrictive inclusion criteria that artificially simplify patient populations. Individuals with severe symptoms, multiple chronic conditions, or inconsistent treatment adherence are commonly excluded. Consequently, the resulting evidence may not accurately represent real consumer behavior or actual safety risks encountered during routine supplement use [7].

Without standardized analytical frameworks capable of accounting for such variability, conclusions derived from RWD may remain unstable and potentially misleading. Within the thesis, a biostatistical methodology for standardized analysis of high-dimensional Real-World Data was developed and validated. The proposed framework integrates causal inference techniques with ensemble machine learning models to improve consistency of efficacy and safety assessment for food supplements. Attention was devoted to reducing observational bias and improving reliability of post-market evidence generation. The methodological novelty of the thesis lies in combining Propensity Score Weighting methods with advanced machine learning architectures, including CatBoost and Random Forest algorithms [8, 9]. This integration makes it possible to identify complex non-linear interactions within observational datasets while simultaneously preserving covariate balance between comparison groups.

In contrast to traditional statistical matching procedures, the developed framework supports deeper modeling of heterogeneous clinical patterns and hidden risk relationships. An additional component of the methodology involves the implementation of Multi-Objective Pareto Optimization for benefit-risk evaluation [10, 11]. Instead of relying on arbitrary weighting coefficients, the framework identifies mathematically optimal tradeoffs between efficacy and safety indicators. This makes it possible to compare supplement profiles using an objective and scale-invariant analytical approach suitable for regulatory applications. The overall methodology was constructed according to principles of Target Trial Emulation, allowing observational datasets to approximate conditions associated with randomized comparative studies [5, 12].

Materials and methods

To validate the framework, a dual-track data methodology was employed, encompassing a clinical audit of non-interventional pilot studies and an empirical validation on high-dimensional synthetic RWD.

Fig. 1. Workflow of the proposed biostatistical methodology for the transformation of Real-World Data into Real-World Evidence

Data Acquisition and Characterization

Validation of the proposed methodology was carried out using two complementary data sources that reflected both practical clinical observations and large-scale simulated real-world conditions. The first stage included analysis of two non-interventional pilot studies. The Study 1 dataset represented a prospective observational cohort study involving 50 peri- and postmenopausal women receiving Food supplement 1 over a six-month period. The Study 2 dataset included 199 patients with chronic low back pain who received therapy based on the Food supplement 2 during a 90-day observation interval. These datasets were selected because they reflected typical limitations encountered in routine observational practice, including incomplete records, heterogeneous patient characteristics, and variable treatment adherence.

The second stage involved validation using a high-dimensional dataset containing 6,000 observations designed to reproduce the structural variability of contemporary healthcare information systems. This was artificially created to reproduce the statistical properties and structural complexity of real medical data as shown by the generation algorithm below.

Fig. 2. Conceptual workflow of the synthetic clinical data generation algorithm, incorporating baseline confounding, heterogeneous outcomes, missingness mechanisms, and empirical calibration [16]

The generated dataset combined structured variables, including dosage, treatment duration, demographic characteristics, and clinical outcomes, with semi-structured and unstructured components such as physician notes, adherence descriptions, and socio-behavioral indicators. This made it possible to evaluate the framework under conditions closer to real-world clinical data environments.

Data Preprocessing and Handling Missing Data. Before analytical modeling, preprocessing procedures were applied to improve consistency and reliability of the collected information. Duplicate records were removed, inconsistent terminology was standardized, and clinical coding systems were harmonized through conversion of ICD-9 and ICD-10 identifiers into phone-based groupings [16]. A major challenge involved the large proportion of incomplete observations frequently encountered in Real-World Data. Missing values were processed using Multiple Imputation by Chained Equations (MICE) [17]. Because the datasets originated from observational rather than randomized environments, the missingness mechanism was treated as Missing at Random (MAR). Ten imputation iterations were performed to stabilize parameter estimation and reduce variability of standard errors. The imputation model incorporated demographic variables, treatment-related characteristics, behavioral indicators, and additional derived features associated with text-based clinical descriptions. After completion of the imputation process, predictive outputs from all generated datasets were aggregated according to Rubin's Rules, allowing uncertainty introduced during imputation to be reflected within final estimates and confidence intervals.

The Analytical Framework: Causal Inference. To reduce the influence of selection bias and confounding by indication, propensity scores were estimated using multivariable logistic regression models [9]. Within the framework, the propensity score represented the probability of treatment assignment conditional on observed baseline characteristics.

e(X) = P(T = 1|X)

After estimation of propensity scores, Inverse Probability of Treatment Weighting (IPTW) was applied to construct a weighted pseudo-population in which treatment assignment became statistically independent from measured covariates [9]. Additional stabilization procedures were introduced to reduce the influence of extreme weights. Asymmetric trimming at the 1st and 99th percentiles was performed to prevent instability during machine learning model training [18]. Balance diagnostics were evaluated using Standardized Mean Differences (SMDs). Covariate balance was considered acceptable when SMD values remained below 0.10 across baseline variables, indicating successful approximation of target trial conditions within the observational dataset [13].

Machine Learning Component. After causal balancing procedures had been completed, predictive modeling was performed using ensemble machine learning methods trained on the IPTW-weighted dataset. The analytical framework incorporated Gradient Boosting algorithms based on the CatBoost architecture together with Random Forest models [19]. The selected algorithms were capable of identifying non-linear dependencies, hidden feature interactions, and heterogeneous treatment patterns commonly observed in Real-World Data. Unlike traditional parametric models, ensemble tree-based methods do not require extensive manual feature engineering and remain more adaptable when processing complex observational datasets containing mixed variable types and irregular distributions [8].

A separate stage of the analysis focused on adverse-event prediction. Because the proportion of positive safety outcomes remained relatively low, the classification task was characterized by strong class imbalance. Under such conditions, use of the default probability threshold frequently leads to unstable predictive quality and insufficient sensitivity to rare events. To improve analytical performance, classification thresholds were additionally optimized instead of relying on the standard cutoff value of 0.50. Model assessment emphasized optimization of the F1-score together with analysis of Precision-Recall trade-offs. This approach made it possible to improve detection of weak safety signals and increase stability of adverse-event prediction within highly imbalanced observational datasets.

Multi-Objective KPI Evaluation and Pareto Optimization. Traditional composite indices in clinical evaluation frequently rely on linear scalarization. Conventional composite evaluation indices often combine several statistical indicators into a single scalar metric using predefined weighting coefficients [20]. In clinical and biostatistical applications, this approach may introduce subjective bias because different performance measures possess different mathematical properties and scales. For example, regression-based efficacy indicators such as R² are frequently combined with classification metrics including PR_AUC despite their fundamentally different analytical interpretations.

To reduce dependence on arbitrary weighting procedures, the developed framework incorporated Multi-Objective Pareto Optimization (MOO) [11]. Within the model, supplement interventions were evaluated across a two-dimensional objective space representing efficacy and safety performance simultaneously. The efficacy objective was defined as maximization of the regression performance metric:

Objective 1 (Efficacy): Maximize f₁(x) = R²

The safety objective was defined as maximization of the Precision-Recall Area Under Curve:

Objective 2 (Safety): Maximize f₂(x) = PR_AUC

An intervention was considered Pareto optimal when no alternative configuration could improve efficacy without causing deterioration of the safety profile. This principle allowed identification of mathematically balanced solutions rather than relying on fixed subjective priorities between competing clinical indicators [21]. To support comparative ranking of interventions, the framework additionally calculated the normalized Euclidean distance from each solution to a theoretical "utopian point" represented by the coordinates (1,1). The inverse transformed distance was then converted into a bounded scalar Key Performance Indicator ranging from 0 to 1 [22]. This made it possible to obtain a scale-invariant metric suitable for comparative regulatory evaluation and benefit-risk assessment.

Results

Phase 1: Critique of the Pilot Study Datasets

Analysis of the Study 1 and Study 2 revealed significant structural limitations in current RWE generation methodologies. The pilot studies demonstrated profound "clinical rigidity." By excluding high-risk populations (e.g., patients with severe symptoms or those on antidepressants), they failed to represent the complex consumers who actually utilize these supplements. Safety reporting was entirely reactive, and the use of Complete Case Analysis (deleting dropouts) introduced severe survivorship bias, masking true intolerance rates.

Fig. 3. Comparison of methodological and generalizability limitations in non-interventional supplement studies

Table 2. Methodological limitations were identified during the analysis of non-interventional pilot studies

Phase 2: Validation on High-Dimensional RWD

Missingness Profile and Imputation Diagnostics. Evaluation of the synthetic validation dataset (N = 6000) revealed a substantial proportion of incomplete observations within several treatment-related variables. The highest level of missingness was identified in the Dose_mg and Duration_Days parameters, where incomplete records accounted for 83.33% of observations. Such data sparsity reflected conditions commonly encountered in large-scale Real-World Data environments. To address this issue, Multiple Imputation by Chained Equations (MICE) was applied. Convergence diagnostics based on trace plot analysis demonstrated stable iterative behavior throughout the imputation process. Additional comparison between observed and reconstructed value distributions indicated that the Missing at Random (MAR) assumption remained acceptable for the analyzed dataset. As a result, incomplete patient records associated with treatment discontinuation and irregular follow-up were retained within the analytical pipeline instead of being excluded from subsequent modeling stages.

Causal Balancing Diagnostics. Before application of weighting procedures, several baseline covariates demonstrated Standardized Mean Difference (SMD) values above 0.25, indicating the presence of clinically significant imbalance between comparison groups and substantial confounding by indication. After implementation of stabilized Inverse Probability of Treatment Weighting with asymmetric trimming, covariate imbalance was markedly reduced. Across all measured baseline variables, SMD values decreased below the predefined threshold of 0.10. Overlap analysis additionally demonstrated sufficient common support between weighted treatment groups. This made it possible to construct a balanced pseudo-population suitable for subsequent predictive and comparative analysis.

Efficacy Prediction Results. Predictive models were trained using the weighted pseudo-population to estimate changes in Efficacy_Score_Improvement. Among the evaluated approaches, the Linear Regression model supplemented with interaction features associated with dosage and treatment duration demonstrated the most stable predictive performance. The model achieved an R² value of 0.3596 with a 95% confidence interval ranging from 0.33 to 0.38. The Mean Absolute Error reached 13.96, indicating moderate predictive deviation under high-dimensional observational conditions. Among evaluated supplement categories, the probiotic formulation based on Bifidobacterium demonstrated the highest average efficacy improvement score, reaching 56.63.

Table 3. Performance metrics of efficacy prediction models within the causally balanced pseudo-population

Safety Results: Signal Detection and Threshold Tuning. Predicting rare Adverse Events in RWD presents a severe class-imbalance challenge (positive class rate < 8%). Consequently, ROC_AUC is misleading. The framework utilized Precision-Recall Area Under Curve (PR_AUC) and the F1-score as definitive metrics. The CatBoost model with optimized classification thresholds demonstrated the highest predictive performance among the evaluated safety detection approaches. The model achieved a PR_AUC value of 0.1230 together with an optimized F1-score of 0.2180, exceeding the performance of baseline logistic regression models under conditions of severe class imbalance. The developed analytical approach also enabled identification of latent adverse-event patterns that may remain undetected within conventional passive reporting systems. This made it possible to improve sensitivity of safety monitoring and strengthen detection of clinically relevant risk signals in observational supplement data. The analytical model identified elevated adverse-event frequencies for several supplement categories. The highest rate was observed for Magnesium 400 mg at 12.47%, followed by Magnesium Glycinate at 9.47% and Vitamin D3 at 8.51%.

Fig. 4. Comparative evaluation of PR values for machine learning and logistic safety prediction models

Fig. 5. Detection of high-risk supplement interventions using algorithm-based safety signal analysis

Table 4. Performance comparison of safety prediction models for detection of rare adverse events

Multi-objective performance evaluation and Pareto optimization of efficacy-safety trade-offs. To reduce the limitations associated with fixed scalar weighting and improve interpretation of benefit-risk relationships, multi-objective Pareto optimization was applied within the analytical framework. The optimization procedure mapped efficacy indicators represented by R² values against safety performance measured through PR_AUC metrics. This approach made it possible to evaluate supplement interventions simultaneously across both clinical effectiveness and safety dimensions without introducing subjective weighting coefficients. As illustrated in Fig. 5, the optimization procedure eliminated reliance on arbitrary weighting schemes by identifying a well-defined Pareto frontier of optimal supplement interventions across competing efficacy and safety objectives:

• Maximum efficacy was observed for Omega3 1000 mg, which demonstrated the highest overall clinical improvement score within the evaluated intervention set.

• Maximum safety signal sensitivity was associated with Magnesium Glycinate, indicating the strongest capacity for proactive adverse-event detection.

Fig. 6. Multi-objective Pareto frontier illustrating trade-offs between efficacy and safety in supplement intervention evaluation

Balanced trade-off solutions were represented by VitD_5000IU and Magnesium_400mg, which occupied the central region ("elbow point") of the Pareto frontier, reflecting efficient compromises between therapeutic efficacy and safety monitoring performance.

Discussion and Future Directions

The results obtained were compiled and sent for Human Interpretation by specialists in biological data analysis.

Interpretation of Results and "Clinical Rigidity". The analysis of the pilot studies highlights a pervasive state of "clinical rigidity" in nutritional science. Although ostensibly designed as non-interventional observational studies, they persist in forcing heterogeneous real-world patients into homogeneous "clean boxes" [6]. By excluding populations with severe symptoms, specific pain etiologies, or those utilizing concomitant medications, these trials create a massive Generalizability Gap. The outputs of such trials represent an idealized physiology, entirely detached from the high-risk, real-world consumer base. Our proposed framework rectifies this by ingesting the totality of the RWD and utilizing IPTW to adjust for such covariates mathematically, rather than excluding them physically [12].

Addressing the Data Continuity Gap. Traditional analytical approaches address messy data and irregular dosing through Complete Case Analysis (data deletion). As seen in the pilot studies, excluding non-adherent patients introduces Survivorship Bias, artificially inflating supplement success rates because only the most tolerant, compliant patients remain [6]. The biostatistical methodology developed herein solves this by instituting Multiple Imputation by Chained Equations (MICE) [17]. Despite the use of MICE, a high proportion of missing values (up to 83.33% in individual variables) remains a significant source of statistical uncertainty. By computationally estimating missing data from dropouts or Missing at Random (MAR), the framework preserves the statistical reality of product intolerance. The successful application of MICE to data with 83.33% missingness made it possible to preserve observations that would otherwise have been excluded using listwise deletion. However, the approach does not eliminate the risk of bias completely, especially if assumptions about the structure of omissions are violated [23].

The Statistical Gap: Linear vs. Advanced Inference. The validation dataset proved the necessity of advanced machine learning methodologies over standard parametric. While a well-engineered Linear Regression captured the continuous baseline efficacy (R² = 0.3596), predicting rare safety events required the non-linear dimensional capability of CatBoost [9]. Traditional statistical models like logistic regression fail to account for non-linear, high-dimensional interactions without massive manual feature engineering. CatBoost's native handling of categorical text and optimized F1 thresholding proved that ensemble tree models are mandatory for proactive signal detection in highly imbalanced real-world datasets [24, 25].

Bridging the Safety Gap: Proactive Monitoring. Currently, the approach to supplement safety is hindered by a reactive reporting system, relying on physicians to notice and manually document adverse events. This passive method misses cumulative, sub-clinical indicators. Our proposed framework disrupts this paradigm by introducing Proactive Safety Signal Detection [26]. By implementing continuous analysis on raw RWD using CatBoost [25, 27], our framework uncovered hidden safety signals such as the 12.47% adverse event rate associated with Magnesium 400 mg that traditional "total quantity" manual reporting logs entirely mask.

Methodological Limitations. While highly effective, the proposed framework possesses inherent limitations. First, the primary empirical validation was conducted on a high-dimensional synthetic dataset. Although intricately designed to mimic the multi-modal complexity and structural noise of modern health data [27], authentic real-world registries are characterized by unpredictable anomalies that synthetic generation processes may struggle to fully replicate [4]. Secondly, causal inference methodologies, specifically the IPTW utilized in this framework, operate on the strict, untestable assumption of "no unmeasured confounding" [6]. If critical variables that influence both supplement choice and outcomes are absent from the RWD, residual bias remains. Furthermore, the validity of causal effect estimates relies heavily on the accurate specification of the propensity score model [26]. Consequently, future deployments of this framework must incorporate robust sensitivity analyses (e.g., E-values) to mathematically quantify the potential impact of unmeasured variables on the observed treatment effects [12].

Comparative Validation with Existing Literature. To substantiate the robustness of the proposed framework, it is imperative to benchmark the results obtained against contemporary biostatistical literature. The success of the IPTW in reducing SMDs to < 0.10 directly corroborates the target trial emulation principles established by Cui et al. [12], who demonstrated that mimicking RCT exchangeability through rigorous propensity score weighting is mathematically superior to traditional covariate adjustment. Furthermore, the superior performance of the CatBoost architecture in detecting rare adverse events aligns with the findings of Reps et al. [26] and Marzano et al. [27], who proved that ensemble models significantly outperform standard parametric designs in risk stratification for imbalanced clinical datasets. Finally, the application of the Pareto Frontier objectively maps trade-offs, ensuring that regulatory evaluations are scale-invariant and immune to the priori weighting bias heavily critiqued in modern multi-criteria decision analysis literature [28].

Regulatory Implications. The developed framework provides a highly structured, foundational prototype for global oversight bodies seeking to evaluate nutraceuticals using real-world data. It successfully maps the necessary biostatistical pathway to transition from passive, reactive pharmacovigilance to proactive, automated post-market monitoring [5]. By providing a scale-invariant, objective heuristic (the Pareto Frontier and Utopian Point distance), regulatory agencies like the FDA, EFSA, and Roszdravnadzor RF can rank and compare disparate products objectively. This empowers domestic clinical research organizations to quantitatively substantiate product portfolios with mathematically unbiased data, fundamentally shifting post-market surveillance from subjective estimation to rigorous, algorithm-backed science [24].

Future Perspectives. Future iterations of this research should include several technological extensions aimed at improving analytical precision and expanding the applicability of the proposed framework across diverse data environments. Utilizing direct APIs, the framework can seamlessly ingest Patient-Generated Health Data (PGHD) from wearable devices. This enables continuous acquisition of physiological telemetry in real time and reduces reliance on retrospective self-reporting. As a result, recall bias associated with traditional patient diaries is minimized, while temporal resolution of health-related measurements is significantly improved [29]. The integration of advanced Large Language Models (LLMs) allows systematic processing of unstructured clinical text, including free-form physician notes and narrative medical records. Such models can be used to extract latent safety-related information and convert unstructured descriptions into structured analytical variables suitable for downstream statistical and machine learning analysis. This makes it possible to improve completeness of safety signal detection and reduce information loss during data preprocessing stages [30]. Further development of the framework should involve deployment across large-scale federated electronic health record (EHR) infrastructures spanning multiple countries. This approach enables evaluation under heterogeneous healthcare systems, differing dietary patterns, and variable genetic backgrounds. Such cross-population validation improves robustness of the methodology and enhances generalizability of results obtained from real-world data analyses [5].

Conclusion

This study developed and validated a hybrid biostatistical framework for evaluating food supplement efficacy and safety using Real-World Data (RWD). The analysis of two non-interventional pilot studies revealed key limitations of conventional statistical approaches in observational settings, particularly related to clinical rigidity and reliance on reactive safety monitoring. Using a high-dimensional validation cohort (N = 6000), the framework applied Multiple Imputation by Chained Equations (MICE) to address extreme missingness (83.33% in key dosage and duration variables), allowing recovery of incomplete records and reducing survivorship bias. Subsequent application of Inverse Probability of Treatment Weighting (IPTW) reduced confounding by indication and improved covariate balance, with all standardized mean differences brought below 0.10 in line with target trial emulation criteria. For outcome modeling, the linear regression approach with interaction terms produced a stable R² of 0.3596. In parallel, ensemble machine learning methods (CatBoost) were used for safety signal detection, achieving a PR_AUC of 0.123 and an optimized F1-score of 0.218. This enabled identification of previously underreported adverse event patterns, including a 12.47% rate associated with Magnesium 400 mg, which is not typically captured through conventional reporting systems. To support integrated benefit-risk evaluation, a multi-objective Pareto optimization approach was introduced, removing the need for subjective weighting of performance metrics and enabling identification of optimal trade-offs between efficacy and safety. Overall, the results indicate that the proposed framework improves consistency and robustness in real-world supplement evaluation and supports more structured post-market assessment of efficacy and safety outcomes.