Gold-Standard Data Download and Benchmark Setup

This vignette shows how to move from dataset discovery to a runnable benchmark manifest for external gold-standard validation.

1. Preconditions

To download open datasets, you need:

  1. bash and wget
  2. DNS/HTTPS access to external hosts (for example, physionet.org, ncbi.nlm.nih.gov, ftp.ncbi.nlm.nih.gov)

In this monorepo, the helper script is:

  • publication/scripts/download_open_medrehab_gold_data.sh
bash publication/scripts/download_open_medrehab_gold_data.sh data/external

Controlled datasets (for example, MIMIC-IV full, eICU, MOST, OAI controlled resources) require account approval and DUA steps before download.

2. Create a benchmark manifest

Use a template and then replace file paths with your real prediction/reference pairs (.csv, .mot, .sto, .trc).

library(PhysioMoCap)

manifest <- benchmarkManifestTemplate(n = 2)
manifest
#>   benchmark_id  prediction_file  reference_file modality units
#> 1      trial_1 prediction_1.csv reference_1.csv    mocap    SI
#> 2      trial_2 prediction_2.csv reference_2.csv    mocap    SI
#>   coordinate_system sampling_rate notes
#> 1               lab           100      
#> 2               lab           100

Write the template to CSV if needed:

tmp_manifest <- tempfile("benchmark_manifest_", fileext = ".csv")
writeBenchmarkManifest(tmp_manifest, n = 2, overwrite = TRUE)
tmp_manifest
#> [1] "/tmp/Rtmp1Kv1zY/benchmark_manifest_115467ebc2b3.csv"

3. Point manifest rows to your downloaded files

Below is a minimal pattern. File names are examples; replace them with paths that exist in your local data_dir.

data_dir <- tempfile("gold_data_")
dir.create(data_dir, recursive = TRUE, showWarnings = FALSE)

# Demo-only placeholder files so the validation example can run end-to-end.
pred <- data.frame(knee_angle = rnorm(200, sd = 0.02),
                   hip_angle = rnorm(200, sd = 0.02))
ref  <- data.frame(knee_angle = pred$knee_angle + rnorm(200, sd = 0.01),
                   hip_angle = pred$hip_angle + rnorm(200, sd = 0.01))

utils::write.csv(pred, file.path(data_dir, "prediction_trial1.csv"), row.names = FALSE)
utils::write.csv(ref,  file.path(data_dir, "reference_trial1.csv"), row.names = FALSE)

manifest <- benchmarkManifestTemplate(n = 1)
manifest$benchmark_id[1] <- "trial1_external"
manifest$prediction_file[1] <- "prediction_trial1.csv"
manifest$reference_file[1] <- "reference_trial1.csv"
manifest$modality[1] <- "mocap"
manifest$units[1] <- "SI"
manifest$sampling_rate[1] <- 100
manifest
#>      benchmark_id       prediction_file       reference_file modality units
#> 1 trial1_external prediction_trial1.csv reference_trial1.csv    mocap    SI
#>   coordinate_system sampling_rate notes
#> 1               lab           100

4. Validate manifest integrity

v <- validateBenchmarkManifest(manifest, data_dir = data_dir)
v
#> Benchmark Manifest Validation
#>   Valid: TRUE 
#>   Rows: 1

If v$valid is FALSE, inspect v$issues and fix file paths or required columns (benchmark_id, prediction_file, reference_file).

5. Run benchmark suite

suite <- runBenchmarkSuite(
  manifest = manifest,
  data_dir = data_dir,
  thresholds = defaultBenchmarkThresholds("balanced"),
  alignment = "truncate"
)

suite$suite_summary
#>   n_trials n_variables overall_pass_rate trial_pass_rate   mean_rmse
#> 1        1           2                 0               0 0.009631669
#>      mean_mae  mean_cor  mean_icc
#> 1 0.007597143 0.8985908 0.8940463
head(suite$metrics)
#>          trial_id   variable   n        rmse         mae          bias
#> 1 trial1_external knee_angle 200 0.010203141 0.008115619 -0.0000878447
#> 2 trial1_external  hip_angle 200 0.009060196 0.007078668 -0.0006012417
#>         cor        r2       icc  loa_width  pass
#> 1 0.8872962 0.7872945 0.8803718 0.04009446 FALSE
#> 2 0.9098854 0.8278914 0.9077207 0.03552596 FALSE

6. Replace placeholders with real external data

  1. Keep the same manifest contract.
  2. Replace the placeholder paths with real downloaded files (.csv, .mot, .sto, .trc).
  3. Re-run validateBenchmarkManifest() and runBenchmarkSuite().
  4. Optionally set report_dir in runBenchmarkSuite() to export reports.

For repository-level data availability and source links, see:

  • publication/data_availability_and_download_guide_ja.md
  • publication/gold_standard_data_search_framework_medrehab_ja.md