Integrating Modalities: Multi-Modal Fusion with PhysioCrossModal

The integration problem

Movement is measured in many modalities — joint kinematics, kinetics (forces), EMG, IMU — and the scientifically interesting question is usually not any one of them but what they share. PhysioCrossModal provides a family of multi-modal fusion methods that take several time-aligned modality blocks and answer complementary questions:

• what is shared across all modalities vs specific to each? — jive(), multipleFactorAnalysis()
• which directions co-vary between two modalities? — cca(), plsBlocks()
• is there a common latent drive? — coupledNMF()
• how related are the modalities as a whole? — rvCoefficient(), distanceCorrelation(), representationalSimilarity()
• one composite index / design matrix / latent model — multimodalGaitIndex(), fuseBlocks(), crossBlockFactor()

Every method takes a list of matrices that share the observation dimension (rows = the same subjects, or gait-cycle points, or time samples), with each modality contributing its own columns (features).

library(PhysioCrossModal)

# two aligned modality blocks: rows = the same 200 observations (e.g. subjects),
# columns = each modality's features
kinematics <- matrix(rnorm(200 * 12), 200, 12)   # e.g. joint-angle features
kinetics   <- matrix(rnorm(200 *  8), 200,  8)   # e.g. ground-reaction-force features

Shared vs modality-specific: JIVE

JIVE (Joint and Individual Variation Explained) splits each block into a low-rank joint structure common to all modalities, low-rank individual structure specific to each, and residual. The joint scores are the integrated representation; the variance table tells you how much of each modality is shared.

fit <- jive(list(kinematics = kinematics, kinetics = kinetics),
            rank_joint = 2, rank_individual = 2)

fit$variance          # joint / individual / residual fraction per modality
fit$joint_scores      # the shared component scores (integrated representation)

# relate the shared component to an outcome (e.g. walking speed):
# cor(fit$joint_scores[, 1], speed)

Multiple Factor Analysis is a block-balanced global PCA — each block is scaled so a large or high-variance modality cannot dominate the shared factors.

mfa <- multipleFactorAnalysis(list(kinematics, kinetics), ncomp = 3)
mfa$explained         # variance per global factor
mfa$contributions     # each block's contribution to each factor

Covarying directions: CCA and PLS

Canonical correlation finds the linear combinations of two modalities whose scores are maximally correlated (auto-regularised when features exceed observations); two-block PLS maximises covariance instead, and is robust in high dimensions.

cc <- cca(kinematics, kinetics)
cc$cor                # canonical correlations (decreasing)
cc$xscores; cc$yscores

pl <- plsBlocks(kinematics, kinetics, ncomp = 2)
pl$correlation        # score correlation per component

A common latent drive: coupled NMF

When two non-negative modalities share a latent drive (e.g. EMG envelopes and a kinematic-speed envelope driven by the same temporal command), coupledNMF() estimates one shared factor W with modality-specific loadings.

emg    <- abs(matrix(rnorm(200 * 6), 200, 6))   # non-negative
imu    <- abs(matrix(rnorm(200 * 4), 200, 4))
cn <- coupledNMF(emg, imu, rank = 2)
cn$shared             # W: the common drive
cn$vaf_x; cn$vaf_y    # fit per modality

Whole-block association

rvCoefficient() is a multivariate correlation between two blocks; distanceCorrelation() also captures non-linear dependence (zero iff independent); representationalSimilarity() compares each modality’s geometry of observations (RSA). All offer permutation p-values.

rvCoefficient(kinematics, kinetics, permutations = 1000)$rv
distanceCorrelation(kinematics, kinetics, permutations = 1000)$dcor
representationalSimilarity(list(kin = kinematics, kinet = kinetics))$similarity

A low RV with a high canonical correlation means the coupling is real but low-dimensional — read several lenses together.

One index, one matrix, one latent model

A composite multi-modal gait index (Gait-Deviation-Index style, but spanning modalities) scores how far a subject’s multi-modal feature vector lies from a normative reference, with a per-modality breakdown:

reference <- matrix(rnorm(300 * 20), 300, 20)     # controls
subjects  <- matrix(rnorm(10 * 20), 10, 20)
blocks    <- list(kinematics = 1:8, kinetics = 9:14, emg = 15:20)
idx <- multimodalGaitIndex(subjects, reference, blocks = blocks)
idx$index          # ~100 = typical; lower = more deviant
idx$by_modality    # which modality drives each subject's deviation

fuseBlocks() produces a single block-balanced design matrix for downstream multi-view machine learning, and crossBlockFactor() fits a common-factor model that flags which latent factors are shared across modalities vs modality-specific.

fused <- fuseBlocks(list(kinematics, kinetics), method = "mfa")$matrix
cf <- crossBlockFactor(list(kinematics, kinetics), nfactors = 2)
cf$shared          # TRUE for factors that load on >= 2 modalities

Choosing a method

Correctness

These methods are cross-validated against independent, established implementations — cca() vs base cancor, distanceCorrelation() vs energy::dcor, rvCoefficient() vs FactoMineR::coeffRV, multipleFactorAnalysis() vs FactoMineR::MFA, representationalSimilarity() vs vegan::mantel, crossBlockFactor() vs stats::factanal, and jive() against its exact algebraic identity — agreeing to machine precision, in addition to the packaged unit tests.

See also

• vignette("coupling-analysis", package = "PhysioCrossModal") — pairwise cross-modal coupling (coherence, phase, PAC, Granger).
• The MultiPhysioExperiment container for storing and aligning modalities.