Skip to contents

IAS-pDT modelling workflow — 5. Model post-processing

Source: vignettes/workflow_5_model_postprocess.Rmd
workflow_5_model_postprocess.Rmd


Post-processing of fitted models within the IAS-pDT workflow is conducted across multiple steps, leveraging both CPU and GPU computations to optimize performance and address memory constraints.


Step 1: CPU

The Mod_Postprocess_1_CPU() function initiates the post-processing phase for each habitat type, automating the following tasks:

Mod_SLURM_Refit() check for unsuccessful model fits
Mod_Merge_Chains() merge MCMC chains and saves fitted model and coda objects to .qs2 or .RData files
Convergence_Plot_All() visualize convergence of rho, alpha, omega, and beta parameters across all model variants; unnecessary for a single variant, this function compares convergence across models with varying thinning values, both with and without phylogenetic relationships, or GPP knot distances
Convergence_Plot() convergence of rho, alpha, omega, and beta parameters for a selected model, offering a detailed view compared to Convergence_Plot_All()
PlotGelman() visualize Gelman-Rubin-Brooks diagnostics for the selected model
Mod_Summary() extract and save a summary of the model
Mod_Heatmap_Beta() generate heatmaps of the beta parameters
Mod_Heatmap_Omega() generates heatmaps of the omega parameter (residual species associations)
Mod_CV_Fit() prepare input data for spatial-block cross-validation model fitting

Previous attempts to prepare response curve data, predict at new sites, and compute variance partitioning using R on CPUs (UFZ Windows server and LUMI HPC) were hindered by memory limitations. Consequently, these tasks are offloaded to GPU-based computations using Python and TensorFlow. The Mod_Postprocess_1_CPU() function invokes the following sub-functions to generate commands for GPU execution:

RespCurv_PrepData() prepare data for predicting latent factors for response curves
Predict_Maps() prepare data for predicting latent factors at new sampling units
VarPar_Compute() prepare data for computing variance partitioning


Prepare commands for GPU computations

Predicting latent factors:

  • Latent factor predictions for response curves and new sampling units are executed via a TensorFlow script located at inst/crossprod_solve.py.
  • For these tasks, the respective R functions export numerous .qs2 and .feather data files to the TEMP_Pred subdirectory, essential for GPU computations. They also generate execution commands saved as LF_RC_Commands_.txt (for response curves) and LF_NewSites_Commands_.txt (for new sites).






Computing variance partitioning:

  • Variance partitioning computations on GPUs are executed using TensorFlow scripts at inst/VP_geta.py, inst/VP_getf.py, and inst/VP_gemu.py.
  • VarPar_Compute() exports required files to the TEMP_VP subdirectory, including numerous .qs2 and .feather files, and generates execution commands saved as VP_A_Command.txt, VP_F_Command.txt, and VP_mu_Command.txt.



Combining commands for GPU computations

After executing Mod_Postprocess_1_CPU() for all habitat types, the Mod_Prep_TF() function consolidates batch scripts for GPU computations across all habitat types:

  • It aggregates script files containing commands for response curves and latent factor predictions, splitting them into multiple scripts (TF_Chunk_*.txt) for batch processing, and generates a SLURM script (LF_SLURM.slurm) for latent factor predictions.



  • It consolidates variance partitioning command files into a single VP_Commands.txt and prepares a SLURM script (VP_SLURM.slurm) for variance partitioning computations.


Step 2: GPU

Latent factor predictions and variance partitioning are computed on GPUs. Batch jobs can be submitted using the sbatch command:

sbatch datasets/processed/model_fitting/TF_postprocess/VP_SLURM.slurm
sbatch datasets/processed/model_fitting/TF_postprocess/LF_SLURM.slurm

Cross-validated models are fitted by submitting corresponding SLURM commands (in preparation):

source datasets/processed/model_fitting/HabX/Model_Fitting_CV/CV_Bash_Fit.slurm

Step 3: CPU

The Mod_Postprocess_2_CPU() function advances the post-processing pipeline for HMSC models on the CPU, automating the following tasks:

RespCurv_PrepData(),
RespCurv_PlotSR(),
RespCurv_PlotSp(),
RespCurv_PlotSpAll() 		continue processing and visualizing response curves
Predict_Maps() predict habitat suitability across climate scenarios, compute model explanatory power (internal evaluation), and prepare maps for the Shiny app
Mod_Predict_Plot() visualize species and species richness predictions as JPEG images
Mod_Plot_LF() visualize spatial variation in site loadings of HMSC models as JPEG images
VarPar_Compute(),
VarPar_Plot() 		continue processing and visualize variance partitioning
Mod_Eval_Plot() visualize explanatory power (internal model evaluation)
In preparation… initiate post-processing of fitted cross-validated models and prepare GPU commands for latent factor predictions

Step 4: GPU

Predicting latent factors for cross-validated models on GPUs (in preparation).

Step 5: CPU

Evaluating the performance of cross-validated models (in preparation).

Previous articles:
↠1. Overview
↠2. Processing abiotic data
↠3. Processing biotic data
↠4. Model fitting