SOCR ≫ DSPA ≫ Topics ≫

1 Assessing Model Performance

Use some of the methods below to do classification, prediction, and model performance evaluation.

Model Learning Task Method Parameters
KNN Classification knn k
Naïve Bayes Classification nb fL, usekernel
Decision Trees Classification C5.0 model, trials, winnow
OneR Rule Learner Classification OneR None
RIPPER Rule Learner Classification JRip NumOpt
Linear Regression Regression lm None
Regression Trees Regression rpart cp
Model Trees Regression M5 pruned, smoothed, rules
Neural Networks Dual use nnet size, decay
Support Vector Machines (Linear Kernel) Dual use svmLinear C
Support Vector Machines (Radial Basis Kernel) Dual use svmRadial C, sigma
Random Forests Dual use rf mtry

\[\textbf{Summary of some machine-learning classification technqiues.}\]

1.1 Model improvement case study

From the course datasets, use the 05_PPMI_top_UPDRS_Integrated_LongFormat1.csv case-study to perform a multi-class prediction. Use ResearchGroup as an outcome response, which includes three classes: “PD”,“Control” and “SWEDD” .

  • Delete the ID column, impute the missing values using feature mean or median (justify your choice)
  • Normalize the covariates
  • Implement automated parameter tuning process and report the optimal accuracy and \(\kappa\)
  • Set arguments and rerun the tuning - try different method and number settings
  • Train a random forest classifier and tune the parameters, report the result and the cross table
  • Use a bagging algorithm and report the accuracy and \(\kappa\)
  • Perform a random Forest classification and report the accuracy and \(\kappa\)
  • Report the accuracy of AdaBoost
  • Finally, give a brief summary about all the model improvement approaches.

Try the procedure on other data in the list of Case-Studies, e.g., Traumatic Brain Injury Study and the corresponding dataset.

SOCR Resource Visitor number Dinov Email