Homework 4 INF 552, Instructor: Mohammad Reza RajatiDisclaimer: This set of homework applies SMOTE to a seriously imbalanced datasetwith a large number of features and data points. SMOTE is essentially a time consumingmethod. You need to start doing this homework early, so that you have enough time to runSMOTE on the full dataset.1. The LASSO and Boosting for Regression(a) Download the Communities and Crime data1from https://archive.ics.uci.edu/ml/datasets/Communities+and+Crime. Use the first 1495 rows of data asthe training set and the rest as the test set.(b) The data set has missing values. Use a data imputation technique to deal withthe missing values in the data set. The data description mentions some featuresare nonpredictive. Ignore those features.(c) Plot a correlation matrix for the features in the data set.(d) Calculate the Coefficient of Variation CV for each feature, where CV =sm, inwhich s is sample variance and m is sample mean..(e) Pick b√128c features with highest CV , and make scatter plots and box plots forthem. Can you draw conclusions about significance of those features, just by thescatter plots?(f) Fit a linear model using least squares to the training set and report the test error.(g) Fit a ridge regression model on the training set, with λ chosen by cross-validation.Report the test error obtained.(h) Fit a LASSO model on the training set, with λ chosen by cross-validation. Reportthe test error obtained, along with a list of the variables selected by the model.Repeat with standardized2features. Report the test error for both cases andcompare them.(i) Fit a PCR model on the training set, with M (the number of principal components)chosen by cross-validation. Report the test error obtained.(j) In this section, we would like to fit a boosting tree to the data. As in classificationtrees, one can use any type of regression at each node to build a multivariateregression tree. Because the number of variables is large in this problem, onecan use L1-penalized regression at each node. Such a tree is called L1 penalizedgradient boosting tree. You can use XGBoost3to fit the model tree. Determineα (the regularization term) using cross-validation.1Question you may encounter: I tried opening the dataset and download it but the file is not readable.How to download the file? Just change .data to .csv. .2In this data set, features are already normalized.3Some hints on installing XGBoost on Windows: http://www.picnet.com.au/blogs/guido/2016/09/22/xgboost-windows-x64-binaries-for-download/.1Homework 4 INF 552, Instructor: Mohammad Reza Rajati2. Tree-Based Methods(a) Download the APS Failure data from: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks . The dataset contains a training set and a testset. The training set contains 60,000 rows, of which 1,000 belong to the positiveclass and 171 columns, of which one is the class column. All attributes are numeric.(b) Data PreparationThis data set has missing values. When the number of data with missing valuesis significant, discarding them is not a good idea. 4i. Research what types of techniques are usually used for dealing with data withmissing values.5 Pick at least one of them and apply it to this data in thenext steps.6ii. For each of the 170 features, calculate the coefficient of variation CV=sm,where s is sample variance and m is sample mean.iii. Plot a correlation matrix for your features using pandas or any other tool.iv. Pick b√170c features with highest CV , and make scatter plots and box plotsfor them, similar to those on p. 129 of ISLR. Can you draw conclusions aboutsignificance of those features, just by the scatter plots? This does not meanthat代写INF 552作业、代做Java实验作业、代写dataset留学生作业、代做Java编程设计作业 代写Web开发|代 you will only use those features in the following questions. We pickedthem only for visualization.v. Determine the number of positive and negative data. Is this data set imbalanced?(c) Train a random forest to classify the data set. Do NOT compensate for classimbalance in the data set. Calculate the confusion matrix, ROC, AUC, andmisclassification for training and test sets and report them (You may use pROCpackage). Calculate Out of Bag error estimate for your random forset and compareit to the test error.(d) Research how class imbalance is addressed in random forests. Compensate forclass imbalance in your random forest and repeat 2c. Compare the results withthose of 2c.(e) Model TreesIn the case of a univariate tree, only one input dimension is used at a tree split.In a multivariate tree, or model tree, at a decision node all input dimensions canbe used and thus it is more general. In univariate classification trees, majoritypolling is used at each node to determine the split of that node as the decisionrule. In model trees, a (linear) model that relies on all of the variables is used4In reality, wehn we have a model and we want to fill in missing values, we do not have access to trainingdata, so we only use the statistics of test data to fill in the missing values. For simplicity, in this exercise,you first fill in the missing values and then split your data to training and test sets.5They are called data imputation techniques.6You are welcome to test more than one method.2Homework 4 INF 552, Instructor: Mohammad Reza Rajatito determine the split of that node (i.e. instead of using Xj > s as the decisionrule, one has PjβjXj > s. as the decision rule). Alternatively, in a regressiontree, instead of using average in the region associated with each node, a linearregression model is used to determine the value associated with that node.One of the methods that can be used at each node is Logistic Regression. Onecan use scikit learn to call Weka7to train Logistic Model Trees for classification.Train Logistic Model Trees for the APS data set without compensation for classimbalance. Use one of 5 fold, 10 fold, and leave-one-out cross validation methodsto estimate the error of your trained model and compare it with the test error.Report the Confusion Matrix, ROC, and AUC for training and test sets.(f) Use SMOTE (Synthetic Minority Over-sampling Technique) to pre-process yourdata to compensate for class imbalance.8 Train a Logistic Model Tree using thepre-processed data and repeat 2e. Do not forget that there is a right and a wrongway of cross validation here. Compare the uncompensated case with SMOTE.3. ISLR 6.8.34. ISLR, 6.8.55. ISLR 8.4.56. ISLR 9.7.37. Extra Practice: ISLR 5.4.2, 6.8.4, 8.4.4, 9.7.2AppendixWeka for Mac users:1. Download JDK 9 from http://www.oracle.com/technetwork/java/javase/downloads/index.html2. Add environment variables in Terminal using : vi~/.bash_profile(a) export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.4.jdk/Contents/Home(b) export PATH=$JAVA_HOME/bin:$PATH3. Restart Terminal4. Get brew (package installer for Mac, if you don’t have it) and install python (notnecessary)7http://fracpete.github.io/python-weka-wrapper/install.html. may help.8If you did not start doing this homework on time, downsample the common class to 6,000 so that youhave 12,000 data points after applying SMOTE. Remember that the purpose of this homework is to applySMOTE to the whole dataset.3Homework 4 INF 552, Instructor: Mohammad Reza Rajati5. brew install pkg-config6. brew install graphviz7. pip install javabridge8. pip install python-weka-wrapperAnd you should be able to use WEKA in your Jupyter Notebooks.转自：http://ass.3daixie.com/2019030458441000.html