Task1_DataPreprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Task 1: Data preprocessing\n",
    "#### Data Analytics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The dataset named *task1_data* (*task1_data.csv*) has around 130K samples and 65 features.\n",
    "\n",
    "The main objective of the task is to preprocess the train data, designing a complete preprocessing scheme, and test it on test data. \n",
    "\n",
    "You must take into account that this is not a toy dataset, and its size could be relevant.\n",
    "\n",
    "The function \"automatic_scoring\" provides a way for comparing different schemes using a classifier, by means of 10-Fold CV and using AUC as metric. You will need to put the right seed as requested. Notice that the function just needs inputs (X) and target (y) arrays as input. NOTE: AUC takes values in [0, 1], being the higher the better.\n",
    "\n",
    "If you try anytime several options it is important to show the results of those discarded trials, because what is not visible cannot be evaluated.\n",
    "\n",
    "The function \"automatic_testing\" trains the model on the train data and applies it to the test data. Do not change the classification algorithm, its parameters and the scoring choice. Those are fixed and their optimization (model selection) is out of the scope of this task.\n",
    "\n",
    "The deliverable of this task is this Jupyter Notebook containing the code, plus some short answers in markdown cells if required. All the cells in the notebook should be run. You should also upload the downloaded html and pdf formats (*File > Export > Files or selection to HTML... (.html)*)\n",
    "\n",
    "NOTE: Keep in mind that some functions accept both Pandas dataframes and Numpy arrays, but some others only one of them. Nevertheless, we should know how to pass from one to the other and viceversa.\n",
    "\n",
    "NOTE: Keep in mind that some functions will take some time to run. You can continue working on other cells during the run to avoid wasting time waiting.\n",
    "\n",
    "NOTE: If you work in pairs, please add the name of your partner in brackets besides yours in the *Name & Surnames* field.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Exercises:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "* (i) Split the data into 4 parts, i.e. train inputs and target (xtr, ytr) and test inputs and target (xte and yte), in such a way that the proportion of the classes is kept constant in train and test parts. The size of the training set must be double of the size of the test data. ***The random seed must be kept during all the task in any possible place***. [5%] <br><br><br>\n",
    "\n",
    "* (ii) Checking for missing values and outliers. If any, considering (a), (b) and (c) below, treat the data however you consider better, arguing your decisions. [20%] <br>\n",
    "    - (a) Is there any missing value? If so, regarding the characteristics of the data, decide what to do arguing your answer. Modify your data according to your answer if necessary.  <br>\n",
    "    - (b) Is there any collective outlier? If so, regarding the characteristics of the data, decide what to do arguing your answer.  Modify your data according to your answer if necessary. <br>\n",
    "    - (c) From now on, this is your basic data. Therefore, it is save to overwrite the names of the data parts.\n",
    "<br><br>\n",
    "\n",
    "* (iii) The feature selection algorithm *SelectPercentile* (sklearn.feature_selection.SelectPercentile) uses different scores (f_classif, mutual_info_classif, chi2, f_regression, etc) in order to select the most relevant features. In the Scikit-Learn documentation\n",
    "(https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectPercentile.html#sklearn.feature_selection.SelectPercentile)\n",
    "you have the function info and an example of use of *chi2* score. Use the feature selection method SelectPercentile with the *mutual_info_classif* score, and *percentile* parameter 20. [30%] <br>\n",
    "    - (a) Which is the compression ratio you obtained? (Note: Compression ratio is the proportion of the original variables kept after the selection). <br>\n",
    "    - (b) Compare the performance (with *automatic_scoring*) with and without feature selection with the right scheme and function. Is selecting those variables a good idea? Argue your response. <br>\n",
    "    - (c) Regarding the answer to (b), get your current data in order to continue preprocessing.\n",
    "<br><br>\n",
    "\n",
    "* (iv) Apply principal component analysis to your data for compression, capturing at least 95% of the cumulative variance. How many extracted variables do you have? Which reduction percentage would you get if you apply it? Compare the performance with the one of your current non-compressed data. Would you use the pca compression here? Act consequently with your answer, and keep the data overwriting the names. [15%] <br> \n",
    "<br>\n",
    "\n",
    "* (v) Check the balance of your current dataset. Which is its imbalance ratio? Discuss if it makes sense to apply imbalanced data treatments or not, considering the size of the data and the performance you have obtained for the data you currently have after (iv). Act in consequence, with total freedom on the sampling method to use if decide you need any.\n",
    "\n",
    "__Hint__:  We can understand the imbalance ratio both as the number of times the majority class is bigger than the minority class, or the proportion of the samples that are from minority class. If imbalance ratio is 49 to 1, then it is equivalent to having 2% of minority class samples. [20%] <br>\n",
    "<br>\n",
    "\n",
    "* (vi) Once you are here, you have final preprocessed data using the definitive preprocessing scheme you have reasonably chosen. Check now the performance using the test data (with *automatic_testing*). Comment on the result you have obtained for test data compared to the one for train data in (v). [10%]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Auxiliar functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "\n",
    "seed = 44837\n",
    "\n",
    "\n",
    "def automatic_scoring(x, y):\n",
    "    average_score = cross_val_score(estimator=RandomForestClassifier(n_estimators=100, random_state=seed), X=x, y=y, cv=5, scoring='roc_auc').mean()\n",
    "    return average_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "\n",
    "def automatic_testing(xtr, ytr, xte, yte):\n",
    "    auc_score = roc_auc_score(yte, RandomForestClassifier(n_estimators=100, random_state=seed).fit(xtr, ytr).predict_proba(xte)[:,1])\n",
    "    return auc_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Solution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>43632.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.009305</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.096014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  Y\n",
       "count  43632.000000\n",
       "mean       0.009305\n",
       "std        0.096014\n",
       "min        0.000000\n",
       "25%        0.000000\n",
       "50%        0.000000\n",
       "75%        0.000000\n",
       "max        1.000000"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# (i)\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import roc_auc_score\n",
    "import pandas as pd\n",
    "\n",
    "# Read data\n",
    "init_data = pd.read_csv('task1_data.csv')\n",
    "init_data.describe()\n",
    "\n",
    "# Split x and y\n",
    "x = init_data.values[:,:-1]\n",
    "y = init_data.values[:,-1]\n",
    "\n",
    "# 2/3 train, 1/3 test\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=1/3, random_state=seed)\n",
    "\n",
    "# Convert to data frame\n",
    "columnas = ['X1','X2','X3','X4','X5','X6','X7','X8','X9','X10','X11','X12','X13','X14','X15','X16','X17','X18','X19','X20',\n",
    "            'X21','X22','X23','X24','X25','X26','X27','X28','X29','X30','X31','X32','X33','X34','X35','X36','X37','X38','X39','X40',\n",
    "            'X41','X42','X43','X44','X45','X46','X47','X48','X49','X50','X51','X52','X53','X54','X55','X56','X57','X58','X59','X60',\n",
    "            'X61','X62','X63','X64','X65']\n",
    "\n",
    "x_train_df = pd.DataFrame(x_train, columns=columnas)\n",
    "y_train_df = pd.DataFrame(y_train, columns=['Y'])\n",
    "x_test_df = pd.DataFrame(x_test, columns=columnas)\n",
    "y_test_df = pd.DataFrame(y_test, columns=['Y'])\n",
    "\n",
    "y_test_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X1</th>\n",
       "      <th>X2</th>\n",
       "      <th>X3</th>\n",
       "      <th>X4</th>\n",
       "      <th>X5</th>\n",
       "      <th>X6</th>\n",
       "      <th>X7</th>\n",
       "      <th>X8</th>\n",
       "      <th>X9</th>\n",
       "      <th>X10</th>\n",
       "      <th>...</th>\n",
       "      <th>X56</th>\n",
       "      <th>X57</th>\n",
       "      <th>X58</th>\n",
       "      <th>X59</th>\n",
       "      <th>X60</th>\n",
       "      <th>X61</th>\n",
       "      <th>X62</th>\n",
       "      <th>X63</th>\n",
       "      <th>X64</th>\n",
       "      <th>X65</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "      <td>87263.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>61.176479</td>\n",
       "      <td>26.493558</td>\n",
       "      <td>0.187255</td>\n",
       "      <td>1.814486</td>\n",
       "      <td>18.264829</td>\n",
       "      <td>0.000790</td>\n",
       "      <td>0.208647</td>\n",
       "      <td>1.112092</td>\n",
       "      <td>-73.837701</td>\n",
       "      <td>768.048879</td>\n",
       "      <td>...</td>\n",
       "      <td>0.338574</td>\n",
       "      <td>2.937087</td>\n",
       "      <td>1823.106158</td>\n",
       "      <td>0.028960</td>\n",
       "      <td>0.540669</td>\n",
       "      <td>0.319887</td>\n",
       "      <td>-64.810811</td>\n",
       "      <td>472.465190</td>\n",
       "      <td>0.475946</td>\n",
       "      <td>0.260109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>19.032381</td>\n",
       "      <td>4.529705</td>\n",
       "      <td>1.275754</td>\n",
       "      <td>32.549068</td>\n",
       "      <td>81.313910</td>\n",
       "      <td>1.028372</td>\n",
       "      <td>1.481605</td>\n",
       "      <td>21.432100</td>\n",
       "      <td>27.254139</td>\n",
       "      <td>525.730088</td>\n",
       "      <td>...</td>\n",
       "      <td>17.965777</td>\n",
       "      <td>60.906770</td>\n",
       "      <td>1698.770121</td>\n",
       "      <td>1.181238</td>\n",
       "      <td>1.667215</td>\n",
       "      <td>9.728382</td>\n",
       "      <td>36.899961</td>\n",
       "      <td>406.227381</td>\n",
       "      <td>1.068878</td>\n",
       "      <td>0.187573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2.680000</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>-3.770000</td>\n",
       "      <td>-125.500000</td>\n",
       "      <td>-1082.000000</td>\n",
       "      <td>-6.120000</td>\n",
       "      <td>-2.860000</td>\n",
       "      <td>-83.000000</td>\n",
       "      <td>-1082.000000</td>\n",
       "      <td>-716.300000</td>\n",
       "      <td>...</td>\n",
       "      <td>-133.000000</td>\n",
       "      <td>-319.000000</td>\n",
       "      <td>-668.000000</td>\n",
       "      <td>-7.780000</td>\n",
       "      <td>-10.000000</td>\n",
       "      <td>-63.000000</td>\n",
       "      <td>-322.000000</td>\n",
       "      <td>-509.200000</td>\n",
       "      <td>-10.000000</td>\n",
       "      <td>-0.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>47.890000</td>\n",
       "      <td>23.570000</td>\n",
       "      <td>-0.570000</td>\n",
       "      <td>-17.500000</td>\n",
       "      <td>-13.000000</td>\n",
       "      <td>-0.650000</td>\n",
       "      <td>-0.600000</td>\n",
       "      <td>-9.500000</td>\n",
       "      <td>-86.000000</td>\n",
       "      <td>389.500000</td>\n",
       "      <td>...</td>\n",
       "      <td>-10.000000</td>\n",
       "      <td>-24.000000</td>\n",
       "      <td>863.400000</td>\n",
       "      <td>-0.710000</td>\n",
       "      <td>-0.540000</td>\n",
       "      <td>-5.000000</td>\n",
       "      <td>-82.000000</td>\n",
       "      <td>174.900000</td>\n",
       "      <td>-0.160000</td>\n",
       "      <td>0.140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>62.400000</td>\n",
       "      <td>25.780000</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>11.500000</td>\n",
       "      <td>0.040000</td>\n",
       "      <td>0.030000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>-69.500000</td>\n",
       "      <td>644.200000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1436.700000</td>\n",
       "      <td>0.080000</td>\n",
       "      <td>0.400000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-55.000000</td>\n",
       "      <td>378.000000</td>\n",
       "      <td>0.560000</td>\n",
       "      <td>0.260000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>75.340000</td>\n",
       "      <td>28.570000</td>\n",
       "      <td>0.820000</td>\n",
       "      <td>19.500000</td>\n",
       "      <td>40.000000</td>\n",
       "      <td>0.700000</td>\n",
       "      <td>0.770000</td>\n",
       "      <td>10.500000</td>\n",
       "      <td>-56.500000</td>\n",
       "      <td>1019.700000</td>\n",
       "      <td>...</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>2335.050000</td>\n",
       "      <td>0.820000</td>\n",
       "      <td>1.460000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>-38.000000</td>\n",
       "      <td>669.000000</td>\n",
       "      <td>1.210000</td>\n",
       "      <td>0.390000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>100.000000</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>50.380000</td>\n",
       "      <td>1059.500000</td>\n",
       "      <td>3355.000000</td>\n",
       "      <td>5.990000</td>\n",
       "      <td>72.280000</td>\n",
       "      <td>973.500000</td>\n",
       "      <td>-23.000000</td>\n",
       "      <td>4423.900000</td>\n",
       "      <td>...</td>\n",
       "      <td>171.000000</td>\n",
       "      <td>2502.000000</td>\n",
       "      <td>64129.400000</td>\n",
       "      <td>5.570000</td>\n",
       "      <td>18.850000</td>\n",
       "      <td>146.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4197.900000</td>\n",
       "      <td>6.600000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 65 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 X1            X2            X3            X4            X5  \\\n",
       "count  87263.000000  87263.000000  87263.000000  87263.000000  87263.000000   \n",
       "mean      61.176479     26.493558      0.187255      1.814486     18.264829   \n",
       "std       19.032381      4.529705      1.275754     32.549068     81.313910   \n",
       "min        2.680000     12.000000     -3.770000   -125.500000  -1082.000000   \n",
       "25%       47.890000     23.570000     -0.570000    -17.500000    -13.000000   \n",
       "50%       62.400000     25.780000      0.100000      1.000000     11.500000   \n",
       "75%       75.340000     28.570000      0.820000     19.500000     40.000000   \n",
       "max      100.000000    100.000000     50.380000   1059.500000   3355.000000   \n",
       "\n",
       "                 X6            X7            X8            X9           X10  \\\n",
       "count  87263.000000  87263.000000  87263.000000  87263.000000  87263.000000   \n",
       "mean       0.000790      0.208647      1.112092    -73.837701    768.048879   \n",
       "std        1.028372      1.481605     21.432100     27.254139    525.730088   \n",
       "min       -6.120000     -2.860000    -83.000000  -1082.000000   -716.300000   \n",
       "25%       -0.650000     -0.600000     -9.500000    -86.000000    389.500000   \n",
       "50%        0.040000      0.030000      0.500000    -69.500000    644.200000   \n",
       "75%        0.700000      0.770000     10.500000    -56.500000   1019.700000   \n",
       "max        5.990000     72.280000    973.500000    -23.000000   4423.900000   \n",
       "\n",
       "       ...           X56           X57           X58           X59  \\\n",
       "count  ...  87263.000000  87263.000000  87263.000000  87263.000000   \n",
       "mean   ...      0.338574      2.937087   1823.106158      0.028960   \n",
       "std    ...     17.965777     60.906770   1698.770121      1.181238   \n",
       "min    ...   -133.000000   -319.000000   -668.000000     -7.780000   \n",
       "25%    ...    -10.000000    -24.000000    863.400000     -0.710000   \n",
       "50%    ...      0.000000      0.000000   1436.700000      0.080000   \n",
       "75%    ...     10.000000     27.000000   2335.050000      0.820000   \n",
       "max    ...    171.000000   2502.000000  64129.400000      5.570000   \n",
       "\n",
       "                X60           X61           X62           X63           X64  \\\n",
       "count  87263.000000  87263.000000  87263.000000  87263.000000  87263.000000   \n",
       "mean       0.540669      0.319887    -64.810811    472.465190      0.475946   \n",
       "std        1.667215      9.728382     36.899961    406.227381      1.068878   \n",
       "min      -10.000000    -63.000000   -322.000000   -509.200000    -10.000000   \n",
       "25%       -0.540000     -5.000000    -82.000000    174.900000     -0.160000   \n",
       "50%        0.400000      0.000000    -55.000000    378.000000      0.560000   \n",
       "75%        1.460000      5.000000    -38.000000    669.000000      1.210000   \n",
       "max       18.850000    146.000000      0.000000   4197.900000      6.600000   \n",
       "\n",
       "                X65  \n",
       "count  87263.000000  \n",
       "mean       0.260109  \n",
       "std        0.187573  \n",
       "min       -0.550000  \n",
       "25%        0.140000  \n",
       "50%        0.260000  \n",
       "75%        0.390000  \n",
       "max        1.000000  \n",
       "\n",
       "[8 rows x 65 columns]"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "# (ii)\n",
    "# a) There are many columns with only 1 NaN maximum, I will replace it with the mean because it is difficult that 1 row could alter the result.\n",
    "x_train_df.describe()\n",
    "x_train_df.isna().sum().sum()\n",
    "\n",
    "imp = SimpleImputer(strategy='mean')\n",
    "x_train_df_nan = pd.DataFrame(data=imp.fit_transform(x_train_df),columns=columnas)\n",
    "y_train_df_nan = y_train_df\n",
    "x_train_df_nan.describe()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outlier positions:  8727\n",
      "Outlier positions:  8727\n"
     ]
    }
   ],
   "source": [
    "# b) Using the EllipticEnvelope class to detect the positions of the outliers and delete them\n",
    "\n",
    "from sklearn.covariance import EllipticEnvelope\n",
    "elip_env = EllipticEnvelope().fit(x_train_df_nan)\n",
    "detection = elip_env.predict(x_train_df_nan)\n",
    "outlier_positions_mah = [x for x in range(x_train_df_nan.shape[0]) if detection[x] == -1]\n",
    "if detection is []:\n",
    "    print(\"There aren't outliers.\")\n",
    "else:\n",
    "    print('Outlier positions: ', len(outlier_positions_mah))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>78536.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.004826</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.069301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  Y\n",
       "count  78536.000000\n",
       "mean       0.004826\n",
       "std        0.069301\n",
       "min        0.000000\n",
       "25%        0.000000\n",
       "50%        0.000000\n",
       "75%        0.000000\n",
       "max        1.000000"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>78536.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.004826</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.069301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  Y\n",
       "count  78536.000000\n",
       "mean       0.004826\n",
       "std        0.069301\n",
       "min        0.000000\n",
       "25%        0.000000\n",
       "50%        0.000000\n",
       "75%        0.000000\n",
       "max        1.000000"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# c) New datasets x_train_free_data, and y_train_free_data\n",
    "\n",
    "x_train_free_data = x_train_df_nan\n",
    "y_train_free_data = y_train_df_nan\n",
    "\n",
    "x_train_free_data.drop(x_train_free_data.index[outlier_positions_mah], inplace=True)\n",
    "y_train_free_data.drop(y_train_free_data.index[outlier_positions_mah], inplace=True)\n",
    "\n",
    "x_train_free_data.describe() # 78536\n",
    "y_train_free_data.describe() # 78536"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# (iii)\n",
    "from sklearn.feature_selection import SelectPercentile, mutual_info_classif\n",
    "\n",
    "x_train_free_data_ndarray = x_train_free_data.to_numpy()\n",
    "y_train_free_data_ndarray = y_train_free_data.to_numpy()\n",
    "\n",
    "per = SelectPercentile(mutual_info_classif, percentile=20)\n",
    "x_train_new = per.fit_transform(x_train_free_data_ndarray, y_train_free_data_ndarray)\n",
    "x_train_new.shape\n",
    "y_train_new = y_train_free_data_ndarray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of columns before:  65\n",
      "Number of columns after:  13\n",
      "0.2\n",
      "Compression ratio:  20.0 %\n"
     ]
    }
   ],
   "source": [
    "# a) Compression ratio\n",
    "ratio = x_train_new.shape[1]/x_train_free_data_ndarray.shape[1]\n",
    "print('Number of columns before: ', x_train_free_data_ndarray.shape[1]) # 65\n",
    "print('Number of columns after: ', x_train_new.shape[1]) # 13\n",
    "print(ratio)\n",
    "print('Compression ratio: ', (x_train_new.shape[1]/x_train_free_data_ndarray.shape[1])*100, '%') # 20% compression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before compression:  0.95280259820858\n"
     ]
    }
   ],
   "source": [
    "# b) Performance comparation using automatic_scoring method\n",
    "\n",
    "y_train_free_data_ndarray_flatten = y_train_free_data_ndarray.ravel()\n",
    "print('Before compression: ', automatic_scoring(x_train_free_data_ndarray, y_train_free_data_ndarray_flatten)) # 0.9516467981849747"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After compression:  0.9241574332891569\n"
     ]
    }
   ],
   "source": [
    "y_train_new = y_train_free_data_ndarray\n",
    "y_train_new_flatten = y_train_new.ravel()\n",
    "print('After compression: ', automatic_scoring(x_train_new, y_train_new_flatten)) # 0.93101975560227\n",
    "\n",
    "# Better without compression, but it is not a huge difference, I think it is better with fewer variables (here from 65 to 13)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There have been selected 6 principal components.\n",
      "After PCA:  0.6875731402686054\n"
     ]
    }
   ],
   "source": [
    "# (iv) Applying Principal Component Analysis\n",
    "\n",
    "from sklearn.decomposition import PCA\n",
    "pca = PCA(n_components=0.95)\n",
    "\n",
    "pca.fit(x_train_free_data_ndarray)\n",
    "X_reduced_raw = pca.transform(x_train_free_data_ndarray)\n",
    "raw_pca_data = pd.DataFrame(data=X_reduced_raw)\n",
    "\n",
    "print(\"There have been selected \" + str(X_reduced_raw.shape[1]) + \" principal components.\") # 6 PCA\n",
    "\n",
    "print('After PCA: ',automatic_scoring(X_reduced_raw,y_train_new_flatten)) # 0.7146784271941337\n",
    "\n",
    "# Better without PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Zeros:  78157 \n",
      "Ones:  379\n",
      "Minority percentage:  0.4825812366303351 %\n",
      "Majority percentage:  99.51741876336968 %\n"
     ]
    }
   ],
   "source": [
    "# (v) Balance\n",
    "\n",
    "from collections import Counter  # For checking the imbalance ratio\n",
    "\n",
    "RANDOM_STATE = seed\n",
    "\n",
    "count = Counter(y_train_new_flatten)\n",
    "print('Zeros: ', count[0.0], '\\nOnes: ', count[1.0]) # 78157, 379\n",
    "print('Minority percentage: ', (count[1.0]/(count[0.0]+count[1.0]))*100, '%') # 0.4825812366303351\n",
    "print('Majority percentage: ', (count[0.0]/(count[1.0]+count[0.0]))*100, '%') # 99.51741876336968\n",
    "# There is a big imbalance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# (vi) Performance check\n",
    "\n",
    "x_test_nan = pd.DataFrame(data=imp.transform(x_test_df),columns=columnas)\n",
    "x_test_nan.describe()\n",
    "\n",
    "# We had some problems with the size of the data, so we didn't remove the outliers nor the percentile to reduce the data of the test data set\n",
    "\n",
    "'''\n",
    "test_detection = elip_env.predict(x_test_nan)\n",
    "outlier_positions_mah_test = [x for x in range(x_test_nan.shape[0]) if test_detection[x] == -1]\n",
    "if test_detection is []:\n",
    "    print(\"No hay outliers.\")\n",
    "else:\n",
    "    print('Posiciones outliers: ', len(outlier_positions_mah_test))\n",
    "'''\n",
    "\n",
    "x_test_free_data = x_test_nan\n",
    "y_test_free_data = y_test_df\n",
    "\n",
    "#x_test_free_data.describe()\n",
    "#y_test_free_data.describe()\n",
    "\n",
    "#x_test_free_data.drop(x_test_free_data.index[outlier_positions_mah_test], inplace=True, index = 0)\n",
    "#y_test_free_data.drop(y_test_free_data.index[outlier_positions_mah_test], inplace=True, index = 0)\n",
    "\n",
    "x_test_free_data_ndarray = x_test_free_data.to_numpy()\n",
    "y_test_free_data_ndarray = y_test_free_data.to_numpy()\n",
    "\n",
    "print(automatic_testing(x_train_free_data,y_train_free_data_ndarray_flatten,x_test_free_data_ndarray,y_test_free_data_ndarray))\n",
    "# 0.9736808591526858"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}