Data Preprocessing and Data prediction using Scikit learn
Data Preprocessing and Data prediction using Scikit learn
Machine learning is the study of statistical models and algorithms by the computer to perform the task without programing explicitly. You can say, its an application of Artificial intelligence (AI), which focuses on relying patterns and inference to learn from data.
How Machine Learning works:
It works on training the algorithmic models to predict the patterns from the collection of data (Examples: images, speed recognition, system logs, and etc. Mathematically, it is totally based on the mathematical equations to get the result.
Linear regression: Y = a + b.x
Multiple Linear regression: Y = a + b1.x1 + b2.x2 + b3.x3
Polynomial regression: Y = a + b1.x1 + b2.x2^2 + b3.x3^3
- Product recommendations based on customer behavior
- Potentially fraudulent transactions
- Translate languages in text or audio
- Predict weather patterns.
- Enable software to accurately respond to voice commands
- Categorize images (such as MRI studies, photos, or satellite imagery)
Data flow diagram:
Collecting row data=> Manipulate the raw data which can be analysis(data preparation)=>Run the model (ML engine)=>Result ( predict the Analytical data).
Scikit learn python library
: It is an open source python library for data analysis and data mining. It is based on the machine learning library “Numpy“.
Let’s say, we have row data in the data.csv file, which has the salary of employees with years of experience in a company. Now, we need to build a model which predicts the salary for new employees based on the salary and year of experience which company offers.
Now, we need to process this row data which can be used for analytics and Data prediction. For this task, we will use Scikit learn python library.
Steps for Data Preparation/preprocessing:
- Import the library and the data set.
- Dependent variables and Independent variables: Identify which data is dependent, Move to depend matrix (Let’s say x) and which are independent data, move to independent matrix (Let’s say Y).
- Handling Missing data if any: If there is any missing data in any column or multiple column, then we need to handle it explicitly. Let’s say taking MEAN of a column.
- Categorical Data if any: If there is Categorical data present, Let’s say Delhi, Mumbai, pune. Then, we can create categories of them like 1 (for Delhi), 2 (for Mumbai), 3 (for Pune). Here, we need to create a dummy variable too.
- Splitting dataset into training set and Test set: Split the entire data set in training and test sets in ration of 80% and 20%. NOTE: It totally depends on the size of your data set.
- Feature Scaling/Normalization if only required: If the range of independent variables are large, then to normalize the range, we need to use feature scaling.
Before starting with the code, please make sure to set your working directory where you have saved the Data.csv file.
Create a file “Data_preprocessing.py”
#Importing Libraries import numpy as np import matplotlib.pyplot as plt import pandas as pd #Importing Datasets dataset=pd.read_csv('Data.csv') #Creating matrix of dependent and independent variable # independent variables representing with X X=dataset.iloc[:,:-1].values Y=dataset.iloc[:,3] #Taking of missing data with help of SimpleImputer class from sklearn.impute import SimpleImputer as si imputer=si(missing_values=np.NaN, strategy="mean") imputer=imputer.fit(X[:,1:3]) X[:,1:3]=imputer.transform(X[:,1:3]) #Encoding the Categorical data from dataset from sklearn.preprocessing import LabelEncoder, OneHotEncoder labelencoder_X=LabelEncoder() X[:,0]=labelencoder_X.fit_transform(X[:,0]) onehotencoder=OneHotEncoder(categories='auto') X=onehotencoder.fit_transform(X).toarray() labelencoder_Y=LabelEncoder() Y=labelencoder_Y.fit_transform(Y) #Splitting the dataset into training set and test set from sklearn.model_selection import train_test_split X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size=0.2, random_state=0) #Feature scaling from sklearn.preprocessing import StandardScaler scale_X=StandardScaler() X_train=scale_X.fit_transform(X_train) X_test=scale_X.transform(X_test)
Once the data processing gets done, we are ready to analyze the test set and predict.
Data prediction with Simple Learn Regression method
Before starting with the code, please make sure to set your working directory where you have saved the Salary_Data.csv file.
Create a file “simple_linear_regression.py”
# Importing the libraries import numpy as np import matplotlib.pyplot as plt import pandas as pd # Importing the dataset dataset = pd.read_csv('Salary_Data.csv') X = dataset.iloc[:, :-1].values y = dataset.iloc[:, 1].values # Splitting the dataset into the Training set and Test set from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 1/3, random_state = 0) # Feature Scaling """from sklearn.preprocessing import StandardScaler sc_X = StandardScaler() X_train = sc_X.fit_transform(X_train) X_test = sc_X.transform(X_test) sc_y = StandardScaler() y_train = sc_y.fit_transform(y_train)""" # Fitting Simple Linear Regression to the Training set from sklearn.linear_model import LinearRegression regressor = LinearRegression() regressor.fit(X_train, y_train) # Predicting the Test set results y_pred = regressor.predict(X_test) # Visualising the Training set results plt.scatter(X_train, y_train, color = 'red') plt.plot(X_train, regressor.predict(X_train), color = 'blue') plt.title('Salary vs Experience (Training set)') plt.xlabel('Years of Experience') plt.ylabel('Salary') plt.show() # Visualising the Test set results plt.scatter(X_test, y_test, color = 'red') plt.plot(X_train, regressor.predict(X_train), color = 'blue') plt.title('Salary vs Experience (Test set)') plt.xlabel('Years of Experience') plt.ylabel('Salary') plt.show()
So, we have successfully built a model, which can predict the salary of new employees which company should offer based on the result of the statistical test set..