A hash table collision happens when two different keys hash to the same value. Two data cannot be stored in the same slot in array.
To avoid hash table collision there are many techniques, here we list out two:
It uses the data structure to store multiple items that hash to the same slot.
It searches for other slots using a second function and store item in first empty slot that is found
In KNN imputation, the missing attribute values are imputed by using the attributes value that are most similar to the attribute whose values are missing. By using a distance function, the similarity of two attributes is determined.
Map-reduce is a framework to process large data sets, splitting them into subsets, processing each subset on a different server and then blending results obtained on each.
An n-gram is a contiguous sequence of n items from a given sequence of text or speech. It is a type of probabilistic language model for predicting the next item in such a sequence in the form of a (n-1).
A correlogram analysis is the common form of spatial analysis in geography. It consists of a series of estimated autocorrelation coefficients calculated for a different spatial relationship. It can be used to construct a correlogram for distance-based data, when the raw data is expressed as distance rather than values at individual points.
Hierarchical clustering algorithm combines and divides existing groups, creating a hierarchical structure that showcase the order in which groups are divided or merged.
In computing, a hash table is a map of keys to values. It is a data structure used to implement an associative array. It uses a hash function to compute an index into an array of slots, from which desired value can be fetched.
Criteria for a good data model includes:
During imputation we replace missing data with substituted values.
The types of imputation techniques involve are:
Hot-deck imputation: A missing value is imputed from a randomly selected similar record by the help of punch card
Cold deck imputation: It works same as hot deck imputation, but it is more advanced and selects donors from another datasets
Mean imputation: It involves replacing missing value with the mean of that variable for all other cases
Regression imputation: It involves replacing missing value with the predicted values of a variable based on other variables
Stochastic regression: It is same as regression imputation, but it adds the average regression variance to regression imputation
Unlike single imputation, multiple imputation estimates the values multiple times
Clustering is a classification method that is applied to data. Clustering algorithm divides a data set into natural groups or clusters.
Properties for clustering algorithm are:
A data scientist must have the following skills:
Big Data Knowledge
Logistic regression is a statistical method for examining a dataset in which there are one or more independent variables that defines an outcome.
The outlier is a commonly used terms by analysts referred for a value that appears far away and diverges from an overall pattern in a sample.
There are two types of Outliers:
Tools used in Big Data includes:
Statistical methods that are useful for data scientist are:
Time series analysis can be done in two domains, frequency domain and the time domain. In Time series analysis the output of a particular process can be forecast by analyzing the previous data by the help of various methods like exponential smoothening, log-linear regression method, etc.
Various steps in an analytics project include:
Although single imputation is widely used, it does not reflect the uncertainty created by missing data at random. So, multiple imputation is more favorable then single imputation in case of data missing at random.
Some of the common problems faced by data analyst are:
To become a data analyst:
To deal the multi-source problems:
Hadoop and MapReduce is the programming framework developed by Apache for processing large data set for an application in a distributed computing environment.
Some of the best practices for data cleaning includes:
KPI: It stands for Key Performance Indicator, it is a metric that consists of any combination of spreadsheets, reports or charts about business process
Design of experiments: It is the initial process used to split your data, sample and set up of a data for statistical analysis
80/20 rules: It me that 80 percent of your income comes from 20 percent of your clients.
Usually, methods used by data analyst for data validation are:
Collaborative filtering is a simple algorithm to create a recommendation system based on user behavioral data. The most important components of collaborative filtering are users- items- interest.
A good example of collaborative filtering is when you see a statement like “recommended for you” on online shopping sites that’s pops out based on your browsing history.
The difference between data mining and data profiling is that:
Data profiling: It targets on the instance analysis of individual attributes. It gives information on various attributes like value range, discrete value and their frequency, occurrence of null values, data type, length, etc.
Data mining: It focuses on cluster analysis, detection of unusual records, dependencies, sequence discovery, relation holding between several attributes, etc.
K mean is a famous partitioning method. Objects are classified as belonging to one of K groups, k chosen a priori.
In K-mean algorithm:
Data cleaning also referred as data cleing, deals with identifying and removing errors and inconsistencies from data in order to enhance the quality of data.
Responsibility of a Data analyst include:
The missing patterns that are generally observed are: