Descriptive statistics and its types

Descriptive statistics involve summarizing and organizing data to be easily understood. It's the first step in data analysis, providing a way to describe the main features of a collection of data quickly.

Two types of descriptive statistics:

1. Central tendency (mean, median and mode) and 

2. variability (variance and std deviation)

1. Measures of Central Tendency

In the realm of statistics, measures of central tendency play a crucial role in summarizing data and understanding its distribution. They provide valuable insights into the "typical" or central value around which data points tend to cluster. Let's delve into the three primary measures of central tendency—mean, median, and mode—and explore how they not only describe data but also aid in handling outliers and missing values in data science and data cleaning processes.

Central tendency - it refers to the measure used to determine the centre of the distribution of the data..

mean - sensitivity to outliers,, it means the value of mean drastically changes with big number outlier...outlier nothing but basically a data point that is different from rest of the datapoints.

remember outlier has an major impact on the entire distribution, because the distribution of the central data shifts... distribution basically means  how your data looks..there are various ways, we use pdfs, bar graphs, histograms etc

Application Examples:

Handling Outliers:

Consider a dataset of salaries where an outlier (CEO's salary) skews the data: $\{3000, 2500, 4000, 3500, 6000, 2800, 3200\}$

• Mean: $\text{Mean} = \frac{23000}{7} \approx 3285.71$
  
• Median: Arrange in ascending order: $\{2500, 2800, 3000, 3200, 3500, 4000, 6000\}$, median = $3200$
  
• Mode: No mode in this dataset.

Data Cleaning: Handling NaN Values:

Consider a dataset of student scores with missing values: $\{85, 92, NaN, 78, 90, NaN, 88\}$

• Mean: $\text{Mean} = \frac{85 + 92 + 78 + 90 + 88}{5} = 86.6$
  • Replace NaN with $86.6$.
• Median: Arrange values: $\{78,85,88,90,92\},\; median\; =$$88$
  
  • Replace NaN with $88$.

2. Measures of Variance or variability

Measures of variability quantify the extent to which data points differ from each other and from the central tendency. Here are the primary measures of variability typically used in statistical analysis:

Measure of dispersion - Variance and Std deviation, also coeff of varaition(learn about this)....dispersion means how well data is spread, from two sets of dataset, lets say we get same mean value but how do i identify these two datasets are different/? at that point of time we use variance (formula shown below for variance with xbar is mean for sample data and mu basically used as mean for population data) and std deviation. 

with the help of variance we can decide how the data is spread and using standard deviation, we can measure how far the element can be...

Also, note in machine learning, the measure of dispersion termed as L1 and L2 norms detailed in below screenshot

Application Example: Handling Outliers and NaN Values in Data Science

You are working with a dataset of daily temperature readings (in Celsius) over a month from multiple weather stations. The dataset contains outliers and missing values (NaN), and you need to use measures of variability to manage these issues effectively.

1. Data Preparation:

   • Collect the dataset of daily temperature readings, noting outliers and missing values (NaN).

   Example Dataset: $\{20, 18, 22, 19, 25, 23, NaN, 21, 24, 17, 16, 30, 18, 20, 26, 22, 19, 27, 28, 15\}$
   

2. Calculating Measures of Variability:

   • Compute the range, variance, standard deviation, and interquartile range (IQR) to understand the dispersion and variability of temperature data.
   

3. Handling Outliers and NaN Values:

   • Outlier Removal: Use IQR to identify and remove outliers:
     • Calculate $Q1$ (25th percentile) and $Q3$ (75th percentile).
     • Identify outliers as values outside $Q1 - 1.5 \times \text{IQR}$ and $Q3 + 1.5 \times \text{IQR}$.
   • Filling NaN Values: Replace NaN values with the mean, median, or mode of the dataset to maintain data integrity.

Conclusion:

Measures of variability—range, variance, standard deviation, and interquartile range (IQR)—are essential in data science for understanding data dispersion, detecting outliers, and managing missing values. By applying these measures effectively, analysts can ensure data quality and reliability, facilitating accurate analysis and decision-making in diverse fields ranging from meteorology to finance and beyond.

3. Understanding Percentile, Quartiles, Five Number Summary, and Box Plot

In data analysis, percentiles, quartiles, the five-number summary, and box plots are essential tools for summarizing and visualizing the distribution of data. These concepts provide valuable insights into the spread, central tendency, and variability of datasets, facilitating effective data exploration and interpretation across various domains.

Application Example

Scenario:

You have a dataset of test scores from a class of students, and you want to analyze the distribution using these concepts.

Example Dataset: Test scores:{65,70,72,75,80,85,90,95,100}

  

Box Plot Visualization

• Draw a box from $Q1$(70) to $Q3$(90).
• Draw whiskers from the box to the minimum (65) and maximum (100) values.
• Display outliers, if any, as individual points outside the whiskers.

Conclusion

Understanding percentiles, quartiles, the five-number summary, and box plots enhances data analysis by providing insights into data distribution, variability, and outlier identification. These tools are fundamental in various fields for summarizing data characteristics and making informed decisions based on data patterns and trends.

Covariance

Understanding Covariance and Correlation: An Example

Covariance and correlation are both measures of how two variables move together. However, they have key differences:

• Covariance quantifies the direction of the linear relationship between variables. Its value can range from negative infinity to positive infinity.
• Correlation standardizes the measure of the relationship by scaling it between -1 and 1, making it easier to interpret. It shows the strength and direction of the linear relationship.

Let's go through an example to understand both concepts and how they differ.

Example: Calculating Covariance and Correlation

Step 1: Gather Data

Consider two datasets representing variables $X$ and $Y$.

$$X = [2, 4, 6, 8, 10]$$ $$Y = [1, 3, 2, 5, 4]$$

Step 2: Calculate the Mean of $X$ and $Y$

$$\text{Mean of } X (\bar{X}) = \frac{2 + 4 + 6 + 8 + 10}{5} = 6$$ $$\text{Mean of } Y (\bar{Y}) = \frac{1 + 3 + 2 + 5 + 4}{5} = 3$$

Step 3: Calculate the Deviations from the Mean

$$\text{Deviations for } X: [2-6, 4-6, 6-6, 8-6, 10-6] = [-4, -2, 0, 2, 4]$$ $$\text{Deviations for } Y: [1-3, 3-3, 2-3, 5-3, 4-3] = [-2, 0, -1, 2, 1]$$

Step 4: Multiply the Deviations and Sum the Products

$$\text{Products of Deviations: } [-4 \cdot (-2), -2 \cdot 0, 0 \cdot (-1), 2 \cdot 2, 4 \cdot 1] = [8, 0, 0, 4, 4]$$ $$\text{Sum of Products: } 8 + 0 + 0 + 4 + 4 = 16$$

Step 5: Calculate the Covariance

$$\text{Covariance (Cov(X, Y))} = \frac{\sum (X_i - \bar{X})(Y_i - \bar{Y})}{N}$$

Where $N$ is the number of data points.

$$\text{Covariance (Cov(X, Y))} = \frac{16}{5} = 3.2$$

Calculating Correlation

Step 6: Calculate the Standard Deviations of $X$ and $Y$

$$\text{Standard Deviation of } X (\sigma_X) = \sqrt{\frac{\sum (X_i - \bar{X})^2}{N}} = \sqrt{\frac{16 + 4 + 0 + 4 + 16}{5}} = \sqrt{8} = 2.83$$ $$\text{Standard Deviation of } Y (\sigma_Y) = \sqrt{\frac{\sum (Y_i - \bar{Y})^2}{N}} = \sqrt{\frac{4 + 0 + 1 + 4 + 1}{5}} = \sqrt{2} = 1.41$$

Step 7: Calculate the Correlation

$$\text{Correlation (Corr(X, Y))} = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}$$ $$\text{Correlation (Corr(X, Y))} = \frac{3.2}{2.83 \cdot 1.41} = \frac{3.2}{3.99} \approx 0.80$$

Interpretation and Comparison

Covariance:

• The covariance value is 3.2.
• This positive value indicates that as $X$ increases, $Y$ also tends to increase.
• The magnitude of covariance depends on the scale of the variables, making it difficult to interpret without context.

Correlation:

• The correlation value is 0.80.
• This value indicates a strong positive linear relationship between $X$ and $Y$.
• The correlation is dimensionless and standardized, making it easier to interpret. It ranges from -1 to 1, where values close to 1 or -1 indicate strong relationships, and values near 0 indicate weak or no linear relationship.

Conclusion

Covariance gives a raw measure of how two variables change together, but its scale can make interpretation challenging. Correlation, on the other hand, standardizes this relationship, providing a clear measure of the strength and direction of the linear relationship, making it more useful for comparing different datasets.