What is Data Cleaning?

Data cleaning is the process of fixing or removing incorrect, corrupted, incorrectly formatted, duplicate, or incomplete data within a dataset. When combining multiple data sources, there are many opportunities for data to be duplicated or mislabeled.

https://cdn.technologyadvice.com/wp-content/uploads/2022/06/Data-Cleaning-scaled.jpeg

Data cleaning is a crucial step in the data science process, as the quality of your data directly influences the accuracy and reliability of your analysis and models.

Here are some common tasks involved in data cleaning:

1. Handling Missing Values:
- Identify and analyze missing data in the dataset.
- Decide on an appropriate strategy for handling missing values, such as imputation, deletion, or interpolation.

2. Dealing with Duplicates:
- Identify and remove duplicate rows or records from the dataset.
- Ensure that data uniqueness is maintained where necessary.

3. Correcting Inconsistent Data:
- Standardize formats for categorical variables (e.g., converting ‘Male’ and ‘M’ to a consistent format).
- Correct inaccuracies or inconsistencies in data entry.

4. Handling Outliers:
- Identify and analyze outliers that may adversely affect analysis or modeling.
- Decide whether to remove, transform, or impute outliers based on the context of the data.

5. Data Type Conversion:
- Ensure that data types are appropriate for analysis. For example, convert string representations of numbers to actual numeric types.

6. Addressing Typos and Inconsistencies:
- Look for and correct typos, inconsistent capitalization, or naming conventions in categorical variables.

7. Normalizing and Scaling:
- Normalize or scale numerical variables if needed, especially when using algorithms sensitive to the scale of variables (e.g., distance-based algorithms).

8. Handling Incomplete Data:
- Understand the nature of incomplete data and decide on appropriate methods for dealing with it (e.g., handling time series data with missing timestamps).

9. Data Transformation:
- Apply transformations to variables if required (e.g., log transformations for skewed distributions).

10. Checking and Ensuring Consistency:
- Validate relationships between variables to ensure consistency.
- Check for logical inconsistencies within the data.

11. Quality Assurance:
- Implement checks to ensure data quality throughout the cleaning process.
- Document all the changes made during data cleaning.

12. Exploratory Data Analysis (EDA):
- Perform exploratory data analysis to gain insights into the data distribution, relationships, and potential patterns.

13. Versioning and Logging:
- Keep track of different versions of the dataset during the cleaning process.
- Maintain a log of all the cleaning operations performed on the data.

14. Handling Categorical Data:
- Convert categorical variables into numerical representations using techniques like one-hot encoding or label encoding.

15. Dealing with Skewed Data:
- Address skewed distributions in variables, especially for machine learning models that may be sensitive to imbalances.

16. Renaming Meaningless Column Names:
- Identify columns with unclear or ambiguous names that do not convey the intended information.
- Rename columns to be more descriptive and indicative of the data they represent.
- Use clear, concise, and consistent naming conventions for better understanding and maintainability.

https://d3caycb064h6u1.cloudfront.net/wp-content/uploads/2021/06/datacleaning-2048x1399-1.jpg

Data cleaning is an iterative process, and it’s essential to continually assess the impact of cleaning operations on the data and the downstream analyses. Each dataset may have unique challenges, and the cleaning process should be tailored to the specific characteristics and goals of the analysis.

import pandas as pd

def clean_data(data):
    # Handling Missing Values
    data.fillna(data.mean(), inplace=True)  # Example: Impute missing values with the mean
    
    # Dealing with Duplicates
    data.drop_duplicates(inplace=True)
    
    # Correcting Inconsistent Data
    data['column_name'] = data['column_name'].apply(lambda x: x.lower())  # Example: Convert text to lowercase
    
    # Handling Outliers
    data['numeric_column'] = data['numeric_column'].clip(lower=data['numeric_column'].quantile(0.05),
                                                      upper=data['numeric_column'].quantile(0.95))
    
    # Data Type Conversion
    data['numeric_column'] = pd.to_numeric(data['numeric_column'], errors='coerce')  # Example: Convert to numeric
    
    # Addressing Typos and Inconsistencies
    data['category_column'] = data['category_column'].replace({'incorrect_value': 'correct_value'})
    
    # Normalizing and Scaling
    data['numeric_column'] = (data['numeric_column'] - data['numeric_column'].mean()) / data['numeric_column'].std()
    
    # Handling Incomplete Data
    data['time_series_column'].fillna(method='ffill', inplace=True)  # Example: Forward fill for time series data
    
    # Data Transformation
    data['skewed_column'] = data['skewed_column'].apply(lambda x: x**0.5)  # Example: Square root transformation
       
    # Handling Categorical Data
    data = pd.get_dummies(data, columns=['categorical_column'])  # Example: One-hot encoding
    
    # Dealing with Skewed Data
    data['skewed_numeric'] = data['skewed_numeric'].apply(lambda x: x if x > 0 else 1)  # Example: Handling negative values
    
    # Renaming Meaningless Column Names
    data.rename(columns={'old_column_name': 'new_descriptive_name'}, inplace=True)
    
    return data

Source: Internet