Automated Data Cleaning & Feature Engineering

Overview: Real-world tabular datasets are noisy, incomplete, and riddled with formatting errors. Doushi's preprocessing engine performs automated cleaning and intelligent feature extraction so you never have to write custom pandas pipelines.

Automated Cleaning Pipeline

When a dataset is submitted, Doushi executes a five-stage cleaning pipeline inside the isolated container sandbox:

[Raw Dataset]
      │
      ▼
1. Type & Anomaly Audit   (Schema detection, corrupted characters, mixed types)
      │
      ▼
2. Imputation Engine      (Median, mode, iterative kNN, missingness indicators)
      │
      ▼
3. Outlier Clipping       (Interquartile Range IQR & Isolation Forest filtering)
      │
      ▼
4. Target Leakage Guard   (Mutual Information correlation & chronological checks)
      │
      ▼
5. Feature Engineering    (Ratios, datetime expansion, interactions, frequency)

---

1. Missing Value Imputation

Rather than blindly dropping rows with missing cells, Doushi preserves statistical sample size:

• Numerical Missingness: Imputed with the feature median (less sensitive to skew than mean) and appends a binary indicator column is_missing_<feature> so tree models can leverage missingness signals.
• Categorical Missingness: Replaced with a designated "__missing__" label or the most frequent category depending on category dispersion.

---

2. Target Leakage Prevention

Target leakage occurs when training data contains features that directly reveal the target or would not be accessible in real production at the moment of prediction.

Common leakage scenarios caught by Doushi:

• Duplicate Recording: An account status column that changes synchronously with the churn flag (e.g. cancellation_survey_date).
• Post-Outcome Metrics: Transaction refund amounts recorded after chargeback disputes have settled.
• Perfect Mutual Information: Columns with a Mutual Information score exceeding 0.98 trigger a warning and are automatically isolated from the candidate feature pool.

---

3. Automated Feature Engineering

Tree-based models excel with nonlinear relationships, but often struggle with ratios and composite metrics without explicit feature synthesis. Doushi generates:

• Numerical Ratios: Automatically computes relevant ratios (e.g. debt_to_income, spend_per_session).
• Logarithmic Transforms: Applied to heavy long-tail distributions (such as annual revenue or page views) to stabilize gradient updates.
• Frequency Encoding: High cardinality columns (e.g. zip_code, merchant_id) are mapped to their empirical occurrence frequencies.
• Datetime Decomposition: Datetime columns are transformed into cyclical sine/cosine components (hour of day, day of week) and elapsed durations.

---

Inspection & Transparency

You can inspect the exact list of dropped columns, engineered features, and imputation strategies applied to your dataset in the Data Audit tab of your project dashboard.