Lakehouse | Databricks | Pipelines | Spark | Visualization
An end-to-end data engineering lakehouse platform built on top of multi-format enterprise marketplace data shares using Apache Spark, Delta Lake, and Databricks.
This implements a comprehensive Three-Tier Medallion Architecture, optimizing raw, multi-structured, siloed web logs and checkout events into high-value executive business intelligence marts.
Approach
- Ingestion: Staged E-Commerce Click stream raw multi-format dataset from Databricks Marketplace over a period of 2 weeks (19/06/2020 to 04/07/2020).
Engineering Challenges
1. Fault-Tolerant Array Indexing (silver_events)
- Problem: Click stream JSON event tracking streams logged empty array states
[] during non-commercial page navigation (e.g., main, faq). Standard direct array reference index pointers completely crashed on empty objects.
2. Embedded JSON String Literal Transformation (silver_sales)
- Problem: Transaction transaction sets packed structural checkout items as a stringified text representation of an array block inside flat pipe-separated files (
|), making direct field aggregation impossible.
3. Irregular Tokenization & Space Separations (silver_users)
- Problem: User account lists were saved with arbitrary tab spaces and multiple dynamic empty intervals, resulting in standard comma-based parsers completely breaking and cramming strings into single messy data blocks.
4. Microsecond Timeline Realignment
- Problem: System activity times were captured down to precise microseconds (10^6), creating sixteen-digit large numeric structures (
long). Straight loading caused evaluation functions to interpret micro-ticks as whole seconds, placing events hundreds of thousands of years in the future.
5. Architectural Fan-Out Management
- Problem: Deciding whether to split checkout items arrays in the intermediate transformation phase vs the analytical aggregations layer.
Key Highlights
Fig. E-Commerce Click Stream & Sales Analysis
Key Insights Identified