Lakehouse | Databricks | Pipelines | Spark | Visualization

An end-to-end data engineering lakehouse platform built on top of multi-format enterprise marketplace data shares using Apache Spark, Delta Lake, and Databricks.

This implements a comprehensive Three-Tier Medallion Architecture, optimizing raw, multi-structured, siloed web logs and checkout events into high-value executive business intelligence marts.

Approach

• Ingestion: Staged E-Commerce Click stream raw multi-format dataset from Databricks Marketplace over a period of 2 weeks (19/06/2020 to 04/07/2020).

Engineering Challenges

1. Fault-Tolerant Array Indexing (silver_events)

• Problem: Click stream JSON event tracking streams logged empty array states [] during non-commercial page navigation (e.g., main, faq). Standard direct array reference index pointers completely crashed on empty objects.
• Solution: Extracted properties safely from nested event tracks by using PySpark's bracket extraction syntax (col().get(0)). This prevents out-of-bound errors on empty arrays [] by gracefully routing missing values to a NULL state without breaking pipeline execution.

2. Embedded JSON String Literal Transformation (silver_sales)

• Problem: Transaction sets packed structural checkout items as a stringified text representation of an array block inside flat pipe-separated files (|), making direct field aggregation impossible.
• Solution: Defined a strict structural layout using a nested StructType schema. Applied PySpark’s native from_json function to programmatically deserialize raw text columns back into relational, queryable array objects for clean aggregation.

3. Irregular Tokenization & Space Separations (silver_users)

• Problem: User account lists were saved with arbitrary tab spaces and multiple dynamic empty intervals, resulting in standard comma-based parsers completely breaking and cramming strings into single messy data blocks.
• Solution: Leveraged the native split function backed by a robust regular expression pattern (r"\s+"). This dynamically tokenizes multi-character gaps and irregular tabs in a single execution step, mapping them into predictable, indexable fields.

4. Microsecond Timeline Realignment

• Problem: System activity times were captured down to precise microseconds (10^6), creating sixteen-digit large numeric structures (long). Straight loading caused evaluation functions to interpret micro-ticks as whole seconds, placing events hundreds of thousands of years in the future.
• Solution: Applied mathematical division arithmetic inside the column select transformations. Scaled the microsecond digits down to epoch seconds by dividing by 1,000,000 inside a sequential from_unixtime and to_timestamp pipeline block.

5. Architectural Fan-Out Management

• Problem: Deciding whether to split checkout items arrays in the intermediate transformation phase vs the analytical aggregations layer.