Documents an end-to-end data pipeline that handles real-time stream ingestion and analytics for NYC Yellow Taxi trip data.
The architecture bridges the gap between infrastructure deployment (Terraform), real-time message brokering (Kafka), storage virtualization (GCS), optimized data warehousing (BigQuery), modern analytics engineering (dbt), and rapid data application development (Streamlit).
Focus: Evolving infrastructure from manual setups to an automated cloud data warehouse pipeline. Inspirations: Applied Data Engineering best practices.
Key Architectural Implementation:
To ensure complete environment reproducibility and prevent "click-ops" configuration drift, the entire cloud infrastructure is managed programmatically. • Tooling: Terraform • Resources Managed: GCS Storage Buckets, BigQuery Datasets, and Identity & Access Management (IAM) roles.
Simulates real-time telemetry updates from active taxi rides, processing messages with low latency. • Tooling: Apache Kafka, Docker, Docker Compose • Workflow: A containerized Python Kafka producer chunks and streams raw NYC taxi datasets into a messaging topic. A companion Python consumer listens to this topic, batches incoming streams, and flushes structured payloads out into a raw storage layer.
Serves as the central repository for raw, immutable data ingest before heavy transformations occur. • Tooling: Google Cloud Storage (GCS) • Design Pattern: Acts as a landing zone / Bronze layer, decoupling the ingestion compute limits from storage billing.
Transitions data from cold storage into a high-performance, structured analytical engine. • Tooling: Google BigQuery • Performance Tuning: Data is loaded utilizing Partitioning (segmented by ride pickup datetime) and Clustering (grouped by geographic location IDs and payment type). This limits full-table scans, directly dropping query execution costs and increasing scan performance.
Cleanses, structures, and enforces business logic on top of raw table schemas. • Tooling: dbt Core (Data Build Tool) • DAG Design: Abstracted into a modular, multi-tier layout: ◦ Staging: Schema casting, column renaming, and initial structural validation. ◦ Marts: High-value production tables optimized for business intelligence tools.
Provides an intuitive interface for decision-makers to track ride patterns, pricing outliers, and monthly trends. • Tooling: Streamlit • Implementation: A Python web application pulling highly aggregated metrics straight from the BigQuery dbt marts, ensuring minimal runtime latency.