Storage layer

The storage layer consists of three S3 buckets, where each conforms to a zone outlined in the reference architecture diagram:

• A raw bucket, where the raw incoming data to the Kinesis Firehose Delivery Stream is backed up to (partitioned by event_date)
• A cleaned bucket, where the data is stored by the Kinesis Firehose Delivery Stream (partitioned by domain_name, event_date and event_type)
• A curated bucket, where the aggregated pageviews and visitors data are stored on a daily level (partitioned by domain_name and event_date, in the stats prefix), on an hourly level (partitioned by event_date and event_hour, in the daily-stats prefix), as well as the aggregated and filtered events (partitioned by domain_name, event_date and event_name, in the events prefix). Furthermore, the complete historical data is stored in a DuckDB database that can be found at the duckdb/data.duckdb key.