The problem
Move raw public data into a controlled cloud workflow
The project used transportation data from the National Highway Traffic Safety Administration as the source for a cloud-based ingestion and transformation exercise.
Boston University · Cloud Data Engineering
Microsoft Azure ETL Pipeline
A reproducible cloud workflow for provisioning resources, ingesting transportation data, validating transformations, and documenting deployment through infrastructure templates.
My contribution
Provision, ingest, validate, and document
I configured the Azure resource group and storage environment, worked through Cloud Shell, prepared ARM deployment files, and verified that the workflow could be reproduced from documented infrastructure settings.
Workflow
ETL and infrastructure steps
1. Provision
Created and organized Azure resources, including a resource group and storage account, using named infrastructure components.
2. Ingest and transform
Moved source data into the cloud environment and prepared it for structured analysis and validation.
3. Validate and reproduce
Checked outputs, documented the resource configuration, and used ARM template and parameter files to support repeatable deployment.
Engineering practices demonstrated
- Cloud resource organization and naming.
- Data ingestion and transformation planning.
- Storage configuration and data validation.
- Infrastructure-as-code concepts with ARM templates.
- Separation of reusable templates from deployment parameters.
- Documentation for reproducibility and troubleshooting.
Quality checks
- Confirm source files arrive in the expected location.
- Validate schemas and field types before analysis.
- Check for missing, duplicated, or malformed records.
- Verify deployment parameters match the target environment.
- Document assumptions and failure points for future runs.
Technology