Description

Trayectoria iniciada como programador a los 15 años, especializado en el ecosistema Big Data desde sus primeras etapas. Más de 10 años de experiencia en arquitecturas de datos cloud, diseño de pipelines ETL/ELT complejos y migración de sistemas legacy hacia soluciones modernas utilizando Spark, Databricks, Snowflake y servicios cloud (AWS, Azure, GCP).

Experto en desarrollo de Data Lakes, optimización de costes y mejora del rendimiento mediante migraciones cloud nativas. Experiencia demostrada en sectores críticos como seguridad pública, retail y tecnología legal. Especialista en Apache Spark (PySpark, Scala), Snowflake, Databricks y metodologías ágiles.

PROYECTOS PERSONALES

• Análisis de Criminalidad en Londres - Pipeline ETL completo con Snowflake y visualización interactiva Streamlit

GitHub:

• Libro Técnico Publicado - Autor especializado en tecnologías de datos

Amazon:

Languages

Portuguese
Native or bilingual
Spanish
Fluent
English
Fluent

Workplace preferences

Remote only

Primarily works remotely

Prodesp/DIPOL
Data Engineer / Data Scientist
February 2022 - Today (4 years and 5 months)
Developed routines in Databricks (PySpark) for processing and analyzing datasets related to Missing Persons and Death Records.
Built routines in Databricks (PySpark) to profile kidnappers.
Created web crawlers to find images of criminals online and compare them with a local database.
Established Data Factory pipelines to execute Databricks notebooks.
Designed routines for cross-referencing local data (SQL Server and Data Lake) with Public Safety Department APIs.
Developed Heat Maps for robbery and theft crimes using BigQuery and Looker.
Created scripts in PySpark (Dataproc) and Apache Beam (Dataflow) for migrating crime data from IBM DB2 to BigQuery for a heat map project.
Migrated data project from GCP to Snowflake
Created databases, schemas, and stages
Copied data from a Google Cloud Storage stage
Bulk loading from Google Cloud Storage
Created jobs in Snowpark
Developed dashboards using Streamlit
Google Cloud Platform (GCP) Dataproc Dataflow Databricks Azure Datalake
Smartfit
Data Engineer
May 2021 - February 2022 (9 months)
Developed routines to monitor Alteryx ETL jobs
Created migration tasks using DMS
Configured Kinesis Data Streams as a target for DMS
Restructured Data Lake Zones
Migrated legacy jobs in Glue to Spark (PySpark) running on EKS
Migrated routines from EMR to Spark on EKS (Fargate)
Implemented pipelines using Airflow (MWAA)
Developed ETL routines with Spark (PySpark) on EKS for loading into parquet (S3/Athena) and Redshift
Conducted a PoC for migrating Tableau to Elasticsearch/Kibana
Migrated data project from AWS to Snowflake
Created databases, schemas, and stages
Copied data from S3 into staging table
Loaded data from staging table to target table
Created jobs in Snowpark
Developed dashboards using Streamlit
JBS/Seara (Project-Based)
Data Engineer
February 2021 - May 2021 (3 months)
Implemented data pipelines with Composer (Airflow).
Developed ETL routines with PySpark (Dataproc) for loading datasets into BigQuery.
Migrated Oracle databases to BigQuery.
Developed web crawlers (Web scraping) for extracting data from government portals.
Dataproc PySpark Oracle Database BigQuery Airflow