Julian M.

Development, update and optimization of different components in Scala/Spark to migrate to the cloud the on premise datalake of an important bank with Amazon Web Services, as well as the creation of the new data platform based on cloud principles.

Responsibilities:

- Creation, maintenance and enhancement of different components developed in Scala and involved in the migration process to the AWS cloud of the client's on premise datalake.

- Implementation of ETL processes in Scala and Spark.

- Monitoring, launching and validation tasks of processes in production oriented to ensure their quality and reliability.

- Management of the main Big Data solutions of the AWS platform: use Amazon S3 or Amazon Redshift as data source or destination; Amazon EMR for the execution of processes developed in Scala/Spark or update of some of the lambdas deployed through AWS Lambda.

- Use of SQL and HiveQL in development (using Spark SQL) and review processes (using Apache Zeppelin).

- Proposal and implementation of additional systems for the improvement of existing processes, anticipating future needs and in order to reduce development and operation times.

- Adoption of Agile methodologies, specifically Scrum, in order to plan and follow up the work with the use of Sprints, Planning meetings and Retrospectives.

- Agile development following the principle of TDD and design patterns that are generally transparent by the abstraction level of Scala and the application of mechanisms such as Companion Object, For Comprehension, Pattern Matching or High Order Functions.

- Use of continuous integration tools such as Jenkins or Git. Use of Nexus, Jira, Confluence.

- Code review to ensure its quality.

- Supervision and follow-up of new recruits to accelerate their adaptation and boost their performance.

- Belonging to the innovation team of a large bank whose main objective is to find the fastest and most efficient way to solve the problem of the different ingest teams, administration, etc.

-Functional knowledge of the main services in Cloudera distributions.

-Scripting in Python and Bash.

-Development of different projects in Scala, Spark, python, go, using different technologies such as Hive, Hadoop, HDFS, prometheus.

-Use of Grafana for visualization of metrics obtained from Cloudera Manager

-Scripting en Python y Bash.

−Implementation of ETL processes in Scala and Spark using the banking core itself.

−Management of the main Big Data solutions of the AWS platform: use Amazon S3 'Amazon EMR for the execution of the processes developed in Scala / Spark or update

of some of the lambdas deployed through AWS Lambda ' Amazon Athena to perform data queries in SQL.

−Agile development following the principle of TDD and design patterns that are generally transparent by the level of abstraction of Scala.

−Use of SQL and HiveQL in development processes (through Spark SQL) and revision (with Apache Zeppelin).