Senior Consultant

Job Description

Senior Data Engineer - Databricks (Azure/AWS)

Role Overview:

We are looking for a hands-on Senior Data Engineer experienced in migrating and building large-scale data pipelines on Databricks using Azure or AWS platforms. The role will focus on implementing batch and streaming pipelines, applying the bronze-silver-gold data lakehouse model, and ensuring scalable and reliable data solutions.

Required Skills and Experience:

- 6+ years of hands-on data engineering experience, with 2+ years specifically working on Databricks in Azure or AWS.
- Proficiency in building and optimizing Spark pipelines (batch and streaming).
- Strong experience implementing bronze/silver/gold data models.
- Working knowledge of cloud storage systems (ADLS, S3) and compute services.
- Experience migrating data from RDBMS (Oracle, SQL Server) or Hadoop ecosystems.
- Familiarity with Airflow, Azure Data Factory, or AWS Glue for orchestration.
- Good scripting skills (Python, Scala, SQL) and version control (Git).

Preferred Qualifications:

- Databricks Certified Data Engineer Associate or Professional.
- Experience with Delta Live Tables (DLT) and Databricks SQL.
- Understanding of cloud security best practices (IAM roles, encryption, ACLs).

Key Responsibilities:

- Design, develop, and operationalize scalable data pipelines on Databricks following medallion architecture principles.
- Migrate and transform large data volumes from traditional on-prem systems (Oracle, Hadoop, Exadata) into cloud data platforms.
- Develop efficient Spark (PySpark/Scala) jobs for ingestion, transformation, aggregation, and publishing of data.
- Implement data quality checks, error handling, retries, and data validation frameworks.
- Build automation scripts and CI/CD pipelines for Databricks workflows and deployment.
- Tune Spark jobs and optimize cost and performance in cloud environments.
- Collaborate with data architects, product owners, and analytics teams.

Attributes for Success:

- Strong analytical and problem-solving skills.
- Attention to scalability, resilience, and cost efficiency.
- Collaborative attitude and passion for clean, maintainable code.

Diversity and Inclusion:An Oracle career can span industries, roles, Countries, and cultures, allowing you to flourish in new roles and innovate while blending work life in. Oracle has thrived through 40+ years of change by innovating and operating with integrity while delivering for the top companies in almost every industry. To nurture the talent that makes this happen, we are committed to an inclusive culture that celebrates and values diverse insights and perspectives, a workforce that inspires thought leadership and innovation. Oracle offers a highly competitive suite of Employee Benefits designed on the principles of parity, consistency, and affordability. The overall package includes certain core elements such as Medical, Life Insurance, access to Retirement Planning, and much more. We also encourage our employees to engage in the culture of giving back to the communities where we live and do business.At Oracle, we believe that innovation starts with diversity and inclusion and to create the future we need talent from various backgrounds, perspectives, and abilities. We ensure that individuals with disabilities are provided reasonable accommodation to successfully participate in the job application, and interview process, and in potential roles. To perform crucial job functions. That's why we're committed to creating a workforce where all individuals can do their best work. It's when everyone's voice is heard and valued that we're inspired to go beyond what's been done before.