Celebal Technologies

7 days ago

Pune, Maharashtra, India CELEBAL TECHNOLOGIES PRIVATE LIMITED Full time ₹ 20,00,000 - ₹ 25,00,000 per year

DATA ENGINEER (Databricks & AWS)

Overview :

As a Data Engineer, you will work with multiple teams to deliver solutions on the AWS Cloud using core cloud data engineering tools such as Databricks on AWS, AWS Glue, Amazon Redshift, Athena, and other Big Data-related technologies. This role focuses on building the next generation of application-level data platforms and improving recent implementations. Hands-on experience with Apache Spark (PySpark, SparkSQL), Delta Lake, Iceberg, and Databricks is essential.

Responsibilities :

- Define, design, develop, and test software components/applications using AWS-native data services : Databricks on AWS, AWS Glue, Amazon S3, Amazon Redshift, Athena, AWS Lambda, Secrets Manager

- Build and maintain ETL/ELT pipelines for both batch and streaming data.

- Work with structured and unstructured datasets at scale.

- Apply Data Modeling principles and advanced SQL techniques.

- Implement and manage pipelines using Apache Spark (PySpark, SparkSQL) and Delta Lake/Iceberg formats.

- Collaborate with product teams to understand requirements and deliver optimized data solutions.

- Utilize CI/CD pipelines with DBX and AWS for continuous delivery and deployment of Databricks code.

- Work independently with minimal supervision and strong ownership of deliverables.

Must Have :

years of experience in Data Engineering on AWS Cloud.

- Hands-on expertise in :

1. Apache Spark (PySpark, SparkSQL)

2. Delta Lake / Iceberg formats

3. Databricks on AWS

4. AWS Glue, Amazon Athena, Amazon Redshift

- Strong SQL skills and performance tuning experience on large datasets.

- Good understanding of CI/CD pipelines, especially using DBX and AWS tools.

- Experience with environment setup, cluster management, user roles, and authentication in Databricks.

- Certified as a Databricks Certified Data Engineer Professional (mandatory).

Good To Have :

- Experience migrating ETL pipelines from on-premise or other clouds to AWS Databricks.

- Experience with Databricks ML or Spark 3.x upgrades.

- Familiarity with Airflow, Step Functions, or other orchestration tools.

- Experience integrating Databricks with AWS services in a secured, production-ready environment.

- Experience with monitoring and cost optimization in AWS.

Key Skills :

- Languages : Python, SQL, PySpark

- Big Data Tools : Apache Spark, Delta Lake, Iceberg

- Databricks on AWS

- AWS Services : AWS Glue, Athena, Redshift, Lambda, S3, Secrets Manager

- Version Control & CI/CD : Git, DBX, AWS CodePipeline/CodeBuild

- Other : Data Modeling, ETL Methodology, Performance Optimization

  • Celebal Technologies

    5 days ago

    Pune, Maharashtra, India CELEBAL TECHNOLOGIES PRIVATE LIMITED Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Job Description : years of experience as an DevOps engineer, deploying, debugging & automating the solutions. - Experience in Troubleshooting Cloud Based (AWS, Azure & GCP), On Prem Applications. - Strong communication skills, team collaboration, problem-solving skills, with experience in working in cross-functional teams and managing stakeholder...

  • Celebal Technologies

    2 days ago

    Pune, Maharashtra, India CELEBAL TECHNOLOGIES PRIVATE LIMITED Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Primary Responsibilities : - Provide expertise in the design and programming of all kinds of Oracle database components logical and physical schemas, tables, indexes, queries, stored procedures, functions, materialized views, oracle jobs, Collections, AQ tables, etc. in a complex Oracle-based software-based modeling execution platform, from Development to...

  • Full Stack Engineer

    4 hours ago

    Pune, Maharashtra, India Celebal Technologies Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Immediate Joiners requiredPosition: Full Stack Developer with DatabricksLocations: Noida, Pune, Jaipur, Bengaluru, HyderabadDuration: FulltimeExperience: 10+ YearsJob Description:a) Backend: Python (preferred) OR strong expertise in with basic Python knowledge and ability to pick it up quicklyb) Frontend: Key Responsibilities:• Design and develop a...

  • Data Architect

    1 week ago

    Pune, Maharashtra, India, Maharashtra Celebal Technologies Full time

    Technical Skills • Databricks Lakehouse Platform: o Medallion Architecture o Unity Catalog (data governance, lineage) o Delta Lake & DLT Pipelines o PySpark Workbooks o Spark SQL & SQL Warehouse • Programming: Python, SQL, PySpark • AWS Cloud Services: IAM, S3, Lambda, EMR, Redshift, Bedrock • Other: o Familiarity with DevOps and CI/CD processes o...

  • Celebal Technologies

    4 weeks ago

    Pune, India Celebal Technologies Full time

    Responsibilities : Platform Expertise : - Act as a trusted advisor by leveraging in-depth knowledge of the Databricks Platform to guide NextGen customers (domain teams) on data and analytics implementations- Design and validate well-architected solutions and plans for productiongrade implementations based on NextGen best practices.- Discover new use case...

  • Data engineer/ Migration Specialist-Databricks

    2 days ago

    Bengaluru, Jaipur, Pune, India Celebal Technologies Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Job Title: Teradata Migration SpecialistCompany: Celebal TechnologiesLocation: Bangalore, Hyderabad, Pune, Noida, JaipurExperience Required: 5+ YearsEmployment Type: Full-timeAbout UsCelebal Technologies is a leading Solution Services company providing expertise in Data Science, Big Data, Enterprise Cloud, and Automation. We are at the forefront of...

  • Celebal Technologies

    4 weeks ago

    Pune, India Celebal Technologies Full time

    Job Description : - 5+ years of experience as an DevOps engineer, deploying, debugging & automating the solutions.- Experience in Troubleshooting Cloud Based (AWS, Azure & GCP), On Prem Applications.- Strong communication skills, team collaboration, problem-solving skills, with experience in working in cross-functional teams and managing stakeholder...

  • Data Architect

    1 week ago

    pune, India Celebal Technologies Full time

    Technical Skills • Databricks Lakehouse Platform: o Medallion Architecture o Unity Catalog (data governance, lineage) o Delta Lake & DLT Pipelines o PySpark Workbooks o Spark SQL & SQL Warehouse • Programming: Python, SQL, PySpark • AWS Cloud Services: IAM, S3, Lambda, EMR, Redshift, Bedrock • Other: o Familiarity with DevOps and CI/CD processes o...

  • Data Architect

    1 week ago

    Pune, India Celebal Technologies Full time

    Technical Skills • Databricks Lakehouse Platform: o Medallion Architecture o Unity Catalog (data governance, lineage) o Delta Lake & DLT Pipelines o PySpark Workbooks o Spark SQL & SQL Warehouse • Programming: Python, SQL, PySpark • AWS Cloud Services: IAM, S3, Lambda, EMR, Redshift, Bedrock • Other: o Familiarity with DevOps and CI/CD processes o...

  • Data Architect

    6 days ago

    Pune, India Celebal Technologies Full time

    Technical Skills • Databricks Lakehouse Platform: o Medallion Architecture o Unity Catalog (data governance, lineage) o Delta Lake & DLT Pipelines o PySpark Workbooks o Spark SQL & SQL Warehouse • Programming: Python, SQL, PySpark • AWS Cloud Services: IAM, S3, Lambda, EMR, Redshift, Bedrock • Other: o Familiarity with DevOps and CI/CD processes o...

  • Data Architect

    1 week ago

    Pune, India Celebal Technologies Full time

    Technical Skills • Databricks Lakehouse Platform: o Medallion Architecture o Unity Catalog (data governance, lineage) o Delta Lake & DLT Pipelines o PySpark Workbooks o Spark SQL & SQL Warehouse • Programming: Python, SQL, PySpark • AWS Cloud Services: IAM, S3, Lambda, EMR, Redshift, Bedrock • Other: o Familiarity with DevOps and CI/CD processes o...

  • Data Architect

    1 week ago

    Pune, India Celebal Technologies Full time

    Technical Skills• Databricks Lakehouse Platform:o Medallion Architecture o Unity Catalog (data governance, lineage)o Delta Lake & DLT Pipelineso PySpark Workbooks o Spark SQL & SQL Warehouse• Programming: Python, SQL, PySpark• AWS Cloud Services: IAM, S3, Lambda, EMR, Redshift, Bedrock• Other:o Familiarity with DevOps and CI/CD processeso Experience...

  • Data Architect

    1 week ago

    Pune, India Celebal Technologies Full time

    Technical Skills• Databricks Lakehouse Platform:o Medallion Architecture o Unity Catalog (data governance, lineage)o Delta Lake & DLT Pipelineso PySpark Workbooks o Spark SQL & SQL Warehouse• Programming: Python, SQL, PySpark• AWS Cloud Services: IAM, S3, Lambda, EMR, Redshift, Bedrock• Other:o Familiarity with DevOps and CI/CD processeso Experience...

  • Data Architect

    1 week ago

    Pune, India Celebal Technologies Full time

    Technical Skills • Databricks Lakehouse Platform: o Medallion Architecture o Unity Catalog (data governance, lineage) o Delta Lake & DLT Pipelines o PySpark Workbooks o Spark SQL & SQL Warehouse • Programming: Python, SQL, PySpark • AWS Cloud Services: IAM, S3, Lambda, EMR, Redshift, Bedrock • Other: o Familiarity with DevOps and CI/CD processes o...