Data Engineer

3 days ago

Anywhere in IndiaMultiple Locations Codesmith Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Description :


We are seeking a strong Data Engineer with advanced expertise in Databricks and PySpark. The successful candidate will be a key contributor to critical projects, including migrating Palantir data transformation pipelines to Databricks Notebooks, designing and implementing incremental data pipelines, and orchestrating workflows in Azure Databricks.

Key Responsibilities :


- Migrate Palantir data pipelines to Databricks Notebooks, leveraging PySpark for complex transformations.

- Replace proprietary Palantir libraries with open source or custom Pyspark implementations

- Design, build, and maintain incremental data load pipelines to handle dynamic updates from various sources, ensuring scalability and efficiency.

- Develop robust data ingestion pipelines to load data into the Databricks Bronze layer from relational databases, APIs, and file systems.

- Implement incremental data transformation workflows to update silver and gold layer datasets in near real-time, adhering to Delta Lake best practices.

- Integrate Airflow with Databricks to orchestrate end-to-end workflows, including dependency management, error handling, and scheduling.

- Understand business and technical requirements, translating them into scalable Databricks solutions.

- Optimize Spark jobs and queries for performance, scalability, and cost-efficiency in a distributed environment.

- Implement robust data quality checks, monitoring solutions, and governance frameworks within Databricks.

- Collaborate with team members on Databricks best practices, reusable solutions, and incremental loading strategies.

Required Qualifications :


- Bachelors degree in computer science, Information Systems, or a related discipline.

- 6+ years of hands-on experience with Databricks, including expertise in PySpark.

- Proven experience in incremental data loading techniques into Databricks, leveraging Delta Lake's features (e.g., time travel, MERGE INTO).

- Strong understanding of data warehousing concepts, including data partitioning, and indexing for efficient querying.

- Solid knowledge of Azure Cloud Services, particularly Azure Databricks and Azure Data Lake Storage.

- Familiarity with version control systems (e.g., Git) and CI/CD pipelines for data engineering workflows.

- Excellent analytical and problem-solving skills with a focus on detail-oriented development.

Preferred Qualifications :


- Proficiency in Palantir and experience in migrating Palantir data pipelines to Databricks.

- Expertise in Airflow integration for workflow orchestration, including designing and managing DAGs.

- Familiarity with advanced Airflow features, such as SLA monitoring and external task dependencies.

- Advanced knowledge of Delta Lake optimizations, such as compaction, Z-ordering, and vacuuming.

- Experience with real-time streaming data pipelines using tools like Kafka or Azure Event Hubs.

- Experience with building, updating, deploying, finetuning ML models

- Certifications such as Databricks Certified Associate Developer for Apache Spark or equivalent.

- Experience in Agile development methodologies.

  • Data Engineer

    1 day ago

    Anywhere in India/Multiple Locations HireVeda Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Description : About the Role : We are looking for a skilled Data Engineer to design, build, and maintain efficient and scalable data pipelines that support business intelligence, analytics, and data science initiatives. You will work closely with cross-functional teams to ensure reliable data flow, quality, and accessibility across multiple...

  • Data Engineer

    5 days ago

    Anywhere in India/Multiple Locations Vikgol Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Job Title : Remote Data Engineer - Azure Databricks Location : Remote Job Type : Full-time Hiring 3 Data Engineers (Remote) Vikgol is looking for experienced Data Engineers to design, build, and maintain our cloud data infrastructure. This is a 100% remote, full-time position where you will work with cutting-edge technologies like Azure...

  • Azure Data Engineer/Data Engineer

    7 days ago

    Anywhere in India/Multiple Locations TESTQ Technologies Limited Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    About the Opportunity : We are looking for a skilled Azure Data Engineer / Data Engineer with hands-on experience in Azure Data Factory, Snowflake, Databricks, and DBT to architect and implement large-scale data integration and transformation pipelines. The ideal candidate will bring deep technical expertise in ETL/ELT design, data modeling, and big...

  • Data Engineer

    3 days ago

    Anywhere in India/Multiple Locations Firstcareercentre Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Essential Duties & Responsibilities : - Design, develop, and optimize scalable data pipelines for ETL/ELT processes. - Develop and maintain Python-based data processing scripts and automation tools. - Write and optimize complex SQL queries (preferably in Snowflake) for data transformation and analytics. - Work with Jenkins or other CI/CD tools to...

  • Data Engineer

    1 day ago

    Anywhere in India/Multiple Locations Techno Wise Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Data Engineer (SAS DQ Conversion) Location : PAN INDIA (Preferred Hyderabad) - Hybrid Experience : 4+ YearsAbout the Role : We are seeking a motivated and detail-oriented Data Engineer to join our team. The ideal candidate will have hands-on experience with SQL, particularly translating between different SQL dialects, and a strong background or...

  • Data Engineer

    7 days ago

    Anywhere in India/Multiple Locations vrushank tech solutions Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Location: Work from homeRole Summary: Were looking for a hands-on Data Engineer to own the data layer of customer go-lives ensuring migrations are validated, analytics pipelines are hardened, and business dashboards are powered by accurate, performant data. Youll be responsible for validating and signing off on end-to-end data migrations, building...

  • Data Engineer

    6 days ago

    Anywhere in India/Multiple Locations Vriba Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Description : Role : Data Engineer Location : Remote Experience Required : 5 to 10 yearsAbout the Role : We are seeking an experienced Data Engineer to design, build, and maintain robust data pipelines and infrastructure. The ideal candidate will have a strong background in data integration, real-time and batch processing, and a deep...

  • Lead Data Engineer

    3 days ago

    Anywhere in India/Multiple Locations ATOM Systems Pvt Ltd Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Description : Name of the position : Data Engineer Location : Coimbatore / Remote Of resources needed : 01 Mode : Contract to Hire Years of experience : 15 Years About the Role : We are seeking a highly skilled and driven Data Engineering Lead to lead our data engineering team. The ideal candidate combines strong leadership and technical expertise...

  • Senior Data Engineer

    3 days ago

    Anywhere in India/Multiple Locations Timus consulting Services Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Description : About the Role : We are seeking an experienced and highly skilled Senior Data Engineer Azure Databricks to join our data engineering team. In this role, you will be responsible for designing, building, and maintaining scalable data pipelines and solutions on Azure using Databricks, with a strong focus on performance, reliability, and...

  • Senior Data Engineer

    1 week ago

    Anywhere in India/Multiple Locations Techno Wise Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Data Engineer Location: Multiple Locations in IndiaJob Type: [Full-Time / Hybrid]Experience required: 3+ years in Data EngineeringJob Description: We are seeking a skilled Data Engineer with strong expertise in Snowflake and SQL to join our data team. The ideal candidate will have hands-on experience designing and developing robust ETL/ELT...