Data Engineer

6 days ago

Greater Kolkata Area, India Codesmith Full time ₹ 5,00,000 - ₹ 25,00,000 per year

Description
We are seeking a strong Data Engineer with advanced expertise in Databricks and PySpark. The successful candidate will be a key contributor to critical projects, including migrating Palantir data transformation pipelines to Databricks Notebooks, designing and implementing incremental data pipelines, and orchestrating workflows in Azure Databricks.

Key Responsibilities

  • Migrate Palantir data pipelines to Databricks Notebooks, leveraging PySpark for complex transformations.
  • Replace proprietary Palantir libraries with open source or custom Pyspark implementations
  • Design, build, and maintain incremental data load pipelines to handle dynamic updates from various sources, ensuring scalability and efficiency.
  • Develop robust data ingestion pipelines to load data into the Databricks Bronze layer from relational databases, APIs, and file systems.
  • Implement incremental data transformation workflows to update silver and gold layer datasets in near real-time, adhering to Delta Lake best practices.
  • Integrate Airflow with Databricks to orchestrate end-to-end workflows, including dependency management, error handling, and scheduling.
  • Understand business and technical requirements, translating them into scalable Databricks solutions.
  • Optimize Spark jobs and queries for performance, scalability, and cost-efficiency in a distributed environment.
  • Implement robust data quality checks, monitoring solutions, and governance frameworks within Databricks.
  • Collaborate with team members on Databricks best practices, reusable solutions, and incremental loading strategies.

Required Qualifications

  • Bachelors degree in computer science, Information Systems, or a related discipline.
  • 6+ years of hands-on experience with Databricks, including expertise in PySpark.
  • Proven experience in incremental data loading techniques into Databricks, leveraging Delta Lake's features (e.g., time travel, MERGE INTO).
  • Strong understanding of data warehousing concepts, including data partitioning, and indexing for efficient querying.
  • Solid knowledge of Azure Cloud Services, particularly Azure Databricks and Azure Data Lake Storage.
  • Familiarity with version control systems (e.g., Git) and CI/CD pipelines for data engineering workflows.
  • Excellent analytical and problem-solving skills with a focus on detail-oriented development.

Preferred Qualifications

  • Proficiency in Palantir and experience in migrating Palantir data pipelines to Databricks.
  • Expertise in Airflow integration for workflow orchestration, including designing and managing DAGs.
  • Familiarity with advanced Airflow features, such as SLA monitoring and external task dependencies.
  • Advanced knowledge of Delta Lake optimizations, such as compaction, Z-ordering, and vacuuming.
  • Experience with real-time streaming data pipelines using tools like Kafka or Azure Event Hubs.
  • Experience with building, updating, deploying, finetuning ML models
  • Certifications such as Databricks Certified Associate Developer for Apache Spark or equivalent.
  • Experience in Agile development methodologies.

)

  • Lead Data Engineer

    5 days ago

    Greater Kolkata Area, India Eucloid Data Solutions Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Job DescriptionEucloid is looking for a senior/ lead Data Engineer to join our Data Platform team supporting various business applications. The ideal candidate will support development of data infrastructure on Databricks for our clients by participating in activities which may include starting from up- stream and down-stream technology selection to...

  • Data Engineer

    5 days ago

    Greater Kolkata Area, India ZenYData Technologies Private Limited Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    We're Hiring – Data Engineer - Google Cloud Platform (GCP) –ZenYData Technologies Private LimitedAt the forefront of Data Automation & Data Management in Kolkata, we are on the lookout for passionate, innovative, and experienced Data Engineers ready to take on exciting challenges in Google Cloud Platform (GCP). Job Title: Data Engineer – Google Cloud...

  • Data Engineer

    2 weeks ago

    Greater Kolkata Area, India MatchMove Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    You Will Get ToDesign, build, and maintain high-performance data pipelines that integrate large-scale transactional data from our payments platform, ensuring data quality, reliability, and compliance with regulatory requirements.Develop and manage distributed data processing pipelines for both high-volume data streams and batch processing workflows in a...

  • Lead Data Engineer

    4 days ago

    Greater Kolkata Area, India Atom Systems Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    DescriptionName of the position :Data EngineerLocation :Coimbatore / RemoteOf resources needed :01Mode :Contract to HireYears of experience :15+ YearsAbout The RoleWe are seeking a highly skilled and driven Data Engineering Lead to lead our data engineering team. The ideal candidate combines strong leadership and technical expertise with the ability to...

  • Data Engineer

    5 days ago

    Greater Kolkata Area, India HIC Global Solutions Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Key ResponsibilitiesDesign, develop, test, and maintain robust data architectures, pipelines, and ETL processes.Ensure data quality, integrity, and security across systems and workflows.Optimize data systems for performance, scalability, and cost-efficiency.Collaborate with cross-functional teams to gather requirements and enable data-driven analytics and...

  • Data Engineer

    2 weeks ago

    Greater Kolkata Area, India The IT Firm Full time ₹ 5,00,000 - ₹ 8,00,000 per year

    ResponsibilitiesDesign, build, and maintain data pipelines and ETL workflows on Google Cloud Platform.Work with BigQuery, Dataflow, Pub/Sub, Dataproc, and Cloud Storage to enable scalable data solutions.Develop and optimize data models, transformations, and analytics layers.Write efficient Python/SQL scripts for data processing and automation.Collaborate...

  • Lead Data Engineer

    5 days ago

    Greater Kolkata Area, India Codesmith Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    DescriptionJob Title : Lead Data Consultant / Senior Data EngineerExperience Level : 10+ Years (with deep Azure experience)Employment Type : Full-time / ContractRole OverviewAs a Lead Data Consultant / Senior Data Engineer, you will design, develop, and lead the delivery of enterprise-scale, cloud-native data platforms for our clients, with a particular...

  • Senior Data Engineer

    2 weeks ago

    Greater Kolkata Area, India Omni Reach Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    We are seeking a Senior Data Engineer to build scalable, cloud-native data platforms and enable end-to-end MLOps workflows. You will design ETL/ELT pipelines, manage data lakes/warehouses/feature stores, and ensure high-performance, secure, and cost-efficient pipelines for AI/ML and analytics. This role blends Data Engineering + MLOps to deliver...

  • Senior Data Engineer

    4 days ago

    Greater Kolkata Area, India DoctusTech Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    DescriptionRole Overview :We are seeking an experienced Senior Data Engineer with a strong background in data security, privacy, and compliance to join our growing engineering team. This individual will play a critical role in designing, building, and maintaining secure data pipelines, ensuring end-to-end compliance across all data systems.The ideal...

  • ML Architect/Data Engineer, 8 YOE in Data Science/Engineering

    5 days ago

    Greater Kolkata Area, India ZenYData Technologies Private Limited Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Experience: 8+ yearsJob Location: REMOTE/CHENNAINotice Period: 30 daysML Architect/Data Engineer, 8 YOE in Data Science/Engineering & 2 YOE as Solution Architect - REMOTEWe are seeking an experienced ML Architect to design, lead, and implement advanced artificial intelligence and machine learning solutions. The ideal candidate will have deep expertise in...