Data Pipelines Specialist

1 day ago

Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000
Job Description

We are seeking an experienced Data Engineer to join our team. The ideal candidate will have a strong focus on designing, developing, and maintaining scalable and reliable data pipelines using PySpark.

This role involves leveraging the Databricks Lakehouse Platform and its advanced AI/ML features to unlock data-driven insights and power our business intelligence and machine learning initiatives.

Key Responsibilities
  1. Design and develop efficient and scalable ETL/ELT pipelines on the Databricks platform using PySpark, SQL, and Delta Live Tables (DLT).
  2. Implement and manage data solutions within the Databricks Lakehouse Platform, ensuring best practices for data storage, governance, and management using Delta Lake and Unity Catalog.
  3. Write high-quality, maintainable, and optimized PySpark code for large-scale data processing and transformation tasks.
  4. Collaborate with data scientists to productionize machine learning models. Utilize Databricks AI features such as the Feature Store, MLflow for model lifecycle management, and AutoML for accelerating model development.
  5. Implement robust data quality checks and validation frameworks to ensure data accuracy, completeness, and reliability within the delta tables.
  6. Monitor, troubleshoot, and optimize the performance of Databricks jobs, clusters, and SQL warehouses to ensure efficiency and cost-effectiveness.
  7. Work closely with data analysts, data scientists, and stakeholders to understand their data requirements and deliver effective solutions.
  8. Create and maintain comprehensive technical documentation for data pipelines, architectures, and processes.
Required Skills & Qualifications
  1. 3-5 years of hands-on experience in a data engineering role.
  2. Proven, in-depth experience with the Databricks platform, including Databricks Workflows, Notebooks, Clusters, and Delta Live Tables.
  3. Strong proficiency in Python and extensive hands-on experience with PySpark for data manipulation and processing.
  4. Solid understanding of modern data architectures, including the Lakehouse paradigm, Data Lakes, and Data Warehousing.
  5. Hands-on experience with Delta Lake, including schema evolution, ACID transactions, and time travel features.
  6. Excellent SQL skills and the ability to write complex queries for data analysis and transformation.
  7. Practical experience with Databricks AI/ML capabilities, particularly MLflow and the Feature Store.
  8. Experience working with at least one major cloud provider (AWS, Azure, or GCP).
  9. Strong analytical and problem-solving skills with the ability to debug complex data issues.
  10. Excellent verbal and written communication skills.
Preferred Qualifications
  1. Databricks Certified Data Engineer Associate/Professional certification.
  2. Experience with CI/CD tools (e.g., Jenkins, Azure DevOps, GitHub Actions) for data pipelines.
  3. Familiarity with streaming technologies like Structured Streaming.
  4. Knowledge of data governance tools and practices within Unity Catalog.

  • Data Pipeline Specialist

    6 days ago

    Bengaluru, Karnataka, India beBeeDataPipeline Full time ₹ 12,00,000 - ₹ 25,00,000

    Job Opportunity: Data Pipeline SpecialistWe are seeking a skilled data pipeline expert to validate and optimize our data transformations.ETL testing methodologies and best practicesDebugging data issues and resolving discrepanciesWorking with large datasets in cloud environmentsThe ideal candidate should have hands-on experience in Databricks, Python, SQL,...

  • Data Pipeline Specialist

    7 days ago

    Bengaluru, Karnataka, India beBeeDataPipeline Full time

    Job Title: Data Pipeline SpecialistA leading IT consulting and software solutions provider is seeking a highly skilled professional to design, build, and maintain scalable and reliable data pipelines.This role involves developing ETL processes to ensure data quality, integrity, and consistency. The ideal candidate will collaborate with analysts, scientists,...

  • Data Pipeline Specialist

    2 weeks ago

    Bengaluru, Karnataka, India beBeeDataPipeline Full time US$ 1,00,000 - US$ 1,50,000

    Job Overview- We are seeking a highly skilled Data Pipeline Specialist to join our team. The ideal candidate will have experience building and productionizing data pipelines using industry-leading technologies.- A strong background in working with complex SQL datasets is required, as well as expertise in building efficient SQL scripts using BigQuery Syntax-...

  • Data Pipeline Specialist

    2 days ago

    Bengaluru, Karnataka, India beBeeData Full time ₹ 15,00,000 - ₹ 28,00,000

    Job Title : Data Pipeline SpecialistKey Responsibilities :Develop and manage scalable data pipelines using PySpark and SQL to drive business growth.Collaborate with cross-functional teams to implement data-driven solutions leveraging AWS services.Utilize Databricks for advanced data processing and analytics, working closely with data scientists and analysts...

  • Senior Data Pipeline Specialist

    1 week ago

    Bengaluru, Karnataka, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    We are seeking a highly skilled Senior Data Pipeline Specialist to join our team.As a Senior Data Pipeline Specialist, you will be responsible for designing and developing real-time and batch data pipelines using tools like Apache NiFi, Apache Kafka, and Apache Airflow.Design and develop scalable data pipelines that meet the needs of various business...

  • Data Pipeline Architect

    1 week ago

    Bengaluru, Karnataka, India beBeeAzure Full time ₹ 15,00,000 - ₹ 25,00,000

    Freelance Azure Data Factory SpecialistWe are seeking an experienced Freelance Azure Data Factory (ADF) Specialist to support our data integration and ETL needs.The ideal consultant will have strong hands-on expertise in Azure ADF pipelines, data orchestration, and transformations to ensure smooth, scalable, and secure data flows across multiple sources and...

  • Data Pipeline Engineer

    2 weeks ago

    Bengaluru, Karnataka, India beBeeData Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    The Role of Senior Data Integration SpecialistJob Overview:We are seeking an experienced Senior Data Integration Specialist to join our team. This is a key position that will play a vital role in the smooth operation of our enterprise data integration platform.As a Senior Data Integration Specialist, you will be responsible for designing, implementing, and...

  • Senior Data Pipeline Specialist

    3 days ago

    Bengaluru, Karnataka, India beBeeDataPipelineSpecialist Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    We are seeking an experienced Senior Data Pipeline Specialist to lead the development and maintenance of complex data pipelines and enterprise-grade data systems.Job DescriptionThe ideal candidate will have 6-15 years of experience in ETL testing, software development, and data engineering. They must be proficient in Python, SQL, DWH, CI/CD, AWS, Automation,...

  • Data Pipeline Specialist

    1 week ago

    Bengaluru, Karnataka, India beBeeBackend Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Backend Developer Job DescriptionWe are seeking a skilled Backend Developer to join our team. The ideal candidate will have a strong background in developing scalable data collection pipelines and implementing automated web data extraction solutions.The successful candidate will be responsible for designing, developing, and maintaining large-scale data...

  • Data Pipeline Engineer

    9 hours ago

    Bengaluru, Karnataka, India beBeeData Full time ₹ 22 - ₹ 28

    Job OpportunityWe are seeking a skilled professional to fill the role of Big Data Specialist. This position requires expertise in designing, developing, and maintaining large-scale data pipelines using AWS services such as Glue, Lambda, and DMS.The ideal candidate will have hands-on experience with Apache Spark (Py Spark) for processing and transforming big...