Data Pipeline Architect

2 days ago

Kolkata, West Bengal, India beBeeDataSpecialist Full time ₹ 18,00,000 - ₹ 25,00,000

We are seeking a Data Specialist to join our organization, responsible for designing and developing robust data pipelines in Databricks PySpark. The ideal candidate will possess exceptional skills in ETL processes and the ability to collaborate with cross-functional teams.

About This Role

The Data Specialist will play a crucial role in ensuring efficient data extraction, transformation, and loading into the data lake and data warehouse. Key Responsibilities:

  • ETL Development:
    • Design and develop ETL processes using Databricks PySpark to extract, transform, and load data from heterogeneous sources into the data lake and data warehouse.
    • Optimize ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes.
    • Implement robust error handling and monitoring mechanisms to detect and resolve issues within ETL processes.
  • Data Pipeline Management:
    • Develop and maintain data pipelines using Databricks PySpark, ensuring data quality, integrity, and reliability throughout the ETL lifecycle.
    • Collaborate with data engineering, data science, and business intelligence teams to translate data requirements into efficient ETL workflows and pipelines.
  • Data Analysis and Query Optimization:
    • Write and optimize complex SQL queries for data manipulation, aggregation, and analysis within Databricks PySpark applications.
  • Project Coordination and Continuous Improvement:
    • Participate in project planning and coordination activities to ensure timely delivery of ETL solutions.
    • Stay updated on the latest developments in Databricks PySpark, Spark SQL, and related technologies, recommending and implementing best practices and optimizations.

Required Qualifications

  • Bachelor's degree in Computer Science, Engineering, or a related field.
  • Minimum of 3 years of experience as an ETL developer, with a strong focus on Databricks PySpark development.
  • Proficiency in Python programming, with extensive experience in developing and debugging Databricks PySpark applications.

Preferred Qualifications

  • Experience with cloud platforms, preferably AWS.
  • Experience with data platform tools such as DataBricks, Snowflake, and Tableau.

  • Chief Data Pipeline Architect

    4 days ago

    Kolkata, West Bengal, India beBeeDataEngineer Full time ₹ 1,00,00,000 - ₹ 1,50,00,000

    Job SummaryWe seek a skilled Data Engineer to develop and maintain ETL pipelines utilizing cutting-edge technologies.This role involves transforming data, supporting ingestion and quality checks during client onboarding, optimizing Snowflake queries, and collaborating with the QC team.Key Responsibilities:Develop data transformations for HighTouch...

  • Chief Data Pipeline Specialist

    2 days ago

    Kolkata, West Bengal, India beBeeData Full time ₹ 12,00,000 - ₹ 15,00,000

    Job Title: Data ArchitectWe are seeking a skilled and motivated Data Engineer to design, develop and maintain scalable data pipelines that process large volumes of structured and unstructured data.Key Responsibilities:Design and implement data architectures for processing structured and unstructured data.Build and maintain ETL/ELT workflows for data...

  • Senior Data Pipeline Architect

    2 days ago

    Kolkata, West Bengal, India beBeePlatformEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    About this Role:Lead Python Platform EngineerWe are seeking a highly skilled Lead Python Platform Engineer to join our team. This individual will be responsible for designing and implementing scalable and efficient data pipelines using Apache Beam and Databricks.This role requires strong expertise in Python, with at least 7 years of experience in performance...

  • Data Solution Architect

    2 days ago

    Kolkata, West Bengal, India beBeeDataSolution Full time ₹ 15,00,000 - ₹ 20,00,000

    Data Solution ArchitectAs a Data Solution Architect, you will be responsible for designing and developing data solutions that facilitate data generation, collection, and processing.Key Responsibilities:Designing and implementing efficient data pipelines to support data processing and analytics.Collaborating with cross-functional teams to understand data...

  • Senior Data Architect

    4 days ago

    Kolkata, West Bengal, India beBeeData Full time US$ 1,25,000 - US$ 1,75,000

    Job Summary:We are seeking a highly skilled Data Architect to design, build, and maintain complex data pipelines using AWS Glue, Lambda, and DBT.About the Role:Develop and optimize data models in Snowflake for scalability and performance.Write efficient and optimized SQL queries for data extraction, transformation, and loading.Collaborate with business...

  • Chief Architect for Scalable Data Pipelines

    3 days ago

    Kolkata, West Bengal, India beBeeDataEngineer Full time ₹ 20,25,000 - ₹ 25,27,500

    Job Title: Senior Data EngineerAt a leading organization, we seek a highly skilled individual with expertise in Microsoft Azure to lead the design, build, and optimization of scalable data pipelines and architectures.Key Responsibilities:Design and develop robust, secure, and high-performance data solutions using Azure Data Factory and Azure Synapse...

  • Data Architect

    2 weeks ago

    Kolkata, West Bengal, India Tata Consultancy Services Full time

    TCS present an excellent opportunity for Data architect Job Description: Skills: AWS, Glue, Redshift, PySpark Location: Pune / Kolkata Experience: 10 to 12 Years Strong hands-on experience in Python programming and PySpark. Experience using AWS services (RedShift, Glue, EMR, S3 & Lambda) Experience working with Apache Spark and Hadoop ecosystem. ...

  • Chief Data Infrastructure Architect

    2 days ago

    Kolkata, West Bengal, India beBeeData Full time ₹ 1,00,00,000 - ₹ 1,50,00,000

    Job Summary:As a senior data engineer, you will be responsible for leading the optimization and scalability of data delivery pipelines to ensure performance and reliability for cross-functional stakeholders.You will architect, build, and maintain robust, high-throughput data pipelines and infrastructure to support advanced analytics and machine learning...

  • Senior Data Infrastructure Architect

    22 hours ago

    Kolkata, West Bengal, India beBeeSnaplogic Full time ₹ 1,04,000 - ₹ 1,30,878

    Job DescriptionWe are seeking an experienced professional to lead and architect our Snaplogic infrastructure. The ideal candidate will have a strong background in designing and implementing complex data pipelines, as well as experience with error handling, scheduling tasks, and fault tolerance.

  • Chief Data Architect

    1 day ago

    Kolkata, West Bengal, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Senior Data EngineerAbout UsWe're seeking a highly skilled Senior Data Engineer to join our team. In this role, you will be responsible for designing and implementing modern data warehouse solutions using Snowflake and Azure Data Factory.Key Qualities of a Successful Team MemberNatural curiosity and creative mind to ask questions, connect the dots, and...