Building Complex Data Pipelines

4 days ago

Thāne, Maharashtra, India beBeeEngineering Full time ₹ 1,80,00,000 - ₹ 2,50,00,000
Data Engineering Specialist

We are seeking a highly skilled data engineering specialist to design, develop, and maintain ETL/ELT data pipelines to ingest data from various sources into our data warehouse.

Key Responsibilities:
  • Data Pipeline Development: Design, develop, and maintain ETL/ELT data pipelines to ingest data from databases and flat files into our data warehouse.
  • AWS Services Utilization: Utilize AWS services such as Lambda, DMS, and Glue for data ingestion and transformation processes.
  • Snowflake Stored Procedures: Develop and optimize Snowflake stored procedures to implement complex business logic and data transformations.
  • Apache Airflow Management: Manage and monitor scheduled data workflows using Apache Airflow.
  • Data Quality Checks: Implement and maintain robust data quality checks to ensure accuracy and reliability of all delivered datasets and reports.
  • Project Delivery Lifecycle: Adhere to the full project delivery lifecycle, including requirements gathering, design, implementation, documentation, and obtaining user acceptance testing (UAT) approvals.
  • Data Set Delivery: Deliver data sets to other AWS accounts and support the creation of Power BI reports.
Requirements:
  • Experience: Proven experience as a Data Engineer or in a similar role.
  • SQL Scripting: Strong proficiency in SQL scripting.
  • Python: Proficiency in Python for data manipulation and automation.
  • Apache Airflow: Experience with Apache Airflow for workflow orchestration.
  • AWS Cloud Services: Hands-on experience with AWS Cloud services, including Lambda, DMS, and Glue.
  • Data Warehousing: Solid understanding of data warehousing concepts, with practical experience using Snowflake.
  • Problem-Solving Skills: Excellent problem-solving skills and the ability to design and implement solutions based on business requirements.
  • SDLC and Project Documentation: Experience with the full software development lifecycle (SDLC) and project documentation.
  • Data Quality Best Practices: Familiarity with data quality best practices.
  • Communication and Collaboration: Strong communication and collaboration skills.

  • Data Pipeline Lead

    3 days ago

    Thāne, Maharashtra, India beBeeDataEngineer Full time ₹ 1,80,00,000 - ₹ 2,20,00,000

    Job OverviewWe are seeking a seasoned data engineer to lead our team in designing and developing scalable data pipelines. This role requires expertise in cloud computing, data engineering, and ETL/ELT processes.Key ResponsibilitiesDesign, develop, and optimize complex data pipelines using Apache Spark, AWS Glue, and Azure Data Factory.Collaborate with...

  • Senior Data Pipeline Developer

    2 days ago

    Thāne, Maharashtra, India beBeeData Full time US$ 80,000 - US$ 1,50,000

    Job SummaryThe ideal candidate will have expertise in developing and maintaining scalable data pipelines using big data technologies such as PySpark, AWS, and Databricks.Responsibilities include designing, building, and optimizing ETL/ELT pipelines, performing data profiling and validation, and collaborating with analytics teams to surface trustworthy KPIs...

  • Senior Data Pipeline Engineer

    2 days ago

    Thāne, Maharashtra, India beBeeAzure Full time ₹ 10,00,000 - ₹ 15,00,000

    Job OpportunityThreats to businesses and organizations are a growing concern in the digital age.A proactive approach to cybersecurity involves continuous monitoring and testing of digital environments to identify vulnerabilities.Job DescriptionSeeking an experienced Freelance Azure Data Factory Specialist to support data integration and ETL needs. The ideal...

  • Python Data Pipeline Lead

    3 days ago

    Thāne, Maharashtra, India beBeeEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    About the RoleWe are seeking a seasoned Python engineer to lead our data pipeline initiatives.The ideal candidate will have hands-on experience with Apache Beam and Databricks, with a strong background in designing and building scalable data processing systems.

  • Data Pipeline Architect

    1 week ago

    Thāne, Maharashtra, India beBeeCloudDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Enterprise Data Engineering LeaderWe are seeking a seasoned data engineering professional to lead the development of secure, scalable, and high-performance data pipelines using Snowflake.Design and implement end-to-end data solutions from data ingestion to analytics and AI projects.Leverage expertise in cloud infrastructure, Terraform, and Pulumi to...

  • Data Pipeline Specialist

    6 days ago

    Thāne, Maharashtra, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 25,00,000

    Data Engineering RoleJob Overview:We are seeking a skilled Data Engineer with expertise in Informatica DEI to design, develop, and implement efficient data pipelines.Main Responsibilities:Design and develop scalable data engineering solutions using Informatica DEICollaborate closely with cross-functional teams to ensure seamless data integration and delivery...

  • Senior Pipeline Developer

    4 days ago

    Thāne, Maharashtra, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,10,000

    Job Title: Data EngineerWe are seeking an experienced professional to build scalable pipelines, work with streaming data, and collaborate with data scientists and stakeholders to deliver high-quality solutions.Key Responsibilities:Design and optimize data pipelines and ETL workflows.Manage and process large datasets using Spark, PySpark, and SQL.Build and...

  • Databricks Engineer with Expertise in Data Pipelines

    5 days ago

    Thāne, Maharashtra, India beBeeDataIngestion Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Data Ingestion SpecialistWe are seeking a seasoned professional to design, develop, and optimize data ingestion pipelines that integrate multiple data sources into Databricks.This role requires a strong understanding of data ingestion processes, focusing on integrating various data sources into Databricks for efficient workflow management.The...

  • Big Data Engineer: Optimizing Scalable Data Pipelines

    3 days ago

    Thāne, Maharashtra, India beBeeData Full time ₹ 15,00,000 - ₹ 30,00,000

    Big Data Engineer PositionThe role is for a highly skilled Big Data Engineer with strong expertise in Scala, Spark, Python, NiFi, and Apache Kafka. The ideal candidate will have hands-on experience in building, scaling, and optimizing big data pipelines.We are seeking a talented individual who can design, develop, and optimize large-scale data pipelines and...

  • Building Intelligent Infrastructure

    1 day ago

    Thāne, Maharashtra, India beBeeData Full time US$ 12,00,000 - US$ 18,00,000

    Senior Data Engineer OpportunityWe are seeking a highly skilled Senior Data Engineer to join our product development and delivery team. This is an exciting opportunity to build the intelligence infrastructure that powers enterprise transformation in complex telecommunications and IT Asset Management (ITAM) environments.Design and implement robust ETL...