Lead Data Pipeline Developer

3 days ago

Thrissur, Kerala, India beBeeEngineer Full time ₹ 15,00,000 - ₹ 25,00,000
Job Title: Data Engineer

Our organization is seeking a skilled and motivated data engineer to design, develop, and maintain scalable data pipelines that process and transform large volumes of structured and unstructured data. The ideal candidate will have a strong background in data engineering, cloud technologies, and data architecture, and will collaborate closely with data scientists, analysts, and business teams to deliver high-quality data solutions.

  • Design and implement scalable data pipelines for processing structured and unstructured data, utilizing tools such as Apache Spark and Cloud Storage.
  • Develop and maintain ETL/ELT workflows for data ingestion from diverse sources including APIs, databases, files, and cloud platforms.
  • Ensure data quality, integrity, and governance across all data pipelines by implementing robust data validation and error handling mechanisms.
  • Collaborate with cross-functional teams to understand and fulfill data requirements, fostering open communication and transparency throughout the process.
  • Optimize data workflows for performance and reliability, leveraging expertise in data optimization techniques and emerging technologies.

Key Responsibilities:

Programming & Data Engineering:

  • Proficiency in Python, PySpark, and SQL programming languages.
  • Strong understanding of CI/CD practices and version control systems like Git.

Streaming & Orchestration:

  • Experience with Kafka and GCP Composer/Apache Airflow.

Cloud Tools:

  • Proficient in Google Cloud Platform (Dataproc, BigQuery, Looker) with experience in designing and deploying scalable data architectures.

Data Architecture:

  • Knowledge of data warehousing and data modeling concepts.

  • Leader in Data Pipeline Development

    3 days ago

    Thrissur, Kerala, India beBeeArtificial Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Data ScientistAs a vital member of our organization, you will play a key role in driving the adoption of Generative AI (GenAI) across our operations. With 5-7 years of experience in data science and a strong background in Python programming, you will design and implement robust data pipelines using industry-standard tools.Key...

  • Technical Lead

    3 days ago

    Thrissur, Kerala, India beBeeEngineering Full time US$ 1,50,000 - US$ 2,50,000

    Senior Pipeline Engineer LeadPure Storage seeks a Senior Pipeline Engineer Lead to oversee the build-test pipelines in Observability & Reliability. This role requires a hands-on leader with a passion for improving efficiency, streamlining development cycles.As a Senior Pipeline Engineer Lead, you will:Lead a team of talented software engineers improving...

  • Chief Data Pipeline Architect

    7 days ago

    Thrissur, Kerala, India beBeeDataEngineer Full time ₹ 10,00,000 - ₹ 20,00,000

    Key Data Engineer PositionWe are seeking a highly skilled data professional with 4+ years of experience in designing, building, and optimizing ETL pipelines for large-scale data processing workflows. The ideal candidate will have hands-on experience working with Azure Cloud services including Azure Data Factory, Databricks, and related...

  • Chief Data Pipeline Architect

    4 days ago

    Thrissur, Kerala, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our growing team of consultants.Design, develop, and maintain scalable and robust data pipelines that collect, process, and store large volumes of structured and unstructured data.Architect and implement data warehouses, data lakes, and other storage solutions that support...

  • Expert Data Pipeline Architect

    2 days ago

    Thrissur, Kerala, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Freelance Data EngineerWe are seeking a Freelance Data Engineer with expertise in designing, building, and optimizing large-scale data pipelines using Python, SQL, Linux/Bash, PySpark, AWS, and Databricks.Job Description:Data Engineering is a key component of our organization's ability to drive business outcomes through data-driven decision making. We are...

  • High-Performance Data Pipeline Architect

    4 days ago

    Thrissur, Kerala, India beBeeDataEngineer Full time ₹ 1,80,00,000 - ₹ 2,20,00,000

    We're seeking a highly skilled Big Data Engineer to help us scale and optimize our data processing capabilities.As a key member of our team, you'll design, build, and maintain high-performance data pipelines that process terabytes of data. Your focus will be on optimizing existing systems, identifying performance bottlenecks, and implementing solutions to...

  • ETL Specialist for Cloud-Based Data Pipelines

    5 days ago

    Thrissur, Kerala, India beBeeDatastage Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: ETL Developer - DataStage, AWS, SnowflakeAbout the OpportunityWe are seeking an accomplished ETL Developer to join our team. The ideal candidate will have a strong background in data engineering and experience working with IBM DataStage (on Cloud Pak for Data), AWS Glue, and Snowflake.Key Responsibilities:Design, develop, and maintain scalable ETL...

  • Lead Data Solutions Architect

    4 days ago

    Thrissur, Kerala, India beBeeData Full time ₹ 1,00,00,000 - ₹ 1,50,00,000

    We are currently seeking a skilled Data Architect to design, develop, and govern our data architecture and pipelines.The ideal candidate will be collaborative, well-organised, and able to think outside the box. Most importantly, this role is for an individual who is passionate about making a difference through healthcare.Key Responsibilities:Data...

  • Big Data Engineering Lead

    4 days ago

    Thrissur, Kerala, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 25,00,000

    Data Scientist - Big Data EngineerWe are looking for a seasoned professional to lead our data engineering initiatives. As a key member of the team, you will be responsible for designing, building, and maintaining large-scale data processing systems.Design and develop scalable data pipelines using Python, PySpark, and AWS services.Collaborate with...

  • Senior Data Developer

    2 days ago

    Thrissur, Kerala, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Senior Data DeveloperJob Description:We are seeking a highly skilled and experienced data developer to join our data platform team. As a key member of our team, you will play a vital role in designing, developing, and maintaining ETL processes to ensure efficient extraction, transformation, and loading of data from various sources into our data...