Senior Data Pipeline Developer

19 hours ago

Bellary, Karnataka, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000
Job Title:

Data Solutions Engineer


About the Job

As a key member of our team, you will play a pivotal role in designing and developing scalable data pipelines and ETL processes to support downstream analytics and AI applications. Your expertise will help ensure seamless integration with machine learning models and workflows.

Key Responsibilities Include:

  • Designing and implementing data solutions to meet client requirements through collaborative efforts with cross-functional teams.
  • Working closely with Machine Learning Engineers to integrate data solutions into machine learning models and workflows, ensuring optimal performance and efficiency.
  • Ensuring data quality, integrity, and security across all projects by adhering to industry best practices and standards.
  • Managing and optimizing data storage solutions in cloud environments, including AWS, Azure, and GCP, to ensure scalability and reliability.
  • Utilizing Databricks for data processing and analytics tasks, leveraging its capabilities to drive business growth and innovation.
  • Monitoring the performance of data pipelines and identifying bottlenecks or failures to implement corrective measures and prevent downtime.
  • Implementing best practices for data engineering, including documentation, testing, and version control, to maintain high-quality code and ensure reproducibility.
Qualifications
  • Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
  • 10+ years of experience as a Data Engineer or in a similar role.
  • Strong proficiency in SQL, Python, and other relevant programming languages.
  • Hands-on experience with Databricks and its ecosystem, including Spark and Delta Lake.
  • Familiarity with major cloud environments and their data services, including AWS S3, Azure Blob Storage, and Google Cloud Storage.
  • Experience with data warehousing solutions like Snowflake, Redshift, or BigQuery.
  • Comfortable working with various SQL, NoSQL, and graph databases.
  • Knowledge of data integration tools, including Fivetran and Matillion.
  • Understanding of data modeling, data architecture, and database design principles.
  • Excellent problem-solving skills and attention to detail.
  • Strong communication and collaboration skills, with the ability to work effectively with cross-functional teams.
Desired Skills
  • Experience with real-time data processing frameworks, such as Apache Kafka and Apache Storm.
  • Knowledge of data visualization tools, including Tableau and Power BI.
  • Familiarity with machine learning concepts and frameworks, including TensorFlow and PyTorch.

  • Senior Data Pipeline Developer

    5 days ago

    Bellary, Karnataka, India beBeeetl Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Job Title: ETL Specialist - Data Pipeline DeveloperExperience: 5–7 YearsLocation: RemoteJob Type: Full-timeAbout the RoleWe are seeking an experienced ETL specialist to join our data engineering team. The ideal candidate will have a strong background in designing, developing, and maintaining large-scale data pipelines using IBM DataStage, AWS Glue, and...

  • Data Pipeline Tester

    5 days ago

    Bellary, Karnataka, India beBeeSenior Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Summary:We are seeking a skilled Senior Test Engineer to join our team. The ideal candidate will have experience in ETL and Power BI testing, with a strong background in Azure Data Factory, Python, and Pytest.The successful candidate will be responsible for designing and developing comprehensive test cases to validate data flows, transformations, and...

  • Data Pipeline Architect

    1 week ago

    Bellary, Karnataka, India beBeeDataPipeline Full time ₹ 8,00,000 - ₹ 15,00,000

    Job Title: Data Pipeline ArchitectKey Responsibilities:We're seeking an experienced Data Pipeline Architect to design, develop and optimize data ingestion pipelines for integrating multiple sources into Databricks.The successful candidate will implement and maintain CI/CD pipelines for data workflows.Deploy and manage containerized applications using...

  • Data Pipelines Specialist

    2 weeks ago

    Bellary, Karnataka, India beBeeTechnicalConsultant Full time ₹ 80,00,000 - ₹ 1,20,00,000

    Job OverviewThe role involves configuring, integrating, and optimizing financial data workflows within a platform. Collaboration with cross-functional teams is essential for designing reusable and scalable solutions, implementing automation, and enhancing system performance across key investment operations.This position demands hands-on expertise in tools...

  • Data Pipeline Specialist

    6 days ago

    Bellary, Karnataka, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Data Engineering Role OverviewWe are seeking a skilled Data Engineer to join our organization.As a Data Engineer, you will be responsible for designing, developing, and maintaining efficient data pipelines to ingest data from various sources into a data warehouse.Design and develop optimized data pipelines to extract data from databases and flat...

  • Data Pipeline Specialist

    3 days ago

    Bellary, Karnataka, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 22,00,000

    Job SummaryWe are seeking a talented Data Engineer to join our team. As a Data Engineer, you will design, build, and maintain production-level data pipelines that handle large amounts of structured and unstructured data.Design and implement scalable data pipelines using AWS services such as S3, EC2, RDS, DynamoDB, Athena, and Redshift.Collaborate with...

  • Building Scalable Data Pipelines

    1 day ago

    Bellary, Karnataka, India beBeeDataEngineer Full time ₹ 12,00,000 - ₹ 20,00,000

    Cloud Data EngineerWe are seeking a highly skilled Cloud Data Engineer to join our team. In this role, you will design and build scalable data pipelines using Google Cloud Platform (GCP) services.The ideal candidate will have experience with GCP data processing technologies such as Cloud Functions, Cloud Run, Pub/Sub, BigQuery, and Cloud Storage. You will...

  • Senior Data Solutions Developer

    3 days ago

    Bellary, Karnataka, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    **Job Opportunity:** Senior Data EngineerWe are seeking a skilled and experienced data engineer to join our dynamic team. In this role, you will design, develop, and maintain scalable data pipelines and data warehouse solutions on Google Cloud Platform (GCP). You will play a critical role in ensuring the seamless flow of data across systems, enabling...

  • Scalable Data Pipeline Specialist

    2 weeks ago

    Bellary, Karnataka, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    As a Data Engineer, you will be responsible for designing and developing scalable data pipelines that fuel real-time insights.We are seeking an experienced professional with a strong understanding of data lake and warehouse design principles to join our team.You will be responsible for creating and maintaining cloud-based data lakes, warehouses, and ETL/ELT...

  • Expert Large-Scale Data Pipeline Specialist

    4 days ago

    Bellary, Karnataka, India beBeeDataEngineer Full time ₹ 10,00,000 - ₹ 25,00,000

    Data Engineering OpportunityWe are seeking a seasoned Freelance Data Engineer with expertise in designing, building, and optimizing large-scale data pipelines.This role involves developing and maintaining ETL/ELT pipelines using PySpark, AWS, and Databricks. Additionally, the successful candidate will perform data profiling, validation, and transformation...