Senior Data Pipeline Specialist
5 days ago
As a seasoned Data Engineer, you will be responsible for designing and building scalable data pipelines to support business intelligence initiatives. Your expertise in Spark, PySpark, and SQL will enable the efficient processing of large datasets.
Key Responsibilities- Design and Build Scalable Pipelines: Develop and maintain data pipelines using Spark and PySpark to deliver high-quality solutions.
- Manage Large Datasets: Process and manage large datasets with expertise in Spark, PySpark, and SQL to ensure data quality and performance.
- Real-Time Streaming Applications: Develop and deploy real-time streaming applications using Spark Streaming/Kafka to integrate AI/ML models into production.
- Collaborate with Teams: Work with data scientists and product teams to integrate AI/ML models into production and ensure seamless integration.
- Deploy on Cloud Platforms: Deploy and manage Spark workloads on cloud platforms such as AWS, Azure, GCP, and Databricks to ensure scalability and performance.
- Education: Bachelor's or Master's degree in Computer Science, Computer Engineering, or related field.
- Experience: Minimum 4 years of professional experience as a Data Engineer.
- Skills: Strong expertise in Python, Spark, PySpark, and knowledge of data ingestion tools (Kafka, Flume, Kinesis) and data formats (Parquet, Avro, ORC).
- Tools: Proficiency in Airflow/Prefect for scheduling and orchestration, and familiarity with performance tuning in Spark (partitioning, caching, broadcast joins).
- Deployment: Experience deploying on Databricks, AWS EMR, Azure HDInsight, or GCP Dataproc, and exposure to testing and CI/CD for data pipelines (pytest, Jenkins, GitHub Actions).
-
Data Pipeline Architect
4 days ago
Mangalore, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 30,00,000Senior Data Engineer">">Our company is seeking an experienced Senior Data Engineer to join our team. This role will play a critical part in building scalable data pipelines and integrating healthcare data.">
-
Senior Data Integration Specialist
2 days ago
Mangalore, Karnataka, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000Senior Data Integration SpecialistWe are seeking a highly skilled Senior Data Integration Specialist to lead the design, build, and maintenance of data pipelines across multiple sources and destinations.The ideal candidate will have strong hands-on expertise in data orchestration, transformations, and ETL/ELT processes to ensure smooth, scalable, and secure...
-
Data Pipeline Specialist
7 days ago
Mangalore, Karnataka, India beBeeDataEngineering Full time ₹ 14,10,000 - ₹ 20,30,000Data Engineer Role Summary:">We are seeking a skilled Data Engineer to join our team. As a Data Engineer, you will be responsible for designing and implementing scalable data pipelines that provide real-time insights for our maritime operations.">Key Responsibilities:">Build & Optimize Data Systems:Create and maintain data highways that fuel AI/ML...
-
Data Pipeline Architect
2 days ago
Mangalore, Karnataka, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000Senior Data EngineerAbout the Role:We are seeking a highly skilled Senior Data Engineer to join our team. As a key member of our engineering group, you will be responsible for designing, developing, and implementing large-scale data processing systems.Key Responsibilities:Design and develop scalable ETL workflows using Python and Pandas.Implement and manage...
-
Senior Data Integration Specialist
2 days ago
Mangalore, Karnataka, India beBeedataintegration Full time ₹ 18,00,000 - ₹ 25,00,000About the RoleWe are seeking a skilled Data Integration Specialist to join our team. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines using IBM DataStage, AWS Glue, and Snowflake.
-
Senior Data Pipeline Architect
17 hours ago
Mangalore, Karnataka, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000Key ResponsibilitiesWe're looking for a skilled professional to lead the development of scalable data pipelines.Design and Implement Efficient ETL ProcessesCollaborate with Data TeamsImplement Data Governance and Security StandardsOptimize Data Workflows for Scalability and PerformanceRequired Skills & QualificationsTo succeed in this role, you'll...
-
Senior Python Pipeline Architect
3 days ago
Mangalore, Karnataka, India beBeeEngineer Full time ₹ 1,80,00,000 - ₹ 2,40,00,000Senior Python Engineer PositionAbout This Role:We are seeking a highly skilled Senior Python Engineer to lead our team in developing scalable data pipelines using Apache Beam and Databricks.Key ResponsibilitiesPipeline ArchitectureCreate a shared component library/SDK for ingestion, parsing/extraction, validation, enrichment, and publishing.Define standard...
-
Data Ingestion Specialist
1 week ago
Mangalore, Karnataka, India beBeeDataIngestion Full time ₹ 1,00,00,000 - ₹ 2,00,00,000Job Title: Data Ingestion SpecialistWe are seeking a skilled data ingestion professional to join our organization. As a data ingestion specialist, you will play a critical role in designing, developing, and optimizing data ingestion pipelines for integrating multiple sources into cloud-based platforms.Key Responsibilities:
-
Cloud Data Pipeline Architect
23 hours ago
Mangalore, Karnataka, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000Job Opportunity: Data Engineer">We are seeking a skilled professional to lead our data engineering efforts.Our team requires someone who can design, develop and maintain robust data pipelines using cloud services such as GCP, AWS or Azure.">Main Responsibilities:Develop scalable data pipelines for real-time data processing.Implement data ingestion,...
-
High-Performance Data Pipeline Engineer
2 days ago
Mangalore, Karnataka, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000Job Overview">This role is focused on designing, building, and maintaining high-performance data pipelines to process large volumes of data.
