Data Engineer

8 hours ago

Mangalore, Karnataka, India R Systems Full time
Job Title: Data EngineerContract Period: 12 MonthsLocation: Offshore candidates accepted (Singapore Based Company)Work Timing : 6.30 AM to 3.30 PM or 7.00 AM to 4.00 PM (IST - India timing)ExperienceMinimum 4+ years as a Data Engineer or similar role.(Please don't apply if less than 4 years exp in Data Engineer)Proven experience in Python, Spark, and Py Spark (non-negotiable).(mandatory*)Hands-on in building ETL pipelines, real-time streaming, and data transformations.Worked with data warehouses, cloud platforms (AWS/Azure/GCP), and databases. Technical SkillsSpark Core API: RDDs, transformations/actions, DAG execution.Spark SQL: Data Frames, schema optimization, UDFs.Streaming: Structured Streaming, Kafka integration.Data Handling: S3, HDFS, JDBC, Parquet, Avro, ORC.Orchestration: Airflow/Prefect.Performance Tuning: Partitioning, caching, broadcast joins.Cloud Deployment: Databricks, AWS EMR, Azure HDInsight, GCP Dataproc.CI/CD: Pytest/unittest for Spark jobs, Jenkins, Git Hub Actions. Education & Soft SkillsBachelor's/Master's in Computer Science, Computer Engineering, or equivalent.Strong analytical, problem-solving, and communication skills.About the RoleWe are seeking an experienced Data Engineer to join our team and support data-driven initiatives.
This role involves building scalable pipelines, working with streaming data, and collaborating with data scientists and business stakeholders to deliver high-quality solutions.Key ResponsibilitiesDesign, build, and optimize data pipelines and ETL workflows.Manage and process large datasets using Spark, Py Spark, and SQL.Build and maintain real-time streaming applications with Spark Streaming/Kafka.Collaborate with data scientists and product teams to integrate AI/ML models into production.Ensure data quality, scalability, and performance in all pipelines.Deploy and manage Spark workloads on cloud platforms (AWS, Azure, GCP, Databricks).Automate testing and deployment of Spark jobs via CI/CD pipelines.RequirementsBachelor's/Master's degree in Computer Science, Computer Engineering, or related field.Minimum 4 years of professional experience as a Data Engineer.Strong expertise in Python, Spark, Py Spark.Hands-on experience with Spark SQL, Data Frames, UDFs, DAG execution.Knowledge of data ingestion tools (Kafka, Flume, Kinesis) and data formats (Parquet, Avro, ORC).Proficiency in Airflow/Prefect for scheduling and orchestration.Familiarity with performance tuning in Spark (partitioning, caching, broadcast joins).Experience deploying on Databricks, AWS EMR, Azure HDInsight, or GCP Dataproc.Exposure to testing and CI/CD for data pipelines (pytest, Jenkins, Git Hub Actions).BudgetOnshore: As per market standards.

  • Data Engineer

    3 days ago

    Mangalore, Karnataka, India R Systems Full time

    Job Title: Data EngineerContract Period: 12 MonthsLocation: Offshore candidates accepted (Singapore Based Company)Work Timing : 6.30 AM to 3.30 PM or 7.00 AM to 4.00 PM (IST - India timing)ExperienceMinimum 4+ years as a Data Engineer or similar role.(Please don't apply if less than 4 years exp in Data Engineer)Proven experience in Python, Spark, and PySpark...

  • Data Engineer Opportunity

    4 days ago

    Mangalore, Karnataka, India beBeeData Full time ₹ 1,25,00,000 - ₹ 2,25,00,000

    Job SummaryWe are seeking a highly skilled data engineer to join our team. As a key member of our engineering team, you will be responsible for designing, developing and implementing large-scale data solutions.This role involves working with various technologies such as distributed computing frameworks, data quality tools and scalable architectures to design...

  • Cloud Data Engineer

    4 days ago

    Mangalore, Karnataka, India beBeeData Full time US$ 70,000 - US$ 1,00,000

    Key Data Engineer RoleThe ideal candidate will have experience with cloud-based technologies, including GCP, Python, and ETL tools. They should be able to design, develop, and maintain large-scale data processing systems.Responsibilities include:Designing and developing data pipelines using GCP services such as BigQuery, Cloud Storage, and Cloud...

  • Senior Data Engineer

    6 days ago

    Mangalore, Karnataka, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Data Engineering LeaderJob DescriptionThe successful candidate will lead the design, development, and maintenance of scalable, secure, and high-performance data solutions on Databricks.He/She will collaborate with business stakeholders, data scientists, and analysts to gather requirements and deliver actionable data solutions.This role requires a...

  • Data Engineering Lead

    6 hours ago

    Mangalore, Karnataka, India beBeeDataEngineering Full time ₹ 8,00,000 - ₹ 19,00,000

    Data Engineering LeadDesigning and implementing data pipelines using Airflow and Python for large-scale data processing systems.Developing and maintaining large-scale data processing systems in Snowflake, ensuring high-quality data delivery and scalability.Maintaining open communication with cross-functional teams to ensure alignment on data requirements and...

  • Data Engineering Specialist

    6 hours ago

    Mangalore, Karnataka, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Data Engineering SpecialistAbout the Role:We are seeking an experienced Data Engineering Specialist to design, develop and maintain high-performance data pipelines that support our business intelligence solutions.The ideal candidate will collaborate with cross-functional teams to architect and implement data models, ETL processes and data...

  • Mastering Data Engineering

    2 days ago

    Mangalore, Karnataka, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Senior Data EngineerWe are seeking a skilled and experienced GCP data engineer to join our dynamic team. In this role, you will design, develop, and maintain data pipelines and data warehouse solutions on Google Cloud Platform (GCP). You will play a critical role in ensuring the seamless flow of data across systems, enabling data-driven...

  • Data Engineer

    6 days ago

    Mangalore, Karnataka, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Seeking an accomplished Data Engineer to craft, build and oversee our data infrastructure and pipelines.This role is perfect for someone who has a proven track record of delivering multiple projects using GCP-Big Query, Data Proc technologies.The ideal candidate will be a collaborative team player with excellent problem-solving skills, strong communication...

  • Data Engineer Architect

    2 days ago

    Mangalore, Karnataka, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    We're seeking a seasoned professional with expertise in Data Engineering and Generative AI.The ideal candidate will have 12–20 years of experience in data engineering, with at least 3 years (last 2 years acceptable) of hands-on work in Generative AI.This role will involve collaborating with Presales Managers to design technical solutions, leading client...

  • Leading Data Engineering Solutions

    4 days ago

    Mangalore, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 30,00,000

    Job Title: Senior Data Engineer is a pivotal role in our data engineering team.As a seasoned and proficient Senior Python Data Engineer, you will play a crucial role in designing, implementing, and optimizing data pipelines ensuring seamless integration with cloud platforms. You will work closely with data scientists, analysts, and other teams to understand...