Data Engineer

4 weeks ago

Shimoga, Karnataka, India RapidBrains Full time

Job Title: Data Engineer

Experience: 5+ Years

Location: Remote

Contract Duration: Long Term

Work Time: UK Shift

Job Description

We are seeking a highly skilled Data Engineer to design and implement solutions for moving and managing large datasets (60–100GB file sizes) efficiently. The ideal candidate will have strong experience in building reliable, scalable data pipelines, developing user-friendly interfaces for data transfer, and ensuring robust error handling. This role requires excellent problem-solving skills, attention to detail, and the ability to deliver production-ready solutions with minimal downtime.

Key Responsibilities

  • Design, develop, and maintain efficient data transfer pipelines for large files (60–100GB).
  • Build a user-friendly interface to simplify dataset movement and monitoring for end-users.
  • Implement strong error handling, logging, and automated recovery mechanisms.
  • Optimize file transfer performance while ensuring data integrity and security.
  • Collaborate with stakeholders to gather requirements and deliver tailored solutions.
  • Monitor, troubleshoot, and resolve issues in production environments.
  • Maintain clear and comprehensive documentation for workflows and tools.

Required Skills & Experience

  • Proven experience in data engineering with large-scale datasets.
  • Strong proficiency in Python, Java, or other relevant programming languages.
  • Hands-on experience with data transfer protocols, APIs, and cloud storage services (AWS S3, Azure Blob, GCP Storage).
  • Experience with UI development for data tools (React, Angular, or similar frameworks).
  • Strong understanding of distributed computing, parallel processing, and data pipeline optimization.
  • Familiarity with ETL tools, workflow orchestration (Airflow, Luigi, etc.), and automation scripts.
  • Knowledge of database systems (SQL and NoSQL).
  • Excellent debugging and performance tuning skills.

Nice-to-Have Skills

  • Experience with big data technologies (Hadoop, Spark).
  • Familiarity with containerization (Docker, Kubernetes).
  • Understanding of data encryption and security best practices.

Soft Skills

  • Strong attention to detail and commitment to high-quality deliverables.
  • Ability to work independently and proactively solve problems.
  • Clear communication skills and a collaborative mindset.

  • Aws Sr. Engineer

    7 days ago

    Shimoga, Karnataka, India NTT DATA Full time

    Req ID 304645NTT DATA strives to hire exceptional innovative and passionate individuals who want to grow with us If you want to be part of an inclusive adaptable and forward-thinking organization apply now We are currently seeking a AWS Sr Engineer to join our team in Remote Karnxc4x81taka IN-KA India IN Basic Qualifications 3 years ...

  • Data Engineer

    4 hours ago

    Shimoga, Karnataka, India Bahwan CyberTek Full time

    Job Title: Data Engineer – Google Cloud Platform (GCP) Job Summary We are seeking a skilled and motivated Data Engineer with hands-on experience in building scalable data pipelines and cloud-native data solutions on Google Cloud Platform. The ideal candidate will be proficient in GCP services like Pub/Sub, Dataflow, Cloud Storage, and BigQuery, with a...

  • Data Engineer

    2 days ago

    Shimoga, Karnataka, India Talogy Full time

    Data EngineerThis opportunity is ideal for a determined and proactive individual who has a wide range of skills in a variety of database administration, reporting and dashboarding disciplines. This role requires an analytical thinker who pays significant attention to detail in their work.We are looking for an experienced data engineer with extensive...

  • Master Data Engineer

    6 days ago

    Shimoga, Karnataka, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Opportunity:We are seeking a seasoned data engineer with expertise in cloud-based services and big data technologies to craft scalable data solutions.Key Responsibilities Include:Design, develop, and maintain large-scale data pipelines using cloud-based tools like AWS and Databricks.Build and optimize ETL/ELT processes for complex structured and...

  • Cloud Data Engineer

    10 hours ago

    Shimoga, Karnataka, India beBeeData Full time US$ 80,000 - US$ 1,50,000

    Are you a skilled professional with expertise in cloud-based technologies?Azure Data Engineer is a key role that drives business transformation through deep engineering expertise and innovative solutions.Main Responsibilities:Develop mastery over AWS/Azure/SAP technologiesELT - Master the process of extracting, loading, and transforming dataData Modeling -...

  • Senior Data Engineer

    12 hours ago

    Shimoga, Karnataka, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    We are seeking a senior data engineer to join our team. As a key member of our technology hub, you will play a vital role in driving technological innovation and product development across our organization.The ideal candidate will work closely with product owners to understand business needs, collaborate with architects and data modelers to design technical...

  • Data Engineering Specialist

    1 day ago

    Shimoga, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Opportunity: Data EngineerAs a data engineer, you will be responsible for designing and developing scalable data pipelines using Spark, Snowflake, and Azure services.

  • Data Engineering Leader

    5 days ago

    Shimoga, Karnataka, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Senior Data EngineerThe role of a Senior Data Engineer involves designing, developing and maintaining data pipelines and data warehouse solutions. This position plays a critical role in ensuring the seamless flow of data across systems.This enables data-driven decision-making for our organization and clients.Key Responsibilities:Design scalable...

  • Data Engineering Leader

    1 week ago

    Shimoga, Karnataka, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    Lead Data Engineering ExpertWe are seeking an accomplished data engineering professional to spearhead our company's data initiatives.As a senior member of the team, you will be responsible for leading and mentoring junior engineers in developing, implementing, and maintaining large-scale ETL pipelines. You will utilize industry-leading tools such as...

  • Chief Data Engineer

    10 hours ago

    Shimoga, Karnataka, India beBeeDataEngineering Full time ₹ 1,20,00,000 - ₹ 2,50,00,000

    Data Engineering Professional OpportunityWe are seeking a seasoned Data Engineer to design and develop data engineering solutions using Snowflake DB, ETL tools, and data modeling techniques.The ideal candidate will have experience with data pipelines, data warehouse design, OLAP/Cube design, and Tabular Model/Cube design using Kimball Method best...