Senior Data Engineer

3 weeks ago


Bangalore, Karnataka, India TecHHire Global Full time

Job Description :


We are looking for a Senior Data engineer-GCP who can design and build production data pipelines from ingestion to consumption within a big data architecture using Python, Scala, PySpark, Java implementing data engineering, ingestion, and curation functions on GCP :

  • Must be able to write quality code and build secure, highly available systems.
  • Assemble large, complex data sets that meet functional / non-functional business requirements.
  • Identify, design, and implement internal process improvements: automating manual processes, optimising data delivery, re-designing infrastructure for greater scalability, etc with the guidance.
  • Create data tools for analytics and data scientist team members that assist them in building and optimising our product into an innovative industry leader.
  • Monitoring performance and advising any necessary infrastructure changes.
  • Defining data retention policies.
  • Implementing the ETL process and optimal data pipeline architecture
  • Build analytic tools that utilise the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.
  • Create design documents that describe the functionality, capacity, architecture, and process.
  • Develop, test, and implement data solutions based on finalised design documents.
  • Work with data and analytics experts to strive for greater functionality in our data systems.
  • Proactively identify potential production issues and recommend and implement solutions

Requirements :

  • Should have at least 2 years of relevant experience as GCP Data Engineer
  • Educational Qualifications - Bachelor's/Master's
  • Good understanding of optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and GCP big data' technologies
  • Strong in SQL and good understanding and experience in writing stored procedures
  • Good experience in GCP
  • Good experience in Hadoop, Big data
  • Proficient understanding of distributed computing principles
  • Experience in working with batch processing/ real-time systems using various open-source technologies like NoSQL, Spark, Pig, Hive, Apache Airflow.
  • Implemented complex projects dealing with the considerable data size (PB).
  • Optimisation techniques (performance, scalability, monitoring, etc.)
  • Experience with integration of data from multiple data sources preferably AWS
  • Experience with NoSQL databases, such as HBase, Cassandra, MongoDB, etc.
  • Knowledge of various ETL techniques and frameworks, such as Flume
  • Experience with various messaging systems, such as Kafka or RabbitMQ
  • Good understanding of Lambda Architecture, along with its advantages and drawbacks
  • Creation of DAGs for data engineering
  • Expert at Python /Scala programming, especially for data engineering/ ETL purposes
(ref:hirist.tech)
  • Senior Data Engineer

    3 weeks ago


    Bangalore, Karnataka, India GLAXOSMITHKLINE ASIA PVT. LTD Full time

    A new world-leading consumer health company. Shaped by all who join us. Together, we're improving everyday health for billions of people. By growing and innovating our global portfolio of category-leading brands including Sensodyne, Panadol, Advil, Voltaren, Theraflu, Otrivin, and Centrum through a unique combination of deep human understanding and...


  • Bangalore, Karnataka, India PROVAB TECHNOSOFT PVT LTD Full time

    Position : Senior Machine Learning Engineer Experience : 6+ YearsNotice period : Immediate joinerLocation : Bengaluru (Hybrid) Job Summary :We are looking for an experienced Senior Machine Learning Engineer/Data Scientist with a proven track record in building and deploying machine learning models using NVIDIA technologies. Specifically, we value expertise...

  • Senior Data Architect

    1 month ago


    Bangalore, Karnataka, India HuQuo Consulting Pvt. Ltd. Full time

    Job Title : Senior Data ArchitectExperience : 10-15 yearsJob Description :We are seeking a highly skilled and experienced Senior Data Architect to join our dynamic team. The ideal candidate will play a crucial role in designing and implementing robust data solutions for our clients on cloud and on-premises platforms. As a Senior Data Architect, you will be...

  • Data Engineer

    2 weeks ago


    Bangalore, Karnataka, India WIZSTAFFING PRIVATE LIMITED Full time

    Job Description :We at Captain Fresh, are building smart supply chain to deliver the highest quality seafood and meat for the Indian consumer.Our innovations in process management and workforce orchestration along with strong industry credentials are enabling us to deliver the fastest harvest-to-fork in the industry. Our endeavor is to leverage experience...

  • Data Engineer

    1 month ago


    Bangalore, Karnataka, India D Square Consulting Service Full time

    Role : Sr. Data Engineer DOMOExperience : 5+ YearsLocation : Bangalore- Candidates who can join within 30 days, may only applyWorking Mode : Work From OfficeJob Highlights :Must-Have skills :- 5+ years' experience in ELT systems (Snowflake and DOMO)- Must have experience in digital marketing projects.- Strong knowledge of digital marketing toolsPreferred...

  • Python Developer

    1 month ago


    Bangalore, Karnataka, India IT Full time

    Job Overview :We are looking for a seasoned Python Developer with over 10 years of experience in IT, with a focus on data engineering. The ideal candidate will have extensive knowledge of Python, Pyspark, Azure Databricks, ADF, SQL, and data warehouse concepts. This role requires strong technical skills, project management capabilities, and excellent...


  • Bangalore, Karnataka, India Indium Software Full time

    Title : Senior/Lead Automation Test Engineer. Experience : 6-12 Years. Work Location : Bangalore. Work Mode : Work from Office. Requirements : - Good in communication.- Strong experience in Automation Testing + Functional Testing.- Experience in Java Selenium - Page Object Model (POM) + BDD Cucumber Framework.- Experience in DB Testing , Writing Advanced...


  • Bangalore, Karnataka, India Freelancer Recruiter Full time

    Desired Skills & Qualifications :- Deep understanding of algorithms and data structures.- Should have worked with MVC frameworks, web / client based server architecture, structured and unstructured databases.- Ability to translate product spec to engineering actionables.- Ability to navigate and structure code bases.- Rudimentary skills around service...

  • Senior Data Analyst

    2 weeks ago


    Bangalore, Karnataka, India Yitro business consultants Full time

    Job Description:Powerful predictors of the future, our Analytics team blends data science and business intelligence to enable extra foresight for our platform offerings. They connect insights to results for our clients by working closely with senior stakeholders, staying in tune with market trends, and with the power of our big data assets. By leveraging new...

  • Juego Studios

    3 weeks ago


    Bangalore, Karnataka, India Juego Studios Full time

    As a Python Data Engineer for Test at Client you will be at the forefront of ensuring the quality and reliability of our data systems. You will play a critical role in analyzing raw data, building test frameworks, and validating data products using Python. Collaborating closely with data analytics experts and stakeholders, you will contribute to the...


  • Bangalore, Karnataka, India Pluralsight Full time

    Job Description : Working at Pluralsight Founded in 2004 and trusted by Fortune 500 companies, Pluralsight is the technology skills platform organizations and individuals in 150+ countries count on to create progress for the world. Our platform helps technologists master their craft and take control of their careers. We empower businesses everywhere to build...


  • Bangalore, Karnataka, India THE RRKS Full time

    Immediate Joiner : Experience : Only 3 years to 5 years. Location : Bengaluru. Budget : 10 LPA to 15 LPA.Job Description :- Experience on Handling Implementation on Netapp Storage - FAS Series, AFF Series.- Data Migration in Netapp.- Upgradation of Netapp.- Netapp storage mapping to servers.- File server creation.Candidate should have knowledge on different...

  • Akaike Technologies

    2 weeks ago


    Bangalore, Karnataka, India Akaike Technologies Full time

    Job Description:Hey, you data rockstars and number nerds! If you're tired of swimming in the same old data pool and want to dive into the craziest data adventure this side of the digital universe, then welcome to Akaike! We're not just an AI company; we're the trendsetters, the disruptors, and the mischief-makers of the AI world. So, strap on...


  • Bangalore, Karnataka, India Freelancer Recruiter Full time

    Mandatory requirements:- 1-2 yrs managing an engineering team- Tech skills : AWS, Android, React, MySQL, MongoDB- Java is only a preferred skill- Techno-managerial exp- Only from product organisationsResponsibilities :- Hands-on coding experience- Directly works with the founding team- Involved in Strategy, prioritisation and Product Planning: Direct,...

  • Data Engineer

    3 weeks ago


    Bangalore/Hyderabad, Karnataka, India Indium Software Full time

    Job Description:We are seeking a talented and motivated Data Engineer to join our team. The ideal candidate should have a strong background in SQL, Python, ETL, and data visualization, with experience working closely with stakeholders and providing data support to Data Analysts. As a Data Engineer, you will play a key role in developing data pipelines,...


  • Bangalore, Karnataka, India BOOTLABS TECHNOLOGIES PRIVATE LIMITED Full time

    Responsibility :- At least 4 years of experience in AWS technologiesThe documented ability of delivering innovative solutions in an applied environment related to at least one of the following areas :- ETL, Data pipeline, Workflow- Strong knowledge of programming in Python, Java, or Scala.- High level of competence in global and cross-skilled collaborative...

  • Engineering Manager

    1 month ago


    Bangalore, Karnataka, India Freelancer Recruiter Full time

    Mandatory requirements :- Apache Kafka- Elastic search- Spark Spring- NoSql- Swift- Distributed systems and Java (preferred)- Only from product organisationsResponsibilities : - Manage software development projects by setting requirements, goals and timelines. - Create clear career paths for team members and help them grow with regular & deep mentoring....


  • Bangalore, Karnataka, India BOOTLABS TECHNOLOGIES PRIVATE LIMITED Full time

    Job Description :- At least 3 years of experience in AWS technologies- The documented ability of delivering innovative solutions in an applied environment related to at least one of the following areas : ETL, Data pipeline, Workflow- Strong knowledge of programming in Python, Java, or Scala.- High level of competence in global and cross-skilled collaborative...

  • Principal Engineer

    1 month ago


    Bangalore, Karnataka, India XANDER CONSULTING AND ADVISORY PRIVATE LIMITED Full time

    What Success Will Look Like :- Design, build and maintain the stream processing, time-series analysis system which is at the heart of Data Platform's products- Responsible for architecture of the platform- Build features, enhancements, new services and bug fixing in Scala and Rust on a Jenkins based pipeline to be deployed as Docker containers on...


  • Bangalore, Karnataka, India Snapmint Full time

    Job Description :Senior Product Analyst @ Snapmint Location: Bangalore Marathali office.Snapmint is on a mission of democratizing installment purchases for next 350 Mn Indians. Of the 300 million credit eligible consumers in India, less than 30 Million actively use credit cards. Snapmint is reinventing credit for the next 200M consumers by providing them...