Lead Data Engineer

4 weeks ago

Bangalore, Karnataka, India Ally-eXecutive.com Full time

Brief Description of Role :

- Develop solutions for the company that will deliver high quality personalized recommendations across different channels to our customers

- Working with Data science team to ensure seamless integration and support of machine learning models.

- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS 'big data' technologies.

- Develop end-to-end (Data/Dev/MLOps) pipelines based on in-depth understanding of cloud platforms, AI/ML lifecycle, and business problems to ensure solutions are delivered efficiently and sustainably.

- Collaborate with other members of the team to ensure high quality deliverables

- Learning and implementing the latest design patterns in software engineering

Qualifications :


Data Management :


- Experience with both structured and unstructured data, and Hadoop, Apache Spark, or similar technologies

- Good understanding of Data Modeling, Data Warehouse, Data Catalog concepts and tools

- Experience with Data Lake architectures, and with combining structured and unstructured data into unified representations

- Able to identify, join, explore, and examine data from multiple disparate sources and formats

- Ability to reduce large quantities of unstructured or formless data and get it into a form in which it can be analyzed

- Ability to deal with data imperfections such as missing values, outliers, inconsistent formatting, etc.

- Ability to manipulate large datasets, (millions of rows, thousands of variables)

Software Development :


- Ability to write code in programming languages such as Python/NodeJs, PySpark and shell script on Linux

- Familiarity with software development methodology such as Agile/Scrum

- Love to learn new technologies, keep abreast of the latest technologies within the cloud architecture, and drive your organization to adapt to emerging best practices

Architecture and Infrastructure :


- Architectural design experience on AWS/Azure.

- Architectural design for application with high transactional volume

- Experience in delivering software with AWS EC2, S3, EMR/Glue, Lambda, Data Pipeline, CloudFormation, Redshift/Snowflake etc.

- Experience in Azure Data Factory, Azure data bricks, Azure Data stage

- Good knowledge of working in UNIX/LINUX systems

- Experience designing and building large scale enterprise systems

Qualifications :


- Bachelor's Degree in Engineering and related field with 10+ years of similar experience

- Tech Stack : Python or NodeJS, PySpark, Micro services, Docker, Serverless Frameworks & Databricks.

- Hands on experience building ETL workflows/data pipelines

- Experience in relational and non-relational databases and SQL (NoSQL is a plus).

- Experience with Cloud technologies (AWS or Azure)

- Experience in Designing and building API's for high transactional volume

- Experience building Data and CI/CD/MLOps pipelines

- Familiarity with Airflow and MLFlow tools

- Familiarity with automated unit/integration test frameworks

- Experience working on AdTech or MarTech technologies is added advantage

- Knowledge of machine learning algorithms and concepts and implementation will be a plus

- Good written and spoken communication skills, team player.

- Strong analytic thought process and ability to interpret findings

In addition, the candidate should have strong business acumen, and interpersonal and communication skills, yet also be able to work independently. He/she should be able to communicate findings and the way techniques work in a manner that all stakeholders, both technical and non-technical, will understand.

(ref:hirist.tech)

  • Solution Engineer

    1 month ago

    Bangalore, Karnataka, India HEVO DATA Full time

    Overview :. We're seeking a talented Solutions Engineer with strong background in Data Domain to join our growing Sales team. You'll work one-on-one with the sales team to help discover our customers' business and technical needs and demonstrate how Hevo's platform can solve their real-time data replication needs. In this role, you will have...

  • Lead Data Engineer

    1 month ago

    Bangalore, Karnataka, India Sentienz Full time

    Job Description : As a Lead Data Engineer, you will play a pivotal role in driving our data-driven initiatives. You will be responsible for designing, developing, and maintaining robust data engineering solutions that support our analytics and decision-making processes.Responsibilities :- Lead and mentor a team of data engineers, providing technical guidance...

  • Technical Lead

    1 month ago

    Bangalore, Karnataka, India FxConsulting Full time

    Job Title : Lead Data EngineerWe are seeking a highly skilled Lead Data Engineer with expertise in Big Data technologies to join our dynamic team. As a Lead Data Engineer, you will play a pivotal role in designing, building, and optimizing large-scale data pipelines and systems to drive business insights and innovation.Key Responsibilities :1. Design,...

  • MakeMyTrip - Lead Data Engineer

    1 month ago

    Bangalore, Karnataka, India MakeMyTrip Full time

    Our main focus is on Scala , Spark , Spark Streaming .MakeMyTrip - Lead Data EngineerAbout the Role :MakeMyTrip, India's leading online travel company, is searching for a seasoned Lead Data Engineer to join our Data Platform Team. You will play a pivotal role in architecting, developing, and managing our data infrastructure and pipelines, empowering...

  • Technical Lead

    1 month ago

    Bangalore, Karnataka, India HireVeda Full time

    HireVeda has been mandated to hire a Technical Lead (Data Engineer) for a B2B SaaS Pioneer, backed by renowned Investors, which is revolutionizing SaaS : - Proven experience of 5-9 years in data engineering, including data pipeline design, ETL processes, data warehousing, and managing data at the terabyte scale - Minimum 4 years of experience in Product...

  • Lead Data Engineer

    1 month ago

    Bangalore, Karnataka, India Rigel networks Full time

    Job Title : Lead Data EngineerLocation : BangaloreNotice Period : Immediate - 15 DaysAbout the Role :We are seeking a highly skilled Lead Data Engineer with extensive experience in data engineering and a strong background in Big Data technologies. The ideal candidate will have 7+ years of relevant experience, with expertise in Python, SQL, and AWS, and a...

  • Software Development Engineer II

    1 month ago

    Bangalore, Karnataka, India HEVO DATA Full time

    About Hevo : Ever wondered, what's common between Zepto, Napster, Curefit, Groww, Jumio, Cars24 ? They all super-charge their business growth with Hevo Data Pipelines. How? 2 of every 3 modern day companies are unable to take data driven decision making due to fragmented data across 100s business software. Hevo is a simple, intuitive, and powerful...

  • Lead Data Engineer

    1 month ago

    Bangalore, Karnataka, India Rigel Networks Pvt Ltd Full time

    Lead Data Engineer :Top Skills :- Python- SQL- AWS- Spark- 7+ years relevant work experience in the Data Engineering field- 5+ years of experience working with Hadoop and Big Data processing frameworks (Hadoop, Spark, Hive, Flink, Airflow etc.)- 5+ years of Strong experience with relational SQL and at least one programming language such as Python, Scala, or...

  • Lead Data Engineer

    1 month ago

    Bangalore, Karnataka, India TheBriminc Full time

    Position : Senior Data Engineer/ Lead Data Engieer. Location : Bangalore. Experience Required:.- Minimum 5+ years' Experience in Data Engineering.- Must have good knowledge and experience in Python.- Must have good Knowledge of Pyspark.- Mush have good Knowledge of Databricks.- Must have good experience in AWS (Glue, Athena, Redshift, EMR).- Typically...

  • Perfios Software Solutions

    1 month ago

    Bangalore, Karnataka, India PERFIOS SOFTWARE SOLUTIONS PRIVATE LIMITED Full time

    Job Description : Tech Lead Data EngineerPosition Overview : We are looking for a seasoned Tech Lead Data Engineer with a robust technical background and leadership experience to spearhead our data engineering efforts. This role demands a deep expertise in Python, PySpark, and Shell scripting, along with a comprehensive understanding of AWS cloud services....

  • Head - Data Engineering

    4 weeks ago

    Bangalore, Karnataka, India Merck Group Full time

    Job Description : Join us to contribute to the org's enterprise priority of driving innovation through data & digital. In R&D, the promise of bringing more medicines to more patients faster is closely linked with the ability to maximize the use of our data for improved decision making using data analytics & AI. For this, having high-quality data readily...

  • Big Data Engineer/Lead

    4 weeks ago

    Bangalore, Karnataka, India TekPillar Full time

    Join our dynamic team in Bangalore as a Big Data Engineer contribute to our key data engineering initiatives! You will play a crucial role in managing and prioritizing projects while driving strategies and executing data engineering solutions.Job Title : Big Data Engineer/ LeadExperience : 4 to 9 YearsLocation : BangaloreKey Responsibilities : - Collaborate...

  • Data Engineer

    1 month ago

    Bangalore, Karnataka, India Coders Brain Technology Private Limited Full time

    Role : Data Engineer (no one over 10 years of experience will be considered)Qualifications :- Bachelor's degree in Computer Science/Engineering or similar- 5 to 7 years of data engineering experience with focus on designing and implementing scalable big data transformations and pipelines- Proficiency with Cloud Platforms like Azure(preferred), AWS,...

  • Senior Data Engineering

    1 month ago

    Bangalore, Karnataka, India FxConsulting Full time

    Job Title : Lead Data EngineerWe are seeking a highly skilled Lead Data Engineer with expertise in Big Data technologies to join our dynamic team. As a Lead Data Engineer, you will play a pivotal role in designing, building, and optimizing large-scale data pipelines and systems to drive business insights and innovation.Key Responsibilities : - Design,...

  • KredX - Senior/Lead Data Integration Engineer

    1 month ago

    Bangalore, Karnataka, India Kredx Finance Private Limited Full time

    Job Description - Senior/Lead Data Integration EngineerPosition - Senior/Lead Data Integration EngineerExperience - 4-7 yearsWork Location - BangaloreCompany Overview :KredX is a supply chain finance powerhouse which emerged in India, and now has an evolving global footprint. With a new-age plug-and-play ecosystem of working capital finance and cash...

  • Infiniti Research

    1 month ago

    Bangalore, Karnataka, India Infiniti Research Limited Full time

    Company : Quantzig Analytics. Location : Kadubeesanahalli, Bangalore. Experience : 6+ years minimum is required. Notice : Preferably 0 to 30days. Job Description : - Seeking a self-driven professional with a strong ownership mentality and the ability to work independently without constant guidance. - The ideal candidate will have an agile mindset, capable...

  • MLOps Lead

    1 month ago

    Bangalore, Karnataka, India Selsoftinc Full time

    Job Title : ML Ops leadExperience Level : 8 - 10 yearsLocation : BangaloreJob Summary : We are seeking a skilled Machine Learning Operations, or MLOps Lead who would be responsible for streamlining the process of deploying and maintaining machine learning (ML) models and oversee unification of ML system development (Dev) and ML system deployment and...

  • Data Engineer

    1 month ago

    Bangalore, Karnataka, India Coders Brain Technology Private Limited Full time

    Qualifications : - Bachelor's degree in Computer Science/Engineering or similar - 5-7 years of data engineering experience with focus on designing and implementing scalable big data transformations and pipelines - Proficiency with Cloud Platforms like Azure(preferred), AWS, and/or Google cloud, including knowledge of core services like Azure: Data...

  • Cloudcover - Engineering Manager - Data Engineering & Data Science

    4 weeks ago

    Bangalore, Karnataka, India Cloudcover Consultancy Private Limited Full time

    Job brief : We are seeking a highly skilled and experienced Engineering Manager Data Insights to join our team who has at least 10 years of experience in data engineering and data science, with significant experience in managing teams and delivering large scale projects. This role will be pivotal in driving the strategy, design, and implementation of our...

  • Senior Data Engineer

    1 month ago

    Bangalore, Karnataka, India MonoSpear Technologies Pvt Ltd Full time

    Senior Data Engineer :- 9+ years' work experience and bachelor's degree in Information Science, Computer Science, Mathematics, Statistics or a quantitative discipline in science, business, or social science.- Hands-on engineer who is curious about technology, should be able to quickly adopt to change and one who understands the technologies...