Big Data Engineering Lead

7 days ago


Hyderabad, Telangana, India Enable Data Incorporated Full time

Key Responsibilities

  • Design and develop scalable and robust data solutions in the cloud using Apache Spark and Databricks.
  • Gather and analyze data requirements from business stakeholders and identify opportunities for data-driven insights.
  • Build and optimize data pipelines for data ingestion, processing, and integration using Spark and Databricks.
  • Ensure data quality, integrity, and security throughout all stages of the data lifecycle.
  • Collaborate with cross-functional teams to design and implement data models, schemas, and storage solutions.
  • Optimize data processing and analytics performance by tuning Spark jobs and leveraging Databricks features.
  • Provide technical guidance and expertise to junior data engineers and developers.
  • Stay up-to-date with emerging trends and technologies in cloud computing, big data, and data engineering.
  • Contribute to the continuous improvement of data engineering processes, tools, and best practices.

Requirements

  • Bachelor's or Master's degree in computer science, engineering, or a related field.
  • 10+ years of experience as a Data Engineer, Software Engineer, or similar role, with a focus on building cloud-based data solutions.
  • Strong experience with cloud platforms such as Azure or AWS.
  • Proficiency in Apache Spark and Databricks for large-scale data processing and analytics.
  • Experience in designing and implementing data processing pipelines using Spark and Databricks.
  • Strong knowledge of SQL and experience with relational and NoSQL databases.
  • Experience with data integration and ETL processes using tools like Apache Airflow or cloud-native orchestration services.
  • Good understanding of data modeling and schema design principles.
  • Experience with data governance and compliance frameworks.
  • Excellent problem-solving and troubleshooting skills.
  • Strong communication and collaboration skills to work effectively in a cross-functional team.
  • Relevant certifications in cloud platforms, Spark, or Databricks are a plus.


  • Hyderabad, Telangana, India data Remote Work Freelance Full time

    What can you tell your friends when they ask you what you do? We're looking for an experienced Big Data Engineer who can create innovative new products in the analytics and data space. You will participate in the development that creates the world's #1 mobile app analytics service. Together with the team, you will build out new product features and...


  • Hyderabad, Telangana, India data Remote Work Freelance Full time

    What can you tell your friends when they ask you what you do? "As a Data Analyst at data.ai, I help analyze, monitor, and deliver comprehensive and high-quality app market data to app professionals. I am empowered to work independently with the support of team members spread across APAC, AMER, and EMEA. I am the key contributor to the new and existing...

  • Big Data Engineer

    2 days ago


    Hyderabad, Telangana, India SMARTWORK IT SERVICES Full time

    Job Title : Big Data Engineer (Dynatrace)Location : Chennai, Bangalore, HyderabadExperience : 6-10 YearsRequired skills : Big Data, DynatraceJob Description : We are looking for an experienced Big Data Engineer (Dynatrace) with building scalable data pipelines and ensuring performance optimization through Dynatrace monitoring. This role requires a seasoned...


  • Hyderabad, Telangana, India Micron Semiconductor Asia Operations Pte. Ltd Full time

    We are looking for leaders with strong technical experience in Big Data and Cloud Data warehouse technologies. This role will work primarily in Cloud data warehouse like Snowflake and GCP platforms, monitoring solutions such as Splunk and automation and machine learning using Python.Key Skills and Qualifications:10+ years developing, delivering, and/or...


  • Hyderabad, Telangana, India Incedo Inc. Full time

    Join Incedo Inc.'s innovative team as a Sr. Lead Data Engineer – Big Data. This role requires a seasoned professional with a strong background in big data technologies and software development.We're looking for someone who can architect and implement robust data pipelines using CDH On-premise, Spark, and other cutting-edge tools. The ideal candidate will...


  • Hyderabad, Telangana, India Talescope Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Talescope. As a key member of our engineering team, you will be responsible for leading the design and implementation of complex, scalable data pipelines using AWS services. You will work closely with cross-functional teams to understand their data needs and translate them...

  • Big data engineer

    2 weeks ago


    Hyderabad, Telangana, India Virtusa Full time

    Big data engineer - CREQ189284 Description Position: Big data engineer Primary: big data concepts, aws, pyspark Location: HYD Create Trigger based automation framework for Data Migration Identify roles/access needed for data migration from federated bucket to managed bucket and Build APIs for the same Integrate CDMS framework with Lake and Data bridge...


  • Hyderabad, Telangana, India Tata Consultancy Services Full time

    About the Role:We are seeking an experienced GCP Data Engineer to lead our big data engineering efforts. The successful candidate will design, develop, and deploy scalable data processing systems on GCP, leveraging technologies such as Dataflow, Bigtable, and Cloud Storage.You will work closely with cross-functional teams to deliver high-quality solutions...


  • Hyderabad, Telangana, India Incedo Inc. Full time

    Job DescriptionSr. Lead Data Engineer – Big DataOverviewIncedo Inc. is a leading company in the data analytics space, and we are seeking a highly skilled Sr. Lead Data Engineer to lead our big data engineering efforts.Key ResponsibilitiesLeverage strong technical, analytical, and problem-solving skills to design and develop large-scale data...

  • Big Data Engineer

    1 week ago


    Hyderabad, Telangana, India Questsphere Private ltd. Full time

    Our client is looking to deploy big data engineer at one of the worlds top product company R&D centre based out of Hyderabad. As a Big Data Engineer, you are going to work with cross-functional collaboration and communication with members from other teams.As an Engineer, you are responsible to analyze large data sets to identify errors . You are responsible...


  • Hyderabad, Telangana, India Live Connections Full time

    Job DescriptionWe are seeking a seasoned Data Engineering Manager with extensive experience in Python and Big Data technologies to lead our analytics team in Hyderabad. As a key member of our organization, you will be responsible for driving data strategies that empower our business intelligence capabilities and contribute to data-driven decision-making...

  • Big Data Engineer

    4 days ago


    Hyderabad, Telangana, India Apollo HealthAxis Full time

    Job DescriptionAs a Big Data Engineer at Apollo HealthAxis, you will be responsible for designing and implementing large-scale data processing systems using distributed computing technologies. You will work closely with cross-functional teams to understand business requirements and translate them into effective data architecture and...


  • Hyderabad, Telangana, India Techwave Consulting Inc Full time

    Job DescriptionWe're seeking a Data Engineer to support AI & Machine Learning initiatives by building and optimizing data pipelines, databases, and data processing frameworks. The ideal candidate will have strong expertise in SQL, MySQL, Azure SQL, NoSQL like MongoDB & Cassandra, along with experience in big data, cloud computing, and AI-driven data...


  • Hyderabad, Telangana, India PRI Global Full time

    About the RolePRI Global is looking for an experienced Senior Big Data Engineer to lead our big data initiatives. In this role, you will be responsible for designing and implementing large-scale data architectures and systems.Develop and Implement Data ArchitecturesCollaborate with Engineering TeamsEvaluate System PerformanceTroubleshoot Complex IssuesKey...

  • Big Data Engineer

    3 weeks ago


    Hyderabad, Telangana, India Folks Palette LLP Full time

    Job Description : We are seeking an experienced Big Data Engineer with PySpark Developer to join our dynamic team. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining scalable data solutions in a big data ecosystem. The ideal candidate will have a strong background in big data technologies,...


  • Hyderabad, Telangana, India ValueLabs Full time

    Company OverviewValueLabs is a leading data engineering and analytics company that delivers innovative solutions to its clients. We are passionate about leveraging big data technologies to drive business growth and improvement.


  • Hyderabad, Telangana, India Enable Data Incorporated Full time

    We are looking for a Senior Data Engineer to join our team at Enable Data Incorporated. Our company is a leading provider of advanced application, data, and cloud engineering services that leverage modern solutions to drive increased value across our clients' business ecosystem.In this role, you will design, develop, and maintain scalable and efficient data...


  • Hyderabad, Telangana, India Info Services Full time

    Job DescriptionWe are looking for a Senior Big Data Engineer to join our team at Info Services. As a key member of our engineering team, you will be responsible for designing, developing, and maintaining large-scale data pipelines that span across Airflow, Databricks, and Snowflake.You will work closely with our cross-functional teams to understand business...


  • Hyderabad, Telangana, India Incedo Inc. Full time

    Incedo Inc. is seeking a highly skilled Data Engineer for Big Data to join our team. The ideal candidate will have strong technical, analytical, and problem-solving skills.We are looking for someone with experience in designing and building big data pipelines using CDH On-premise, Spark, PySpark, Scala, Python, NiFi, Hive, and NoSql DBs.The candidate should...

  • Big Data Engineer

    2 weeks ago


    Hyderabad, Telangana, India TechSophy Full time

    Job Description: Roles&Responsibilities : 1. As an Engineer, you are responsible to analyze large data sets to identify errors in the map. 2. As an Engineer, you are responsible to design and implement a complex algorithm for resolving the issue, review the solution with a team of engineers and analysts, and integrate the resulting solution into the...