Big Data Engineer

1 week ago

Pune, Maharashtra, India Pixeldust Technologies Full time

Job Descriptions: We are seeking an experienced and highly skilled Data Engineer to join our dynamic team. The ideal candidate will be proficient in designing, building, and operationalizing large-scale data processing systems. This role requires expertise in a wide range of technologies and methodologies, with a strong emphasis on delivering scalable and robust data solutions. Key Responsibilities: Design, develop and implement data solutions using Cloudera technologies such as Hadoop, Spark and Hive Collaborate with Data Engineers to optimize data pipelines and data processing workflows. Work closely with Data Analysts and Data Scientists to ensure data quality and integrity. Troubleshoot and resolve issues with data processing and data storage systems. Stay up-to-date on the latest trends and best practises in the Cloudera development. Participate in code reviews and provide feedback to team members. Implement expert-level coding in SQL, Python, Scala and Spark to manage complex data transformation and processing tasks. Utilize Hive/Impala and NoSQL databases to support application development and data analytics. Use technologies like GCP dataflow/Apache Beam to create robust data pipelines. Utilize streaming data platforms like Kafka or Spark Streaming for real-time data processing. Engage in data modeling to understand and represent data relationships, flows, and structures effectively. Employ continuous integration and deployment (CI-CD) practices and Git for version control to ensure code quality and efficiency. Mandatory Skills: Proficiency in working with Spark . Understanding of Spark's architecture and fault tolerance mechanisms. Proficiency in using Spark DataFrames and Spark SQL for querying structured data. Experience in optimizing Spark execution plan is a plus. Skills in performing Extract, Transform, and Load (ETL) processes using Spark. Experience with integrating Spark Streaming with other technologies like Kafka is an advantage. Familiarity with the Hadoop ecosystem, including tools such as HDFS, Hive (Cloudera stack can be of advantage. Experience with deploying and managing Spark applications on a Hadoop cluster or GCP Dataproc. Strong knowledge of Python and experience with Java. Strong knowledge of DevOps tools and practices (CI/CD, Docker). Hands-on experience in GCP services: Dataproc, Cloud Function, Cloud Run, Pub/Sub, BigQuery. Good to Have: Experience as a Big Data Engineer (6 or more years). Experience with containerisation technologies such as Docker and Kubernetes. Background in the Network or NATCO/TELCO domain. Bigdata Certifications. Expertise in: SQL, Python, Spark, Hive/Impala, NoSQL databases, Oozie/Airflow/Cloud Composer, CI-CD/Git, Cloud technologies (GCP preferred), Data Analysis, Kafka/Spark Streaming. Proven experience as an Cloudera Developer or similar roles. Solid understanding of Cloudera technologies such as Hadoop, Spark and hive. Experience with data modelling, data warehousing and data integration. Strong programming skills like Java, Python and Scala.

  • Big Data Engineer

    2 weeks ago

    Pune, Maharashtra, India Kapstag Workforce Solutions Private Limited Full time

    Job DescriptionDescriptionWe are seeking a highly skilled Big Data Engineer to join our team. The ideal candidate should have 4-9 years of experience in the field of big data engineering and should be able to design, develop, and maintain large-scale data processing systems. The candidate should also have a strong understanding of data architecture and data...

  • Big Data Engineer

    4 days ago

    Pune, Maharashtra, India ACL Digital Full time

    Role SummaryWe are seeking a skilled Big Data Engineer to join our team at ACL Digital. In this role, you will be responsible for designing and implementing our big data technology stack, ensuring that it meets the needs of our analytics team. The ideal candidate will have strong experience with big data technologies, ETL pipelines, and SQL. A deep...

  • Big Data Engineer

    6 days ago

    Pune, Maharashtra, India LTI - Larsen & Toubro Infotech Full time

    LTI - Larsen & Toubro Infotech is a global technology consulting and digital solutions company. We are seeking an experienced Big Data Engineer to join our team.Job Description:We are looking for a highly skilled professional with expertise in designing and developing big data platforms using Apache Spark, Scala, and Hadoop ecosystem technologies.The ideal...

  • Big Data Engineer

    2 days ago

    Pune, Maharashtra, India Deutsche Bank Full time

    About the RoleWe are seeking a highly motivated Big Data Engineer to join our team. As a key member of our organization, you will be responsible for designing and implementing enterprise-level big data solutions using Cloudera, Hadoop, Spark, and other related technologies.Key Responsibilities:Implement project requirements on the enterprise analytics...

  • Big Data Engineering Lead

    4 days ago

    Pune, Maharashtra, India HyrEzy Talent Solutions Full time

    Job DescriptionWe are seeking a highly skilled Big Data Engineering Lead to lead our big data engineering team. In this role, you will be responsible for leading the design, development, and maintenance of large-scale big data frameworks with AWS or other cloud providers. The ideal candidate will have 1 year of experience developing modern, industry-standard...

  • Cloud Big Data Engineer

    1 day ago

    Pune, Maharashtra, India IT Full time

    We are hiring Cloud Big Data Engineer.Location : Experience level : 5+ years.Required Skills : GCP, Big Data, Pyspark, Python, Spark, Kafka, Hadoop, Hive, Scala/Django, DFS Elastic search.Key Responsibilities :- Design, develop, and implement big data solutions on the Google Cloud Platform (GCP).- Develop and maintain data pipelines for data ingestion,...

  • Big Data Engineer Leader

    21 hours ago

    Pune, Maharashtra, India Recro Full time

    About the RoleThe Lead Big Data Engineer will play a pivotal role in designing and implementing scalable solutions to address complex Big Data challenges. The successful candidate will collaborate closely with Product Management and Engineering leadership to drive innovation and technical excellence.ResponsibilitiesCollaborate with Product Management and...

  • Senior Big Data Engineer

    3 days ago

    Pune, Maharashtra, India LTIMindtree Full time

    Job Overview:In a fast-paced, dynamic environment at LTIMindtree, we are looking for an experienced Senior Big Data Engineer to lead our team in delivering innovative solutions using Apache Spark and the Big Data Hadoop Ecosystem. With a strong background in data engineering and excellent problem-solving skills, you will be responsible for designing,...

  • Big Data Engineer

    3 days ago

    Pune, Maharashtra, India Autodesk Full time

    Job DescriptionWe are seeking an experienced Big Data Engineer to build and scale data infrastructure that powers batch and real-time data processing of billions of records daily. The ideal candidate will have strong experience in AWS Cloud Automation, Python programming, and automation of cloud infrastructure services.

  • Big Data Engineering Specialist

    6 days ago

    Pune, Maharashtra, India Undisclosed Full time

    **Qualifications**We are seeking a big data engineering specialist with 4-6 years of experience in data engineering. You should have a solid understanding of ETL frameworks, data integration, and data architecture.A bachelor's degree in computer science, information technology, or a related field is required. Experience with relational databases, big data...

  • Big Data Engineer

    4 days ago

    Pune, Maharashtra, India LTI - Larsen & Toubro Infotech Full time

    Company Overview: Greetings from LTIMindtree, a leading global technology consulting and digital solutions company. We are committed to delivering innovative and effective solutions to our clients.Job Description: We are seeking an experienced Scala Spark Developer to join our team. As a key member of our Big Data practice, you will be responsible for...

  • Cloud Big Data Engineer

    4 weeks ago

    Pune, Maharashtra, India IT Full time

    We are hiring Cloud Big Data Engineer. Location : Experience level : 5+ years. Required Skills : GCP, BigData, Pyspark, Python, Spark, Kafka, Hadoop, Hive, Scala/Django, DFS Elastic search. Key Responsibilities : - Design, develop, and implement big data solutions on the Google Cloud Platform (GCP). - Develop and maintain data pipelines for data ingestion,...

  • Lead Big Data Engineer

    5 days ago

    Pune, Maharashtra, India Recro Full time

    Job Description :The Lead Big Data Engineer will play a crucial role in designing and implementing solutions to address complex Big Data challenges. The successful candidate will work closely with Product Management and Engineering leadership to drive innovation and technical excellence.Responsibilities :Collaborate closely with Product Management and...

  • Senior Big Data Engineer

    5 days ago

    Pune, Maharashtra, India Recro Full time

    We are seeking an experienced Lead Big Data Engineer in Recro with at least 9 years of experience. Our ideal candidate will possess a strong understanding of the Big Data ecosystem and the ability to design and implement scalable data pipelines.Key Responsibilities:Collaborate closely with Product Management and Engineering leadership to devise and build the...

  • Senior Data Engineer

    3 weeks ago

    Pune, Maharashtra, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology...

  • Big Data Engineer Lead

    4 days ago

    Pune, Maharashtra, India Upraised Full time

    We're looking for an experienced Data Engineer to lead our big data efforts. The ideal candidate will have a strong background in designing and implementing scalable data architectures using Azure big data tools such as Azure Databricks, Azure Synapse, and HDInsight.About the Role:Leverage your expertise in relational SQL and NoSQL databases to build robust...

  • Big Data Engineering Expert

    3 days ago

    Pune, Maharashtra, India LTIMindtree Full time

    Job DescriptionWe are seeking a highly skilled Big Data Engineer to join our team at LTIMindtree. As a key member of our data engineering team, you will be responsible for designing, implementing, and maintaining large-scale data pipelines to meet business requirements.Key Responsibilities:1. Design and implement complex PySpark code to convert business...

  • Big Data Systems Engineer

    4 days ago

    Pune, Maharashtra, India CORTEX Consultants Full time

    Job OverviewAs a Big Data Systems Engineer at CORTEX Consultants, you will be responsible for designing, managing, and overseeing all aspects of a Production Environment for Big Data Platforms.Key Responsibilities- Plan, manage, and oversee all aspects of a Production Environment for Big Data Platforms.- Define strategies for Application Performance...

  • Big Data Engineer Lead

    1 day ago

    Pune, Maharashtra, India CGT Solutions Full time

    About the RoleThis is an exciting opportunity to work as a Big Data Engineer Lead at CGT Solutions, where you will lead the development of big data solutions using Apache Hadoop, Pyspark, and Apache Spark.You will be responsible for designing, implementing, and maintaining large-scale data processing systems.Key Responsibilities:Lead the development of big...

  • Senior Big Data Engineer

    7 days ago

    Pune, Maharashtra, India Synechron Technologies Private Limited Full time

    About the RoleWe are seeking an experienced Sr. Big Data Engineer to join our team at Synechron Technologies Private Limited.The ideal candidate will have a strong background in Big Data technologies, Java programming, and a solid understanding of data architecture principles.Main Responsibilities:Design, develop, and maintain large-scale data processing...