Procal - Big Data Engineer - Scala/Python

1 month ago

Hyderabad, India procal tech Full time

Job Title : Big Data Engineer

No. of Positions : 1

Experience : 7-15 Years

Job Type : Full-time

Job Overview :

The Sr. Big Data Engineer is a pivotal role in our team, responsible for designing and implementing scalable processing and storage solutions for large datasets. This position requires proficiency in big data technologies, containerization, cloud services, and software development, ensuring that our data solutions are robust, scalable, and efficient.

Key Responsibilities :

- Design and implement big data system architecture to support scalable processing and storage of large datasets.

- Containerize applications using Docker and manage deployment on Kubernetes.

- Develop solutions using Python and Scala programming languages.

- Implement and manage Elasticsearch queries through both programmatic and console methods.

- Work with AWS services including S3 and Lambda Functions, ensuring optimal integration and performance.

- Understand and apply Apache Spark fundamentals, working proficiently with Hive.

- Ensure stability and troubleshoot performance issues by leveraging extensive knowledge on Spark internals.

- Manage and maintain streaming frameworks such as Apache Kafka, ensuring the reliability of messaging queues.

- Work efficiently with Core Java, Linux, and PostgreSQL in development environments.

Key skillsets :

Programming Languages :

- Python and Scala: Advanced programming skills in Python and Scala, with the ability to write efficient, maintainable code.

- Core Java: Solid background in Core Java, understanding of object-oriented design, and experience in multi-threaded programming.

Big Data Technologies :

- Apache Spark: Comprehensive knowledge of Apache Spark internals, experience in troubleshooting performance issues, and ensuring stability of data processing jobs.

- Hive: Proficiency in using Hive for querying and managing large datasets, understanding of HiveQL and its optimization.

Database and Querying :

- Elasticsearch: In-depth knowledge in Elasticsearch, ability to construct and manage queries via programmatic and console methods.

- PostgreSQL: Experience with PostgreSQL, understanding of relational databases, and ability to optimize queries for performance.

Operating Systems :

- Linux: Strong command over Linux environment, comfortable with shell scripting and system administration.

Attributes :

- Proactive problem solver with strong analytical skills.

- Effective communicator with the ability to translate technical concepts to non-technical stakeholders.

- Strong team player with the ability to provide technical leadership.

- Detail-oriented and committed to delivering high-quality software solutions.

This Sr. Big Data Engineer role requires a seasoned professional with a comprehensive skill set in Big Data / Spark development, a deep understanding of software architecture, and the ability to collaborate effectively within a dynamic development : Bachelor's degree in computer science, Information Technology OR Equivalent degree is required.

Work Location : Hyderabad, India

(ref:hirist.tech)

  • Scala Developer

    4 weeks ago

    Hyderabad, India Trinity Infotech INC Full time

    Job Description : - 5+ years of experience working in a software product development organization building modern and scalable big data applications.- Expertise with Scala functional programming (intermediate/advanced level is a mandate).- Data structures and algorithms in Scala.- Fluency in Hadoop technologies around handling big data processing e.g....

  • Scala Developer

    4 weeks ago

    Hyderabad, India Trinity Infotech INC Full time

    Job Description :- 5+ years of experience working in a software product development organization building modern and scalable big data applications.- Expertise with Scala functional programming (intermediate/advanced level is a mandate).- Data structures and algorithms in Scala.- Fluency in Hadoop technologies around handling big data processing e.g. Spark.-...

  • Scala Developer

    1 week ago

    Hyderabad, Telangana, India Trinity Infotech INC Full time

    Job Description :- 5+ years of experience working in a software product development organization building modern and scalable big data applications.- Expertise with Scala functional programming (intermediate/advanced level is a mandate).- Data structures and algorithms in Scala.- Fluency in Hadoop technologies around handling big data processing e.g. Spark.-...

  • Scala Developer

    1 week ago

    Hyderabad, India Trinity Infotech INC Full time

    Job Description :- 5+ years of experience working in a software product development organization building modern and scalable big data applications.- Expertise with Scala functional programming (intermediate/advanced level is a mandate).- Data structures and algorithms in Scala.- Fluency in Hadoop technologies around handling big data processing e.g. Spark.-...

  • Scala Developer

    4 weeks ago

    Hyderabad, Telangana, India Trinity Infotech INC Full time

    Job Description :- 5+ years of experience working in a software product development organization building modern and scalable big data applications.- Expertise with Scala functional programming (intermediate/advanced level is a mandate).- Data structures and algorithms in Scala.- Fluency in Hadoop technologies around handling big data processing e.g. Spark.-...

  • Scala Developer

    1 week ago

    Hyderabad, India Trinity Infotech INC Full time

    Job Description : - 5+ years of experience working in a software product development organization building modern and scalable big data applications.- Expertise with Scala functional programming (intermediate/advanced level is a mandate).- Data structures and algorithms in Scala.- Fluency in Hadoop technologies around handling big data processing e.g....

  • Data Engineer

    2 weeks ago

    hyderabad, India programming.com Full time

    Job Description : Role Overview : As a Data Engineer , you will be responsible for designing, building, and maintaining efficient, scalable data pipelines and infrastructure. You will work closely with data scientists, analysts, and other stakeholders to ensure data integrity, accessibility, and usability. Your expertise in SQL, Python or Scala, Spark,...

  • Data Engineer

    2 weeks ago

    Hyderabad, India programming.com Full time

    Job Description :Role Overview :As a Data Engineer , you will be responsible for designing, building, and maintaining efficient, scalable data pipelines and infrastructure. You will work closely with data scientists, analysts, and other stakeholders to ensure data integrity, accessibility, and usability. Your expertise in SQL, Python or Scala, Spark, Hadoop,...

  • Data Engineer

    1 week ago

    Hyderabad, Telangana, India programming.com Full time

    Job Description :Role Overview :As a Data Engineer , you will be responsible for designing, building, and maintaining efficient, scalable data pipelines and infrastructure. You will work closely with data scientists, analysts, and other stakeholders to ensure data integrity, accessibility, and usability. Your expertise in SQL, Python or Scala, Spark, Hadoop,...

  • Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India programming.com Full time

    Job Description :Role Overview :As a Data Engineer , you will be responsible for designing, building, and maintaining efficient, scalable data pipelines and infrastructure. You will work closely with data scientists, analysts, and other stakeholders to ensure data integrity, accessibility, and usability. Your expertise in SQL, Python or Scala, Spark, Hadoop,...

  • Data Engineer

    2 weeks ago

    Hyderabad, India programming.com Full time

    Job Description : Role Overview : As a Data Engineer , you will be responsible for designing, building, and maintaining efficient, scalable data pipelines and infrastructure. You will work closely with data scientists, analysts, and other stakeholders to ensure data integrity, accessibility, and usability. Your expertise in SQL, Python or Scala, Spark,...

  • Big Data Engineer

    1 month ago

    Hyderabad/Bangalore/Indore, India Nucleusteq Full time

    Job Description : Role Requirements :- Profound programming language experience in Java/Scala and Python.- Knowledge of Hadoop ecosystem and strong hands-on experience on Hive and Spark.- Understanding on Modernisation, Informatica migration- Must have understanding on messaging services and their integrations.- Good to have experience in streaming...

  • Big Data Engineer

    1 week ago

    Hyderabad/Bangalore/Indore, India Nucleusteq Full time

    Job Description : Role Requirements :- Profound programming language experience in Java/Scala and Python.- Knowledge of Hadoop ecosystem and strong hands-on experience on Hive and Spark.- Understanding on Modernisation, Informatica migration- Must have understanding on messaging services and their integrations.- Good to have experience in streaming...

  • Data Engineer

    1 month ago

    Bangalore/Chennai/Hyderabad, India Cortex Consulting Pvt. Ltd. Full time

    Responsibilities :- Design, construct, install, test, and maintain highly scalable data management systems.- Develop and implement ETL processes to collect, store, and process data from various sources.- Build and optimize data pipelines for efficient data ingestion and transformation.- Collaborate with data analysts and data scientists to understand data...

  • Data Engineer

    1 week ago

    Bangalore/Chennai/Hyderabad, India Cortex Consulting Pvt. Ltd. Full time

    Responsibilities :- Design, construct, install, test, and maintain highly scalable data management systems.- Develop and implement ETL processes to collect, store, and process data from various sources.- Build and optimize data pipelines for efficient data ingestion and transformation.- Collaborate with data analysts and data scientists to understand data...

  • Revrr - Data Engineer - Scala/Spark

    2 weeks ago

    Hyderabad, India REVRR TECHNOLOGIES PRIVATE LIMITED Full time

    Company : Revrr TechnologiesEmployment Type : Full-TimeRole : Software Development EngineerPosition : Developer, Senior Developer, and Lead DeveloperProject : Maps and Advertising Platforms ProductExperience: Sr SDE: 4-8 YearsKey Skills : Spark Core/Streaming with Any programming language Scala/ Java/ Python, Good Understanding of Data Structures,...

  • Revrr - Data Engineer - Scala/Spark

    2 weeks ago

    Hyderabad, India REVRR TECHNOLOGIES PRIVATE LIMITED Full time

    Company : Revrr TechnologiesEmployment Type : Full-TimeRole : Software Development EngineerPosition : Developer, Senior Developer, and Lead DeveloperProject : Maps and Advertising Platforms ProductExperience: Sr SDE: 4-8 YearsKey Skills : Spark Core/Streaming with Any programming language Scala/ Java/ Python, Good Understanding of Data Structures,...

  • Big Data/acd/pdfa/6445

    4 days ago

    Hyderabad, India Innova Solutions Full time

    Bigdata/Data AnalyticsHyderabad Required Experience - 5 - 8 Years - Posted On - 20 Nov 2023 Basic Section- Grade- L5A- Designation- Principal Software Engineer- Shift Details- Day A (9:00 AM-6:00 PM)- Organisational- Country- India- City- Hyderabad- Work Location- Raheja- SkillsSkill - SCALA- BIG DATAEducation Qualification No data...

  • Bigdata Scala Spark Data Engineer

    2 months ago

    Hyderabad, India LTIMindtree Full time

    Primary Skill –Scala SparkTotal Exp – 5.2 Years to 13 YearsNotice Period – ONLY 0 to 60 Days JoinersJob Location – Pune, Mumbai, Chennai, Hyderabad, Bangalore, Kolkata, Coimbatore, DelhiJob Description:Job Description:JD: Spark + scala Tech LeadJob DescriptionWe are looking for a Spark Scala Tech Lead with 5+ Years experience who knows how to fully...

  • AWS Data Engineer

    1 month ago

    Hyderabad, India Sureminds Solutions Pvt. Ltd. Full time

    Job Description :Experience in architecting and delivering highly scalable, distributed, cloud-based enterprise data solutionsStrong expertise in end-to-end implementation of Cloud data engineering solutions like Enterprise Data Lake, Data hub in AWSProficient in Lambda or Kappa ArchitecturesShould be aware of Data Management concepts and Data...