Procal - Big Data Engineer - Scala/Python

1 month ago

Hyderabad, Telangana, India procal tech Full time

Job Title : Big Data Engineer

No. of Positions : 1

Experience : 7-15 Years

Job Type : Full-time

Job Overview :

The Sr. Big Data Engineer is a pivotal role in our team, responsible for designing and implementing scalable processing and storage solutions for large datasets. This position requires proficiency in big data technologies, containerization, cloud services, and software development, ensuring that our data solutions are robust, scalable, and efficient.

Key Responsibilities :

- Design and implement big data system architecture to support scalable processing and storage of large datasets.

- Containerize applications using Docker and manage deployment on Kubernetes.

- Develop solutions using Python and Scala programming languages.

- Implement and manage Elasticsearch queries through both programmatic and console methods.

- Work with AWS services including S3 and Lambda Functions, ensuring optimal integration and performance.

- Understand and apply Apache Spark fundamentals, working proficiently with Hive.

- Ensure stability and troubleshoot performance issues by leveraging extensive knowledge on Spark internals.

- Manage and maintain streaming frameworks such as Apache Kafka, ensuring the reliability of messaging queues.

- Work efficiently with Core Java, Linux, and PostgreSQL in development environments.

Key skillsets :

Programming Languages :

- Python and Scala: Advanced programming skills in Python and Scala, with the ability to write efficient, maintainable code.

- Core Java: Solid background in Core Java, understanding of object-oriented design, and experience in multi-threaded programming.

Big Data Technologies :

- Apache Spark: Comprehensive knowledge of Apache Spark internals, experience in troubleshooting performance issues, and ensuring stability of data processing jobs.

- Hive: Proficiency in using Hive for querying and managing large datasets, understanding of HiveQL and its optimization.

Database and Querying :

- Elasticsearch: In-depth knowledge in Elasticsearch, ability to construct and manage queries via programmatic and console methods.

- PostgreSQL: Experience with PostgreSQL, understanding of relational databases, and ability to optimize queries for performance.

Operating Systems :

- Linux: Strong command over Linux environment, comfortable with shell scripting and system administration.

Attributes :

- Proactive problem solver with strong analytical skills.

- Effective communicator with the ability to translate technical concepts to non-technical stakeholders.

- Strong team player with the ability to provide technical leadership.

- Detail-oriented and committed to delivering high-quality software solutions.

This Sr. Big Data Engineer role requires a seasoned professional with a comprehensive skill set in Big Data / Spark development, a deep understanding of software architecture, and the ability to collaborate effectively within a dynamic development : Bachelor's degree in computer science, Information Technology OR Equivalent degree is required.

Work Location : Hyderabad, India

(ref:hirist.tech)

  • Scala Developer

    3 weeks ago

    Hyderabad, Telangana, India Trinity Infotech INC Full time

    Job Description :- 5+ years of experience working in a software product development organization building modern and scalable big data applications.- Expertise with Scala functional programming (intermediate/advanced level is a mandate).- Data structures and algorithms in Scala.- Fluency in Hadoop technologies around handling big data processing e.g. Spark.-...

  • Scala Developer

    5 days ago

    Hyderabad, Telangana, India Trinity Infotech INC Full time

    Job Description :- 5+ years of experience working in a software product development organization building modern and scalable big data applications.- Expertise with Scala functional programming (intermediate/advanced level is a mandate).- Data structures and algorithms in Scala.- Fluency in Hadoop technologies around handling big data processing e.g. Spark.-...

  • Data Engineer

    1 week ago

    Hyderabad, Telangana, India programming.com Full time

    Job Description :Role Overview :As a Data Engineer , you will be responsible for designing, building, and maintaining efficient, scalable data pipelines and infrastructure. You will work closely with data scientists, analysts, and other stakeholders to ensure data integrity, accessibility, and usability. Your expertise in SQL, Python or Scala, Spark, Hadoop,...

  • Data Engineer

    5 days ago

    Hyderabad, Telangana, India programming.com Full time

    Job Description :Role Overview :As a Data Engineer , you will be responsible for designing, building, and maintaining efficient, scalable data pipelines and infrastructure. You will work closely with data scientists, analysts, and other stakeholders to ensure data integrity, accessibility, and usability. Your expertise in SQL, Python or Scala, Spark, Hadoop,...

  • Engineering Manager

    5 days ago

    Hyderabad, Telangana, India Questsphere Private ltd. Full time

    Job Description :Skills Required : - Experience in project management and Agile/Scrum product development life cycles- Work with product management, design, support and other dev teams to create high-quality solutions- Manage best-of-class engineering teamRequirement (To be successful in this role you have) :- 10+ years of experience with technologies...

  • Software Engineer

    1 week ago

    Hyderabad, Telangana, India Whitefield Careers Full time

    Notice Period : Max 30 daysWe are searching for a talented Software Engineer to join our growing team. In this role, you will be responsible for designing, developing, and maintaining large-scale data processing applications using Scala and Spark. You will work closely with data scientists and engineers to develop solutions that transform raw data into...

  • Software Engineer

    5 days ago

    Hyderabad, Telangana, India Whitefield Careers Full time

    Notice Period : Max 30 daysWe are searching for a talented Software Engineer to join our growing team. In this role, you will be responsible for designing, developing, and maintaining large-scale data processing applications using Scala and Spark. You will work closely with data scientists and engineers to develop solutions that transform raw data into...

  • CarbyneTech - Data Science Engineer - Machine/Deep Learning Models - R/Python/Scala

    4 weeks ago

    Hyderabad, Telangana, India CarbyneTech India Pvt Ltd Full time

    Job Requirements :- Experience in research and development. Conference papers at reputed AI/ML platforms (ICML, NIPS, AAAI) will be a big plus,- Experience in Time series modeling (classical methods like ARIMA, STL Forecast). Understanding and implementing active research in Time series forecasting (Bayesian Forecasting, Hierarchical Forecasting),- Hands-on...

  • CarbyneTech - Data Science Engineer - Machine/Deep Learning Models - R/Python/Scala

    5 days ago

    Hyderabad, Telangana, India CarbyneTech India Pvt Ltd Full time

    Job Requirements :- Experience in research and development. Conference papers at reputed AI/ML platforms (ICML, NIPS, AAAI) will be a big plus,- Experience in Time series modeling (classical methods like ARIMA, STL Forecast). Understanding and implementing active research in Time series forecasting (Bayesian Forecasting, Hierarchical Forecasting),- Hands-on...

  • Senior Data Engineer

    1 week ago

    Hyderabad, Telangana, India Warner Bros. Discovery Full time

    Role : Senior Data Engineer 5+ Years, Description :What You Will Do : - Design and develop the data platform to efficiently and cost effectively address various data needs across the business.- Build software across our entire cutting-edge data platform, including event driven and batch data processing, storage, and serving through various data integration...

  • Senior Data Engineer

    5 days ago

    Hyderabad, Telangana, India Warner Bros. Discovery Full time

    Role : Senior Data Engineer 5+ Years, Description :What You Will Do : - Design and develop the data platform to efficiently and cost effectively address various data needs across the business.- Build software across our entire cutting-edge data platform, including event driven and batch data processing, storage, and serving through various data integration...

  • Lead/Software Development Engineer

    1 month ago

    Hyderabad, Telangana, India MarketScope Full time

    Key Skills : - Spark Core/Streaming with Any programming language Scala/ Java/ Python- Good Understanding of Data Structures, Algorithms, Logical and problem solving skills- Data Transformation, Data Ingestion, Optimization mechanism/techniques- Good understanding of Big Data (Hadoop, MapReduce, Kafka, Cassandra) Technologies, AWS, Postgresql.Education :...

  • Lead/Software Development Engineer

    5 days ago

    Hyderabad, Telangana, India MarketScope Full time

    Key Skills : - Spark Core/Streaming with Any programming language Scala/ Java/ Python- Good Understanding of Data Structures, Algorithms, Logical and problem solving skills- Data Transformation, Data Ingestion, Optimization mechanism/techniques- Good understanding of Big Data (Hadoop, MapReduce, Kafka, Cassandra) Technologies, AWS, Postgresql.Education :...

  • Azure Data Engineer

    6 days ago

    Hyderabad, Telangana, India ZF Group Full time

    Req ID 70402 | Hyderabad, India, ZF India Pvt. Ltd. Primary skills: Azure Data Engineer, ETL/ELT, Data Lake, ADF, Data Bricks, Azure Synapse- Python/Pyspark, SQL/Mongo DB/Azure Cosmos DB/POWER BIBachelor's degree in Computer Science, Data Engineering, or a related field. Master's degree is a plus. Proven experience as a Data Engineer, preferably in a...

  • Azure Data Engineer

    5 days ago

    Hyderabad, Telangana, India ZF Group Full time

    Req ID 70402 | Hyderabad, India, ZF India Pvt. Ltd. Primary skills: Azure Data Engineer, ETL/ELT, Data Lake, ADF, Data Bricks, Azure Synapse- Python/Pyspark, SQL/Mongo DB/Azure Cosmos DB/POWER BIBachelor's degree in Computer Science, Data Engineering, or a related field. Master's degree is a plus. Proven experience as a Data Engineer, preferably in a...

  • Engineering Manager

    1 month ago

    Hyderabad, Telangana, India Questsphere Private ltd. Full time

    Job Description :Skills Required : - Experience in project management and Agile/Scrum product development life cycles- Work with product management, design, support and other dev teams to create high-quality solutions- Manage best-of-class engineering teamRequirement (To be successful in this role you have) :- 10+ years of experience with technologies...

  • Python/PySpark Developer

    3 days ago

    Hyderabad, Telangana, India Selsoft Full time

    Job role : Python Pyspark developerExperience : 5+ yearsLocation : Hyderabad, WFOResponsibilities :- Ability to design and build Python-based code generation framework and runtime engine by reading Business Rules repository in order to.Requirements :- Minimum 3 years of experience in build & deployment of Bigdata applications using Spark SQL, Spark Streaming...

  • Staff Big data Devops Engineer

    2 weeks ago

    Hyderabad, Telangana, India ServiceNow Full time

    Company DescriptionAt ServiceNow, our technology makes the world work for everyone, and our people make it possible. We move fast because the world can't wait, and we innovate in ways no one else can for our customers and communities. By joining ServiceNow, you are part of an ambitious team of change makers who have a restless curiosity and a drive for...

  • Senior Big data Devops Engineer

    4 days ago

    Hyderabad, Telangana, India ServiceNow Full time

    Company DescriptionAt ServiceNow, our technology makes the world work for everyone, and our people make it possible. We move fast because the world can't wait, and we innovate in ways no one else can for our customers and communities. By joining ServiceNow, you are part of an ambitious team of change makers who have a restless curiosity and a drive for...

  • Staff Big data Devops Engineer

    5 days ago

    Hyderabad, Telangana, India ServiceNow Full time

    Company DescriptionAt ServiceNow, our technology makes the world work for everyone, and our people make it possible. We move fast because the world can't wait, and we innovate in ways no one else can for our customers and communities. By joining ServiceNow, you are part of an ambitious team of change makers who have a restless curiosity and a drive for...