Big Data Engineer

1 month ago

india ATech Full time

Designation: BIG DATA ENGINEER

Job Description:

Your Role and Responsibilities:

- Understand a data warehousing solution and able to work independently in such an environment

- Responsible in Project development and delivery experience of a few good size projects

- Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements.

- Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization.

- Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too.

- Experiencing developing scalable Big Data applications or solutions on distributed platforms

- Able to partner with others in solving complex problems by taking a broad perspective to identify innovative solutions

- Strong skills building positive relationships across Product and Engineering.

- Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders

- Able to quickly pick up new programming languages, technologies, and frameworks

- Experience working in Agile and Scrum development process

- Experience working in a fast-paced, results-oriented environment

- Experience in Amazon Web Services (AWS) or other cloud platform tools

- Experience working with Data warehousing tools, including Dynamo DB, SQL, Amazon Redshift, and Snowflake

- Experience architecting data product in Streaming, Serverless and Microservices Architecture and platform.

- Experience working with Data platforms, including EMR, Data Bricks etc

- Experience working with distributed technology tools, including Spark, Presto, Scala, Python, Databricks, Airflow

- Developed the Pysprk code for AWS Glue jobs and for EMR.. Worked on scalable distributed data system using Hadoop ecosystem in AWS EMR, MapR distribution..

- Developed Python and pyspark programs for data analysis.. Good working experience with python to develop Custom Framework for generating of rules (just like rules engine).

- Developed Hadoop streaming Jobs using python for integrating python API supported applications..

- Developed Python code to gather the data from HBase and designs the solution to implement using Pyspark. Apache Spark DataFrames/RDD's were used to apply business transformations and utilized Hive Context objects to perform read/write operations..

- Re - write some Hive queries to Spark SQL to reduce the overall batch time

Required Technical and Professional Expertise:

- First and most important- Sound understanding of data structures & SQL concepts and experience in writing complex SQL especially around OLAP systems

- Sound knowledge of the ETL tool like informatica, 5+ years of experience, Big Data technologies'' like Hadoop ecosystem, its various components, along with different tools including Spark, Hive, Sqoop,etc..

- In-depth knowledge of MPP/distributed systems

Preferred Technical and Professional Expertise:

- The ability to write precise, scalable, and high-performance code

- The ability to write precise, scalable, and high-performance code

- Knowledge/Exposure in data modeling with OLAP (Optional)

(ref:hirist.tech)

  • Big Data Engineer

    4 weeks ago

    India TekJobs Full time

    This role is for a Data Engineering Lead to work on the Vanguard Big Data Platform. The team is responsible for the maintenance and development of leading Big Data initiatives and use cases providing business value.Job Background/context:The technology team is responsible for building and maintaining a multitude of applications including:Real-time...

  • Big Data Engineer

    4 weeks ago

    india TekJobs Full time

    This role is for a Data Engineering Lead to work on the Vanguard Big Data Platform. The team is responsible for the maintenance and development of leading Big Data initiatives and use cases providing business value. Job Background/context: The technology team is responsible for building and maintaining a multitude of applications including: Real-time...

  • Big Data Engineer

    3 days ago

    india NeoStats Full time

    Job Description About Neostats Analytics: NeoStats is a new-age, Data & Analytics firm offering contemporary solutions and infinite possibilities. Our mission is to create lasting competitive advantage for our clients by transforming them into world-class, data-driven organizations. Established in 2022 to provide End to End Data & Analytics Services, we...

  • Data Engineer

    2 days ago

    india Evnek Technologies Pvt Ltd Full time

    Job Description This is a remote position. Role: Data Engineer  Experience: 2+Years Notice period: Immediate Location: Anywhere from India Job Type: Full Time Responsibilities: Designing & building data architectures including the design & set-up of scalable (big) data lakes (full cloud or hybrid cloud), scheduled transformations &...

  • AWS Big Data Engineer

    1 month ago

    india Bosch Full time

    Job Description Technical skills ETL development experience with overall 4 to 7 years in AWS , Big data engineering related application Minimum 2 end to end ETL project implementation Proficient in PySpark, Python4. Proficient in Big Data technologies - Hadoop, Spark, Hive, Databricks5. Expert in AWS to develop, deploy and maintain ETL...

  • Data Engineer –

    4 weeks ago

    india HARMAN International Full time

    HARMAN’s engineers and designers are creative, purposeful and agile. As part of this team, you’ll combine your technical expertise with innovative ideas to help drive cutting-edge solutions in the car, enterprise and connected ecosystem. Every day, you will push the boundaries of creative design, and HARMAN is committed to providing you with the...

  • Senior Data Engineer

    1 month ago

    india SWAI TECHNOLOGIES PRIVATE LIMITED Full time

    Job Description : We are seeking a highly skilled Senior Data Engineer with 5 - 12 years of experience in building scalable data solutions. The ideal candidate should be proficient in Python programming and SQL, with extensive experience in utilizing various AWS services for data processing and storage. Experience with big data processing environments,...

  • Big Data Engineer

    1 month ago

    india Careerfit.ai Full time

    Responsibilities :Solution Design :- Design, develop, and maintain scalable big data solutions and data pipelines for efficient data ingestion, processing, storage, and analysis.- Collaborate with data scientists, analysts, and business stakeholders to identify data requirements, use cases, and business objectives.- Translate business needs into technical...

  • Big Data Architect

    3 weeks ago

    India Bloom Consulting Services Full time

    NA Experience **12 - 16 years** Offered Salary Notice Period **Not Disclosed** **Role: Big Data Architect** **Location: Bangalore** **Skills required**: Total 12 to 16 years of experience in BI, analytics, visualization, **data** integration, designing and developing **data **warehouses (ETL, **data**bases, front-end tools). 9+ years of...

  • Big Data Architect

    3 weeks ago

    India Bloom Consulting Services Full time

    Not Disclosed Experience **12 - 16 Years** Offered Salary **Lakh** Notice Period **Not Disclosed** **Role: Big Data Architect** **Location: Bangalore** **Skills required**: Total 12 to 16 years of experience in BI, analytics, visualization, **data** integration, designing and developing **data **warehouses (ETL, **data**bases, front-end...

  • Big Data Quality Assurance Engineer

    4 weeks ago

    Bangalore/Anywhere in India/Multiple Locations, IN Leadway Resource Full time

    We are seeking a highly skilled and experienced Big Data QA Engineer to join our team. You will play a critical role in ensuring the quality of our big data platform and data solutions. You will leverage your expertise in Big Data testing methodologies, automation frameworks, and scripting languages to design, develop, and execute comprehensive test...

  • Big Data Quality Assurance Engineer

    1 month ago

    Bangalore,Anywhere in India,Multiple Locations Leadway Resource Full time

    We are seeking a highly skilled and experienced Big Data QA Engineer to join our team. You will play a critical role in ensuring the quality of our big data platform and data solutions. You will leverage your expertise in Big Data testing methodologies, automation frameworks, and scripting languages to design, develop, and execute comprehensive test...

  • Big Data Analyst

    1 month ago

    india Royal Cyber ​​Inc. Full time

    1) 5+ years of experience managing and deploying CDP or HDP clusters.2) Experience configuring and setup of big data environments.3) Experience with Kerberos security and LDAP setup.4) Experience in setting up monitoring and alerting pipelines for big data clusters.

  • Big Data Developer

    3 weeks ago

    india RiskInsight Consulting Pvt Ltd Full time

    We are seeking a skilled Data Engineer with 4-8 years of experience, proficient in Big Data technologies and data processing frameworks. The ideal candidate will have strong expertise in Hadoop ecosystem components such as HDFS, HiveQL, and experience in developing and optimizing data processing workflows using PySpark. Proficiency in SQL for data...

  • DigiHelic - Senior Data Architect - Big Data Technologies

    1 month ago

    india Digihelic Solutions Pvt. Ltd. Full time

    Job Summary :We are seeking a highly skilled and motivated Product Solution Architect with extensive expertise in designing and implementing cutting-edge big-data solutions on the Amazon Web Services (AWS) platform.As a key member of our dynamic team, you will play a pivotal role in driving the success of our data-driven initiatives, leveraging AWS services...

  • Lead Big Data Engineer

    3 weeks ago

    Anywhere in India,Multiple Locations Grownex HR Solution Full time

    Role : Lead Big Data Engineer Work Mode : Remote(5 Days working) The Opportunity - Development team is establishing a new offshore product development team for its partner , that is building next-generation Big Data, Cloud-Based Business Operation Support technology for utilities, retail energy suppliers and Community Choice Aggregators (CCA). The...

  • Lead Big Data Engineer

    3 weeks ago

    Anywhere in India/Multiple Locations, IN Grownex HR Solution Full time

    Role : Lead Big Data EngineerWork Mode : Remote(5 Days working)The Opportunity - Development team is establishing a new offshore product development team for its partner , that is building next-generation Big Data, Cloud-Based Business Operation Support technology for utilities, retail energy suppliers and Community Choice Aggregators (CCA). The candidate...

  • Big Data Engineer

    4 weeks ago

    Bangalore/Anywhere in India/Multiple Locations, IN Connectio IT Pvt Ltd Full time

    Key Skills : Java, Apache Spark, Apache Kafka, Hadoop, AWSJob Description :- Designing, installing, testing, and maintaining scalable data management systems- Ensuring systems meet business requirements and industry standards- Creating custom software components and analytics applications- Research data acquisition opportunities and new uses for existing...

  • Big Data Engineer

    1 month ago

    Bangalore,Anywhere in India,Multiple Locations Connectio IT Pvt Ltd Full time

    Key Skills : Java, Apache Spark, Apache Kafka, Hadoop, AWS Job Description : - Designing, installing, testing, and maintaining scalable data management systems- Ensuring systems meet business requirements and industry standards- Creating custom software components and analytics applications- Research data acquisition opportunities and new uses for existing...

  • Senior AWS Data Engineer

    4 days ago

    india LevelUP HCS Full time

    AWS Data Engineer- Kafka We are looking for a Data Engineer (AWS Data Platform) to join our busy and dynamic business intelligence – data platform team. If you are a passionate Data Engineer you have the chance to be part of a team that looks to build out our next-generation data platform, leveraging new technologies and techniques to maximize the...