Sr Data Engineer

2 months ago

Bengaluru, Karnataka, India Leading-Biotechnology-Company Full time

Senior Data Engineer

JOB DUTIES:

  • Design and architect data lake solutions on AWS S3 and Snowflake, considering scalability, automation, security, and performance requirements.
  • Responsible for architecting, building, and optimizing data lake to store, process and analyze large volumes of structure and unstructured data.
  • Requirements gathering and analysis, development planning and co-ordination, data testing, documentation and ensuring high levels of data quality is maintained.
  • documentation.
  • Work with project leads, stakeholders, and business SMEs to define technical specifications to develop data integration requirements, code design and functional and technical documentation.
  • Work with specifications to design, develop, and maintain data infrastructure.
  • Work with cross functional teams and process owners on the development of test cases and scripts, test models and solutions to verify that requirements are met and ensuring high levels of data quality.
  • Promote and apply quality assurance best practices.
  • Understand and comply with data governance and compliance practices as defined for risk management. This includes data encryption practices, RBAC and security policies.
  • Promote and apply metadata management best practices supporting enterprise data catalogs.
  • Excellent problem-solving skills and ability to troubleshoot complex data engineering issues.
  • Benchmark application operational performance periodically, track (metrics) and fix issues.
  • Support change and release management processes.
  • Support incident and response management including problem solving and root cause analysis, documentation.
  • Support automation and on-call processes (Tier 1 / Tier 2).

SPECIFIC SKILLS OR OTHER REQUIREMENTS:

  • Requires 5 + years of experience with-
  • Primary Experience: Data Lakes and Data Integrations

Data Engineering:

  • AWS: 4+ years of required expertise with AWS services like S3, Glue, Lambda, EMR, Athena and proficiency in programming languages/ tools like Python, Scala, SQL.
  • Apache Software: 2+ years of required expertise in data processing with Spark & Flink, message broking using Kafka, orchestration using Airflow, processing high data volumes in open table formats like Iceberg.
  • Snowflake: 3+ years of required expertise with Snowflake SnowSQL, Snowpipe, (integrated with AWS S3), Streams and Tasks, Stored Procedures (expertise working with merge statements), Compute and Storage usage optimizations, Performance optimizations.
  • HVR (Fivetran): Near Real-time Data Replications with HVR is a plus.
  • SnapLogic: Data integrations with SnapLogic integration platform is a plus.
  • Certifications: AWS and Snowflake data engineering certifications is a plus.

Data Platforms:

  • Required: AWS S3 data lake store platform, Snowflake cloud data platform
  • Preferred: Denodo, SAP HANA

Programming/ Scripting:

  • Snowflake: SQL Scripting (Snowpipes, Tasks, Streams, Merge Statements, Stored Procedures, Functions, Security Policies DDM, Row-Access), SQL Scripting, Python
  • AWS: Apache spark, Python

Data Orchestration:

  • Preferred: Control-M, Apache Airflow

Cloud Storage Platforms:

  • Required: Amazon Web Services
  • Preferred: Microsoft Azure

Sr Data Engineer

  • Required: Knowledge and experience integrating data from SAP ERP (On premises, Cloud), Salesforce CRM, Workday, Planisware, Team Center PLM, Relational databases, REST APIs
  • Preferred: ServiceNow, MFG MES, IOT/ Sensor data platforms, Unstructured data, Other SaaS Applications

Data Operations: Excellent understanding of Data Ops practices for data management.

Stakeholder Engagement: Ability to take the lead and drive project activities collaborating with Analytics stakeholders and ensure the requirement is completed.

Solution Design: Good understanding of end-to-end solution architecture and design practices, ability to document solutions (maintain diagrams)

Governance: Good understanding of working with companies having regulated systems and processes for data. Adherence to data protection practices using tagging, security policies and data security (object-level, column-level, row-level).

Data Warehousing: Good understand on how data onboarded in the data lakes is used to support data warehouses. Impact of risks and change to a data warehouse. Fundamental concepts of dimensional modeling, experience working on data warehouse solutions, requirement gathering, design & build.

Data As-A-Product: Preferred knowledge and experience working with data treated as data products. Illumina is following a hybrid data mesh architecture which promotes data as a product for data lifecycle management.

Operating Systems: Windows, Linux

EDUCATION & EXPERIENCE:
Bachelors degree equivalent in Computer Science/Engineering or equivalent degree.

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job OpportunityReq ID: 286326 NTT DATA Services is a global leader in IT and business services, committed to helping clients succeed in a rapidly changing world. We are seeking a talented Sr Data Engineer to join our team in Bangalore, India.This role requires a strong background in AWS cloud services, including Lambda, Glue, S3, EMR, SNS, SQS, CloudWatch,...

  • Prod Mgmt Engineering Sr Advisor

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job Title: Digital Engineering Sr Staff EngineerJob DescriptionWe are seeking a highly skilled Digital Engineering Sr Staff Engineer to join our team in Bangalore, India. As a key member of our team, you will be responsible for designing and developing DevOps solutions for automated testing, infrastructure, and config management.Key...

  • Prod Mgmt Engineering Sr Advisor

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job Title: Digital Engineering Sr Staff EngineerReq ID: 275654 We are seeking a highly skilled Digital Engineering Sr Staff Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). As a key member of our team, you will be responsible for designing and developing DevOps solutions for automated testing, infrastructure, and config management.Key...

  • Salesforce CPQ Sr. Developer Position

    2 days ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job Title: Salesforce CPQ Sr. DeveloperWe are seeking a highly experienced Salesforce CPQ Sr. Developer to join our team at NTT DATA Services. This role will involve implementing, customizing, and maintaining Salesforce solutions using advanced technical skills.Key Responsibilities:Implement and customize Salesforce CPQ solutions to meet client...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job Title: Data EngineerJob Summary:We are seeking a highly skilled Data Engineer to join our team at NTT DATA. As a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data systems and pipelines using Azure data factory and Azure databricks.Key Responsibilities:Design and develop innovative data pipelines using Azure...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Aezion, Inc Full time

    Sr. Data EngineerAziona, Inc is seeking a highly skilled Sr. Data Engineer to join our team. The ideal candidate will have a strong background in designing, developing, and deploying data extraction pipelines in Python to process both structured and unstructured data.Key Responsibilities:Design and develop data extraction pipelines in Python to process...

  • Senior Engineering Manager

    2 weeks ago

    Bengaluru, Karnataka, India HEVO DATA Full time

    About HevoWe're revolutionizing the way companies harness data to drive user experience, growth, and business processes. With the exponential growth of data, companies are recognizing its value as a competitive advantage.The RoleThis is a unique opportunity to create a significant impact on our existing and future customers through technology innovation. If...

  • Digital Solution Consultant Sr. Analyst

    1 week ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job Title: Digital Solution Consultant Sr. AnalystAbout the Role:We are seeking a skilled Digital Solution Consultant Sr. Analyst to join our team in Bangalore, India. As a key member of our QA Test team, you will play a crucial role in ensuring the digital technology and related services that NTT DATA builds for our clients are valuable, intuitive, and...

  • Digital Solution Consultant Sr. Analyst

    5 days ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job Title: Digital Solution Consultant Sr. AnalystAbout the Role:NTT DATA Services is seeking a Digital Solution Consultant Sr. Analyst to join our team in Bangalore, India. As a key member of our QA Test team, you will play a crucial role in ensuring the digital technology and related services that we build for our clients are valuable, intuitive, and...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Aezion, Inc Full time

    Job Title: Sr. Data EngineerWe are seeking a highly skilled Sr. Data Engineer to join our team at Aezion, Inc.Job Summary:As a Sr. Data Engineer, you will be responsible for designing, developing, and deploying data extraction pipelines in Python to process both structured and unstructured data. You will work closely with our data team to build transactional...

  • Cloud Data Engineer

    1 week ago

    Bengaluru, Karnataka, India AT&T Full time

    Job Title: Cloud Data Engineer - Sr SpecialistJob Summary:AT&T's Chief Data Office (CDO) is harnessing data and making AT&T's data assets and ground-breaking AI functionality accessible to employees across the firm. In addition, our talented employees are a significant component that contributes to AT&T's place as the U.S. company with the sixth most...

  • Cloud Data Engineer

    5 days ago

    Bengaluru, Karnataka, India AT&T Full time

    Job Title: Cloud Data Engineer - Sr SpecialistJob Summary:AT&T's Chief Data Office (CDO) is harnessing data and making AT&T's data assets and ground-breaking AI functionality accessible to employees across the firm. In addition, our talented employees are a significant component that contributes to AT&T's place as the U.S. company with the sixth most...

  • Digital Solution Consultant Sr. Analyst

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job Title: Digital Solution Consultant Sr. AnalystWe are seeking a highly skilled Digital Solution Consultant Sr. Analyst to join our team in Bangalore, India. As a key member of our QA Test team, you will play a crucial role in ensuring the digital technology and related services that we build for our clients are valuable, intuitive, and impactful.Key...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job Title: Senior Data EngineerJob Summary:We are seeking a highly skilled Senior Data Engineer to join our team in Bangalore, India. The successful candidate will have a strong background in AWS cloud services, expertise in SQL and relational databases, and proficiency in Python programming.Key Responsibilities:Design and implement data engineering...

  • Salesforce Digital Solution Consultant Sr. Analyst

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job Title: Salesforce Digital Solution Consultant Sr. AnalystAbout the Role:We are seeking a highly skilled Salesforce Digital Solution Consultant Sr. Analyst to join our team in Bangalore, India. As a key member of our team, you will be responsible for delivering high-quality Salesforce solutions to our clients.Key Responsibilities:Develop and implement...

  • Data Visualization Engineer

    6 days ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job Title: Data Visualization EngineerJob Summary:Data Visualization Engineer - NTT DATAData visualization engineers utilize expertise in information visualization and design to create and deploy new platforms and services for visualization initiatives. Visualization specialists build interactive and complex data visualizations and analysis tools. They are...

  • Cloud Data Engineer

    1 week ago

    Bengaluru, Karnataka, India AT&T Full time

    Job Title: Cloud Data Engineer - Sr SpecialistJob Summary:AT&T's Chief Data Office (CDO) is harnessing data and making AT&T's data assets and ground-breaking AI functionality accessible to employees across the firm. In addition, our talented employees are a significant component that contributes to AT&T's place as the U.S. company with the sixth most...

  • Data Engineer

    5 days ago

    Bengaluru, Karnataka, India NTT DATA Services Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at NTT DATA Services. As a Data Engineer, you will be responsible for designing and developing data pipelines utilizing Cloud technologies Azure/AWS/GCP.Key ResponsibilitiesDesign and develop data pipelines using Cloud technologies Azure/AWS/GCPLead a team of Data Engineers and act...

  • SAP BODS Developer

    1 week ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Job Title: Sr BODS DeveloperJob Summary:We are seeking a highly skilled Sr BODS Developer to join our team in Bangalore, India. The successful candidate will be responsible for executing data migration blueprints, contributing to documentation, and providing technical support.Key Responsibilities:Understand and execute data migration blueprints, including...

  • Data Analyst Sr Associate

    2 weeks ago

    Bengaluru, Karnataka, India AT&T Full time

    Job Title: Sr Associate Data AnalystJob Summary:We are seeking a highly skilled Sr Associate Data Analyst to join our team at AT&T. As a Sr Associate Data Analyst, you will be responsible for developing, understanding, and enhancing code in traditional data warehouse environments, data lake, and cloud environments like Snowflake, Azure, Databricks.Key...