Data Engineer

1 day ago

Erode, India Luxoft Full time

Project description

As a Senior Engineer in the Data Engineering & Analytics team, you will develop data & analytics solutions that sit atop vast datasets gathered by retail stores, restaurants, banks, and other consumer-focused companies. The challenge will be to create high-performance algorithms, cutting-edge analytical techniques including machine learning and artificial intelligence, and intuitive workflows that allow our users to derive insights from big data that in turn drive their businesses. You will have the opportunity to create high-performance analytic solutions based on data sets measured in the billions of transactions and front-end visualizations to unleash the value of big data. You will have the opportunity to develop data-driven innovative analytical solutions and identify opportunities to support business and client needs in a quantitative manner and facilitate informed recommendations/decisions through activities like building ML models, automated data pipelines, designing data architecture/schema, performing jobs in big data cluster by using different execution engines and program languages such as Hive/Impala, Python, Java, Kafka, Spark, R, etc.


Responsibilities

-Hands-on developer who writes good quality, secure code that is modular, functional, and testable.

-Drive the evolution of Data & Services products/platforms with an impact-focused on data science and engineering.

-Design and implement scalable data architecture and data pipelines

-Solving complex problems with multi-layered data sets, as well as optimizing existing machine learning libraries and frameworks.

-Provide support for deployed data applications and analytical models by being a trusted advisor to Data Scientists and other data consumers by identifying data problems and guiding issue resolution with partner Data Engineers and source data providers.

-Ensure proper data governance policies are followed by implementing or validating Data Lineage, Quality checks, classification, etc.

-Discover, ingest, and incorporate new sources of real-time, streaming, batch, and API-based data into our platform to enhance the insights we get from running tests and expand the ways and properties on which we can test Experiment with new tools to streamline the development, testing, deployment, and running of our data pipelines.

-Participate in the development of data and analytic infrastructure for product development

-Continuously innovate and determine new approaches, tools, techniques & technologies to solve business problems and generate business insights & recommendations

-Partner with roles across the organization including consultants, engineering, and sales to determine the highest priority problems to solve

-Evaluate trade-offs between many possible analytics solutions to a problem, taking into account usability, technical feasibility, timelines, and differing stakeholder opinions to make a decision

-Break large solutions into smaller, releasable milestones to collect data and feedback from product managers, clients, and other stakeholders

-Evangelize releases to users, incorporating feedback, and tracking usage to inform future development

-Work with small, cross-functional teams to define the vision, establish team culture and processes

-Consistently focus on key drivers of organization value and prioritize operational activities accordingly

-Escalate technical errors or bugs detected in project work

-Maintain awareness of relevant technical and product trends through self-learning/study, training classes, and job shadowing.

-Support the building of scaled machine learning production systems by designing pipelines and engineering infrastructure.


Skills

Must have

-Working proficiency in using Python/Scala, Spark (tuning jobs), SQL, Hadoop platforms to build Big Data products & platforms.

-Good programming skills in Java and spring boot and Junit.

-Knowledge in software development test approaches & frameworks

-Familiarity with RESTful APIs and micro-services architectures

-Experience in working with CI/CD

-Experience in working with SQL database like Postgres, Oracle

-Preferably with hands-on experience with Hadoop big data tools (Hive, Impala, Spark)

-Experience with data pipeline and workflow management tools: NIFI, Airflow.

-Comfortable in developing shell scripts for automation.

-Good troubleshooting and debugging skills.

-Proficient in standard software development, such as version control, testing, and deployment

-Demonstrated basic knowledge of statistical analytical techniques, coding, and data engineering

-Ability to quickly learn and implement new technologies

-Ability to Solve complex problems with multi-layered data sets

-Ability to innovate and determine new approaches & technologies to solve business problems and generate business insights & recommendations.

-Ability to multi-task and strong attention to detail

-Flexibility to work as a member of a matrix based diverse and geographically distributed project teams

-Good communication skills both verbal and written and strong relationship, collaboration skills, and organizational skills.


Nice to have

-Experience with performance Tuning of Database Schemas, Databases, SQL, ETL Jobs, and related scripts

-Experience in working with Cloud APIs (e.g., Azure, AWS)

-Experience participating in complex engineering projects in an Agile setting e.g. Scrum

  • Data Engineer

    3 days ago

    Erode, India S P A Enterprise Full time

    Hi, Job Title: Data Engineer (Permanent)Location: Bangalore and PuneJob Description:Required Skills:data engineers 6+ years’ experiencePython proficiency and it's data structures Libraries: NumPy, pandas, beautiful soup, selenium, pdfplumber, Requests etc. SQL programming DevOps AWS S3 and Azure data bricks

  • Data Engineer

    3 weeks ago

    Erode, India Algoscale Full time

    Exp: 4+ YearsLocation: Noida/ remoteWork Timings: 10:30 AM to 7:30 PM; Mon - FriKey Responsibilities:Design, develop, and optimize data pipelines and workflows using Java and Apache Flink to process and transform large volumes of data.Build and maintain data infrastructure and frameworks to support high-performance data processing.Collaborate with data...

  • Snowflake Data Engineer

    1 month ago

    Erode, India DNA Technology Services Full time

    Job Title: Data/Platform Engineer (SNOWFLAKE)Location: India (Remote)Experience Required: 7-12 YearsResponsibilities:Design, develop, and maintain data ingestion methods and patterns with tools like Fivetran, Snowflake Snowpipe, and Snowflake Secure Data Share.Build reliable, high-quality software by implementing best practices for full SDLC and CI/CD,...

  • Data Engineer

    3 days ago

    Erode, India Luxoft Full time

    Project descriptionProvide End to End Development solution to one of the largest investment banking products by scaling up the latest technology stack by adopting the cloud-based services under Payment Settlement stream.ResponsibilitiesWrite effective, scalable code using Python.In-depth working knowledge of data management activities like data extraction,...

  • Senior Azure Data Engineer

    4 days ago

    Erode, Tamil Nadu, India Alp Consulting Ltd. Full time

    About the RoleWe are seeking an experienced Senior Azure Data Engineer to join our team at Alp Consulting Ltd. The successful candidate will design, implement, and maintain large-scale data processing solutions using Azure services.Key Responsibilities:Design and develop ETL processes using Azure Data Factory and SSIS to extract, transform, and load data...

  • Data Engineering Specialist

    5 days ago

    Erode, India BT Group Full time

    Why this job matters?The purpose of this role is to provide technology expertise to drive the delivery of outcomes as well as coach and lead of a team of Software Engineers. Leads a team composed of technical practitioners to plan and design converged technology solutions, providing technical expertise to support. The role is focused on best practices,...

  • Data Analytics Engineer

    1 week ago

    Erode, Tamil Nadu, India Celito Tech, Inc. Full time

    At Celito Tech, Inc., we are seeking a highly skilled Data Analytics Engineer to join our team as a Spotfire Administrator.About the RoleThis is a remote opportunity in India and requires a strong background in data analysis, data engineering, analytics development, and administration of data analytics platforms.The ideal candidate will have expertise in...

  • Bhavitha Tech, CMMi Level 3 Company | AWS Data Engineer

    1 day ago

    Erode, India Bhavitha Tech, CMMi Level 3 Company Full time

    Company DescriptionBhavitha Tech, a CMMi Level 3 Company located in Bengaluru, specializes in digital transformation services including Application Modernization, Custom Product Development, AWS Cloud Consulting, Industrial 4.0 IoT, Consumer IoT, and ERP software implementation & support. With a diverse client portfolio ranging from enterprise companies like...

  • Enterprise Cloud Architect with Data Analytics Expertise

    4 days ago

    Erode, Tamil Nadu, India Evernorth Health Services Full time

    About Evernorth Health ServicesWe are a leading health services company that creates innovative solutions to improve health and increase vitality.Role Title: Architecture Senior AdvisorPosition Summary:You will be part of a team transforming healthcare and improving the lives and vitality of millions of members we serve. We leverage cutting-edge data and...

  • Marketing Data Scientist

    4 weeks ago

    erode, India Maino.ai Full time

    Job Description:At Maino.ai, we are seeking talented folks with 2-3 years of experience in AI/ML with comprehensive understanding of the ML models ecosystem. This role offers a unique opportunity for someone passionate about pushing the boundaries of technology and thriving in a dynamic environment.Qualifications:Must-Have:Strong experience with Large...

  • Software Engineer

    5 days ago

    Erode, India TalentBridge Full time

    Software Engineer / ArchitectIndia (Remote)6 months of contract ( it will keep on renewing )RoleWe are seeking a Senior Engineer with a strong focus on API development and integrations and a background in eCommerce & supply chain. This role will be responsible for designing and implementing both internal and external APIs, leading integration efforts, and...

  • Generative AI Engineer

    16 hours ago

    Erode, India Datadroves Full time

    Company Description Datadroves is a pioneering data intelligence company founded in 2023, specializing in developing advanced solutions that transform complex data into actionable insights for businesses across all industries. Our platforms empower organizations to uncover hidden patterns, optimize operations, and make informed decisions. We prioritize...

  • Senior Azure Data Technical Lead

    1 week ago

    Erode, Tamil Nadu, India SourceBae Full time

    Job Title: Senior Azure Data Technical LeadWe are seeking an experienced Senior Azure Data Technical Lead to manage and oversee the implementation of cutting-edge data solutions using Azure technologies. The ideal candidate should have extensive experience in data engineering, strong leadership skills, and excellent communication abilities.Key...

  • Vulnerability Automation Engineer

    3 weeks ago

    Erode, India Servsys Corporation Full time

    Job Title: Vulnerability Automation EngineerLocation: Remote (India)Duration: 9+ MonthsKey Responsibilities:Vulnerability Automation:Develop and maintain automation scripts in Python to enhance the efficiency of vulnerability detection, reporting, and remediation workflows.Use libraries such as Pandas for data analysis and manipulation to streamline...

  • Modeling Engineer

    3 days ago

    Erode, India Tata Technologies Full time

    About the Company - Tata Technologies is a global engineering and digital solutions company that serves automotive, industrial heavy machinery and aerospace industries.About the Role - Looking for CAS & CLASS A Team Lead - Digital Modeling (Onsite NCR region)Responsibilities -Ability to handle team of 5 digital designers and co-ordinate between team,...

  • Full Stack Engineer

    6 days ago

    Erode, India RevX Full time

    Position description:As a Software Engineer at RevX, you will take part in the implementation and delivery of robust features/products. You will work closely with software engineers, data engineers, engineering managers and product managers to build and ship new features that optimize customer engagement, and operational efficiency, and drive growth.Required...

  • Integration Engineer

    1 month ago

    Erode, India Collabera Digital Full time

    Job Title:- Azure Integration Engineer/ LeadLocations:- Bangalore, Pune, Chennai, Hyderabad (Hybrid work from office. 2 days in a week.) Job Description:- Job Overview: As an Azure Integration Specialist, you will be responsible for designing, implementing, and managing integration solutions using Azure services. You will work closely with various teams to...

  • Sr. Design Engineer/Design Manager

    2 weeks ago

    Erode, India HappieHire Full time

    Role: Sr. Design Engineer/Design Manager - Network Cabling/Data CentersJob Type: Full-timeLocation: Remote/Hybrid(Noida, Bengaluru, Hyderabad, Mumbai, Pune, Chennai)Experience: 5- 8 yearsWork Week: 6 days (Mon-Sat)Job Description:We seek dynamic and results-oriented Design/Senior Design Engineers to join our growing team of 225 professionals to do...

  • Proposal Engineer

    3 months ago

    Erode, India BOOM HR SOLUTION Full time

    Need*Proposal Engineer*forour InstrumentationIndustry!Position:ProposalEngineerQualification:Diploma/B.E(Instrumental)Experience:2YearsSalary:/(BASED ONEXPERIENCE)Skill: Instrumentation background with MSOfficeNatureof work: Technical Offer Preparation & Payment FollowupwithclientsLanguagesknown:TamilEnglishWorkLocation...

  • Aws data engineer

    2 weeks ago

    Erode, India Tata Consultancy Services Full time

    Note- This will be a Walkin Interview @ Chennai on 7-11-24PositionJDLocationDevOps,Terraform, Ansible, Linux/Dockers6+ Years of Experience in DevOps Related projectsHyderabad Should have good knowledge in AWS and should have automation experience using TerraformShould have working experience on Linux Environment and automations using Shell ScritpingShould...