Senior Data Engineer

6 days ago

Dindigul, India Luxoft Full time

Project Description:

As a Senior Engineer in the Data Engineering & Analytics team, you will develop data & analytics solutions that sit atop vast datasets gathered by retail stores, restaurants, banks, and other consumer-focused companies. The challenge will be to create high-performance algorithms, cutting-edge analytical techniques including machine learning and artificial intelligence, and intuitive workflows that allow our users to derive insights from big data that in turn drive their businesses. You will have the opportunity to create high-performance analytic solutions based on data sets measured in the billions of transactions and front-end visualizations to unleash the value of big data. You will have the opportunity to develop data-driven innovative analytical solutions and identify opportunities to support business and client needs in a quantitative manner and facilitate informed recommendations/decisions through activities like building ML models, automated data pipelines, designing data architecture/schema, performing jobs in big data cluster by using different execution engines and program languages such as Hive/Impala, Python, Java, Kafka, Spark, R, etc.


Responsibilities:

• Hands-on developer who writes good quality, secure code that is modular, functional, and testable.

• Drive the evolution of Data & Services products/platforms with an impact-focused on data science and engineering.

• Design and implement scalable data architecture and data pipelines

• Solving complex problems with multi-layered data sets, as well as optimizing existing machine learning libraries and frameworks.

• Provide support for deployed data applications and analytical models by being a trusted advisor to Data Scientists and other data consumers by identifying data problems and guiding issue resolution with partner Data Engineers and source data providers.

• Ensure proper data governance policies are followed by implementing or validating Data Lineage, Quality checks, classification, etc.

• Discover, ingest, and incorporate new sources of real-time, streaming, batch, and API-based data into our platform to enhance the insights we get from running tests and expand the ways and properties on which we can test Experiment with new tools to streamline the development, testing, deployment, and running of our data pipelines.

• Participate in the development of data and analytic infrastructure for product development

Continuously innovate and determine new approaches, tools, techniques & technologies to solve business problems and generate business insights & recommendations

• Partner with roles across the organization including consultants, engineering, and sales to determine the highest priority problems to solve

• Evaluate trade-offs between many possible analytics solutions to a problem, taking into account usability, technical feasibility, timelines, and differing stakeholder opinions to make a decision

Break large solutions into smaller, releasable milestones to collect data and feedback from product managers, clients, and other stakeholders

• Evangelize releases to users, incorporating feedback, and tracking usage to inform future development

Work with small, cross-functional teams to define the vision, establish team culture and processes

Consistently focus on key drivers of organization value and prioritize operational activities accordingly

• Escalate technical errors or bugs detected in project work

• Maintain awareness of relevant technical and product trends through self-learning/study, training classes, and job shadowing.

• Support the building of scaled machine learning production systems by designing pipelines and engineering infrastructure.


Mandatory Skills:

• Working proficiency in using Python/Scala, Spark (tuning jobs), SQL, Hadoop platforms to build Big Data products & platforms.

• Good programming skills in Java and spring boot and Junit.

• Knowledge in software development test approaches & frameworks

• Familiarity with RESTful APIs and micro-services architectures

• Experience in working with CI/CD

• Experience in working with SQL database like Postgres, Oracle

• Preferably with hands-on experience with Hadoop big data tools (Hive, Impala, Spark)

• Experience with data pipeline and workflow management tools: NIFI, Airflow.

• Comfortable in developing shell scripts for automation.

• Good troubleshooting and debugging skills.

• Proficient in standard software development, such as version control, testing, and deployment

• Demonstrated basic knowledge of statistical analytical techniques, coding, and data engineering

• Ability to quickly learn and implement new technologies

• Ability to Solve complex problems with multi-layered data sets

• Ability to innovate and determine new approaches & technologies to solve business problems and generate business insights & recommendations.

• Ability to multi-task and strong attention to detail

• Flexibility to work as a member of a matrix based diverse and geographically distributed project teams

• Good communication skills - both verbal and written - and strong relationship, collaboration skills, and organizational skills


Nice-to-Have Skills:

• Experience with performance Tuning of Database Schemas, Databases, SQL, ETL Jobs, and related scripts

• Experience in working with Cloud APIs (e.g., Azure, AWS)

• Experience participating in complex engineering projects in an Agile setting e.g. Scrum


Languages:

  • English: C1 Advanced

  • Senior Data Engineer

    2 days ago

    Dindigul, India RapidBrains Full time

    Job Title: Senior Data EngineerExperience: 5-9 YearsLocation: BangaloreNotice Period: 0 - 30 DaysJob Description:As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines, working with cutting-edge tools like Python, DBT, and Databricks. Your primary focus will be transforming raw data into...

  • Senior Data Engineer

    1 month ago

    Dindigul, India Apptad Inc. Full time

    Role - Senior Data EngineerExp - 7+ yrsLocation - Bangalore/Hyderabad* Proficient in Python to design, develop, and maintain scalable data pipelines and ETL processes. * Has knowledge in pulling data using R via API's.* Extensive experience in building and optimizing data architectures, as well as strong skills in data modeling and SQL. * Capable enough to...

  • Senior Associate Data Engineering L2 DE

    3 weeks ago

    Dindigul, India Publicis Sapient Full time

    Company DescriptionPublicis Sapient is a digital transformation partner helping established organizations get to their future, digitally enabled state, both in the way they work and the way they serve their customers. We help unlock value through a start-up mindset and modern methods, fusing strategy, consulting and customer experience with agile engineering...

  • Data Engineer--PD

    4 weeks ago

    Dindigul, India Programming.com Full time

    Position: Data EngineerExperience:-5+ YearsLocation:- ChennaiNotice period: Immediate JoinerJob Description● Build, test, scale and maintain highly reliable data pipelines from a variety of batch datasources and real-time streams● Contribute to the data infrastructure and platform used to build our data pipelines● Serve as a core member of the data...

  • Data Engineer

    2 weeks ago

    dindigul, India Bits In Glass (India & UK) Full time

    Title: Data Engineer Practice: Data Engineering & AI Location: Office (Pune / Mohali / Hyderabad) Job Description As a Data Engineer, you will be an integral part of our team, working on data pipelines, data warehousing, and data integration for various analytics and AI use cases. You will collaborate closely with Delivery Managers, ML Engineers and other...

  • Senior Databricks Engineer

    2 weeks ago

    dindigul, India InfoStride Full time

    As Senior Data Platform Engineer , you'll maintain and build out our Databricks based platform for our solution teams. You’ll help to identify meaningful improvements to our infrastructure and pipelines to reduce costs, increase user access without violating HIPAA contracts, and streamline our deployment processes. You’ll provide day to day...

  • Data Engineer--PD

    2 weeks ago

    Dindigul, India Programming.com Full time

    Position: Data EngineerDuration: 6 months and extendableLocation: ChennaiExperience: 3+ yearsJob roleLooking for Data Engineer. The engineer will be responsible for expanding and optimizing our data and data pipeline infrastructure, as well as optimizing data flow and collection for cross-functional teams. The candidate should be experienced in building data...

  • Senior Deep Learning Engineer

    1 month ago

    Dindigul, India Staffing Ninja Full time

    Role: Senior Deep Learning EngineerLocation: RemoteResponsibilities: Provide technical leadership in the development and deployment of advanced AI models for generating realistic virtual environments and interactions.Oversee the integration of new AI models into our platform's infrastructure, ensuring seamless deployment, scalability, and efficient resource...

  • Senior Software Engineer

    3 weeks ago

    Dindigul, India Quidich Innovation Labs Full time

    Quidich Innovation Labs is a global company headquartered in Mumbai, India that pioneers in customized technology solutions for Sports Broadcast. From the outset, we have believed in the power of the story that sport tells as a tool to bring people together; and that this story best reaches fans, through the way it is broadcast. Building on this thinking, we...

  • Senior Technical Architect

    13 hours ago

    Dindigul, India Markovate Full time

    We are looking for a highly skilled and experienced Senior Technical Architect with over 15 years of expertise in software development, architecture, and strategic technology leadership, including a deep focus on AI technologies for one of our clients. The ideal candidate will have a strong background in AI/ML frameworks, data engineering, and cloud-based AI...

  • Senior Project Engineer

    2 weeks ago

    Dindigul, India Water Sanitation and Hygiene Institute (WASH Institute) Full time

    Water, Sanitation and Hygiene Institute (WASH Institute) Reddiarchatram, Dindigul, Tamilnadu About WASH Institute Water, Sanitation and Hygiene Institute (WASH Institute), established in 2008, is a non-profit organization, dedicated to providing practical solutions to tackle a wide range of water, sanitation, hygiene and environmental issues in India and the...

  • Performance Test Engineer

    2 weeks ago

    dindigul, India Wipro Full time

    Dear Applicant, Wipro is Hiring for performance testing !! Please share below information along with your latest resume and passport size photo to Location : only Chennai, who is willing to attend F2F interview only. EXP-5 to 12 years and should join within 30-60 days. Job Description Senior Performance Engineer conducts a wide range of quality...

  • Senior Project Engineer

    2 weeks ago

    Dindigul, India Water Sanitation and Hygiene Institute (WASH Institute) Full time

    Water, Sanitation and Hygiene Institute (WASH Institute) Reddiarchatram, Dindigul, Tamilnadu About WASH Institute Water, Sanitation and Hygiene Institute (WASH Institute), established in 2008, is a non-profit organization, dedicated to providing practical solutions to tackle a wide range of water, sanitation, hygiene and environmental issues in...

  • Senior Project Engineer

    2 weeks ago

    Dindigul, India Water Sanitation and Hygiene Institute (WASH Institute) Full time

    Water, Sanitation and Hygiene Institute (WASH Institute)Reddiarchatram, Dindigul, TamilnaduAbout WASH InstituteWater, Sanitation and Hygiene Institute (WASH Institute), established in 2008, is a non-profit organization, dedicated to providing practical solutions to tackle a wide range of water, sanitation, hygiene and environmental issues in India and the...

  • Senior Project Engineer

    2 days ago

    dindigul, India Water Sanitation and Hygiene Institute (WASH Institute) Full time

    Water, Sanitation and Hygiene Institute (WASH Institute)Reddiarchatram, Dindigul, TamilnaduAbout WASH InstituteWater, Sanitation and Hygiene Institute (WASH Institute), established in 2008, is a non-profit organization, dedicated to providing practical solutions to tackle a wide range of water, sanitation, hygiene and environmental issues in India and the...

  • Data Protection Automation Engineer

    2 weeks ago

    Dindigul, India PURVIEW Full time

    Role: Data Protection Automation ArchitectLocation: Pune or Hyderabad (Hybrid)Job Type: Long Term Contract (Permanent to Purview Services)NOTE: Client is looking for immediate joiners or Minimum 15 Days Notice periodClient is looking for 10+ years experience candidatesIn this role, you will:Design, develop, test, and support automation for the data...

  • Senior Python/AI/API Developer

    5 days ago

    Dindigul, India ZydeSoft Full time

    Company Overview:ZydeSoft is a premier software development company based in Chennai, India. We provide exceptional software services and solutions to companies worldwide. We are currently hiring a Senior Python/AI/API Developer to join our team. This is a challenging role that requires expertise in AI manipulation, large datasets, API development, and...

  • Senior Software Engineer

    3 weeks ago

    Dindigul, India Resourcequick Full time

    Job Title: Senior Software Engineer (Frontend) Experience Required: 5 to 8 YearsLocation: BangaloreBudget: 32 LPA to 35 LPANotice period: 15-30 daysSenior Software Engineer in Frontend development, you'll be at the forefront of creating an intuitive and dynamic user experience. You'll play a pivotal role in building and refining our React-based application,...

  • Lead Data Scientist

    2 weeks ago

    Dindigul, India Quess Corp Limited Full time

    Hiring Lead Data Scientist (Senior Consultant) 8 - 10 yearsExperience: 8 - 10 yearsCompany Name:- Quess Corp Ltd.Location: Bangalore - HybridNotice: Immediate - 15 days is considerable.Model: HybridJD:Translate client needs intro bust end tend analytical solutions.Design and implement production data pipelines using distributed computing and cloud...

  • AI/ML Engineer

    2 weeks ago

    dindigul, India Lingaro Full time

    AI/ML Engineer – Senior Consultant AI Engineering Group is part of Data Science & AI Competency Center and is focusing technical and engineering aspects of DS/ML/AI solutions. We are looking for experienced AI/ML Engineers to join our team to help us bring AI/ML solutions into production, automate processes, and define reusable best practices and...