Big Data Engineer Consultant II

At Allstate, great things happen when our people work together to protect families and their belongings from life's uncertainties. And for more than 90 years our innovative drive has kept us a step ahead of our customers' evolving needs. From advocating for seat belts, air bags and graduated driving laws, to being an industry leader in pricing sophistication, telematics, and, more recently, device and identity protection. Job DescriptionWe are seeking a Data Engineer to support the design, development, and maintenance of robust data pipelines and infrastructure that enable data-driven decision-making. This role requires hands-on experience in Azure Fabric, Oracle SQL, and PySpark, along with a strong understanding of modern data engineering practices. The ideal candidate will collaborate with team members and stakeholders to deliver efficient, high-quality data solutions that support analytics and reporting initiatives across the organization.Key ResponsibilitiesData Pipeline Development: Design, develop, and maintain ETL/ELT pipelines using PySpark, Oracle SQL, and Azure Fabric components to move and transform data from diverse sources.Data Integration: Support the integration of structured and semi-structured data across multiple systems, ensuring data consistency, quality, and performance.Cloud-Based Data Engineering: Leverage Azure Data Services (e.g., Azure Data Factory, Azure Synapse, Data Lake Storage) to implement scalable and reliable data workflows.Data Modeling & Querying: Write efficient SQL queries, perform data profiling, and assist in the development of logical and physical data models.Collaboration: Work with data analysts, BI developers, and data scientists to understand data needs and provide well-prepared datasets that support analytics use cases.Data Quality & Testing: Contribute to the implementation of data quality checks, error handling, and validation processes in data pipelines.Documentation: Maintain clear and accurate documentation of data workflows, transformations, and architecture components.Continuous Improvement: Identify areas for performance optimization and recommend improvements to pipeline efficiency, storage design, or processing logic.QualificationsBachelor's degree in computer science, Information Systems, or a related field.3–6 years of experience in data engineering or a similar role.Hands-on experience with:Azure Fabric and Azure-based data servicesOracle SQL (data querying, scripting, and performance tuning)PySpark (for scalable data processing and transformation)Solid understanding of ETL/ELT processes, data warehousing principles, and data lake architectures.Familiarity with data modeling, version control systems (e.g., Git), and CI/CD practices.Primary SkillsAzure Service Fabric, Big Data, Big Data Engineering, Big Data Systems, Big Data Technologies, Data Engineering, Data Modeling, Data Science, ETL Processing, Extract Transform Load (ETL), Influencing Others, Oracle SQL, PySparkShift TimeShift B (India)Recruiter InfoDiksha PrasadAbout AllstateJoining our team isn't just a job — it's an opportunity. One that takes your skills and pushes them to the next level. One that encourages you to challenge the status quo. One where you can shape the future of protection while supporting causes that mean the most to you. Joining our team means being part of something bigger – a winning team making a meaningful impact.The Allstate Corporation is one of the largest publicly held insurance providers in the United States. Ranked No. 84 in the 2023 Fortune 500 list of the largest United States corporations by total revenue, The Allstate Corporation owns and operates 18 companies in the United States, Canada, Northern Ireland, and India. Allstate India Private Limited, also known as Allstate India, is a subsidiary of The Allstate Corporation. The India talent center was set up in 2012 and operates under the corporation's Good Hands promise. As it innovates operations and technology, Allstate India has evolved beyond its technology functions to be the critical strategic business services arm of the corporation. With offices in Bengaluru and Pune, the company offers expertise to the parent organization's business areas including technology and innovation, accounting and imaging services, policy administration, transformation solution design and support services, transformation of property liability service design, global operations and integration, and training and transition.Learn more about Allstate India here.