Senior data engineer

2 weeks ago

Anand, India BrightEdge Full time

Big Data Engineer - Bright Edge About Us Bright Edge is a leading SEO and content performance marketing platform that transforms online content into tangible business results. Our platform processes massive amounts of data to provide actionable insights to our clients. We're looking for a talented Big Data Engineer to join our Professional Services to help us scale and optimize our data processing capabilities. Role Overview As a Big Data Engineer at Bright Edge, you will design, build, and maintain high-performance data pipelines that process terabytes of data. You'll work on optimizing our existing systems, identifying and resolving performance bottlenecks, and implementing solutions that improve the overall efficiency of our platform. This role is critical in ensuring our data infrastructure can handle increasing volumes of data while maintaining exceptional performance standards. Key Responsibilities ● Design and implement scalable batch processing systems using Python and big data technologies ● Optimize database performance, focusing on slow-running queries and latency improvements ● Use Python profilers and performance monitoring tools to identify bottlenecks● Reduce P95 and P99 latency metrics across our data platform ● Build efficient ETL pipelines that can handle large-scale data processing ● Collaborate with data scientists and product teams to understand data requirements ● Monitor and troubleshoot data pipeline issues in production ● Implement data quality checks and validation mechanisms ● Document data architecture and engineering processes ● Stay current with emerging big data technologies and best practices Qualifications Required ● Bachelor's degree in Computer Science, Engineering, or related technical field ● 4+ years of experience in data engineering roles ● Strong Python programming skills with focus on data processing libraries ● Experience with big data technologies (Spark, Hadoop, etc.) ● Proven experience optimizing database performance (SQL or No SQL) ● Knowledge of data pipeline orchestration tools (Airflow, Luigi, etc.) ● Understanding of performance optimization techniques and profiling tools ● Experience with cloud platforms (AWS, GCP, or Azure) ● Ability to translate business requirements into technical specifications ● Ability to work in UK timezone (12 PM IST to 9:30 PM IST) Preferred ● Master's degree in Computer Science or related field ● Experience with SEO data or web crawling systems ● Experience with Clickhouse Database ● Knowledge of distributed systems and microservices architecture ● Familiarity with container orchestration (Kubernetes, Docker) ● Experience with real-time data processing ● Contributions to open-source projects ● Experience with machine learning operations Skills and Abilities ● Strong analytical thinking and problem-solving skills ● Excellent attention to detail and commitment to data quality ● Ability to work effectively in a collaborative team environment ● Good communication skills to explain complex technical concepts ● Self-motivated with the ability to work independently ● Passion for performance optimization and efficiency

  • Senior Data Engineer

    3 weeks ago

    Anand, Gujarat, India JRD Systems Full time

    We are seeking a Senior Data Engineer with deep expertise in AWS Cloud Development. This role is critical to both the development of new data products and the modernization of existing platforms.The ideal candidate is a seasoned data engineer with hands-on experience designing, building, and optimizing large-scale data pipelines and architectures in both...

  • Data Engineer

    4 weeks ago

    Anand, Gujarat, India Insight Global Full time

    100% Remote Data Engineer Required Skills & Experience5+ years of experience as a Data EngineerExpertise in Python, Pyspark and SQLHands-on Python/PySpark coding experienceStrong Big Data experience, GCP preferredTelemetry experienceStreaming experience with analytics and data processingBusiness analytics experience and working directly with business...

  • Senior Data Engineer

    4 weeks ago

    Anand, Gujarat, India Tredence Inc. Full time

    About the CompanyTredence focuses on last-mile delivery of powerful insights into profitable actions by uniting its strengths in business analytics, data science and software engineering. The largest companies across industries are engaging with us and deploying their prediction and optimization solutions at scale. Head quartered in the San Francisco Bay...

  • Senior Engineer

    4 weeks ago

    Anand, Gujarat, India Next Pathway Full time

    Nextpathway | The Automated Cloud Migration CompanyNext Pathway is a technology services company providing clients a pathway from existing to emerging technologies. Our automation technology helps our customers accelerate the migration of complex applications and workloads to the cloud.About the Role:Next Pathway is looking for a highly skilled and...

  • Data Engineer

    4 weeks ago

    Anand, Gujarat, India Aiprus Software Private Limited Full time

    Job Title: Data EngineerRole Overview:We are seeking a skilled and motivated Data Engineer to design, develop, and maintain scalable data pipelines that process and transform large volumes of structured and unstructured data. The ideal candidate will have a strong background in data engineering, cloud technologies, and data architecture, and will collaborate...

  • Lead azure data engineer

    3 weeks ago

    Anand, Gujarat, India RandomTrees Full time

    Immediate #HIRING for a highly motivated and experienced Lead Azure Data Engineer to join our growing team.We're a leading software company specializing in Artificial Intelligence, Machine Learning, Data Analytics, Innovative data solutions, Cloud-based technologiesIf you're passionate about building robust applications and thrive in a dynamic environment,...

  • Data Engineer

    3 weeks ago

    Anand, Gujarat, India Incept Labs Full time

    Position: Software Engineer (Data)Location: Remote, IndiaAt Incept Labs, we believe the future of education and research lies in humans and AI workingtogether side by side. AI brings the ability to process knowledge at scale, while peoplecontribute imagination, values, and lived experience. When combined, they create a partnershipwhere each strengthens the...

  • Data Engineer

    4 weeks ago

    Anand, Gujarat, India IntraEdge Full time

    Job Title: Data EngineerLocation: [Remote]Experience: [5+ years]Employment Type: Full-timeJob Summary:We are seeking a skilled Data Engineer with hands-on experience in Snowflake, AWS (Lambda, Glue), DBT, and SQL to design, build, and optimize scalable data pipelines. The ideal candidate will be responsible for enabling seamless data integration,...

  • Data Engineer

    3 weeks ago

    Anand, Gujarat, India Alp Consulting Ltd. Full time

    9-12 years of Professional experience designing/building/maintaining highly available data and analytics platform.Hands-on experience with AWS or similar cloud platform building data engineering solutions for analytics and science.Must have experience building complex data pipelines – batch and/or real time event-based processing Strong experience in...

  • azure data engineer

    3 weeks ago

    Anand, Gujarat, India InfoSpeed Services, Inc. Full time

    Roles Open – Akkodis (Remote, 2 Months Project)Azure Data Engineer – 7+ years experienceJunior Azure Data Engineer – 5+ years experienceStatus: Open | Location: Remote