Senior Splunk Administrator

We are seeking a seasoned Splunk engineer to join our team. As a key member of our infrastructure team, you will be responsible for designing, implementing, and administering large-scale Splunk solutions in highly available, redundant, distributed computing environments.

You will lead the design and deployment of new Splunk environments, including clustered, multi-site, and large-scale configurations. This involves performing Splunk forwarder deployment, configuration, and troubleshooting across diverse platforms, as well as integrating, curating, and normalizing diverse log sources into Splunk.

Additionally, you will configure and maintain Splunk dashboards, searches, and alerts to meet PCI DSS logging requirements and deliver evidentiary reports to auditors to support compliance verification. You will also develop advanced content for SIEM correlation, including custom correlation searches, dashboards, and alerts.

In this role, you will administer, maintain, and tune Splunk components (Indexers, Search Heads, Forwarders, Cluster Masters, Deployer, Deployment Server, and License Master). You will proactively monitor platform health using internal logs, KPIs, and custom monitoring solutions to identify and address performance bottlenecks.

Furthermore, you will lead capacity planning, storage forecasting, and continuity of operations for large Splunk deployments. You will optimize Splunk performance through configuration tuning, search optimization, and data model acceleration strategies.

As a senior member of our team, you will troubleshoot complex ingestion, performance, and search-related issues, identifying root causes and implementing sustainable fixes or workarounds. You will create, maintain, and enforce Splunk engineering documentation, including SOPs, design diagrams, architecture runbooks, and KB articles.

You will develop custom scripts and automation tools (e.g., Python, Bash, PowerShell) to improve Splunk administration, onboarding, and operational workflows. You will utilize Splunk APIs for integration with enterprise tools and automation frameworks.

Ultimately, you will serve as a technical escalation point for Splunk Engineer I/II and Splunk Admin roles. You will administer, tune, and troubleshoot Splunk Enterprise Security, maintaining data models, correlation searches, and notable events pipeline.

Key Responsibilities:

• Design, implement, and administer large-scale Splunk solutions in highly available, redundant, distributed computing environments
• Lead the design and deployment of new Splunk environments, including clustered, multi-site, and large-scale configurations
• Integrate, curate, and normalize diverse log sources into Splunk
• Configure and maintain Splunk dashboards, searches, and alerts to meet PCI DSS logging requirements
• Develop advanced content for SIEM correlation, including custom correlation searches, dashboards, and alerts
• Administer, maintain, and tune Splunk components (Indexers, Search Heads, Forwarders, Cluster Masters, Deployer, Deployment Server, and License Master)
• Proactively monitor platform health using internal logs, KPIs, and custom monitoring solutions
• Optimize Splunk performance through configuration tuning, search optimization, and data model acceleration strategies
• Troubleshoot complex ingestion, performance, and search-related issues
• Create, maintain, and enforce Splunk engineering documentation
• Develop custom scripts and automation tools to improve Splunk administration, onboarding, and operational workflows
• Utilize Splunk APIs for integration with enterprise tools and automation frameworks
• Serve as a technical escalation point for Splunk Engineer I/II and Splunk Admin roles
• Administer, tune, and troubleshoot Splunk Enterprise Security

To be successful in this role, you will need to possess the following skills and qualifications:

• 8+ years of IT experience in technical engineering, security operations, or infrastructure roles
• 5+ years of direct, hands-on Splunk engineering and administration experience in large-scale, distributed environments
• Expert-level knowledge of Splunk Enterprise and Splunk Enterprise Security, including architecture, clustering, and scaling strategies
• Proficiency in Linux/Unix administration and shell scripting
• Strong knowledge of Splunk APIs, including use for automation and tool integrations
• Expertise in regex, field extractions, and key-value parsing
• Strong programming/scripting skills in one or more languages (Python, Bash, PowerShell, Perl, JavaScript)
• Experience with storage systems (DAS, SAN, object storage) and understanding of their performance implications for Splunk indexing
• Solid understanding of networking (switches, routers, firewalls, load balancers, DNS, SSL/TLS) and how it impacts Splunk architecture
• Familiarity with Enterprise Management and automation tools
• Experience with Splunk ITSI (preferred) and other premium Splunk apps
• Strong knowledge of data formats including JSON, XML, and CSV
• Demonstrated experience delivering Splunk-based compliance reporting and audit support
• Strong communication skills for interacting with technical and non-technical stakeholders
• Proven ability to lead projects, mentor team members, and provide architectural guidance

As a member of our team, you will enjoy a range of benefits, including:

• Competitive salary and benefits package
• Opportunities for professional growth and development
• Collaborative and dynamic work environment
• Recognition and rewards for outstanding performance
• Flexible working arrangements and work-life balance