Splunk Administrator

Description

Project Scope This position is part of a team responsible for administering the Splunk system as directed by the Splunk Architect and Enterprise Monitoring manager. This position supports, maintains, and documents the Splunk system which is critical in supporting day to day enterprise system logging for the agency. Deliverables Produce, publish and maintain documentation adhering to department protocol.
Thoroughly document system configuration and changes.
Coordinate with network, database and server administrators to prepare nonproduction and production environments for new or enhanced systems, processes or applications.
Participate in system installation, upgrades/patches, configuration and tuning.
Create and maintain reports, scripts, programs or other tools within the Splunk system to maintain system health and integrity.
Define and maintain system and monitor application security.
Follow the Security Technical Implementation Guide (STIGs) methodology for standardized secure installation and maintenance of computer software and hardware.
Administration of all aspects of all Splunk environments i. Implement and maintain Splunk infrastructures and configurations.
Monitor all system, applications, hosts, data ingestion, forwarder status, system health, etc.
System management for clustering, deployment, licensing, and other server functions.
All forwarder interactions with systems
Manage and create applications for all of the above. Collect and assess system metrics.
Customer interaction to determine requirements and success factors for requests
Assesses customer requests Provides guidance and training to customers
Customer 1st level support, acts as interface between customers and Splunk team
Create and maintain manageable processes and documentation for all aspects of administration and development.
Assist the Splunk team with development of new applications, views, dashboards, reports, alerts, data collection, scheduling of alerts and reports.
Assist with the development and implementation of document configuration standards, policies and procedures for operating, managing and ensuring the security of Splunk infrastructure.
Collaborate with other teams to establish optimally functioning processes for system optimization, incident management, development practices, etc.
Collaborate with architects, designers, requirements analysts, database and system administrators and developers on system and application design.
Promote changes and releases through the various test, quality assurance and production systems.
Export, import, build and deploy various system configurations and components. Enterprise Monitoring systems and applications problem resolutions and production support.
Research and troubleshoot the most complex technical issues causing the unavailability of systems or applications for users and provide expertise for resolution.
Collaborate with vendor and internal support areas.
Document the cause and resolution of outages. Skill Sets Required Strong Linux/Unix background with scripting for automation and troubleshooting.
Advanced knowledge of logging and troubleshooting.
Demonstrated good overall understanding of enterpriselevel architecture.
HTTP/XML programming experience.
Strong interpersonal and communication skills, written and oral.
Strong documentation and organizational skills.
Understanding of Data Security principles and practices.
Excellent written and verbal skills.
Handson experience with Splunk or other data consolidation / data mining technologies.
Scripting experience with Regular Expressions, Python, JavaScript, XML or other languages.