It is a simple component which performs minimal processing on the incoming data streams before forwarding them to an indexer.ĭata transfer is a major problem with almost every tool in the market. Universal Forwarder – You can opt for an universal forwarder if you want to forward the raw data collected at the source. Now, let us understand the different types of Splunk forwarders. You can scale them up to tens of thousands of remote systems easily, and collect terabytes of data with minimal impact on performance. To understand how real time forwarding of data happens, you can read my blog on how Domino’s is using Splunk to gain operational efficiency.Ĭompared to other traditional monitoring tools, Splunk Forwarder consumes very less cpu ~1-2%. You can install them in multiple systems and collect the data simultaneously from different machines in real time. You can configure the forwarders to send data to Splunk indexers in real-time. What if you want to do real-time analysis of the data? Splunk forwarders can be used for that purpose too. In fact, you can install several such forwarders in multiple machines, which will forward the log data to a Splunk Indexer for processing and storage. Suppose, you want to collect logs from a remote machine, then you can accomplish that by using Splunk’s remote forwarders which are independent of the main Splunk instance. Splunk Forwarder is the component which you have to use for collecting the logs. Search Head, is a GUI used for searching, analyzing and reporting.Splunk Indexer, used for Parsing and Indexing the data.Splunk Forwarder, used for data forwarding.If you look at the below image, you will understand the different data pipeline stages under which various Splunk components fall under. The search function also manages the search process. As part of the search function, Splunk software stores user-created knowledge objects, such as reports, event types, dashboards, alerts and field extractions. This stage controls how the user accesses, views, and uses the indexed data. The benefit of Indexing is that the data can be easily accessed during searching. It writes both compressed raw data and the corresponding index file. In Indexing phase, Splunk software writes parsed events to the index on disk.Transforming event data and metadata according to regex transform rules.Annotating individual events with metadata copied from the source-wide keys.Identifying, parsing, and setting timestamps.Breaking the stream of data into individual lines.

It is during this phase that Splunk software breaks the data stream into individual events. In Parsing phase, Splunk software examines, analyzes, and transforms the data to extract only the relevant information.Data Storage Stageĭata storage consists of two phases: Parsing and Indexing. The keys can also include values that are used internally, such as character encoding of the data stream and values that control the processing of data during the indexing stage, such as the index into which the events should be stored. The metadata keys include hostname, source, and source type of the data. In this stage, Splunk software consumes the raw data stream from its source, breaks it into 64K blocks, and annotates each block with metadata keys. There are primarily 3 different stages in Splunk: In case you want more clarity on what is Splunk, refer to the Splunk Certification that will give you an understanding of Splunk and tell you why it is a necessity for companies having a huge infrastructure.īefore I talk about how different Splunk components function, let me mention the various stages of data pipeline each component falls under. I have written this blog to help you understand the Splunk architecture and tell you how different Splunk components interact with one another. The demand for Splunk Certified professionals has seen a tremendous rise, mainly due to the ever-increasing machine-generated log data from almost every advanced technology that is shaping our world today. If you want to implement Splunk in your infrastructure, then it is important that you know how Splunk works internally.

Splunk join by two fields