Splunk Search Basics

Search Basics

SPL is the key to turning Splunkd into a valuable asset. I look at it like SQL, if you have a pile of sales data, its not really any good to anyone until you can display it in ways that makes it useful to somoene in sales.

At first look the syntax may look somewhat intimidating, if its your frist time. Everything on the right is a subset of the data on the left. Splunk SPL uses a PIPE just like PowerShell to segment the query and just remember everything on the left of the PIPE | is passed to the right of the PIPE.

Field Values

When sorting data field values and attempting to perform calculations or comparisons make sure you have the correct datatype. My example was I had a PowerShell script placing "C_Drive_Free" data field values in my index as "34 GB". I decided to change my data field in my script to "C_Drive_Free_GB" and made my values to 34. Now I can properly work with data values and get expected results.

Sample of a single set of data in one event. Our goal is to query what we need and properly format the output to make it more readable.

Searching & Features

A Quick Look and Need to Know !

01

index

Your index is a dataset that you can search within to produce tables, reports, alerts. This is the core source of data in Splunk systems. The default index is "main" but I would suggest making your own indexes to separate out technologies. Examples: IIS, SQL, Event Logs, Firewall, Switches.

03

Search Timeframes

When creating a search make sure to select the appropriate timeframe to search within on the right side of the search textbox. This can greatly increase performance and speed up your query results. Also make sure to keep this in mind because Splunk index can hold millions of events \ records.

02

Universal Forwarder

This is a separate software element downloaded and installed on remote system to collect logs and other information and send it to Splunk. There are other means to get data to Splunk but this is made by Splunk and works quite well especially for new users that are just getting into Splunk.

04

Evaluation Period

Create a Splunk account and have access to download and install Splunk on Lunix or Windows. This also includes access to the Universal Forwarder software. Using Splunk during evaluation you can utilize all enterprise features, index data, search, install apps and more. Great for learning Splunk !

Search Basics

Field Value Pair

The very basics to learn is the search syntax (format) of how queries are built. There are many ways to start a search string so here are some basic examples of Field-value pair matching.

Note: "main" is the default index. If you don't specify your index when sending data it will default to "main".

Define index to search:

index="main"

Refine to only include specific host:

index="main" host="HostNameHere"

Refine Search by index, host, sourcetype, IP address

index="main" host=SERVER1 sourcetype=iis s_ip="192.168.1.145"

Filtering Results

IN

Using the IN search command allow you to put together a list in a string with comma seprated values. Seems to work as the term Match I have used in other program languages.

index="windows" source="WinEventLog:security"

| where Logon_Type IN (2,4)

isnotnull

When assigned to a a Field it filters out all null entries for that field from your resulting dataset.

head \ tail

Sorted by hadr drive free space and used "head 2" to only pull the first two results. Just a proof of concept so I can create a report of systems with the least amount of free hard drive space.

Click Image!

Filtering Results

Wildcards

Wildcard asterisk to return all account names that start with DW.

index="windows" source="WinEventLog:security" Account_Name=DW*

Where

This below lists all account login failures by host where count is above 3.

The breakdown of the query below:

The first line builds the data from the index we want to drill into.

The second does a calculation resulting in a column named "Total" then group by Account_Name.

The third line is a where clause to only show results where "Total" is greater than 3.

index="windows" source="WinEventLog:security" EventCode="4625"

| stats count(eval(EventCode="4625")) as Total by Account_Name

| where Total>3

Click Image!

earliest \ lastest

Allows you to specify the timeframe of the resulting dataset returned. You have to use thier syntax as shown HERE:

dedup

This allows one to return only unique entries. I designed this so I would have index updated each day and then I can return unique entries over the last day.

Click Image!

Formatting Results

upper \ lower

Can format the output to change the case of the data. I prefer to have my hostnames set to all uppercase.

Super Special Note:

In SPL there is a logical constraint on the order you place the commands in the query.

Below returns 0 results becuase the rename and eval are after the table command.

Correct Syntax: This is the exact same code just in a different order and will return the desired results show below.

Click Image!

rename

This allows you to rename a field for display as shown below.

strftime

Allows you to take the _time Splunk timestamp and format it.

Final Note: You notice that _time is in the Fields command but not in the Table command becuase its not needed. The _time is required to be in the Fields command if you want to use eval on this data. Field is like a statement of what data is included in your overall results. If its not in Fields it cannot be formated and placed in Table.