Creating, Validating, and Maintaining a Custom IoC File

This topic describes how to create and validate a custom IoC table you host on a server you manage and link to it via URL in Stellar Cyber as a third-party threat intelligence feed.

The overall procedure is described in Configuring Threat Intelligence and can be summarized as follows:

  1. You configure the URL for the allthreat.tsv file in the System | Threat Intelligence page.

  2. The URL points to an IoC file (allthreat.tsv) hosted on a server you manage.

  3. The IoC file lists IP addresses, domains, and URLs with associated severity scores and conforms to the rules and syntax detailed in this topic.

  4. You use the Python scripts described in this topic to validate the IoC file, as well as merge and remove entries.

See the following sections for details:

Contact Stellar Cyber Customer Support to obtain the scripts.

Creating a Custom IoC File (allthreat.tsv)

You can use the System | Threat Intelligence page to point Stellar Cyber to a threat intelligence feed you host on your own server in a custom allthreat.tsv file. The file is essentially a table listing URLs, IP addresses, and fully-qualified domains with an associated severity score. Stellar Cyber uses this table to generate Emerging Threat alerts on matching values detected in incoming data.

The rules for the customer-maintained file are as follows:

  • File Name: Must be allthreat.tsv.

  • Delimiter: Must be a tab

  • Fields, Field order, and Syntax:

    Each row in the allthreat.tsv file must have fields in the order and syntax shown below. Rows that do not meet this format are ignored.

    FIELD TYPE VALUE SOURCE SCORE
    Purpose Specify the type of address that will be entered in the next column. Specify the address of the threat. This value may be associated with multiple sources. Supply a name to identify the threat. Assign a severity score to the threat.
    Syntax

    url

    ip

    domain

    A url

    An IPv4 address

    A fully qualified domain name

    A-Z a-z 0-9 _ -

    Alphanumeric, underscore, dash. No spaces, no other symbols.

    Integer from 0-100

    Use 90 if you prefer not to tune
    Case Sensitive? yes no no no

Using Python Scripts to Manage allthreat.tsv

Stellar Cyber provides the following Python scripts to validate and maintain the IoC entries in your allthreat.tsv file. The scripts are supported for use with Python v3.0.

Script Purpose
threat_verify.py Validate the allthreat.tsv file.
threat_merge.py

Merge new entries into allthreat.tsv.
threat_remove.py Remove entries from allthreat.tsv.

The threat_verify.py script requires that the validators Python package be installed.

Validating Your IoC File with threat_verify.py

Use this script to perform basic validation of your allthreat.tsv file. Any rows which do not meet requirements are reported on the command line and removed from the file.

Output Command Help

python3 threat_verify.py -h

Verify File

python3 threat_verify.py [--table_in TABLE_IN] [--table_out TABLE_OUT]

Argument Description
--table_in Absolute path and name of the file to be validated
--table_out The name of the output file, which will exclude any rows for which errors were encountered
Example
            # cat allthreat.tsv 
ip	172.xxx.xx.777	TsT_Threat	90
url	https://www.example.com	TsT_Threat	90
domain	example.com	TsT_Threat	90
# python3 threat_verify.py --table_in home/myhome/allthreat.tsv --table_out home/myhome/allthreat_result.tsv
Customer TI threat table parse error: ip	172.xxx.yy.777	TsT_Threat	90
# cat allthreat_result.tsv 
url	https://www.example.com	TsT_Threat	90
domain	example.com	TsT_Threat	90

Merging Entries with threat_merge.py

Use this script to merge entries from a specified .tsv file into your existing, validated allthreat.tsv file. The script ensures no duplicate entries are added.

Both files must obey the standard syntax and rules in Creating a Custom IoC File (allthreat.tsv). Be sure to run threat_verify.py separately on both files prior to using this script to ensure that you are merging valid content.

Output Command Help

python3 threat_merge.py -h

Merge Files

python3 threat_merge.py [--table_new TABLE_NEW] [--table_total TABLE_TOTAL]

Argument Description
--table_new Absolute path and name of the file containing new, validated entries
--table_total Absolute path and name of an already validated file to which you want to merge in the new entries

Example 

            # cat allthreat2.tsv
ip	172.xxx.xx.99	TsT_Threat	90
domain	example.com	TsT_Threat	90
url	http://www.example.com	TsT_Threat	90
# cat allthreat.tsv
url	http://www.example.com	TsT_Threat	90
ip	172.xxx.xx.77	TsT_Threat	90
domain	example2.com	TsT_Threat	90
url	https://www.example2.com	TsT_Threat	90
# python3 threat_merge.py --table_new /home/myhome/allthreat2.tsv --table_total /home/myhome/allthreat.tsv 
# cat allthreat.tsv 
url	http://www.example.com	TsT_Threat	90
ip	172.xxx.xx.77	TsT_Threat	90
domain	example2.com	TsT_Threat	90
url	https://www.example2.com	TsT_Threat	90
ip	172.xxx.xx.99	TsT_Threat	90
domain	example.com	TsT_Threat	90

Removing Entries from your IoC File with threat_remove.py

Use this script to remove entries listed in a specified .tsv file from your existing, validated allthreat.tsv file.

Both files must obey the standard syntax and rules in Creating a Custom IoC File (allthreat.tsv). Be sure to run threat_verify.py separately on both files prior to using this script to ensure that you are removing valid content.

It's a good idea to make a backup copy of your allthreat.tsv file before running threat_remove.py on it.

Output Command Help

python3 threat_remove.py -h

Remove Entries

python3 threat_remove.py [--table_remove TABLE_REMOVE] [--table_total TABLE_TOTAL]

Argument Description
--table_remove Absolute path and name of the validated file containing entries that you want removed
--table_total Absolute path and name of an already validated file from which you want to remove entries.

Example 

            # cat allthreatREMOVE.tsv
ip	192.xxx.xx.99	TsT_Threat	90
domain	example.com	TsT_Threat	90
url	http://www.example.com	TsT_Threat	90
# cat allthreat.tsv 
domain	example.com	TsT_Threat	90
ip	192.xxx.xx.77	TsT_Threat	90
url	http://www.example.com	TsT_Threat	90
domain	example2.com	TsT_Threat	90
ip	192.xxx.xx.99	TsT_Threat	90
url	https://www.example2.com	TsT_Threat	90
# python3 threat_remove.py --table_remove /home/myhome/allthreatREMOVE.tsv --table_total /home/myhome/allthreat.tsv 
# cat allthreat.tsv
ip	192.xxx.xx.77	TsT_Threat	90
domain	example2.com	TsT_Threat	90
url	https://www.example2.com	TsT_Threat	9