Creating, Validating, and Maintaining a Custom IoC File
This topic describes how to create and validate a custom IoC table you host on a server you manage and link to it via URL in Stellar Cyber as a third-party threat intelligence feed.
The overall procedure is described in Configuring Threat Intelligence and can be summarized as follows:
-
You configure the URL for the allthreat.tsv file in the System | Integration | Threat Intelligence page.
-
The URL points to an IoC file (allthreat.tsv) hosted on a server you manage.
-
The IoC file lists IP addresses, domains, and URLs with associated severity scores and conforms to the rules and syntax detailed in this topic.
-
You use the Python scripts described in this topic to validate the IoC file, as well as merge and remove entries.
See the following sections for details:
Contact Stellar Cyber Customer Support to obtain the scripts.
Creating a Custom IoC File (allthreat.tsv)
You can use the System | Integration | Threat Intelligence page to point Stellar Cyber to a threat intelligence feed you host on your own server in a custom allthreat.tsv file. The file is essentially a table listing URLs, IP addresses, and fully-qualified domains with an associated severity score. Stellar Cyber uses this table to generate Emerging Threat alerts on matching values detected in incoming data.
The rules for the customer-maintained file are as follows:
-
File Name: Must be allthreat.tsv.
-
Delimiter: Must be a tab
-
Fields, Field order, and Syntax:
Each row in the allthreat.tsv file must have fields in the order and syntax shown below. Rows that do not meet this format are ignored.
FIELD TYPE VALUE SOURCE SCORE Purpose Specify the type of address that will be entered in the next column. Specify the address of the threat. This value may be associated with multiple sources. Supply a name to identify the threat. Assign a severity score to the threat. Syntax url
ip
domain
A url
An IPv4 address
A fully qualified domain name
A-Z a-z 0-9 _ -
Alphanumeric, underscore, dash. No spaces, no other symbols.
Integer from 0-100
Use 90 if you prefer not to tune
Case Sensitive? yes no no no
Using Python Scripts to Manage allthreat.tsv
Stellar Cyber provides the following Python scripts to validate and maintain the IoC entries in your allthreat.tsv file. The scripts are supported for use with Python v3.0.
Script | Purpose |
---|---|
threat_verify.py | Validate the allthreat.tsv file. |
threat_merge.py |
Merge new entries into allthreat.tsv. |
threat_remove.py | Remove entries from allthreat.tsv. |
The threat_verify.py script requires that the validators Python package be installed.
Validating Your IoC File with threat_verify.py
Use this script to perform basic validation of your allthreat.tsv file. Any rows which do not meet requirements are reported on the command line and removed from the file.
Output Command Help
python3 threat_verify.py -h
Verify File
python3 threat_verify.py [--table_in TABLE_IN] [--table_out TABLE_OUT]
Argument | Description |
---|---|
--table_in | Absolute path and name of the file to be validated |
--table_out | The name of the output file, which will exclude any rows for which errors were encountered |
# cat allthreat.tsv ip 172.xxx.xx.777 TsT_Threat 90 url https://www.example.com TsT_Threat 90 domain example.com TsT_Threat 90 # python3 threat_verify.py --table_in home/myhome/allthreat.tsv --table_out home/myhome/allthreat_result.tsv Customer TI threat table parse error: ip 172.xxx.yy.777 TsT_Threat 90 # cat allthreat_result.tsv url https://www.example.com TsT_Threat 90 domain example.com TsT_Threat 90
Merging Entries with threat_merge.py
Use this script to merge entries from a specified .tsv file into your existing, validated allthreat.tsv file. The script ensures no duplicate entries are added.
Both files must obey the standard syntax and rules in Creating a Custom IoC File (allthreat.tsv). Be sure to run threat_verify.py separately on both files prior to using this script to ensure that you are merging valid content.
Output Command Help
python3 threat_merge.py -h
Merge Files
python3 threat_merge.py [--table_new TABLE_NEW] [--table_total TABLE_TOTAL]
Argument | Description |
---|---|
--table_new | Absolute path and name of the file containing new, validated entries |
--table_total | Absolute path and name of an already validated file to which you want to merge in the new entries |
Example
# cat allthreat2.tsv ip 172.xxx.xx.99 TsT_Threat 90 domain example.com TsT_Threat 90 url http://www.example.com TsT_Threat 90 # cat allthreat.tsv url http://www.example.com TsT_Threat 90 ip 172.xxx.xx.77 TsT_Threat 90 domain example2.com TsT_Threat 90 url https://www.example2.com TsT_Threat 90 # python3 threat_merge.py --table_new /home/myhome/allthreat2.tsv --table_total /home/myhome/allthreat.tsv # cat allthreat.tsv url http://www.example.com TsT_Threat 90 ip 172.xxx.xx.77 TsT_Threat 90 domain example2.com TsT_Threat 90 url https://www.example2.com TsT_Threat 90 ip 172.xxx.xx.99 TsT_Threat 90 domain example.com TsT_Threat 90
Removing Entries from your IoC File with threat_remove.py
Use this script to remove entries listed in a specified .tsv file from your existing, validated allthreat.tsv file.
Both files must obey the standard syntax and rules in Creating a Custom IoC File (allthreat.tsv). Be sure to run threat_verify.py separately on both files prior to using this script to ensure that you are removing valid content.
It's a good idea to make a backup copy of your allthreat.tsv file before running threat_remove.py on it.
Output Command Help
python3 threat_remove.py -h
Remove Entries
python3 threat_remove.py [--table_remove TABLE_REMOVE] [--table_total TABLE_TOTAL]
Argument | Description |
---|---|
--table_remove | Absolute path and name of the validated file containing entries that you want removed |
--table_total | Absolute path and name of an already validated file from which you want to remove entries. |
Example
# cat allthreatREMOVE.tsv ip 192.xxx.xx.99 TsT_Threat 90 domain example.com TsT_Threat 90 url http://www.example.com TsT_Threat 90 # cat allthreat.tsv domain example.com TsT_Threat 90 ip 192.xxx.xx.77 TsT_Threat 90 url http://www.example.com TsT_Threat 90 domain example2.com TsT_Threat 90 ip 192.xxx.xx.99 TsT_Threat 90 url https://www.example2.com TsT_Threat 90 # python3 threat_remove.py --table_remove /home/myhome/allthreatREMOVE.tsv --table_total /home/myhome/allthreat.tsv # cat allthreat.tsv ip 192.xxx.xx.77 TsT_Threat 90 domain example2.com TsT_Threat 90 url https://www.example2.com TsT_Threat 9