Back to Blog January 13, 2022

    How to Use Python to Automate Data Enrichment

    The amount of context that cybersecurity analysts and engineers require for assessing security alerts is overwhelming. But here’s the good news: Data enrichment can help. Practitioners often perform data enrichment to determine the severity and criticality of a security alert.

    Data enrichment is the process of combining disparate data sources together to create a more meaningful story of existing data. For example, retrieving the hostname from a security alert and searching for the hostname in your CMDB, SIEM, and threat intelligence platform is a common data enrichment use case. 

    The challenge that teams often face is proactively presenting enriched data for team members while they’re assessing security alerts. In some situations, technology solutions are missing integrations or are in isolated segments of the network, which makes data enrichment tedious and manual. 

    It’s essential for teams to employ automation on tasks such as data enrichment to prevent burnout amongst team members and to translate team tradecraft into logic. Python is a great resource for security teams to leverage for automating data enrichment. 

    This post is a step-by-step walkthrough of implementing a data enrichment use case using Python. Follow along by starting a free trial with Axonius.

    Define Data Enrichment Use Case

    Before writing code, it’s important to ensure that your team has defined the use case that your Python script will help solve. The example use case in this post is intended to assist team members with data enrichment by providing asset intelligence for security alerts.

    Use Case

    Data Enrichment - Asset Intelligence

    Goal

    Reduce the Mean Time To Respond (MTTR) to security alerts by 10% by providing asset intelligence on security alert assessment.

    Trigger Event

    New issue is created in case management platform (JIRA) with Security Alert as issue type

    Precondition

    Empty Additional Context field in the case

    Postcondition

    Asset Intelligence from Axonius inserted in Additional Context field

    • Hostname(s)
    • Date Last Seen
    • Network Interface Addresses
    Identify Data Sources

    The primary goal when performing data enrichment is combining disparate data sources.  While selecting all the useful data sources is typically the end goal, it’s best to start with a few data sources and add additional sources as needed. While identifying data sources there are a few questions to consider answering:

    1. Does my data source have an API or a mechanism to collect data?
    2. Does my development environment have access to the data source?
    3. Are there any requirements to keep in mind when handling the data collected?
    4. Which attributes and fields are required to successfully implement my use case?

    A goal for our use case is to provide asset intelligence from Axonius when a new JIRA issue is created with “Security Alert” as the issue type. For this use case, two data sources are required.

    Data Source

    Data 

    Connection Type

    Permissions

    Internet Connected?

    JIRA

    Issue Details

    • Name
    • Description
    • Additional Context field

    API

    • Read Security Alert Incidents
    • Edit Security Alert Incidents

    True

    Axonius

    Asset Data

    • Hostname
    • Internal IP Address
    • External IP Address

    API

    • Search Assets
    • Read Asset Data

    True

    Define Security and Programming Logic

    Translating your team’s tradecraft into documentation and automated steps is critical for maturing your security program. The more detailed your security logic, the easier it will be to translate it into Python code. This can be initially done from a high level as documented steps. 

    Security Logic:

    1. Retrieve new Security Alert type issues
    2. Enrich hostname field with Asset Intelligence:
      > Hostname(s)
      > Date Last Seen(es)
      > Network Interface Addresses
    3. Update case Additional Context field with Asset Intelligence

    After your security logic is defined, it’s time to review API documentation to get an understanding of how to access data from the API. The field hostname is a custom field for the Security Alert issue type in JIRA. 

    The API documentation mentions that an additional API call is required to convert the hostname custom field into JIRA’s custom field ID. Depending on your data sources, you may have the option to leverage a Python library to interact with a data source. For example, to leverage Axonius as a data source the Python API Client can be installed to facilitate API calls between the development environment and the Axonius platform.

    Programming Logic:

    1. Retrieve metadata for JIRA project
    2. Retrieve new Security Alert type issues
    3. Search for hostname in Axonius
    4. Format response from Axonius 
    5. Update JIRA issue field, Additional Context field with formatted response
    Setting Up Development Environment

    A development environment in Python is a combination of a text editor and a Python runtime implementation. To set up your Python environment, ensure that you have Python 3 installed and run the following shell commands:

    Screen Shot 2022-01-12 at 2.38.23 PM

    When writing Python code it’s critical that you avoid saving secrets in your code. It’s best to store your secrets in environment variables if you’re not able to pull secrets directly from a secrets vault. Environment variables can also be used to store other static details like the URL of the JIRA instance.

    Screen Shot 2022-01-12 at 3.20.23 PM

    Write Code

    Writing Python code is a continuous cycle of design, implementation, and testing until the intended outcome is consistently achieved. When building Python applications, it’s important to consider the readability of your source code. After reviewing API documentation, outlining your programming logic, and setting up your development environment, you’re ready to begin taking the steps defined in your programming logic and converting them to Python functions.

    1. Import packages and assign static variables
    Screen Shot 2022-01-12 at 3.21.44 PM

    2. Retrieve metadata for JIRA project

    Screen Shot 2022-01-12 at 3.23.19 PM

    3. Retrieve new Security Alert type issues

    Screen Shot 2022-01-12 at 3.29.29 PM

    4. Search for hostname in Axonius

    Screen Shot 2022-01-12 at 3.30.26 PM

    5. Format response from Axonius

    Screen Shot 2022-01-12 at 3.31.29 PM

    6. Update JIRA issue field, Additional Context field with formatted response

    Screen Shot 2022-01-12 at 3.32.23 PM

    7. Combining Functions

    Screen Shot 2022-01-12 at 3.33.16 PM

    Don’t wait. Try Axonius free for 30 days.

    START FREE TRIAL

    Sign up to get first access to our latest resources