Detection database
The problem
Microsoft threat engineers managed their detection database using a web app that tracked details like the severity of a detection and what data it consumes. To manage detections more efficiently, the threat engineers requested new features including bulk updates, content tagging, and improved search.
My role was to add new features to the web app. As the only engineer on the project I knew it would take me too long to provide the users with the improvements they wanted. So, I decided to use Azure DevOps to fill customer needs and I used the migration as an opportunity to define the detection process.
Role: Project manager, engineer, content designer
Team size: 1
Tools: Azure DevOps, Azure Functions, Azure Logic Apps, Python, MITRE
Skills: Planning, UX writing, technical writing, coding, storytelling, automation
Timeline: 6 months
Outcome: Product and process
My process
User research
Understanding requirements
I setup individual user studies where I asked participants about their pain points with the current web app and what their ideal experience was and why. From my research I learned that users wanted a powerful experience that simplified work item tracking and that they were already using Azure DevOps for their team project planning.
I created a proposal for migrating to Azure DevOps rather than continuing with the existing web app (which users considered “clunky” and “frustrating”).
My justifications for using Azure DevOps were:
Supported first party tool - users will have access to product support and the ability to give direct user feedback for improvements.
Consistency - users are already familiar with the product and using it daily.
Customization - users have the freedom to create and customize work item types.
Integration options - has built-in reporting features and a robust API for automation.
Feature library - already includes the features users wanted and more.
Good technical tradeoff - does not require any resources to scale or maintain.
Defining a detection
As I learned about my users existing detection management tool and their process I learned there was a lot of inconsistency and ambiguity around what information a detection entailed. Detection work items included fields like “environment”, “status”, “type”, “workstream”. However, these field definitions were vague and non-existent.
I wanted the new system I built to be simple, concise, and easy to understand. In order to accomplish this, I met with users and stakeholders on a regular basis for 2 months. During these meetings we individually discussed each of the existing fields and use cases for detections as a whole. From my research I was able to remove unhelpful fields, add clarifying fields, and provide clear definitions for all fields in the form of a wiki dictionary and tooltips.
Redacted view of detection field dictionary wiki
Development
Azure DevOps
I setup a custom Azure DevOps organization and created the user experience for the detection work items based off my user research. I defined the fields and organized the detection work item form to allow users to achieve their goals in as few clicks as possible.
Security best practices
Due to the sensitive nature of this detection database, I enabled two factor authentication for access and limited security permissions within the organization.
Automatic data ingestion
In order to stay up to date with external and partner detection data sources, I used Python Azure Functions and Azure Logic Apps to automatically ingest data from the MITRE framework and other detection databases.
Redacted detection database architecture diagram
Documentation
I wrote and socialized documentation that taught users how to use the new detection database as well as the process for requesting changes and updates. I included this documentation in the Azure DevOps organization wiki.
The migration to the new detection database included massive improvements and was incredibly successful. I wrote and designed the walking deck content that leadership now uses to evangelize the detection database story.
Example slide from detection database evangelical content
The product
The resulting product is a robust 1st party service that includes increased security, scalability, integration with additional data sources, automation capabilities, and data story telling through visual dashboards.
Onboarding to a new product that fit the needs of the customers created vastly more future potential for their use cases and is now being used as an example for managing security data within their organization.
Redacted view of final detection database product