[Solutions] Research the lengthy security detection flow of product release, design and propose automating solutions (Fortify on Demand and Black Duck scans)

This automation solutions was my research for the last half year of my time at this company.

We have a lengthy security detection flow which needs to pass each time when there is new feature wants to release on production. The average time of passing this flow is approximately 3 weeks to 1 month, as soon as 2 weeks and the slowest is more than a month.

To complete this flow, we need at first product manager provide detailed information of new/fix feature and prodocu when submitting the release request, then the information will be reviewed by security department, some times need to several meeting calls or comments to ensure the correctness of information.

Then we need engineers to help do scans on each related micro-service manually and provide the evidence of the scan result.

I remember one time we finished 5 features development and all release schedules were slow because of the stuck of the flow.

Since the June of this year (2023), we were committed to research the flow, design and propose the solutions to automate the part of doing scans. We developed the program, helped our team and US team and propose this solution to cross-orgnization teams.

Thanks for the supports from the company, team and my colleagues, they supported us to research and achieve many innovative solutions during this work period.

Here is the research.

Table of Content

Research the whole flow and found the possibility automating steps.
The system design of the first program.
Hackathon: version extension - cloud version and local script optimization.
- Cloud version - FoD and BD APIs system design
- Local version - developing a new UI interface for the local python script
Optimization of features
Conclusion

Research the whole flow and found the possibility automating steps.

We clarified the whole flow by interviewed our product manager, Engineers and QA Engineers.

This is the whole flow and I removed sensitive information.

The Release Request Flow:

Submit the Release Request by creating a Jira Ticket
- Create a Jira Ticket
- Fill Up required information
- Submit for Review
Review
- Review the release content on Jira
  - If information is incorrect, return for revising.
  - If correct, open the Security Detection Ticket.
Security Detection Request
- Black Duck Scan
  - Generate and download report for each micro-service
  - Check and prove there is not In violation in micro-service by screenshot.
  - Package reports and screenshot as attachment and upload to Jira.
  - Prepare for review.
- Fortify On Demand Scan
  - Scan each micro-service or check the last scanning date not over 30 days.
  - Gather the result links and update to Jira.
  - Prepare for review
- … (At least 3 scans, max 5.)

During the whole flow, there are possibilities to cost many days on the request returning and revising.

After clarified the whole flow, we found the possible automating steps in following list.

Create/Clone Release Request Jira Ticket -> Jira API
- Clone previous Jira Ticket, avoid the manual inputting error.
- Create new ticket and paste the cloned content.
Check the status of ticket -> Jira API
- Check the status changes of ticket for the next steps.
- Check any new comment (Need to take action on revising)
Automating Black Duck Scan -> Selenium + Jira API Upload attachment
Automating Fortify On Demand Scan -> Selenium + Jira API Update links to comment
Check any status changes or new comment on Ticket and notify success message to user for deploy features to production. -> Jira API + Python

Finally, we chosen to develop program and automated the Black Duck Scan and Fortify On Demand Scan.

The system design of the first program.

We decided to develop a python script run locally and automating the the Black Duck Scan and Fortify On Demand scans by Selenium, upload/update the reports to Jira Ticket by Jira APIs.

Here is the diagram.

We designed 2 ways for inputting the micro-services list which need to scan.

In our team, we have a standardized document format to record each release information, so the first way is inputting the URL of the document and script will crawl the services list from it.

However, we considered there is not each team uses this format, so we designed the second way which is inputting required scan service list manually.

After getting the service list, the script will automate:

Fortify On Demand scan (trigger Selenium open Chrome to do scan)
Update the gathered result links to Jira comment by using Jira APIs.
Black Duck scan (trigger Selenium open Chrome to do scan)
Upload the packaged reports and screenshots as attachment to specific Jira ticket by Jira APIs.

If you have interests on how to use Jira APIs, you can refer this article

By the way, Selenium supported Headless Mode, so once you set the mode and it will run automating at the backgound, so users can do other things and not interrupt by automating.

So here is the first local script version and it helped us to finish many times release request flow.

Hackathon: version extension - cloud version and local script optimization.

After the first version script has been used officially, we met the 2 months internal Hackathon in our company, our manager propose us to join the competition and extended the cloud version which we wrote in future work.

Future Work: if there is chance in future, I would like to research how to move the automating script to Cloud.

The target of this Hackthon, I want to research and develop the AWS cloud version automating solution. However, we found the difficult point: Authentication which might let us failed in this challenge.

In the normal flow, before we open the scan services of Fortify On Demand (FoD) and Black Duck (BD), we need to pass the series of authentication steps until generate the SAML based on our account and password.

Then we are authorized to visit 2 scan services on website with SAML protocol.

The first version local script can run the automating because before we trigger the script, we have already pass the authentication by ourselves, so website will save the cookie or infomration in localStorage.

If we use cloud version which means, every time when cloud service be ran, everything will initialize and start at 0.

We wanted to use AWS ECS to build container and request users pass the authentication manually at first time, but this didn’t work.
We tried to think if we can develop an AWS Amplify to get the account and password from users, but this against the security, like a phishing site.
Final, after discussed with other team members, we found we can research the APIs of FoD and BD scans, visit 2 scans webiste directly and skip the authentication.
The conservative option is if the cloud version failed, we can still develop an interface for our local script and submit this to final result.

So we planned 3 solutions to research.

[Cloud Version] To research if there is other AWS services can help us overcome the authentication and generate the SAML protocol.
[Cloud Version] Research FoD and BD scan APIs, develop AWS lambda services to do automating scans.
[Local Version] Optimizing the original local script, design an new interface for script, so users don’t need to face the termincal or open IDE to trigger script.

In the end of Hackathon, I want to show the result of the second and third solutions.

The first solution was researched by my colleague, so I won’t put it here.

Cloud version - FoD and BD APIs system design

So after reseaching the APIs of 2 scans in 2-3 weeks, I have found which APIs can achieve our automating steps.

In system design, I designed an API Endpoint by AWS API Gateway allow user request RESTful API with POST method to trigger Lambda and do automating scans, then save the result in S3 bucket.

However, in the steps of BD scan, there is one step requires to provide the screenshot of the In Violation filter of the scan, Lambda can not achieve it, so we design an addtional script with Selenium to help taking screenshot.

The program list in development.

[AWS Lambda] Fortify On Demand Scanner
[AWS Lambda] Black Duck Scanner
[AWS Gateway] RESTful POST api endpoint
[Local Python Script] automating screenshot by Selenium

POST Request Body

// FoD scan api: /api/v1/fortify-on-demand/
{
  "tenet": "",
  "username": "",
  "token": "",
  "serviceList": []

}

// BD scan api: /api/v1/blackduck/
{
  "token": "",
  "serviceList": []

}

About the research of APIs of Fortify On Demand and Black Duck, I will write another article.

Local version - developing a new UI interface for the local python script

In the local version, we decided to develop a new UI interface for users, so users don’t need to use IDE and termincal to triger the original local script.

I considered to connect Python local script more easily, so I planned to use Python GUI package Tkinder to develop the inferface.

The interface I designed by Figma.

The Hompage of the application is choosing the way for inputting the list of services for scanning.

And designed a log page for showing the logs for dynamic running result, includes success and errors.

Final, showing the scan result, users can check the result manually before upload the evidence to Jira Ticket.

When developing, we have some consideration, because the second solution of API endpoint has finished, so I was thinking to instead the part of connecting original local script by requesting API of second cloud version. Keep the Selenium part for doing screenshot in the step of BD scan and this can fill a vacancy which lambda can not take screenshot.

Also using request APIs of BD and FoD scans can raise the the stability of the system.

Optimization of features

We removed update/upload to Jira ticket, because users want to check the result manually before upload automitically to ticket.
Revise broken FoD service name if the name is too long.
Help to gather the BD scan links on the scanned service list.

Conclusion

Final, I finished the second solution which is [Cloud Version] request AWS Gateway RESTful API with POST method to trigger AWS Lambda for automate scans through the APIs of 2 scan services.

The third solution, because our team got the official announcement of team disbanded, so we don’t have chance to continue our research and don’t have time to achieve Hackathon because my last day is early before the end of date of Hackathon, but this still a interesting oppotunity to join the competition.

It’s very happy that I finished the final goal - cloud version automating program and handovered to US Team, I researched and developed this automating solution and it is still helping US Team members to handle the scans.

Thank you for your read.

Sign @MinaYu.

Innovative Solutions