Merge ubuntu-security-tools:iosifache/semgrep-rules-manager into ubuntu-security-tools:master

[George-Andrei Iosif]

Integrates semgrep-rules-manager

This merge proposal targets the auditing capabilities of UST.

As a result of the Semgrep integration in MP #446963 [1], the commits in this merge proposal include a newly constructed Semgrep-related utility, semgrep-rules-manager.

The objective of semgrep-rules-manager [2] is to manage third-party Semgrep rule sources by enabling actions such as downloading, listing, synchronizing, and uninstalling. Beside the open source codebase, there are also a snap [4] and a Python package [5] available.

The merge proposal adds a new workflow:
1. When the first time uaudit is launched, the user is prompted if she wishes to install new Semgrep rules.
2. If yes, the semgrep-rules-manager snap is installed on the local host.
3. Once the installation is complete, all Semgrep rule sources are downloaded to $UST/audits/custom-semgrep-rules/third-party. This allows the user to avoid getting lost in a maze of files by writing her custom rules in the root of $UST/audits/custom-semgrep-rules and having the automatically downloaded ones on $UST/audits/custom-semgrep-rules/third-party.
4. uaudit will then launch Semgrep, which will detect and use the downloaded rules for scanning.

Regardless of the answer supplied in the first step, semgrep-rules-manager can be alternatively launched with the --download-semgrep-rules flag from uaudit.

The commits, in addition to the mentioned behaviour, target documentation files (README.md and CONTRIBUTING.md), which are updated with information about the changes made in this MP.

[1] https://code.launchpad.net/~ubuntu-security/ubuntu-security-tools/+git/ubuntu-security-tools/+merge/446963
[2] https://github.com/iosifache/semgrep-rules-manager
[3] https://snapcraft.io/semgrep-rules-manager
[4] https://pypi.org/project/semgrep-rules-manager

P.S.: The previously generated Semgrep snap is actively utilized by tens of users in our community! The download of semgrep-rules-manager has also begun. So happy about this!

[Spyros Seimenis (sespiros)]

I think we can reuse the logic and flow that is already there for tool installation instead of introducing extra helpers. The new requirement of semgrep-rules-manager for example can be handled in verify_requirements() or maybe even better in uaudit_install_dependencies() directly:

```python
def uaudit_install_dependencies():
    dependencies = {}
    for tool in static_analysis_tools:
        dependencies[tool.name] = tool.install_cmd()

+ dependencies["semgrep-rules-manager"] = ["snap", "install", "semgrep-rules-manager"]

    for tool, cmd_args in dependencies.items():
        if is_program_installed(tool):
            debug(f'{tool} is already installed in the system')
        else:
            install_tool(cmd_args, tool)

```

Then we would only need to prompt for the extra sources via ask_for_custom_semgrep_rules() -> download_custom_semgrep_rules() and remove the ensure_installed_semgrep_rules_manager() method.

review: Needs Fixing

Unmerged commits

78b19f8... by George-Andrei Iosif on 2023-08-02

Updates the documentation with info about semgrep-rules-manager

Signed-off-by: George-Andrei Iosif <email address hidden>

3827dde... by George-Andrei Iosif on 2023-08-02

Integrates semgrep-rules-manager in uaudit

Signed-off-by: George-Andrei Iosif <email address hidden>

diff --git a/audits/.gitignore b/audits/.gitignore
 new file mode 100644
index 0000000..e01805a
--- /dev/null
+++ b/audits/.gitignore
@@ -0,0 +1,2 @@
+custom-semgrep-rules/third-party/*
+!custom-semgrep-rules/third-party/.gitkeep
 \ No newline at end of file
diff --git a/audits/CONTRIBUTING.md b/audits/CONTRIBUTING.md
index 8d59bfa..7c79d99 100644
--- a/audits/CONTRIBUTING.md
+++ b/audits/CONTRIBUTING.md
@@ -20,3 +20,7 @@
     ```
 
 5. If the `summary` member of your `StaticAnalysisTool` object is initialized, its content can be used in reports (e.g. `bug.template` and `workflow.template`) by embedding the `$tool_id>` variable. A frequent usecase is to save the number of created errors and warnings and reference it in the "Static analysis results summary" section of `jira.template`.
+
+## Adding Third-party Semgrep Rules
+
+Please follow the [contribution guide from `semgrep-rules-manager`](https://github.com/iosifache/semgrep-rules-manager/blob/main/CONTRIBUTING.md#add-a-new-source-with-semgrep-rules).
diff --git a/audits/README.md b/audits/README.md
index b34440f..841b25a 100644
--- a/audits/README.md
+++ b/audits/README.md
@@ -18,6 +18,8 @@
     $ uaudit --download-release=precise hello
     ```
 
+If you run `uaudit` for the first time on your system, it will ask if you wish to download additional third-party Semgrep rules. If you intend to scan with Semgrep and opt to download the rules, they will be saved in `./custom-semgrep-rules/third-party`. If you decided not to download them but later changed your mind, execute `uadit --download-semgrep-rules`.
+
 Observe the output of `uaudit`. It uses `umt` to download and build source and binary packages. It also creates an `audits` directory that contains all the reports.
 
 `uaudit` will generate its reports in `<toplevel source>/audits`. You might want to copy these to somewhere safe (eg, `~/ubuntu/security/audits`). Ideas for starting points are in `WORKFLOW.md`. Sift through the various reports, be
diff --git a/audits/custom-semgrep-rules/third-party/.gitkeep b/audits/custom-semgrep-rules/third-party/.gitkeep
 new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/audits/custom-semgrep-rules/third-party/.gitkeep
diff --git a/audits/uaudit b/audits/uaudit
index 96d03d2..f3b8aef 100755
--- a/audits/uaudit
+++ b/audits/uaudit
@@ -46,6 +46,9 @@ class StaticAnalysisToolSource(Enum):
 
 OUTPUT_FILE="$OUTPUT"
 
+def get_custom_semgrep_rules_location() -> str:
+    return os.path.join(os.path.dirname(__file__), "custom-semgrep-rules")
+
 class StaticAnalysisTool(object):
     """A class modeling a static analysis tool
 
@@ -265,7 +268,7 @@ static_analysis_tools = [
             "--config",
             "auto",
             "--config",
-            os.path.join(os.path.dirname(__file__), "custom-semgrep-rules")
+            get_custom_semgrep_rules_location()
         ],
         cmd_txt=["--vim"],
         cmd_json=["--json"],
@@ -506,15 +509,21 @@ def uaudit_setup_env():
 
 
 def query_user_consent_to_install_tool(cmd_args, tool):
-    valid_answers = {"yes": True, "y": True, "no": False, "n": False}
     command = " ".join(cmd_args)
     question = f"{tool} is not present in the system.\n" \
         "The preferred version can be installed by running:\n" \
         f"  {command}\n" \
         "Do you want to run this now to install it? [y/n] "
 
+    return ask_question_with_binary_response(question)
+
+
+def ask_question_with_binary_response(question) -> bool:
+    valid_answers = {"yes": True, "y": True, "no": False, "n": False}
+
     while True:
         sys.stdout.write(question)
+
         choice = input().lower()
         if choice in valid_answers:
             return valid_answers[choice]
@@ -538,12 +547,15 @@ def uaudit_install_dependencies():
     for tool in static_analysis_tools:
         dependencies[tool.name] = tool.install_cmd()
     for tool, cmd_args in dependencies.items():
-        rc, out = cmd(['which', tool])
-        if rc != 0:
-            install_tool(cmd_args, tool)
-        else:
+        if is_program_installed(tool):
             debug(f'{tool} is already installed in the system')
+        else:
+            install_tool(cmd_args, tool)
 
+def is_program_installed(software_name: str) -> bool:
+    rc, _ = cmd(['which', software_name])
+    
+    return (rc == 0)
 
 def verify_requirements():
     uaudit_setup_env()
@@ -1139,6 +1151,55 @@ def create_tags(audit_dir):
 
     msg("Tags file: %s" % (out_fn_rel))
 
+def ask_for_custom_semgrep_rules() -> None:
+    user_not_asked_before = check_create_semgrep_ask_file()
+    if not user_not_asked_before:
+        return
+
+    question = """\
+Because Semgrep is built into uaudit for SAST, you can download extra rules to
+your system to uncover more potential software flaws. Do you wish to install
+semgrep-rules-manager automatically and download all Semgrep rule sources?
+[y/n] """
+
+    response = ask_question_with_binary_response(question)
+    if response:
+        download_custom_semgrep_rules()
+
+
+def check_create_semgrep_ask_file() -> bool:
+    ask_fn = os.path.join(get_3rd_party_semgrep_rules_location(), ".ask")
+
+    if os.path.isfile(ask_fn):
+        return False
+
+    os.mknod(ask_fn)
+
+    return True
+
+
+def get_3rd_party_semgrep_rules_location() -> str:
+    return os.path.join(get_custom_semgrep_rules_location(), "third-party")
+
+
+def download_custom_semgrep_rules() -> None:
+    ensure_installed_semgrep_rules_manager()
+
+    location = get_3rd_party_semgrep_rules_location()
+
+    rc, out = cmd(["semgrep-rules-manager", "--dir", location, "download"])
+    if rc == 0:
+        msg("The third-party Semgrep rules were downloaded!")
+    else:
+        error(f"An error occurred while running semgrep-rules-manager!")
+
+
+def ensure_installed_semgrep_rules_manager() -> None:
+    if not is_program_installed("semgrep-rules-manager"):
+        rc, out = cmd(["snap", "install", "semgrep-rules-manager"])
+        if rc != 0:
+            error(f"An error occurred while installing semgrep-rules-manager!")
+
 
 def get_usage():
     '''Print usage statement'''
@@ -1237,6 +1298,11 @@ if __name__ == "__main__":
                         default=False,
                         help="Disable automatic coverity integration via cov-build",
                         action="store_true")
+    parser.add_argument("--download-semgrep-rules",
+                        dest="download_semgrep_rules",
+                        default=False,
+                        help="Download additonal Semgrep rules and exit",
+                        action="store_true")
     parser.add_argument("--debug",
                         default=False,
                         help="Print debug output",
@@ -1264,6 +1330,12 @@ if __name__ == "__main__":
 
     verify_requirements()
 
+    if args.download_semgrep_rules:
+        download_custom_semgrep_rules()
+        exit()
+    else:
+        ask_for_custom_semgrep_rules()
+
     if not os.path.exists(args.config):
         error("Could not find '%s'" % args.config)
     ust = load_ust_config(args.config)
diff --git a/audits/workflow.template b/audits/workflow.template
index 17735f3..c0b6c2a 100644
--- a/audits/workflow.template
+++ b/audits/workflow.template
@@ -74,7 +74,7 @@ the response within the Launchpad MIR bug.
 ### CVE History
 
 - Run $UCT/scripts/pkg_history $package.
-- Run $UCT/scripts/pkg_status $package
+- Run $UCT/scripts/pkg_status $package.
 - If there are old CVEs,
   - What were they?
   - Were they fixed in a timely fashion?