Package cli_tool_audit

Audit CLI tool version numbers.

cli_tool_audit

Verify that a list of cli tools are available. Like a requirements.txt for cli tools, but without an installer component. Intended to work with cli tools regardless to how they were installed, e.g. via pipx, npm, etc.

If 100% of your tools are installed by the same package manager that can install tools from a list with desired versions, then you don't need this tool.

Some useful scenarios:

  • Validating a developer's workstation instead of an "install everything" script.
    • Validating a CI environment and failing the build when configuration has drifted
  • Validating an end user's environment before running an app where you can't install all the dependencies for them.

How it works

You declare a list of cli commands and version ranges.

The tool will run tool --version for each tool and make best efforts to parse the result and compare it to the desired version range.

The tool then can either output a report with warnings or signal failure if something is missing, the wrong version or can't be determined.

There is no universal method for getting a version number from a CLI tool, nor is there a universal orderable version number system, so the outcome of many check may be limited to an existence check or exact version number check.

Here is an example run.

❯ cli_tool_audit audit
+--------+--------------------------+--------+----------+------------+----------+
|  Tool  |          Found           | Parsed | Desired  |   Status   | Modified |
+--------+--------------------------+--------+----------+------------+----------+
|  java  | openjdk version "17.0.6" | 17.0.6 | >=17.0.6 | Compatible | 01/18/23 |
|  make  |      GNU Make 3.81       | 3.81.0 |  >=3.81  | Compatible | 11/24/06 |
|        |       Copyright (        |        |          |            |          |
| python |      Python 3.11.1       | 3.11.1 | >=3.11.1 | Compatible | 01/13/24 |
+--------+--------------------------+--------+----------+------------+----------+

Installation

You will need to install it to your virtual environment if tools you are looking for are in your virtual environment. If all the tools are global then you can pipx install. It is on the roadmap to support a pipx install for all scenarios.

pipx install cli-tool-audit

Usage

Generate minimal config for a few tools.

cli_tool_audit freeze python java make rustc

Copy result of above into your pyproject.toml. Edit as needed, especially if you don't want snapshot versioning, which is probably too strict.

Audit the environment with the current configuration.

cli_tool_audit audit

All commands

❯ cli_tool_audit --help
usage: cli_tool_audit [-h] [-V] [--verbose] [--demo {pipx,venv,npm}]
                      {interactive,freeze,audit,single,read,create,update,delete} ...

Audit for existence and version number of cli tools.

positional arguments:
  {interactive,freeze,audit,single,read,create,update,delete}
                        Subcommands.
    interactive         Interactively edit configuration
    freeze              Freeze the versions of specified tools
    audit               Audit environment with current configuration
    single              Audit one tool without configuration file
    read                Read and list all tool configurations
    create              Create a new tool configuration
    update              Update an existing tool configuration
    delete              Delete a tool configuration

options:
  -h, --help            show this help message and exit
  -V, --version         Show program's version number and exit.
  --verbose             verbose output
  --demo {pipx,venv,npm}
                        Demo for values of npm, pipx or venv

    Examples:

        # Audit and report using pyproject.toml
        cli_tool_audit audit

        # Generate config for snapshots
        cli_tool_audit freeze python java make rustc

Note. If you use the create/update commands and specify the --version switch, it must have an equal sign.

Here is how to generate a freeze, a list of current versions by snapshot, for a lis tof tools. All tools will be check with --version unless they are well known.

cli_tool_audit freeze python java make rustc

This is for programmatic usage.

import cli_tool_audit

print(cli_tool_audit.validate(file_path="pyproject.toml"))

The configuration file lists the tools you expect how hints on how detect the version.

[tool.cli-tools]
# Typical example
pipx = { version = ">=1.0.0", version_switch = "--version" }
# Restrict to specific OS
brew = { version = ">=0.1.0", if_os="darwin" }
# Pin to a snapshot of the output of `poetry --version`
poetry = {version = "Poetry (version 1.5.1)", schema="snapshot"}
# Don't attempt to run `notepad --version`, just check if it is on the path
notepad = { schema = "existence" }
# Any version.
vulture = { version = "*" }
# Supports ^ and ~ version ranges.
shellcheck = { version = "^0.8.0" }
# Uses semver's compatibility logic, which is not the same as an exact match.
rustc = { version = "1.67.0" }

See semver3 for compatibility logic for versions without operators/symbols.

See poetry for version range specifiers.

See stackoverflow for os names.

Demos

Demos will discover a bunch of executables as installed in the local virtual environment, installed by pipx or installed by npm. It will then assume that we want the current or any version and run an audit. Since we know these files already exist, the failures are centered on failing to execute, failing to guess the version switch, failure to parse the switch or the tool's version switch returning a version incompatible to what the package manager reports.

cli_tool_audit --demo=pipx --verbose
cli_tool_audit --demo=venv --verbose
cli_tool_audit --demo=npm --verbose

How does this relate to package managers, e.g. apt, pipx, npm, choco, etc.

Package managers do far more than check for the existence of a tool. They will install it, at the desired version and make sure that tools and their transitive dependencies are compatible.

What they can't do is verify what other package managers have done.

This captures your desired tools, versions and guarantees you have them by installing them.

# list everything available on one machine
pip freeze>requirements.txt
# install it on another.
pip install -r requirements.txt

This is the same thing, but for windows and .net centric apps.

choco export requirements.txt
choco install -y requirements.txt

There are similar patterns, for apt, brew, npm, and so on.

It would be foolish to try to create a package manager that supports other package managers, so features in that vein are out of scope.

Prior Art

Use Cases

Developer Environment

Goal: Make sure developers have all the tools they need at a suitable version.

Alternative solutions: Docker files, imaged laptops, devcontainers, installing all tools with the same package manager.

Steps

  1. Freeze the versions of the tools you want to use. Developers may have different OS, and getting workstations exactly the same is difficult, so choose "semver" for some flexibility.
cli_tool_audit freeze python java make rustc --version semver
  1. Copy the output into your pyproject.toml.
  2. Edit the "installation_instructions" as needed
  3. Developers run on their machines, or as a early step in the local build process
cli_tool_audit audit

Server/Build Server Configuration Drift Detection

Goal: Know when the tool versions have drifted too far for comfort on the build server.

Alternative solutions: Docker base image that you control and upgrade on your own timeline.

  1. Freeze the versions of the tools you want to use.
cli_tool_audit freeze python java make rustc --version snapshot
  1. Copy the output into your pyproject.toml.
  2. Add to build script.
cli_tool_audit audit

End-User Application "pre-flight" checks

Goal: Make sure the user has the tools they need to run your application.

Alternative solutions: Bundle the other applications wiht your application.

Steps

  1. Freeze the versions of the tools you want to use.
cli_tool_audit freeze python java make rustc --version snapshot
  1. Copy the output into your pyproject.toml or other toml file.
  2. Call programmatically on startup.
import cli_tool_audit

results = cli_tool_audit.validate(file_path="path/to/your/config.toml")
# Display results if there are problems.

Environment Variables

You can modify the apps behavior with environment variables.

NO_COLOR

Disables progress bar, colored tables and color logging.

CI

Same as NO_COLOR

CLI_TOOL_AUDIT_TIMEOUT

This is how long a the application will wait for a tool to reply to a version query, defaults to 15 seconds.

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[3.0.0] - 2024-01-19

Fixed

  • only truncate long version on console output
  • removed three dependencies (toml, hypothesis, importlib-metadata)

Added

  • filter an audit by tags
  • install_docs, install_command to show what to do when there are problems
  • single to validate one tool without a config file
  • .html output

[2.0.0] - 2024-01-19

Fixed

  • Cache now adds gitingore and clears files older than 30 days on startup
  • Audit is now a sub command with arguments.

Changed

  • Gnu options with dashes, no underscores
  • Global opts that apply to only some commands move to subparser
  • check_only_for_existence is now schema type "existence" snapshot_version is now schema type "snapshot"
  • default action is now "audit" with all defaults.

[1.2.0] - 2024-01-17

Added

  • Caching of good results, speeding things up
  • Caching enabled only for batches of 5+ tool checks
  • TODO: command to clear cache/change cache location
  • Formalize support of four schemas, semver, pep440, snapshot and exists.
  • Added about.py

Fixed

  • "wrong OS" no longer flagged as problem
  • Added lock to ThreadPoolExecutor's work items

[1.1.0] - 2024-01-16

Added

  • Interactive config at cli with --interactive
  • Short switches for cli args and aliases for - and _ as connectors
  • Export multiple file formats with --file-format
  • Added subcommand for audit with intention of removing default action.

Fixed

  • Spelling and docs. Make file now lints docs and runs spell check tools.

[1.0.7] - 2024-01-16

Fixed

  • Spelling and docs. Make file now lints docs and runs spell check tools.

[1.0.6] - 2024-01-15

Fixed

  • Add "basic_test.sh" and fix issue found with it.

[1.0.5] - 2024-01-15

Fixed

  • Needs packaging lib in deps

[1.0.4] - 2024-01-14

Added

  • Support for if_os, snapshot
  • Config manager and possibility to do config via cli
  • Freeze command
  • Last modified

Fixed

  • Tested with tox and got passing on 3.9-3.12
  • Possibly fixed dependencies.

[1.0.3] - 2024-01-14

Added

  • Support for ^, ~ and * version ranges.

[1.0.2] - 2024-01-14

Added

  • Color mode actually works now, problems are in red.
  • Logging and verbose switch created

Fixed

  • When calling subprocess, it now checks stderr if nothing returned by stdout

[1.0.1] - 2024-01-13

Added

  • New stress test using npm.
  • Started concept of "known switches"

Fixed

  • All reports print with pretty tables
  • Version switch always defaults to --version
  • Tuple results are now dataclasses
  • validate is now a function and exported

[1.0.0] - 2024-01-13

Added

  • Application created.
Expand source code 
"""
Audit CLI tool version numbers.

.. include:: ../README.md

.. include:: ../docs/UseCases.md

.. include:: ../docs/EnvironmentVariables.md

.. include:: ../CHANGELOG.md
"""
__all__ = ["validate", "process_tools", "read_config", "check_tool_availability", "models", "__version__", "__about__"]

import cli_tool_audit.__about__ as __about__
import cli_tool_audit.models as models
from cli_tool_audit.__about__ import __version__
from cli_tool_audit.call_tools import check_tool_availability
from cli_tool_audit.config_reader import read_config
from cli_tool_audit.views import process_tools, validate

Sub-modules

cli_tool_audit.audit_cache

This module provides a facade for the audit manager that caches results.

cli_tool_audit.audit_manager

Class to audit a tool, abstract base class to allow for supporting different version schemas …

cli_tool_audit.call_and_compatible

Merge tool call and compatibility check results.

cli_tool_audit.call_tools

Check if an external tool is expected and available on the PATH …

cli_tool_audit.compatibility

Functions for checking compatibility between versions.

cli_tool_audit.compatibility_complex

This module contains functions to check if a found version is compatible with a desired version range.

cli_tool_audit.config_manager
cli_tool_audit.config_reader

Read list of tools from config.

cli_tool_audit.freeze

Capture current version of a list of tools.

cli_tool_audit.interactive

Interactively manage tool configurations.

cli_tool_audit.json_utils

JSON utility functions.

cli_tool_audit.known_switches

This file contains a dictionary of known switches for various CLI tools.

cli_tool_audit.logging_config

Logging configuration.

cli_tool_audit.models

This module contains dataclasses for the tool audit.

cli_tool_audit.policy

Apply various policies to the results of the tool checks.

cli_tool_audit.version_parsing
cli_tool_audit.view_npm_stress_test

This module contains a stress test for the cli_tool_audit module …

cli_tool_audit.view_pipx_stress_test

Stress test for the cli_tool_audit package using pipx installed tools as source data.

cli_tool_audit.view_venv_stress_test

Stress test for the cli_tool_audit package using venv as source data.

cli_tool_audit.views

Main output view for cli_tool_audit assuming tool list is in config.

Functions

def check_tool_availability(tool_name: str, schema: SchemaType, version_switch: str = '--version') ‑> ToolAvailabilityResult

Check if a tool is available in the system's PATH and if possible, determine a version number.

Args

tool_name : str
The name of the tool to check.
schema : SchemaType
The schema to use for the version.
version_switch : str
The switch to get the tool version. Defaults to '–version'.

Returns

ToolAvailabilityResult
An object containing the availability and version of the tool.
Expand source code 
def check_tool_availability(
    tool_name: str,
    schema: models.SchemaType,
    version_switch: str = "--version",
) -> models.ToolAvailabilityResult:
    """
    Check if a tool is available in the system's PATH and if possible, determine a version number.

    Args:
        tool_name (str): The name of the tool to check.
        schema (models.SchemaType): The schema to use for the version.
        version_switch (str): The switch to get the tool version. Defaults to '--version'.


    Returns:
        models.ToolAvailabilityResult: An object containing the availability and version of the tool.
    """
    # Check if the tool is in the system's PATH
    is_broken = True

    last_modified = get_command_last_modified_date(tool_name)
    if not last_modified:
        logger.warning(f"{tool_name} is not on path.")
        return models.ToolAvailabilityResult(False, True, None, last_modified)
    if schema == models.SchemaType.EXISTENCE:
        logger.debug(f"{tool_name} exists, but not checking for version.")
        return models.ToolAvailabilityResult(True, False, None, last_modified)

    if version_switch is None or version_switch == "--version":
        # override default.
        # Could be a problem if KNOWN_SWITCHES was ever wrong.
        version_switch = KNOWN_SWITCHES.get(tool_name, "--version")

    version = None

    # pylint: disable=broad-exception-caught
    try:
        command = [tool_name, version_switch]
        timeout = int(os.environ.get("CLI_TOOL_AUDIT_TIMEOUT", 15))
        result = subprocess.run(
            command, capture_output=True, text=True, timeout=timeout, shell=False, check=True
        )  # nosec
        # Sometimes version is on line 2 or later.
        version = result.stdout.strip()
        if not version:
            # check stderror
            logger.debug("Got nothing from stdout, checking stderror")
            version = result.stderr.strip()

        logger.debug(f"Called tool with {' '.join(command)}, got  {version}")
        is_broken = False
    except subprocess.CalledProcessError as exception:
        is_broken = True
        logger.error(f"{tool_name} failed invocation with {exception}")
        logger.error(f"{tool_name} stderr: {exception.stderr}")
        logger.error(f"{tool_name} stdout: {exception.stdout}")
    except FileNotFoundError:
        logger.error(f"{tool_name} is not on path.")
        return models.ToolAvailabilityResult(False, True, None, last_modified)

    return models.ToolAvailabilityResult(True, is_broken, version, last_modified)
def process_tools(cli_tools: dict[str, CliToolConfig], no_cache: bool = False, tags: Optional[list[str]] = None, disable_progress_bar: bool = False) ‑> list[ToolCheckResult]

Process the tools from a dictionary of CliToolConfig objects.

Args

cli_tools : dict[str, CliToolConfig]
A dictionary of tool names and CliToolConfig objects.
no_cache : bool, optional
If True, don't use the cache. Defaults to False.
tags : Optional[list[str]], optional
Only check tools with these tags. Defaults to None.
disable_progress_bar : bool, optional
If True, disable the progress bar. Defaults to False.

Returns

list[ToolCheckResult]
A list of ToolCheckResult objects.
Expand source code 
def process_tools(
    cli_tools: dict[str, models.CliToolConfig],
    no_cache: bool = False,
    tags: Optional[list[str]] = None,
    disable_progress_bar: bool = False,
) -> list[models.ToolCheckResult]:
    """
    Process the tools from a dictionary of CliToolConfig objects.

    Args:
        cli_tools (dict[str, models.CliToolConfig]): A dictionary of tool names and CliToolConfig objects.
        no_cache (bool, optional): If True, don't use the cache. Defaults to False.
        tags (Optional[list[str]], optional): Only check tools with these tags. Defaults to None.
        disable_progress_bar (bool, optional): If True, disable the progress bar. Defaults to False.

    Returns:
        list[models.ToolCheckResult]: A list of ToolCheckResult objects.
    """
    if tags:
        print(tags)
        cli_tools = {
            tool: config
            for tool, config in cli_tools.items()
            if config.tags and any(tag in config.tags for tag in tags)
        }

    # Determine the number of available CPUs
    num_cpus = os.cpu_count()

    enable_cache = len(cli_tools) >= 5
    # Create a ThreadPoolExecutor with one thread per CPU

    if no_cache:
        enable_cache = False
    lock = Lock()
    # Threaded appears faster.
    # lock = Dummy()
    # with ProcessPoolExecutor(max_workers=num_cpus) as executor:
    with ThreadPoolExecutor(max_workers=num_cpus) as executor:
        disable = should_show_progress_bar(cli_tools)
        with tqdm(total=len(cli_tools), disable=disable) as pbar:
            # Submit tasks to the executor
            futures = [
                executor.submit(call_and_compatible.check_tool_wrapper, (tool, config, lock, enable_cache))
                for tool, config in cli_tools.items()
            ]
            results = []
            for future in concurrent.futures.as_completed(futures):
                result = future.result()
                pbar.update(1)
                results.append(result)
    return results
def read_config(file_path: pathlib.Path) ‑> dict[str, CliToolConfig]

Read the cli-tools section from a pyproject.toml file.

Args

file_path : Path
The path to the pyproject.toml file.

Returns

dict[str, CliToolConfig]
A dictionary with the cli-tools section.
Expand source code 
def read_config(file_path: Path) -> dict[str, models.CliToolConfig]:
    """
    Read the cli-tools section from a pyproject.toml file.

    Args:
        file_path (Path): The path to the pyproject.toml file.

    Returns:
        dict[str, models.CliToolConfig]: A dictionary with the cli-tools section.
    """
    # pylint: disable=broad-exception-caught
    try:
        logger.debug(f"Loading config from {file_path}")
        manager = config_manager.ConfigManager(file_path)
        found = manager.read_config()
        if not found:
            logger.warning("Config section not found, expected [tool.cli-tools] with values")
        return manager.tools
    except BaseException as e:
        logger.error(e)
        print(f"Error reading pyproject.toml: {e}")
        return {}
def validate(file_path: pathlib.Path = WindowsPath('pyproject.toml'), no_cache: bool = False, tags: Optional[list[str]] = None, disable_progress_bar: bool = False) ‑> list[ToolCheckResult]

Validate the tools in the pyproject.toml file.

Args

file_path : Path, optional
The path to the pyproject.toml file. Defaults to "pyproject.toml".
no_cache : bool, optional
If True, don't use the cache. Defaults to False.
tags : Optional[list[str]], optional
Only check tools with these tags. Defaults to None.
disable_progress_bar : bool, optional
If True, disable the progress bar. Defaults to False.

Returns

list[ToolCheckResult]
A list of ToolCheckResult objects.
Expand source code 
def validate(
    file_path: Path = Path("pyproject.toml"),
    no_cache: bool = False,
    tags: Optional[list[str]] = None,
    disable_progress_bar: bool = False,
) -> list[models.ToolCheckResult]:
    """
    Validate the tools in the pyproject.toml file.

    Args:
        file_path (Path, optional): The path to the pyproject.toml file. Defaults to "pyproject.toml".
        no_cache (bool, optional): If True, don't use the cache. Defaults to False.
        tags (Optional[list[str]], optional): Only check tools with these tags. Defaults to None.
        disable_progress_bar (bool, optional): If True, disable the progress bar. Defaults to False.

    Returns:
        list[models.ToolCheckResult]: A list of ToolCheckResult objects.
    """
    if tags is None:
        tags = []
    cli_tools = config_reader.read_config(file_path)
    return process_tools(cli_tools, no_cache, tags, disable_progress_bar=disable_progress_bar)