Package sysbot

SysBot - System Test Automation Library

MIT License

Copyright (c) 2024 Thibault SCIRE

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

For complete documentation, see the README.md and CONTRIBUTING.md files included in this package directory.

SysBot

Table of Contents

Overview

SysBot is a system test tool that provides a unified interface for connecting to and testing various systems through different protocols. Built with Robot Framework integration in mind, it offers a modular architecture that simplifies system automation and testing.

Key Features

  • Multi-protocol Support: SSH, HTTP, WinRM, Socket, and more
  • SSH Tunneling: Support for nested SSH tunnels with automatic management
  • Cross-platform: Support for Linux and Windows systems
  • Robot Framework Integration: Built-in support for Robot Framework automation with GLOBAL scope
  • Modular Architecture: Dynamic components loading and discovery (modules and plugins)
  • Connection Management: Robust session caching and lifecycle management
  • Secret Management: Secure storage and retrieval of sensitive data
  • Database Listeners: Store test results in SQLite, MySQL, PostgreSQL, or MongoDB
  • Polarion Integration: Generate Polarion-compatible xUnit reports for ALM/QA integration

Architecture

sysbot/
├── Sysbot.py           # Main SysBot class
├── connectors/         # Protocol-specific connectors
├── plugins/            # Plugins utilities (data, vault)
├── utils/
│   └── engine.py       # Engine class
└── modules/            # Modules

Installation

Prerequisites

  • Python 3.8 or higher
  • pip

Install from PyPI

pip install sysbot

Optional Dependencies

For specific features, you can install additional dependencies:

# Install with all database support
pip install sysbot[all_databases]

# Install with specific database support
pip install sysbot[mysql]        # MySQL support only
pip install sysbot[postgresql]   # PostgreSQL support only
pip install sysbot[mongodb]      # MongoDB support only

# Install with development dependency
pip install sysbot[dev]

Quickstart

Basic SSH Connection

import sysbot

bot = sysbot.Sysbot()

# Open an SSH session to a Linux system
bot.open_session(
    alias="my_linux_server",
    protocol="ssh",
    product="bash",
    host="192.168.1.100",
    port=22,
    login="username",
    password="password"
)

# Execute a command
result = bot.execute_command("my_linux_server", "ls -la")
print(result)

# Close all sessions
bot.close_all_sessions()

SSH Tunneling

# Configure nested SSH tunnels
tunnel_config = [
    {
        "ip": "192.168.1.1",
        "port": 22,
        "username": "user1",
        "password": "pass1"
    },
    {
        "ip": "192.168.2.1", 
        "port": 22,
        "username": "user2",
        "password": "pass2"
    }
]

# Open session through tunnels
bot.open_session(
    alias="tunneled_server",
    protocol="ssh", # or http / winrm / ect...
    product="bash",
    host="192.168.3.100",
    port=22,
    login="final_user",
    password="final_pass",
    tunnel_config=tunnel_config
)

Secret Management

SysBot provides a built-in secret management system for secure storage and retrieval of sensitive data like passwords, tokens, and configuration values. Secrets can be stored directly or loaded from external sources like files or HashiCorp Vault.

import sysbot

bot = sysbot.Sysbot()

# Using plugins with secret management
bot.plugins.data.csv("/path/to/file", key="my_secret")
secret_data = bot.get_secret("my_secret.0.name")

# Secret management without plugin
bot.add_secret("new_secret", "very_secret_value")
bot.get_secret("new_secret")
bot.remove_secret("new_secret")

# Using Vault plugin to dump HashiCorp Vault secrets
bot.plugins.vault.dump_engine(
    token="hvs.CAESIJ...",
    url="https://vault.example.com:8200",
    engine_name="secret",
    key="vault_secrets",
    verify_ssl=False  # Set to True for production with valid certificates
)
# Access Vault secrets using dot notation
db_url = bot.get_secret("vault_secrets.myapp/config.database_url")

Module System

# Import sysbot to access loaded components (all modules/plugins loaded by default)
import sysbot

bot = sysbot.Sysbot()

# Open an SSH session to a Linux system
bot.open_session(
    alias="my_linux_server",
    protocol="ssh",
    product="bash",
    host="192.168.1.100",
    port=22,
    login="username",
    password="password"
)

result = bot.linux.dnf.repolist("my_linux_server")

Session Management

# Close a specific session
bot.close_session("my_linux_server")

# Close all sessions (automatically handles tunnels)
bot.close_all_sessions()

Supported Protocols

SSH

  • Bash: Full support for bash via SSH
  • Powershell: Support for powershell via SSH (requires SSH server)

Local Execution

  • Bash: Execute bash/shell commands locally without SSH
  • Powershell: Execute PowerShell commands locally without SSH or WinRM

SysBot provides local execution connectors that allow running commands directly on the local machine without the overhead of SSH or WinRM connections. This is useful for: - Running commands on the local system during automation - Testing without remote systems - Avoiding connection overhead for local operations

Local Bash Execution:

import sysbot

bot = sysbot.Sysbot()

# Open a local bash session (no actual connection is made)
bot.open_session(
    alias="local_bash",
    protocol="local",
    product="bash",
    host="localhost",  # Required but not used
    port=0  # Required but not used
)

# Execute commands locally
result = bot.execute_command("local_bash", "ls -la")
print(result)

bot.close_session("local_bash")

Local PowerShell Execution:

import sysbot

bot = sysbot.Sysbot()

# Open a local PowerShell session (no actual connection is made)
bot.open_session(
    alias="local_ps",
    protocol="local",
    product="powershell",
    host="localhost",  # Required but not used
    port=0  # Required but not used
)

# Execute PowerShell commands locally
result = bot.execute_command("local_ps", "Get-Process | Select-Object -First 5")
print(result)

bot.close_session("local_ps")

HTTP/HTTPS

SysBot provides a generic HTTP/HTTPS connector with support for 9 authentication methods.

Supported Authentication Methods: 1. API Key (apikey) - API Key authentication via headers or query parameters 2. Basic Auth (basicauth) - Standard HTTP Basic Authentication 3. OAuth 1.0 (oauth1) - OAuth 1.0 authentication (RFC 5849) 4. OAuth 2.0 (oauth2) - OAuth 2.0 Bearer token authentication 5. JWT (jwt) - JSON Web Token authentication with automatic token generation 6. SAML (saml) - SAML assertion/token authentication 7. HMAC (hmac) - HMAC signature-based authentication 8. Certificate (certificate) - Client certificate authentication (mutual TLS) 9. OpenID Connect (openidconnect) - OpenID Connect authentication

Usage Examples:

Basic Authentication:

bot.open_session(
    alias="my_api",
    protocol="http",
    product="basicauth",
    host="api.example.com",
    port=443,
    login="username",
    password="password"
)

result = bot.execute_command("my_api", "/users", options={"method": "GET"})

API Key Authentication:

bot.open_session(
    alias="my_api",
    protocol="http",
    product="apikey",
    host="api.example.com",
    port=443,
    api_key="your-api-key-here",
    api_key_header="X-API-Key"  # Custom header name (optional)
)

result = bot.execute_command("my_api", "/data", options={"method": "GET"})

WinRM

  • Powershell: Native Windows Remote Management support

Socket

  • TCP: Native TCP socket with SSL if needed
  • UDP: Native UDP socket

RobotFramework Usage

SysBot is designed to work seamlessly with Robot Framework, providing powerful automation capabilities with a simple syntax.

Basic Robot Framework Test

*** Settings ***
Library        sysbot.Sysbot
Suite Setup       Call Components    plugins.data.yaml    /path/to/connexion.yml    key=connexion
Suite Teardown    Close All Sessions

*** Variables ***
${HOST}=       192.168.1.112
${PORT}=       22
${USER}=       sysbot
${PASSWORD}=   P@ssw0rd

*** Test Cases ***

Open Session without secret
    Open Session    target    ssh    bash    ${HOST}    ${PORT}   ${USER}    ${PASSWORD}
    Close All Sessions

Open Session with secret
    Open Session    target    ssh    bash    connexion.host    ${PORT}   connexion.username    connexion.password   is_secret=True
    Close All Sessions

Using Modules in Robot Framework

Modules can be loaded and used to perform specific operations on target systems:

*** Settings ***
Library        sysbot.Sysbot    linux.systemd    linux.dnf

*** Test Cases ***

Check System Service
    Open Session    server1    ssh    bash    ${HOST}    ${PORT}    ${USER}    ${PASSWORD}
    ${status}=    Linux Dnf Repolist    server1
    Log    ${status}
    Close All Sessions

Secret Management in Robot Framework

*** Settings ***
Library        sysbot.Sysbot

*** Test Cases ***

Using Secrets
    Add Secret    db_password    MySecretPassword
    ${password}=    Get Secret    db_password
    Log    Using password: ${password}
    Remove Secret    db_password

UnitTest Usage

SysBot can be used in Python unittest for system testing scenarios.

Module Testing with UnitTest

import unittest
import Sysbot

class TestLinuxModules(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        """Set up class fixtures."""
        cls.bot = sysbot.Sysbot("linux.systemd", "linux.dnf")
        cls.bot.open_session(
            alias="linux_server",
            protocol="ssh",
            product="bash",
            host="192.168.1.100",
            port=22,
            login="user",
            password="pass"
        )

    @classmethod
    def tearDownClass(cls):
        """Clean up class fixtures."""
        cls.bot.close_all_sessions()

    def test_systemd_service_status(self):
        """Test checking systemd service status."""
        result = self.bot.linux.systemd.status("linux_server", "sshd")
        self.assertIsNotNone(result)

    def test_dnf_repolist(self):
        """Test listing DNF repositories."""
        result = self.bot.linux.dnf.repolist("linux_server")
        self.assertIsNotNone(result)

if __name__ == '__main__':
    unittest.main()

Listener Usage

SysBot includes Robot Framework listener plugins that can store test results in various databases. Each database type has its own independent, self-contained listener. The listeners create a hierarchical structure: Campaign → Suite → Test Case → Keyword.

Available Listeners

  • SQLite: Lightweight file-based database, perfect for local testing
  • MySQL: Popular relational database for team environments
  • PostgreSQL: Enterprise-grade relational database
  • MongoDB: NoSQL document database for flexible schemas

Usage with Robot Framework

Each listener is used directly with its specific class:

# Store results in SQLite
robot --listener sysbot.utils.robot.listener.sqlite.Sqlite:results.db:MyCampaign your_tests/

# Store results in MySQL
robot --listener sysbot.utils.robot.listener.mysql.Mysql:mysql://user:pass@localhost/testdb:MyCampaign your_tests/

# Store results in PostgreSQL
robot --listener sysbot.utils.robot.listener.postgresql.Postgresql:postgresql://user:pass@localhost/testdb:MyCampaign your_tests/

# Store results in MongoDB
robot --listener sysbot.utils.robot.listener.mongodb.Mongodb:mongodb://localhost:27017/testdb:MyCampaign your_tests/

Listener Parameters

The listener accepts two parameters: 1. Database Connection: Connection string or path to database 2. Campaign Name: Name of the test campaign for organizing results

Data Structure

The listeners store test execution data in a hierarchical format:

  • Campaign: Top-level container for test executions
  • Suite: Test suite information
    • Test Case: Individual test cases
    • Keyword: Keywords executed within tests

Each level stores relevant metadata including: - Execution timestamps - Status (PASS/FAIL) - Error messages - Statistics

Installation Requirements

# Install with all database support
pip install sysbot[all_databases]

# Or install specific database support
pip install sysbot[mysql]        # MySQL support only
pip install sysbot[postgresql]   # PostgreSQL support only
pip install sysbot[mongodb]      # MongoDB support only

Polarion Integration

SysBot includes a Polarion plugin that enables integration with Siemens Polarion ALM/QA for test result management. The plugin provides:

  • xUnit Post-processor: Converts Robot Framework output to Polarion-compatible xUnit XML using rebot
  • Test Case Mapping: Links Robot Framework tests to Polarion test cases via tags
  • Custom Properties: Supports Polarion custom fields and metadata

Linking Tests to Polarion

Use tags in your Robot Framework tests to establish links with Polarion test cases:

*** Test Cases ***
Login Functionality Test
    [Documentation]    Validates user login with valid credentials
    [Tags]    polarion-id:TEST-001    polarion-title:Login Test    polarion-priority:High
    # Test steps...

User Management Test
    [Documentation]    Test user creation and deletion
    [Tags]    polarion-id:TEST-002    polarion-testEnvironment:Production
    # Test steps...

Tag Format

  • polarion-id:TEST-XXX - Links to Polarion test case ID (required for mapping)
  • polarion-title:Test Name - Sets Polarion test case title
  • polarion-{property}:{value} - Custom Polarion properties (e.g., polarion-priority:High, polarion-assignee:jdoe)

Generating Polarion-Compatible xUnit

Using Python API:

from sysbot.utils.robot.polarion import Polarion

polarion = Polarion()
polarion.generate_xunit(
    output_xml='output.xml',
    xunit_file='polarion_results.xml',
    project_id='MYPROJECT',
    test_run_id='RUN-001',
    custom_properties={'environment': 'test', 'version': '1.0'}
)

Using Command Line:

# Run Robot Framework tests
robot --outputdir results your_tests/

# Generate Polarion xUnit using Python
python -c "from sysbot.utils.robot.polarion import Polarion; \
    polarion = Polarion(); \
    polarion.generate_xunit('results/output.xml', 'results/polarion.xml', \
    project_id='PROJ', test_run_id='RUN-001')"

Importing into Polarion

Once you have the Polarion-compatible xUnit file:

  1. Manual Import: Use Polarion's UI to import the xUnit file
  2. Scheduled Import: Configure Polarion's scheduled xUnit importer
  3. API Import: Use tools like dump2polarion or Polarion's REST API

Generated xUnit Content

The generated xUnit file includes: - Test case IDs for proper mapping to existing Polarion test cases - Test execution results (pass/fail/error) - Execution time and timestamps - Custom properties for filtering and reporting - Project and test run associations

Additional Resources

Documentation

SysBot includes comprehensive Google-style docstrings for all modules, classes, and methods.

Online Documentation

The complete documentation is available online at https://joreci2.github.io/sysbot/

Viewing Documentation Locally with pdoc3

Install pdoc3 as a development dependency:

pip install pdoc3

Generate and serve interactive HTML documentation:

# Start a local documentation server (recommended)
pdoc3 --http localhost:8080 sysbot

# Or generate static HTML files
pdoc3 --html --output-dir docs sysbot

Then open your browser and navigate to http://localhost:8080/sysbot to browse the complete API documentation.

The documentation includes: - Module-level docstrings: Purpose and overview of each module - Class documentation: Detailed class descriptions and initialization parameters - Method documentation: Comprehensive Args, Returns, and Raises sections - Package structure: Hierarchical organization of all components

Error Handling

SysBot provides comprehensive error handling:

  • Connection Errors: Detailed error messages for connection failures
  • Tunnel Management: Automatic cleanup on tunnel failures
  • Session Validation: Verification of session validity before operations
  • Module Errors: Clear error messages for module and function calls

License

This project is licensed under the MIT License - see the LICENSE file for details.

Author

Thibault SCIRE - GitHub

Sub-modules

sysbot.connectors

Connectors Package …

sysbot.modules

Modules Package …

sysbot.plugins

Plugins Package …

sysbot.utils

Utilities Package …

Classes

class Sysbot (components=None)

Main Sysbot class for managing system automation and remote connections.

This class provides a unified interface for managing remote system connections, executing commands, and interacting with various system modules and plugins. It supports multiple protocols (SSH, WinRM, Socket, Local) and can manage tunneling configurations for complex network setups.

Attributes

ROBOT_LIBRARY_SCOPE
Robot Framework library scope set to GLOBAL.
ROBOT_LIBRARY_DOC_FORMAT
Documentation format set to reST.

Initialize the Sysbot instance.

Args

components
Optional list of component paths to load. If None, automatically discovers and loads all available modules and plugins. Component paths should be in the format 'modules.name' or 'plugins.name'.

Class variables

var ROBOT_LIBRARY_DOC_FORMAT
var ROBOT_LIBRARY_SCOPE

Methods

def add_secret(self, secret_name: str, value: ) ‑> None

Add or update a secret in the secret cache.

Args

secret_name
Name of the secret to store.
value
Secret value to store (can be any type).
def call_components(self, function_path: str, *args, **kwargs) ‑> 

Dynamically call a function from loaded components.

This method allows calling any function from loaded modules or plugins using a dot-notation path (e.g., 'modules.linux.systemd.is_active').

Args

function_path
Dot-separated path to the function (e.g., 'module.submodule.function'). Must contain at least module.function.
*args
Positional arguments to pass to the function.
**kwargs
Keyword arguments to pass to the function.

Returns

The result returned by the called function.

Raises

ValueError
If the function path format is invalid.
AttributeError
If the module or function is not found.
TypeError
If the target is not a callable function.
Exception
If the function call fails.
def close_all_sessions(self) ‑> None

Close all active sessions and clean up associated resources.

This method closes all open connections, stops all active tunnels, and clears the connection cache.

Raises

Exception
If any session fails to close properly.
def close_session(self, alias: str) ‑> None

Close a specific session identified by its alias.

Args

alias
Session alias identifying the connection to close.

Raises

RuntimeError
If no valid session is found for the alias.
Exception
If the session fails to close properly.
def execute_command(self, alias: str, command: str, **kwargs) ‑> 

Execute a command on a remote session.

Args

alias
Session alias identifying the connection to use.
command
Command string to execute on the remote system.
**kwargs
Additional command execution options specific to the protocol.

Returns

Command execution result. The format depends on the protocol used.

Raises

ValueError
If the specified alias does not exist.
RuntimeError
If no valid session is found for the alias.
Exception
If command execution fails.
def get_secret(self, secret_name: str) ‑> 

Retrieve a secret value from the secret cache.

Args

secret_name
Name of the secret to retrieve.

Returns

The secret value associated with the given name.

def open_session(self,
alias: str,
protocol: str,
product: str,
host: str,
port: int,
login: str = None,
password: str = None,
tunnel_config=None,
is_secret=False,
**kwargs) ‑> None

Open a new remote session with the specified connection parameters.

This method establishes a connection to a remote system using the specified protocol and credentials. It supports direct connections and tunneling through intermediate hosts for complex network configurations.

Args

alias
Unique identifier for the session.
protocol
Connection protocol to use (e.g., 'ssh', 'winrm', 'socket', 'local').
product
Product-specific implementation (e.g., 'bash', 'powershell').
host
Target host IP address or hostname.
port
Target port number.
login
Username for authentication. Optional if is_secret is True.
password
Password for authentication. Optional if is_secret is True.
tunnel_config
Optional tunnel configuration as JSON string or dict for nested tunneling through intermediate hosts.
is_secret
If True, treats host, login, and password as secret keys to retrieve actual values from the secret cache.
**kwargs
Additional protocol-specific connection options.

Raises

Exception
If the session fails to open or tunnel configuration is invalid.
def remove_secret(self, secret_name: str) ‑> None

Remove a secret from the secret cache.

Args

secret_name
Name of the secret to remove.