The 'Right' Way to Add Python Scripting Plugins to a Non-Python Application: External Executable Approach (Blender-Style)

Table of Contents#

1. Understanding the Problem: Embedding Python vs. External Executables
2. The External Executable Approach Explained
3. Key Components of the System
   • 3.1 App-Side: Managing Plugins and IPC
   • 3.2 Plugin-Side: Python Scripts as Clients
4. Step-by-Step Implementation
   • 4.1 Define the App-Plugin Interface
   • 4.2 Choose an IPC Mechanism
   • 4.3 App: Spawn and Manage Python Processes
   • 4.4 App: Expose API Endpoints via IPC
   • 4.5 Plugin: Implement the Python Client
   • 4.6 Test and Debug
5. Best Practices
6. Pitfalls to Avoid
7. Conclusion
8. References

1. Understanding the Problem: Embedding Python vs. External Executables#

Before diving into the external approach, let’s clarify why embedding Python might not be ideal for all cases.

Embedding Python: The Traditional Approach#

Embedding Python involves linking the Python interpreter into your application’s binary. When a user runs a plugin, the script executes within your app’s process.

Pros:

• Tight integration: Plugins can directly access the app’s memory and state (no IPC overhead).
• Fast: No serialization/deserialization of data between processes.

Cons:

• Version conflicts: The app is locked to a specific Python version (e.g., 3.9), but users may want to use 3.11 or libraries requiring newer versions.
• Memory/security risks: A buggy or malicious plugin can crash the entire app or leak memory.
• Deployment complexity: Shipping the Python runtime with your app increases install size and complicates cross-platform support.

The External Executable Approach: Isolation and Flexibility#

In contrast, the external approach treats Python plugins as separate executables. Your app spawns a Python process for each plugin, and they communicate via IPC (e.g., pipes, sockets, or files).

Pros:

• Isolation: Plugins run in their own processes; crashes or memory leaks won’t take down the app.
• Python version flexibility: Users can use any Python version (as long as it supports your IPC protocol).
• Security: Plugins have limited access to the app’s state (only what’s exposed via IPC).
• Simpler deployment: No need to bundle Python with the app—users use their existing Python installations.

Cons:

• IPC overhead: Data must be serialized (e.g., JSON, Protocol Buffers) and sent between processes, adding latency.
• Complex state sync: Keeping the app and plugin in sync (e.g., if the app’s state changes mid-plugin execution) requires careful design.

Why Blender-Style? Blender, the 3D modeling suite, is renowned for its Python plugin ecosystem. While Blender embeds Python, its success lies in a well-defined API, robust error handling, and user-friendly tools for plugin development. The external executable approach aims to replicate this usability and extensibility—with isolation as a bonus.

2. The External Executable Approach Explained#

At a high level, the system works as follows:

1. App (Non-Python): The main application (e.g., a C++ photo editor) exposes a set of API endpoints (e.g., get_selected_layers(), apply_filter()) that plugins can call.
2. Plugin (Python Script): A user-written Python script that acts as a client, sending requests to the app’s API via IPC.
3. IPC Channel: A communication bridge (e.g., pipes, TCP sockets) that carries serialized requests/responses between the app and plugin.

Figure 1: High-level architecture of the external executable plugin system.

3. Key Components of the System#

3.1 App-Side: Managing Plugins and IPC#

The app’s responsibility is to:

• Spawn and manage Python processes for plugins.
• Expose a well-defined API (e.g., "list all open documents", "export a scene").
• Handle IPC (send/receive messages, serialize/deserialize data).
• Validate plugin inputs to prevent abuse.

3.2 Plugin-Side: Python Scripts as Clients#

Plugins are Python scripts that:

• Connect to the app’s IPC channel.
• Send requests to the app’s API (e.g., "give me the current selection").
• Process data locally (e.g., run ML inference on an image).
• Send back results (e.g., "here’s the filtered image data").

4. Step-by-Step Implementation#

Let’s build a minimal example: a C++ app that lets Python plugins query and modify a "document" (e.g., a text file). We’ll use pipes for IPC and JSON for serialization (simple and human-readable).

4.1 Define the App-Plugin Interface#

First, define the API endpoints the app will expose. For our text editor example:

4.2 Choose an IPC Mechanism#

For simplicity, we’ll use anonymous pipes (supported on Windows, Linux, and macOS). Pipes are unidirectional, so we’ll create two: one for the app to send data to the plugin (app→plugin), and one for the plugin to send data to the app (plugin→app).

4.3 App: Spawn and Manage Python Processes#

In C++, use subprocess (or CreateProcess on Windows) to spawn a Python process. We’ll pass the pipe handles to the Python script so it can connect.

// C++ code (App side)
#include <iostream>
#include <cstdio>
#include <cstring>
#include <sys/wait.h> // For waitpid (Linux/macOS)
#include <unistd.h>  // For pipe, fork, execlp (Linux/macOS)
 
int main() {
    int app_to_plugin[2]; // Pipe: app writes, plugin reads
    int plugin_to_app[2]; // Pipe: plugin writes, app reads
    pipe(app_to_plugin);
    pipe(plugin_to_app);
 
    pid_t pid = fork();
    if (pid == 0) { // Child process (Python plugin)
        // Close unused pipe ends
        close(app_to_plugin[1]); // App's write end
        close(plugin_to_app[0]); // App's read end
 
        // Pass pipe file descriptors to Python via environment variables
        setenv("APP_TO_PLUGIN_FD", std::to_string(app_to_plugin[0]).c_str(), 1);
        setenv("PLUGIN_TO_APP_FD", std::to_string(plugin_to_app[1]).c_str(), 1);
 
        // Execute the Python plugin script
        execlp("python3", "python3", "plugin.py", nullptr);
        perror("execlp failed");
        exit(1);
    } else { // Parent process (C++ app)
        // Close unused pipe ends
        close(app_to_plugin[0]); // Plugin's read end
        close(plugin_to_app[1]); // Plugin's write end
 
        // ... (handle IPC, see Section 4.4)
    }
}

4.4 App: Expose API via IPC#

The app must read requests from the plugin, process them, and send responses. Here’s how to handle get_document_text and set_document_text:

// C++ code (continued from Section 4.3)
#include <nlohmann/json.hpp> // Use JSON library (https://github.com/nlohmann/json)
using json = nlohmann::json;
 
std::string g_document_text = "Initial text"; // App's state
 
void handle_request(const json& req, json& res) {
    if (req["cmd"] == "get_text") {
        res["text"] = g_document_text;
    } else if (req["cmd"] == "set_text") {
        g_document_text = req["text"];
        res["status"] = "ok";
    } else {
        res["error"] = "Unknown command";
    }
}
 
int main() {
    // ... (spawn Python process, as above)
 
    char buffer[1024];
    ssize_t bytes_read;
 
    // Read from plugin (plugin_to_app pipe)
    while ((bytes_read = read(plugin_to_app[0], buffer, sizeof(buffer)-1)) > 0) {
        buffer[bytes_read] = '\0';
        json req = json::parse(buffer); // Parse JSON request
        json res;
 
        handle_request(req, res);
 
        // Send response back to plugin (app_to_plugin pipe)
        std::string res_str = res.dump() + "\n"; // Add newline for delimiter
        write(app_to_plugin[1], res_str.c_str(), res_str.size());
    }
 
    // Cleanup: Wait for Python process to exit
    waitpid(pid, nullptr, 0);
    return 0;
}

4.5 Plugin: Implement the Python Client#

The Python plugin connects to the pipes, sends requests, and handles responses. Use the os module to access the pipe file descriptors passed via environment variables:

# plugin.py (Python side)
import os
import json
 
# Get pipe file descriptors from environment
app_to_plugin_fd = int(os.environ["APP_TO_PLUGIN_FD"])
plugin_to_app_fd = int(os.environ["PLUGIN_TO_APP_FD"])
 
# Open pipes as file objects
app_to_plugin = os.fdopen(app_to_plugin_fd, "r")
plugin_to_app = os.fdopen(plugin_to_app_fd, "w")
 
def send_request(cmd, **kwargs):
    req = {"cmd": cmd, **kwargs}
    plugin_to_app.write(json.dumps(req) + "\n")
    plugin_to_app.flush()  # Ensure data is sent immediately
    res = json.loads(app_to_plugin.readline())
    return res
 
# Example: Use the app's API
print("Current text:", send_request("get_text")["text"])  # Output: "Initial text"
send_request("set_text", text="Hello from Python!")
print("New text:", send_request("get_text")["text"])      # Output: "Hello from Python!"

4.6 Testing and Debugging#

• Test IPC: Verify that requests/responses are correctly serialized (e.g., use print in Python or std::cout in C++ to log messages).
• Handle edge cases: What if the plugin sends invalid JSON? The app should return an error response instead of crashing.
• Debugging plugins: Use Python’s pdb debugger or print statements to troubleshoot plugin logic.

5. Best Practices#

Version Your API#

As your app evolves, API endpoints may change. Include a version field in requests (e.g., {"cmd": "get_text", "version": "1.0"}) to ensure compatibility with old plugins.

Validate Plugin Inputs#

Malicious plugins could send oversized data or invalid commands. Sanitize inputs (e.g., limit text length for set_text) and use timeouts to kill unresponsive plugins.

Use Robust Serialization#

JSON is simple, but for binary data (e.g., images), use MessagePack or Protocol Buffers (smaller, faster, and type-safe).

Manage Processes Gracefully#

• Kill orphaned Python processes if the app exits unexpectedly (use SIGTERM on Unix, TerminateProcess on Windows).
• Limit the number of concurrent plugins to avoid resource exhaustion.

Document the API#

Write clear docs for plugin developers, including:

• List of endpoints, request/response formats, and error codes.
• Examples (e.g., "How to filter an image using the app’s API").

6. Pitfalls to Avoid#

Latency from IPC Overhead#

IPC adds latency (e.g., serializing a 1MB image takes time). For real-time apps (e.g., video editors), use binary serialization (e.g., Protocol Buffers) or batch requests.

Orphaned Processes#

If the app crashes, Python processes may linger. Use "process groups" (Unix) or "job objects" (Windows) to ensure child processes are terminated with the app.

Security Risks#

Plugins could send malicious requests (e.g., overwriting system files via set_text). Restrict plugin access: only expose necessary API endpoints, and sanitize all inputs.

Poor API Design#

A vague API (e.g., modify_data()) frustrates developers. Be specific (e.g., crop_image(x1, y1, x2, y2)), and version aggressively.

7. Conclusion#

The external executable approach is a powerful way to add Python scripting to non-Python apps, offering isolation, flexibility, and security. By separating plugins into their own processes and using IPC for communication, you avoid the pitfalls of embedding Python while still enabling rich user customization.

While it requires careful design (defining APIs, choosing IPC, managing processes), the result is a Blender-style plugin system that’s robust, user-friendly, and future-proof.

8. References#

• Blender Python API Documentation – Inspiration for a well-designed plugin API.
• nlohmann/json – Lightweight C++ JSON library used in examples.
• Boost.Process – Cross-platform process management for C++.
• Python subprocess Module – Spawning processes from Python (if your app is Python-based).
• Inter-Process Communication (IPC) – Wikipedia – Overview of IPC mechanisms.
• Protocol Buffers – Efficient binary serialization for high-performance IPC.