API reference
The engine's api/ folder file by file, with every function it defines, and how the website reads the collected data.
"API" means two different things in this project:
| Layer | Where | Covered in |
|---|---|---|
| The telemetry client in the engine | api/ in core | this page |
| The telemetry service the client reports to | telemetry.community-scripts.org | Telemetry service |
For what is sent and how to switch it off, read Telemetry and diagnostics first. This page is the reference for people who work on the engine or call these functions from a script.
How the folder is loaded
api/api.func
is the only file other code sources. It defines the settings below and then
loads the four parts in this order: exitcodes, errorlog, sysinfo,
telemetry.
| Setting | Value | Meaning |
|---|---|---|
TELEMETRY_URL | https://telemetry.community-scripts.org/telemetry | Endpoint. Set the variable before the script starts to override it |
TELEMETRY_TIMEOUT | 5 | Seconds, for the install record and progress |
STATUS_TIMEOUT | 10 | Seconds, for the terminal status |
TELEMETRY_PAYLOAD_VERSION | 3 | Sent as a field and in the user agent |
TELEMETRY_ERROR_MAX_LINES | 60 | Cap on the error trace |
TELEMETRY_ERROR_MAX_BYTES | 10240 | Cap on the error trace |
Each part is found through the engine's resolver when it is loaded
(_cs_source_func), otherwise in COMMUNITY_SCRIPTS_CORE_DIR, otherwise
downloaded from COMMUNITY_SCRIPTS_CORE_URL (default
https://raw.githubusercontent.com/community-scripts/core/main).
Who loads it:
| Caller | How |
|---|---|
| Container scripts | ui/build-ui.func sources api/api.func |
| Proxmox VM scripts | load_api_functions in pve/vm-core.func |
| Incus VM scripts | incus/vm-core.func |
| Inside the container | lxc/install.func loads exitcodes.func for the exit code table and defines its own small post_progress_to_api |
The payload version history, from the comment in api.func:
| Version | Added |
|---|---|
| 1 | The original payload |
| 2 | kernel_version, app_version, failed_command, failed_line; gpu_passthrough means actual passthrough |
| 3 | error_message, error_step, error_phase, failed_function; the trace is limited to the failing step and filtered |
Function index
api/API.txt
lists all 41 functions the folder defines. The workflow
verify-split-files.yml
compares the code against that file, so adding or renaming a function means
updating API.txt in the same pull request.
Names starting with _tm_ or _telemetry_ are internal. Scripts call the
public ones.
exitcodes.func
api/exitcodes.func
is the single copy of the exit code table. It is loaded on the host and inside
the container.
| Function | Arguments | Does |
|---|---|---|
explain_exit_code | <code> | Prints the description of an exit code, for example 100 gives "APT: Package manager error (broken packages / dependency problems)". Unknown codes give "Unknown error" |
categorize_error | <code> [command] | Prints the telemetry category. ERROR_CATEGORY_OVERRIDE wins; then fixed codes; then the failing tool (systemctl, pct, qm, pvesm, pvesh, pveam); then the code. Without a command it reads COMMAND= from the capture |
The full category table is in Telemetry and diagnostics; the codes are in Exit codes.
errorlog.func
api/errorlog.func
turns a failure into the error trace. It reads the capture file the error
handler writes (.errinfo):
EXIT_CODE=<n>
LINE=<n>
COMMAND=<command, one line>
FUNCTION=<function the command ran in>
STEP=<msg_info text of the running step>
MESSAGE=<first msg_error of that step>
PHASE=<host|container>
--- OUTPUT ---
<raw output>| Function | Arguments | Does |
|---|---|---|
json_escape | <text> | Prints the text as the body of a JSON string. Removes ANSI sequences and control characters. Uses jq when present, otherwise awk |
build_error_string | <exit_code> | Prints the finished trace: header line, ---, output. Runs _tm_error_context and _tm_format_error |
get_error_log | none | Prints the failing step's output from the active log, for failures that left no capture |
_tm_error_context | <exit_code> | Sets TM_ERR_CODE, TM_ERR_COMMAND, TM_ERR_LINE, TM_ERR_FUNCTION, TM_ERR_STEP, TM_ERR_MESSAGE, TM_ERR_PHASE, TM_ERR_OUTPUT and TM_ERR_EXPLANATION. A capture is used only when its exit code matches the reported one, or the reported one is the generic 1 or 255 |
_tm_format_error | none | Prints the trace from the TM_ERR_* variables |
_tm_find_errinfo | none | Prints the capture file to use: TELEMETRY_ERRINFO, /tmp/.errinfo-<session>, <BUILD_LOG>.errinfo, <INSTALL_LOG>.errinfo, first one that is not empty |
_tm_pick_logfile | none | Prints the most specific log that exists: the combined log, /tmp/install-<session>-combined.log, /tmp/<app>-<ctid>-<session>.log, INSTALL_LOG, BUILD_LOG, SILENT_LOGFILE |
_tm_clean_log | stdin | Filter: strips ANSI and carriage returns, masks IPv4 addresses to <first octet>.x.x.x, drops noise lines (_TM_NOISE) and blank lines |
_tm_step_output | stdin | Filter: keeps everything from the last [INFO] or [OK] log line on |
_tm_focus | stdin | Filter: _tm_step_output, _tm_clean_log, lines cut to 1000 characters, last TELEMETRY_ERROR_MAX_LINES lines, last TELEMETRY_ERROR_MAX_BYTES bytes |
_tm_explain | <code> [command] | Prints the explanation for a code. Uses the error handler's tool-aware _eh_explanation when it is loaded and a command is known, otherwise explain_exit_code |
_tm_one_line | <text> [max] | Prints the text on one line without colour codes, cut to max characters (default 300) |
sysinfo.func
api/sysinfo.func
finds out where the script came from and what it runs on. detect_repo_source
runs as soon as the file is loaded.
| Function | Sets | How |
|---|---|---|
detect_repo_source | REPO_SOURCE, REPO_SLUG | Keeps an existing REPO_SOURCE (this is how a container inherits the host's value). Otherwise takes owner/repo from COMMUNITY_SCRIPTS_URL, then from the process command line, then from the first 10 lines of the script file |
detect_cpu | CPU_VENDOR, CPU_MODEL | vendor_id and model name from /proc/cpuinfo; vendor is intel, amd, arm or unknown |
detect_gpu | GPU_VENDOR, GPU_MODEL, GPU_PASSTHROUGH | First VGA, XGA, 3D or display controller in lspci; vendor is intel, amd, nvidia or unknown |
detect_ram | RAM_SPEED | dmidecode -t memory, "Configured Memory Speed" or "Speed"; empty without dmidecode |
detect_arm | HAS_ARM | true when dpkg --print-architecture (or uname -m) says arm64 or aarch64 |
telemetry_collect_sysinfo | all of the above plus KERNEL_VERSION, TM_ARCH, TM_PVE_VERSION, TM_PLATFORM | Calls the detectors that have not run yet. TM_PLATFORM is TELEMETRY_PLATFORM when the host passed one, otherwise pve or incus by which command exists |
REPO_SOURCE has three values:
Detected owner/repo | REPO_SOURCE | REPO_SLUG |
|---|---|---|
community-scripts/ProxmoxVE | ProxmoxVE | community-scripts/ProxmoxVE |
community-scripts/DevScripts (or its former name community-scripts/ProxmoxVED) | DevScripts | community-scripts/DevScripts |
nothing found, or community-scripts/core | ProxmoxVE | community-scripts/ProxmoxVE |
| anything else | external | the detected owner/repo |
telemetry.func
api/telemetry.func
builds the payload and sends it.
Public functions
| Function | Arguments | Does |
|---|---|---|
post_to_api | none | Sends the installing record. Once per execution, up to 3 attempts. Starts the duration clock if it is not running. Sets POST_TO_API_DONE |
post_to_api_vm | none | Reads DIAGNOSTICS from /usr/local/community-scripts/diagnostics, sets TELEMETRY_TYPE=vm and CT_TYPE=2, then calls post_to_api |
post_progress_to_api | [status] | Sends a progress record with the full payload, one attempt. Default status configuring |
post_update_to_api | <status> <exit_code> [force] | Sends the terminal record. Status done or success, failed, aborted. Exit code none means 0. Sends once per execution unless the third argument is force. Does nothing when post_to_api was never due |
post_tool_to_api | <name> [status] [exit_code] [reason] | Record for a host tool run, type from TELEMETRY_TOOL_TYPE (default pve) |
post_addon_to_api | <name> [status] [exit_code] [reason] | Record for an add-on run, type addon, OS from /etc/os-release |
init_tool_telemetry | [name] [pve|addon] | One-line setup for tool and add-on scripts: reads the diagnostics file, starts the timer, installs the EXIT trap that reports the result |
telemetry_new_attempt | none | Starts a new execution for a retry: new EXECUTION_ID, resets POST_TO_API_DONE and POST_UPDATE_DONE, clears the previous failure's variables |
start_install_timer | none | Sets INSTALL_START_TIME unless it is already set |
get_install_duration | none | Prints the seconds since INSTALL_START_TIME, or 0 |
A tool script needs one line. This is from tools/addon/add-tailscale-lxc.sh:
declare -f init_tool_telemetry &>/dev/null && init_tool_telemetry "add-tailscale-lxc" "addon"Container scripts do not call any of these themselves. The engine does it at
the right points. VM scripts have no build step that could: they call
post_to_api_vm before the first change to the host, name
post_update_to_api in their signal traps, and end with vm_finish (older
ones with post_update_to_api "done" "none").
post_update_to_api maps what it is given to the status it sends:
| Called with | Sent status |
|---|---|
done or success | success, exit code forced to 0 |
aborted | aborted |
failed with exit code 113, 114, 122, 129, 130, 143, 253 or 254 | aborted |
failed with any other code | failed |
| anything else | unknown |
Internal functions
| Function | Does |
|---|---|
_tm_enabled | True when curl exists, DIAGNOSTICS is not no and RANDOM_UUID is set |
_tm_payload | <status> [exit_code] [error_trace]. Prints the JSON record |
_tm_send | <payload> [timeout] [attempts]. Posts it. Returns 0 on HTTP 2xx, 1 otherwise, and never raises |
_tm_quiet | Runs a sender with errexit, pipefail and the caller's ERR trap off, then restores the trap |
_tm_post_install | Body of post_to_api |
_tm_post_progress | Body of post_progress_to_api |
_tm_post_update | Body of post_update_to_api |
_tm_post_run | <type> <name> <status> <exit_code> [reason]. Body of the tool and add-on senders; gives each run a fresh UUID |
_tm_requirement_code | True for the exit codes that mean a requirement was not met (10, 103, 104, 106, 232, 233, 234, 236, 238). Tool failures with these codes are not reported |
_tm_dialog_exit | True when the failing command was a whiptail dialog that returned 1 or 255, or a read or select that returned 1. Such a run is aborted, not failed |
_telemetry_report_exit | The EXIT trap init_tool_telemetry installs |
Variables the senders read
| Variable | Used for |
|---|---|
DIAGNOSTICS | no or unset disables every sender |
RANDOM_UUID, EXECUTION_ID | random_id, execution_id |
TELEMETRY_TYPE | type, default lxc |
NSAPP | nsapp |
CT_TYPE, DISK_SIZE, CORE_COUNT, RAM_SIZE | guest settings |
var_os, var_version | os_type, os_version |
METHOD | method |
var_gpu, ENABLE_GPU | gpu_passthrough |
APPLICATION_VERSION, var_appversion | app_version |
INSTALL_START_TIME | install_duration |
FAILED_COMMAND, FAILED_LINE, FAILED_FUNCTION, CS_STEP, CS_STEP_DONE, CS_ERROR_MSG | failure fields, when no capture file applies |
ERROR_CATEGORY_OVERRIDE | forces error_category |
TELEMETRY_ERRINFO | path of the capture file pulled from the container |
DEV_MODE | true prints each payload and HTTP status to standard error |
How the website uses the data
The website does not receive telemetry. It reads aggregates from the telemetry service and shows them on the analytics page and as install counts on script pages. Which endpoints it calls is documented in Telemetry service.