VM Implementation Guide

Overview

Use this path for a Linux VM on AWS, GCP, Azure, or an equivalent provider. One systemd-managed OpenTelemetry Collector Contrib instance collects host and systemd metrics, host-local logs, its health metrics, resource metadata, and OTLP from local applications.

Level 1 installs the agent, level 2 proves host collection reaches Tsuga, and level 3 accepts telemetry from local applications. Use the tested installer or a manual operating-system change path. Both expose OTLP only on loopback.

Local application ──OTLP──> 127.0.0.1:4317 (gRPC) ┐
Local application ──OTLP──> 127.0.0.1:4318 (HTTP) ├──> otelcol-contrib ──OTLP/HTTP──> Tsuga
Host metrics, systemd, logs, cloud metadata ───────┘
                                                       └──> 127.0.0.1:13133 health
                                                       └──> 127.0.0.1:8888 self-metrics

This is host observability, not automatic application instrumentation. Instrument applications separately, set a stable service.name, then export them to the local Collector.

Before starting

• Use a systemd-based Linux host: Ubuntu, Debian, Amazon Linux, RHEL, Rocky, AlmaLinux, CentOS Stream, Fedora, openSUSE, or SLES.
• Confirm sudo access, outbound HTTPS to GitHub (package download) and to the Tsuga OTLP endpoint, and enough disk space for Collector checkpoint storage.
• Create an ingestion key and retrieve the endpoint from Tsuga. Store both in the VM bootstrap secret store, never in the Collector YAML or source control.
• Agree SERVICE_NAMESPACE, DEPLOYMENT_ENVIRONMENT_NAME, and the host name that users will recognize. Preserve application-provided service.name. Do not give every application the Collector’s name.
• Decide whether host file logs or journald logs are in scope. Do not enable every available log source blindly: duplicate reads create duplicate logs.
• If cloud tags/labels are needed, allow only useful stable keys such as Name, Environment, and Service. EC2 needs Allow tags in instance metadata for IMDS tag enrichment. GCP label collection requires roles/compute.viewer. Azure tags are attached directly to the VM.

References: Install the OpenTelemetry Collector, Collector configuration, and Collector internal telemetry.

Level 1 - Run the Collector as a host agent

Install one otelcol-contrib instance as a systemd unit. This level gives the VM its own collection point: host, process, and systemd metrics, Collector self-metrics, cloud and host resource identity, the host-local log source, and a loopback OTLP receiver that local applications can use in level 3.

Choose and record one route per fleet. Option A is the default because it validates checksums, startup sequencing, configuration, and health. Use Option B when the customer must review and own each package, file, and systemd change.

Exit criteria: the otelcol-contrib unit is enabled and healthy on the target host, its configuration validates with the same binary that runs it, and the credentials live in a root-only environment file rather than a shell history or a checked-in file.

Option A - Install with the remote script (default)

The installer detects the package family and architecture, verifies the package checksum, blocks startup until configuration exists, writes the environment file and drop-in, validates the configuration, and waits for health.

Download the pinned gist revision and review both files before granting sudo. The revision hash is the tested source. Do not pipe the remote script to a shell. Customers that cannot fetch third-party build-time scripts should mirror this revision in their own artifact store.

curl -fsSL -o tsuga_otel_install.sh \
  'https://gist.githubusercontent.com/nxnarbais/b8de0fa84fe752aee96d75d67ac47038/raw/d023b77d93611d820d439522ab2992bcdbd33409/tsuga_otel_install.sh'
curl -fsSL -o otel-config.yaml \
  'https://gist.githubusercontent.com/nxnarbais/b8de0fa84fe752aee96d75d67ac47038/raw/d023b77d93611d820d439522ab2992bcdbd33409/otel-config.yaml'
chmod 0755 tsuga_otel_install.sh

Set the secret values in the current shell, optionally set the host identity, then run the installer as a normal user with sudo access:

export TSUGA_ENDPOINT='<TSUGA_OTLP_ENDPOINT>'
export TSUGA_API_KEY='<TSUGA_INGESTION_KEY>'
export SERVICE_NAMESPACE='<TEAM_OR_PRODUCT>'
export DEPLOYMENT_ENVIRONMENT='production'
export HOST_ALIAS='<RECOGNIZABLE_HOST_NAME>'

./tsuga_otel_install.sh ./otel-config.yaml

SERVICE_NAMESPACE becomes the OpenTelemetry resource attribute service.namespace: a stable label that groups related services and their supporting VM telemetry, such as payments or checkout. It is not the VM name, environment, or per-instance identifier. Keep service.name specific to each application (for example, payment-api), use HOST_ALIAS for the machine, and avoid volatile values such as hostnames, regions, or build IDs.

The installer automatically chooses the baseline profile and enables Docker or containerd metrics only when the corresponding local endpoint is usable. The default configuration reads /var/log files. Enable journald only when those files do not mirror the same entries, or adjust the configuration to exclude the duplicated source:

export OTELCOL_ENABLE_JOURNALD=true
./tsuga_otel_install.sh ./otel-config.yaml

Useful profile overrides are OTELCOL_CONFIG_PROFILE=baseline, docker, containerd, and docker-containerd. Use an override only when the required Docker socket or containerd Prometheus endpoint is known to be available. Preview the package, architecture, and enabled receivers without changing the host:

OTELCOL_BOOTSTRAP_PLAN=1 ./tsuga_otel_install.sh

The installer stores its credentials in /etc/otelcol-contrib/tsuga.env with mode 0600, installs the active configuration at /etc/otelcol-contrib/config.yaml, and logs the bootstrap to /var/log/cloud-vm-bootstrap/install.log.

Option B - Install and configure manually

Use this route when the customer needs to review and manage each operating-system change themselves. It provides the same baseline as the remote installer: host metrics, systemd state, host-local logs, Collector self-metrics, cloud metadata, local application OTLP, span metrics, and secure Collector-to-Tsuga export.

1. Install the Collector package

Choose a Contrib version that has been validated with the configuration below. The tested version at the time of writing is 0.154.0. Update it deliberately and run the validation command in Level 2 with the exact package version you deploy. The core otelcol distribution is insufficient because this configuration uses Contrib receivers and processors.

export OTELCOL_VERSION='0.154.0'

case "$(uname -m)" in
  x86_64|amd64) export OTELCOL_ARCH='amd64' ;;
  aarch64|arm64) export OTELCOL_ARCH='arm64' ;;
  *) echo 'Unsupported architecture'; exit 1 ;;
esac

. /etc/os-release
case " ${ID} ${ID_LIKE:-} " in
  *' debian '*|*' ubuntu '*) export OTELCOL_PACKAGE='deb' ;;
  *) export OTELCOL_PACKAGE='rpm' ;;
esac

export OTELCOL_FILE="otelcol-contrib_${OTELCOL_VERSION}_linux_${OTELCOL_ARCH}.${OTELCOL_PACKAGE}"
export OTELCOL_RELEASE="https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTELCOL_VERSION}"

# Prevent package hooks from starting the unit before its configuration and
# root-only environment file are present.
sudo systemctl mask otelcol-contrib.service
curl -fLO "${OTELCOL_RELEASE}/${OTELCOL_FILE}"
curl -fsSLO "${OTELCOL_RELEASE}/opentelemetry-collector-releases_otelcol-contrib_checksums.txt"
expected_checksum="$(awk -v file="${OTELCOL_FILE}" '$2 == file { print $1; exit }' opentelemetry-collector-releases_otelcol-contrib_checksums.txt)"
[ -n "${expected_checksum}" ] || { echo 'Checksum not found'; exit 1; }
printf '%s  %s\n' "${expected_checksum}" "${OTELCOL_FILE}" | sha256sum --check --strict

if [ "${OTELCOL_PACKAGE}" = deb ]; then
  sudo dpkg -i "${OTELCOL_FILE}"
else
  sudo rpm -Uvh "${OTELCOL_FILE}"
fi

The package service remains masked until the environment file, configuration, and systemd drop-in are ready:

sudo install -d -m 0750 /var/lib/otelcol-contrib/storage
sudo install -d -m 0755 /etc/otelcol-contrib /etc/systemd/system/otelcol-contrib.service.d
sudo ln -sfn / /hostfs

2. Create the root-only runtime environment

Create /etc/otelcol-contrib/tsuga.env with values supplied through the customer’s secret-management process. It is the only file in this guide that contains the ingestion secret.

sudo tee /etc/otelcol-contrib/tsuga.env >/dev/null <<'EOF'
TSUGA_OTLP_ENDPOINT=<TSUGA_OTLP_ENDPOINT>
TSUGA_API_KEY=<TSUGA_INGESTION_KEY>
SERVICE_NAMESPACE=<TEAM_OR_PRODUCT>
DEPLOYMENT_ENVIRONMENT_NAME=production
HOST_ALIAS=<RECOGNIZABLE_HOST_NAME>
COLLECTOR_LOG_LEVEL=info
EOF
sudo chmod 0600 /etc/otelcol-contrib/tsuga.env
sudo chown root:root /etc/otelcol-contrib/tsuga.env

3. Create the Collector configuration

Create /etc/otelcol-contrib/config.yaml with sudoedit /etc/otelcol-contrib/config.yaml, then paste the baseline below. It tails common host log files and does not enable Docker, containerd, or journald. Add a runtime receiver only after confirming its local endpoint and access permissions. Otherwise, a missing Docker socket can prevent the Collector from starting.

extensions:
  health_check:
    endpoint: 127.0.0.1:13133
  file_storage:
    directory: /var/lib/otelcol-contrib/storage
    create_directory: true

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
      http:
        endpoint: 127.0.0.1:4318
  hostmetrics:
    collection_interval: 10s
    root_path: /hostfs
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      disk:
      filesystem:
        metrics:
          system.filesystem.utilization:
            enabled: true
      load:
      memory:
        metrics:
          system.memory.limit:
            enabled: true
          system.memory.utilization:
            enabled: true
      paging:
        metrics:
          system.paging.utilization:
            enabled: true
      processes:
      process:
        mute_process_all_errors: true
        include:
          match_type: regexp
          names:
            - ^(otelcol-contrib|dockerd|containerd|containerd-shim.*)$
        metrics:
          process.cpu.utilization:
            enabled: true
          process.memory.utilization:
            enabled: true
          process.threads:
            enabled: true
      system:
  systemd:
    collection_interval: 10s
    scope: system
    units:
      - otelcol-contrib.service
      - docker.service
      - containerd.service
      - systemd-journald.service
    metrics:
      systemd.service.restarts:
        enabled: true
  prometheus/self:
    config:
      scrape_configs:
        - job_name: otel-collector
          scrape_interval: 10s
          static_configs:
            - targets: [127.0.0.1:8888]
  filelog/system:
    include:
      - /var/log/syslog
      - /var/log/messages
      - /var/log/*.log
    start_at: end
    storage: file_storage
    include_file_name: true
    include_file_path: true

processors:
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25
  resourcedetection/cloud:
    detectors: [env, gcp, azure, ec2, system]
    override: false
    timeout: 2s
    ec2:
      tags:
        - (?i)^name$
        - (?i)^environment$
        - (?i)^service$
      tags_from_imds: true
    gcp:
      labels:
        - (?i)^name$
        - (?i)^environment$
        - (?i)^service$
    azure:
      tags:
        - (?i)^name$
        - (?i)^environment$
        - (?i)^service$
  resource/common:
    attributes:
      - key: service.namespace
        value: ${env:SERVICE_NAMESPACE}
        action: insert
      - key: deployment.environment.name
        value: ${env:DEPLOYMENT_ENVIRONMENT_NAME}
        action: insert
  resource/collector:
    attributes:
      - key: service.name
        value: vm-otel-collector
        action: insert
  cumulativetodelta: {}
  batch:
    timeout: 1s
    send_batch_size: 1024
    send_batch_max_size: 2048

connectors:
  spanmetrics:
    dimensions:
      - name: http.request.method
        default: GET
      - name: http.response.status_code
      - name: http.route

exporters:
  otlphttp/tsuga:
    endpoint: ${env:TSUGA_OTLP_ENDPOINT}
    compression: gzip
    headers:
      Authorization: Bearer ${env:TSUGA_API_KEY}
    timeout: 10s

service:
  extensions: [health_check, file_storage]
  telemetry:
    resource:
      service.name: vm-otel-collector
      service.namespace: ${env:SERVICE_NAMESPACE}
      deployment.environment.name: ${env:DEPLOYMENT_ENVIRONMENT_NAME}
      host.name: ${env:HOST_ALIAS}
    logs:
      level: ${env:COLLECTOR_LOG_LEVEL:-info}
    metrics:
      level: normal
      readers:
        - pull:
            exporter:
              prometheus:
                host: 127.0.0.1
                port: 8888
  pipelines:
    metrics:
      receivers: [hostmetrics, systemd, otlp, spanmetrics, prometheus/self]
      processors: [memory_limiter, resourcedetection/cloud, resource/common, resource/collector, cumulativetodelta, batch]
      exporters: [otlphttp/tsuga]
    logs:
      receivers: [filelog/system, otlp]
      processors: [memory_limiter, resourcedetection/cloud, resource/common, resource/collector, batch]
      exporters: [otlphttp/tsuga]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection/cloud, resource/common, batch]
      exporters: [spanmetrics, otlphttp/tsuga]

4. Create the systemd drop-in and start the service

The drop-in makes the same environment available to every systemd restart. Do not rely on environment variables exported only in an SSH session.

sudo tee /etc/systemd/system/otelcol-contrib.service.d/tsuga.conf >/dev/null <<'EOF'
[Service]
EnvironmentFile=/etc/otelcol-contrib/tsuga.env
EOF

sudo systemctl unmask otelcol-contrib.service
sudo systemctl daemon-reload
sudo systemctl enable --now otelcol-contrib

Level 2 - Validate host collection

Validate the exact configuration with the same Contrib binary that will run it, then check the local service before searching Tsuga:

sudo otelcol-contrib validate --config=/etc/otelcol-contrib/config.yaml
sudo systemctl status otelcol-contrib --no-pager
curl -fsS http://127.0.0.1:13133/
curl -fsS http://127.0.0.1:8888/metrics | grep '^otelcol_' | head
sudo journalctl -u otelcol-contrib -n 100 --no-pager

Within one collection interval, confirm recent system.cpu.utilization, system.processes.count, systemd.unit.state, and otelcol_exporter_sent_metric_points in Tsuga. Send and search a distinctive host log line. Then send one application request through 127.0.0.1:4317 or 127.0.0.1:4318 and verify its trace, span metrics, and correlated log where IDs are emitted.

Use Validate Telemetry Arrival in Tsuga for sender, trace, metric, log, and correlation checks. Systemd confirms local process state. Complete the level only after the Collector is healthy and a known signal reaches Tsuga.

Exit criteria: recent host, process, systemd, and Collector self-metrics are visible in Tsuga with stable host, cloud, namespace, and environment context. The host appears in the Hosts inventory. The agreed host-local log source arrives once, not twice.

Level 3 - Accept telemetry from local applications

The OTLP receiver listens only on loopback. Point local applications to 127.0.0.1:4317 or 127.0.0.1:4318 and preserve their service.name. The baseline inserts service.namespace and deployment.environment.name without overwriting application identity. The traces pipeline does not apply the Collector’s service.name.

The spanmetrics connector lets traced applications populate their Service page. This level proves transport. Configure application log format, routing, and trace correlation when the host file source does not meet the need.

Containers use a separate network namespace. Route them explicitly to the host loopback receiver through runtime networking, or use a Collector they can reach. 127.0.0.1 inside a container resolves inside that container.

Exit criteria: a local application exports a known trace through the loopback receiver, that trace arrives with the application’s own service.name rather than the Collector’s, and its span metrics appear on the Service page.

Make it operational

Add coverage once the metrics and logs above are reliable, and keep it scoped to the team that owns the VM rather than the applications running on it.

• Host pressure with a response. CPU saturation, memory exhaustion, and filesystem utilization approaching full are the conditions where an on-call engineer can act. Use system.filesystem.utilization per mount rather than a whole-host average. A full /var fills before the root filesystem looks busy.
• Collector health. Persistent export errors, queue pressure, or drops mean the host’s evidence is incomplete during an incident. Alert on the Collector’s own metrics and the otelcol-contrib systemd unit state, and route this to the platform team.
• A host dashboard per fleet. CPU, memory, load, filesystem, and disk I/O with a host filter, plus Collector export health, is enough to triage. Build widgets from an Analytics query you have already validated.
• Alert on the fleet, not on each host. Aggregate by fleet, environment, or role. One monitor per VM does not survive an autoscaling group.

Troubleshooting path

Validate in signal order and stop at the first failing boundary: the unit starts, the configuration validates, the health endpoint answers, the self-metrics show successful exports, then the signal is searchable in Tsuga.

• The service fails immediately: run sudo otelcol-contrib validate --config=/etc/otelcol-contrib/config.yaml. A Contrib configuration must be validated by otelcol-contrib, not core otelcol.
• No data reaches Tsuga: check journalctl for exporter authentication or network errors, then confirm the endpoint is the base OTLP URL and the ingestion key is present in the root-only environment file. Keep the endpoint and key out of diagnostic tickets.
• Applications cannot send OTLP: they must use 127.0.0.1, because the receiver is intentionally loopback-only. A Docker container is not the host network namespace. Use an appropriate container networking arrangement or a dedicated Collector endpoint rather than assuming host loopback works.
• Docker receiver fails at startup: leave Docker metrics disabled unless the Collector service can read a real Docker socket. In the remote route, use the auto-detected profile rather than forcing docker.
• Cloud tags are absent: check EC2 IMDS tag access, GCP compute.viewer, or Azure VM tags. Restrict enrichment to operationally useful keys so telemetry does not gain uncontrolled cardinality or sensitive metadata.
• Logs are duplicated or missing: choose either file logs or journald for each source, check file permissions, and verify the relevant source exists on that distribution. File paths differ between Debian and RPM families.

Completion criterion

Recent host metrics for the VM are visible in Tsuga under the agreed host name and environment, the host’s log source is searchable, and a local application’s trace arrives through the loopback receiver with the application’s own service.name.

Validate with the Tsuga CLI

# Confirm the distinctive host log line arrived with the expected host identity.
tsuga logs search \
  --query "context.host.name:<host> <unique-log-token>" \
  --from -15m \
  --to now \
  --max-results 10

# Confirm a local application exported spans through the loopback receiver.
tsuga traces search \
  --query "context.service.name:<service>" \
  --from -15m \
  --to now \
  --max-results 10

The first command returns the test line with the host identity the team recognizes. If the record arrives without it, check resourcedetection/cloud and HOST_ALIAS before changing the log source. The second returns spans carrying the application’s service.name. The traces pipeline does not insert the Collector’s identity, so an application that never set its own appears as the SDK default unknown_service. Fix that in the application, not in the Collector. For host metric families, use the Hosts inventory and Validate Telemetry Arrival in Tsuga rather than a CLI aggregation query.