rt-analyzer User Guide V1.0.1-beta

Editions

System Requirements

To Run the Binary (Target System)

Required Capabilities

• For static analysis: Readable access to /proc and /sys filesystems
• For measurement (--measure): Root access (sudo) and Linux 4.18+ (for tracefs)
• For capture: Root access is recommended so the bundle includes root-only files

Optional Tools (for --measure)

sudo apt install rt-tests   # Debian/Ubuntu
sudo dnf install rt-tests   # RHEL/Fedora

rt-analyzer degrades gracefully without these — you still get the full static report, and --measure reports exactly which tool was missing rather than failing silently.

Installation

Option 1: Run It Where It Lands (Recommended)

scp rt-analyzer user@target:/opt/rt-analyzer
ssh user@target chmod +x /opt/rt-analyzer
ssh user@target /opt/rt-analyzer --version

Option 2: System-Wide Install

sudo install -m 0755 rt-analyzer /usr/local/bin/rt-analyzer
rt-analyzer --version

Quick Start

Analyze Your System (instant)

./rt-analyzer

Prints a terminal report with overall score, findings, and recommendations. No root required.

Get Detailed Analysis

sudo ./rt-analyzer --measure --duration 60

Includes active latency measurement (requires root and optional tools).

Generate an HTML Report

./rt-analyzer -f html

Writes a self-contained rt-report.html that opens offline in any browser — no default filename to remember, no -o required.

Capture for Analysis Elsewhere

sudo ./rt-analyzer capture -o bundle.tar.gz

Snapshots the system. Copy the bundle anywhere (even Windows) and replay it with analyze bundle.tar.gz.

Basic Usage

Usage Syntax

rt-analyzer [options]
rt-analyzer analyze BUNDLE [options]
rt-analyzer capture [-o FILE] [options]

Main Command (Analyze Live System or Bundle)

./rt-analyzer

sudo ./rt-analyzer --measure --duration 60

Subcommands

capture

Snapshot the current system into a gzipped tarball for offline analysis.

sudo ./rt-analyzer capture -o bundle.tar.gz [--redact]

• --redact: Strip process command lines and the hostname for shareable bundles

analyze

Analyze a captured bundle. The bundle path may also be given as --root.

./rt-analyzer analyze bundle.tar.gz [options]

Command Reference

Source Options

Grading Options

Measurement Options (Require Root)

Output Options

Capture Options

Exit Codes

Common Scenarios

Quick Static Check (No Root, Instant)

./rt-analyzer --interval 0 --no-color -q

Full Analysis With Measurement (Requires Root, ~60s)

sudo ./rt-analyzer --measure --duration 60 -f html
# writes ./rt-report.html

CI/CD Gate (Fail if Score Below 80)

./rt-analyzer --interval 0 -q --fail-under 80

Low-Latency Profile (Not Hard Real-Time)

./rt-analyzer --profile low-latency --target-us 1000

Verbose Debugging (See Evidence Paths)

./rt-analyzer --verbose --no-color | less

JSON Output, Piped Into jq

./rt-analyzer -f json -o - | jq '.score'

JSON to a Specific File

./rt-analyzer -f json -o /tmp/scan.json

Examples: Static Analysis

Example 1: A Well-Tuned System

$ ./rt-analyzer --root fixtures/tuned-rt --interval 0 --no-color --ascii

REAL-TIME SYSTEM ANALYSIS
==============================================================================
Kernel:  6.8.0-rt8-amd64  (PREEMPT_RT)
Profile: hard-rt, target 100 us

  Overall Score:  100/100   excellent
  ##############################

    Kernel & preemption    #################### 100
    CPU isolation          #################### 100
    IRQ affinity           #################### 100
    Thread scheduling      #################### 100
    Frequency & C-states   #################### 100
    System noise           #################### 100

  [OK] PREEMPT_RT enabled
  [OK] CPU isolation detected: CPUs 2,3,6,7
  [OK] nohz_full covers every isolated CPU
  [OK] No device interrupts on isolated CPUs
  [OK] Deepest enabled C-state C1 costs 2.0 us
  ...

------------------------------------------------------------------------------
  Nothing to fix -- no failing or warning checks.

Example 2: Detect a Common Misconfiguration

$ ./rt-analyzer --root fixtures/misconfigured --interval 0 --no-color --ascii -q

  Overall Score:  57/100   marginal
  #################.............

  [XX] Isolated CPUs still take the scheduler tick: 2,3
  [XX] Isolated CPUs share physical cores with non-isolated CPUs 6,7
  [XX] C-State C6 enabled (133 us exit latency)
  [!!] Interrupt eth0 (IRQ 124) lands on isolated CPU 3
      Observed 2.8M since boot.

------------------------------------------------------------------------------
  WHAT TO DO

  1. Set the kernel command line
     4 of the findings above are fixed by boot parameters. Apply them as one
     line: the kernel command line is replaced whole, so adding a parameter on
     its own drops the ones already there. Check the CPU numbers against this
     machine first.

       isolcpus=2,3,6,7 nohz_full=2,3 rcu_nocbs=2,3 irqaffinity=0,1,4-7

     Debian/Ubuntu: append it to GRUB_CMDLINE_LINUX_DEFAULT in
     /etc/default/grub and run update-grub. RHEL/Fedora: grubby
     --update-kernel=ALL --args='...'. Reboot, then check /proc/cmdline.

     It covers:
       - Isolated CPUs still take the scheduler tick: 2,3
       - Isolated CPUs share physical cores with non-isolated CPUs 6,7
       - RCU callbacks still run on isolated CPUs: 2,3
       - Default IRQ affinity includes isolated CPUs 2,3

  2. Interrupt eth0 (IRQ 124) lands on isolated CPU 3
     echo 0,1,4-7 > /proc/irq/124/smp_affinity_list

  Item 2 writes to /proc or /sys. Those are the running kernel, not storage:
  the change takes effect at once and is gone after a reboot. To keep it, run
  the commands from a service that starts before the workload.

    # /etc/systemd/system/rt-tuning.service
    [Unit]
    Description=Real-time tuning
    After=sysinit.target

    [Service]
    Type=oneshot
    RemainAfterExit=yes
    ExecStart=/usr/local/sbin/rt-tuning.sh   # the commands above, one a line

    [Install]
    WantedBy=multi-user.target

    systemctl enable --now rt-tuning.service

Example 3: Analyze a Mounted Image

$ ./rt-analyzer --root /mnt/target --verbose

# Analyzes the tree at /mnt/target instead of live /
# Useful for a mounted system image or a chroot

Examples: Active Measurement

Example 1: Quick Latency Check (30 seconds)

$ sudo ./rt-analyzer --measure --duration 30 --no-color -v

  Worst latency:   47 us
  Average latency: 8.2 us
  Best latency:    2 us

  [OK] Worst-case latency 47 us against a 100 us target

Example 2: Deep Diagnostics When the Target Is Missed

hwlatdetect and rtla osnoise run automatically — only when the target was missed — since each costs another minute and perturbs the machine further.

$ sudo ./rt-analyzer --measure --duration 60 -v

  [XX] Worst-case latency 950 us against a 100 us target
  [XX] Firmware stole the CPU 4 times (worst 187 us)
      This is a BIOS problem, not a kernel one. Disable USB legacy
      emulation, hardware power capping/monitoring and thermal
      polling in firmware setup, and check for a BIOS update.

Example 3: HTML Report With Charts

$ sudo ./rt-analyzer --measure --duration 60 -f html
rt-analyzer: wrote rt-report.html

# rt-report.html contains:
#  - Overall score and per-category bars
#  - Per-CPU latency histogram (log scale)
#  - IRQ distribution heatmap
#  - C-state exit-latency table
#  - Findings with evidence, grouped by severity

Examples: Capture & Replay

Example 1: Capture a System for Analysis Elsewhere

$ sudo ./rt-analyzer capture -o myhost.tar.gz
rt-analyzer: wrote myhost.tar.gz

Example 2: Analyze a Captured Bundle Anywhere

$ ./rt-analyzer analyze myhost.tar.gz -f html
rt-analyzer: wrote rt-report.html

# The C binary itself only runs on Linux, but a bundle it produces
# analyzes identically wherever the binary runs — no root required
# to replay it, and no access to the original machine.

Example 3: A Shareable Bundle (Redacted)

$ sudo ./rt-analyzer capture -o report.tar.gz --redact

# Removes:
#  - Process command lines
#  - Hostname (replaced with "redacted")

# Safe to attach to a support ticket

Example 4: Compare Two Systems

$ ./rt-analyzer analyze system1.tar.gz -f json -o s1.json
$ ./rt-analyzer analyze system2.tar.gz -f json -o s2.json
$ diff s1.json s2.json

# Shows exactly what changed between the two systems

The Round-Trip Guarantee

A bundle replays to identical verdicts — not merely the same score, but the same finding-by-finding result. Every release is tested against this on all three reference systems, so a report produced from a bundle says exactly what a report produced on the machine itself would have said.

Output Formats

Terminal Report (Default)

Human-readable text with ANSI colors (auto-detected; honors NO_COLOR). Goes to stdout. Suitable for CI/CD logs, troubleshooting, and quick checks.

HTML Report

Self-contained, offline-capable file — no CDN, no external font, no JavaScript framework. Defaults to rt-report.html. Contains:

• Visual score gauge and per-category bars
• Findings grouped by severity, expandable for evidence
• Per-CPU IRQ heatmap (inline SVG, log-scaled)
• Latency histogram with a log-scale axis
• C-state exit-latency table
• Dark/light theme support (prefers-color-scheme)

JSON Output

Machine-readable snapshot. Defaults to rt-report.json. Contains:

• Score, verdict, per-category scores
• Every finding: id, category, severity, title, detail, impact, remediation, evidence
• A snapshot section: kernel, topology, C-state floors, IRQs on isolated CPUs, RT task inventory
• Measured latency data, when --measure ran
• Paths that could not be read, with their errno

What It Checks

Seven categories, weighted into a 0–100 score. UNKNOWN and N/A findings are excluded from both the numerator and denominator — an unreadable file can neither deflate nor inflate the result.

Three Things It Gets Right

• effective_affinity_list is the truth. smp_affinity_list is only the request — for MSI-X vectors the chip may collapse it, and managed interrupts ignore it entirely.
• Counters beat masks. /proc/interrupts is sampled twice, so the report distinguishes an interrupt merely routed to an isolated CPU from one actually firing there.
• Managed interrupts cannot be moved. NVMe and multi-queue NIC vectors reject smp_affinity writes with EIO — these get the fix that actually works (isolcpus=managed_irq,…) instead of an echo that fails silently.

Troubleshooting

"cannot open '/path': No such file or directory"

Solution: Check the path. It must be either a directory or a .tar/.tar.gz file.

Many Findings Show UNKNOWN

Solution: Some files require root. Run with sudo for full analysis:

sudo ./rt-analyzer

--measure Reports "no latency tool available"

Solution:

sudo apt-get install rt-tests

The tool degrades gracefully — the static report is still complete without these.

"another tracer is active"

Solution: Check what's already tracing, then override deliberately if it's yours:

cat /sys/kernel/tracing/current_tracer
sudo ./rt-analyzer --measure --duration 30 --force

Capture Bundle Won't Analyze

Solution: The transfer likely corrupted the file — re-copy it in binary mode (scp/rsync, not a text-mode transfer) and check the size matches the source.

Build Fails: "zlib.h: No such file"

Solution: Install the zlib development headers:

sudo apt install zlib1g-dev      # Debian/Ubuntu
sudo dnf install zlib-devel      # RHEL/Fedora

Frequently Asked Questions

Q: Do I need PREEMPT_RT to get a good score?

A: Yes, under the default hard-rt profile. Use --profile low-latency for non-RT systems with millisecond-scale targets — it relaxes several severities without changing which checks run.

Q: What does a score of 100 mean?

A: Every applicable check passed. It does not guarantee zero latency — it means the static configuration is optimal and, if measured, the observed worst case was within target.

Q: Can I run this in a container or VM?

A: Yes. Some checks return UNKNOWN if container/VM restrictions hide the underlying data. --measure needs root and, for the tracefs-based tools, an RT-capable kernel exposed to the container.

Q: Is my data safe in a bundle?

A: A bundle contains a full /proc//sys snapshot, including process command lines and the hostname. Use --redact to strip both before sharing.

Q: What CPU architectures are supported?

A: Linux on x86-64.

rt-analyzer (C edition) v1.0.1-beta — © 2026 Haithem Jebali. All rights reserved.

For support or licensing inquiries, contact: h.jebali@haithemjebali.com