In physics, observation disturbs the system. Measure a particle’s position, lose its momentum. Heisenberg’s uncertainty principle.

In computing, Sun Microsystems said: No.

DTrace observes everything. DTrace disturbs nothing. Production systems, full tracing, zero overhead when disabled.

This is Sun’s second gift. ZFS protects your data. DTrace reveals what your system actually does.

The Problem Before DTrace:

Before DTrace, debugging production systems was archaeology:

Traditional debugging:
1. Problem occurs in production
2. Add printf/logging statements
3. Recompile
4. Redeploy
5. Wait for problem to recur
6. Examine logs
7. Realize you logged the wrong thing
8. Repeat until insane

Adding instrumentation required code changes. Code changes required testing. Testing required time. Production problems do not wait.

DTrace instruments running systems without modification. No recompilation. No redeployment. No waiting.

What DTrace Does:

DTrace provides probes — points where you can observe system behavior:

# Count syscalls by process
dtrace -n 'syscall:::entry { @[execname] = count(); }'

# Trace file opens
dtrace -n 'syscall::open*:entry { printf("%s %s", execname, copyinstr(arg0)); }'

# Time function execution
dtrace -n 'pid$target::malloc:entry { self->ts = timestamp; }
           pid$target::malloc:return /self->ts/ {
               @["malloc (ns)"] = quantize(timestamp - self->ts);
               self->ts = 0;
           }' -p 1234

# Trace kernel function
dtrace -n 'fbt::tcp_input:entry { trace(arg0); }'

Probes exist at:

• syscall: Every system call entry/return
• fbt: Function Boundary Tracing — every kernel function
• pid: User-space function tracing
• io: Disk I/O events
• sched: Scheduler events
• proc: Process creation/destruction
• tcp/udp/ip: Network stack events

When no DTrace script is running, probes have zero overhead. The instructions are not executed. When you enable a probe, DTrace dynamically patches the running system to call your tracing code.

This is not sampling. This is not approximation. This is every event you ask for.

The D Language:

DTrace uses D — a scripting language for tracing (not the systems programming language):

#!/usr/sbin/dtrace -s

/* Trace slow disk I/O */
io:::start
{
    start[arg0] = timestamp;
}

io:::done
/start[arg0]/
{
    this->delta = timestamp - start[arg0];
    @slow[args[1]->dev_statname] = quantize(this->delta);
    start[arg0] = 0;
}

END
{
    printa(@slow);
}

D looks like C but executes in kernel context safely. No loops that could hang the system. No memory allocation that could exhaust resources. DTrace enforces safety.

You cannot crash the system with a DTrace script. You cannot corrupt memory. You cannot deadlock. Sun engineered the constraints.

Aggregations:

DTrace aggregations summarize data efficiently:

/* Distribution of read sizes */
syscall::read:return
{
    @sizes = quantize(arg0);
}

/* Output:
           value  ------------- Distribution ------------- count
              -1 |                                         2
               0 |@@                                       156
               1 |@@@@@@@@@@@@@@@@@@                       1894
               2 |@@@                                      287
               ...
*/

Aggregations run in kernel. Only summaries cross to userspace. Tracing millions of events produces kilobytes of output.

This is how you trace production without drowning in data.

Platform Support:

Platform	Status
illumos/Solaris	Native — where DTrace was born
FreeBSD	Full support in base
macOS	Available (but neglected by Apple)
Linux	Partial — legal gray area, BPF is the alternative
Windows	No

FreeBSD ships DTrace in base. No packages. No configuration. Just dtrace -n '...' and observe.

Linux has licensing issues (CDDL vs GPL, again). Linux developed BPF/eBPF as their answer. BPF is capable but less elegant. DTrace came first. DTrace works now.

Real Debugging Examples:

# Why is this process slow?
dtrace -n 'syscall:::entry /pid == 1234/ { @[probefunc] = count(); }'

# Who is writing to this file?
dtrace -n 'syscall::write:entry /fds[arg0].fi_pathname == "/var/log/messages"/
           { printf("%s", execname); }'

# Kernel memory allocations
dtrace -n 'fbt::kmem_alloc:entry { @[stack()] = sum(arg0); }'

# TCP connection latency
dtrace -n 'tcp:::connect-request { self->ts = timestamp; }
           tcp:::connect-established /self->ts/ {
               printf("%d ms", (timestamp - self->ts) / 1000000);
               self->ts = 0;
           }'

# Context switches per second
dtrace -n 'sched:::on-cpu { @[execname] = count(); } tick-1s { printa(@); clear(@); }'

No log parsing. No config file editing. Just ask the question, get the answer.

Why Sun Built It:

Sun sold servers to enterprises. Enterprises had production problems. Production problems were expensive.

Traditional debugging: “Please reproduce in test environment.” DTrace debugging: “Let’s trace it right now in production.”

DTrace sold Solaris. DTrace sold SPARC. DTrace sold support contracts because Sun engineers could solve problems others could not.

Then Oracle bought Sun and stopped innovating. But DTrace was already open source. FreeBSD adopted it. The gift survives.

DTrace vs Alternatives:

Tool	Mechanism	Overhead	Safety
DTrace	Dynamic instrumentation	Zero when idle	Guaranteed
strace	ptrace (syscalls only)	High — stops process	Safe but slow
ltrace	Library call tracing	High	Process only
perf	Sampling + tracepoints	Low but incomplete	Kernel statistics
eBPF	VM in kernel	Low	Verified bytecode

strace stops your process for every syscall. Do not use strace in production. DTrace instruments without stopping. Use DTrace everywhere.

The Observability Philosophy:

“You cannot improve what you cannot measure.”

Traditional monitoring tells you what: CPU is high, memory is low, disk is busy.

DTrace tells you why: Which function, which call, which path, which data.

Dashboards show symptoms. DTrace reveals causes.

Integration with ZFS:

DTrace + ZFS = complete storage observability:

/* ZFS read latency by dataset */
fbt::zfs_read:entry
{
    self->ts = timestamp;
}

fbt::zfs_read:return
/self->ts/
{
    @[args[0]->v_path] = quantize(timestamp - self->ts);
    self->ts = 0;
}

Your ZFS pool is slow? Trace it. Find the hot files. Find the slow vdevs. No guessing.

Sun built ZFS and DTrace together. They were designed to complement each other.

The Lesson:

Production systems are black boxes — until you have DTrace.

Every syscall, every function, every I/O, every network packet — observable without overhead, without recompilation, without redeployment.

Heisenberg said observation disturbs the system.

Sun said: Not on our watch.

Run FreeBSD. Run DTrace. See everything. Fix anything.

— Kim Jong Rails, Supreme Leader of the Republic of Derails