+ - 0:00:00
Notes for current slide
Notes for next slide

OpenBSD's small steps towards DTrace

EuroBSDcon 2017, Paris

Jasper Lievisse Adriaanse

1

Agenda

  1. Introduction
  2. DTrace
  3. CTF
  4. Current state
  5. Conclusion
2

Introduction

3

jasper@

  • OpenBSD developer since 2006
    • ports (GNOME)
    • octeon
    • urng
    • ctf
4

jasper@

  • OpenBSD developer since 2006
    • ports (GNOME)
    • octeon
    • urng
    • ctf
  • By day
    • UNIX engineer at Snow
  • By night
    • jasperla
    • jasper_la
4

"a tale about DDB and CTF"

A short while ago in source tree, not so far away...

5

DTrace

(the endgame)

6

DTrace

What is it?

  • Dynamic Tracing framework
  • Originated at Sun for Solaris 10
  • CDDL licensed
  • Ported to macOS/FreeBSD/NetBSD/Linux
  • Providers and probes
  • D scripts
7

DTrace

# dtrace -n 'syscall:::entry { trace(execname); }'
dtrace: description 'syscall:::entry ' matched 233 probes
CPU     ID                    FUNCTION:NAME
  0   5654                      ioctl:entry   dtrace
  0   5654                      ioctl:entry   dtrace
  1   5860                lwp_sigmask:entry   qemu-system-x86_
  1   5568                      write:entry   qemu-system-x86_
[..]
8

DTrace

# dtrace -n 'sysinfo:::writech { @bytes[execname] = sum(arg0); }'
dtrace: description 'sysinfo:::writech ' matched 6 probes
^C
  dtrace                                                            1
  tty                                                              11
  uname                                                            11
  cat                                                              53
  zoneadm                                                        1026
  vmadm                                                          1508
  auditd                                                         1684
  sed                                                            3206
  lldpd                                                          3600
  sshd                                                           4957
  bash                                                           5354
  qemu-system-x86_                                            1074704
9

DTrace

What is it?

Why should you care?

  • truss/strace/ktrace/ltrace only give a partial view
  • Addresses the observability problem
  • Fine-grained (short term) inspection
  • Unparalleled inspection *
  • No more printf debugging (less at least)

* "Shouting in the datacenter"

10

DTrace

What is it?

Why should you care?

How does it work?

  • Providers (e.g. syscall or fbt)
  • Probes (e.g. syscall::execve:entry)
  • Function boundary patching/emulating
  • Knowledge of functions and types
11

CTF

12

13

CTF

Compact Type Format

14

CTF

Que?

  • Compacted notation of
    • labels
    • objects
    • functions
    • types
    • strings
  • .SUNW_ctf ELF section
  • converted from DWARF
  • significantly smaller and simpler than DWARF
15

CTF (ctfdump)

# ctfdump -h /bsd
  cth_magic    = 0xcff1
  cth_version  = 2
  cth_flags    = 0x01
  cth_parlabel = (anon)
  cth_parname  = (anon)
  cth_lbloff   = 0
  cth_objtoff  = 8
  cth_funcoff  = 10958
  cth_typeoff  = 203484
  cth_stroff   = 1649404
  cth_strlen   = 462757
#
16

CTF

Que?

DWARF

  • Debugging data format generated by compiler
  • Compilation Unit (CU)
  • Debugging Information Entry (DIE)
    • DW_TAG_variable, etc
  • DWARFv2 supported on OpenBSD
  • DWARFv5 standardized in 2017
17

DWARF

# dwarfdump nfsstat.o
.debug_info
COMPILE_UNIT<header overall offset = 0x00000000>:
< 0><0x0000000b>  DW_TAG_compile_unit
                    DW_AT_producer              OpenBSD clang version 4.0.0 (tags/RELEASE_400/final) (based on LLVM 4.0.0)
                    DW_AT_language              DW_LANG_C99
                    DW_AT_name                  nfsstat.c
                    DW_AT_stmt_list             0x00000000
                    DW_AT_comp_dir              /usr/src/usr.bin/nfsstat
                    DW_AT_low_pc                0x00000000
                    DW_AT_high_pc               0x000011de
LOCAL_SYMBOLS:
< 1><0x0000002e>    DW_TAG_variable
                      DW_AT_name                  nl
                      DW_AT_type                  <0x00000044>
                      DW_AT_external              yes(1)
                      DW_AT_decl_file             0x00000002 /usr/src/usr.bin/nfsstat/nfsstat.c
                      DW_AT_decl_line             0x0000003d
                      DW_AT_location              DW_OP_addr 0x00000000
< 1><0x00000044>    DW_TAG_array_type
                      DW_AT_type                  <0x00000050>
< 2><0x00000049>      DW_TAG_subrange_type
                        DW_AT_type                  <0x000000c0>
                        DW_AT_count                 0x00000002
[...]
18

CTF

Que?

DWARF

Toolchain

  • ctftools
    • ctfconvert
    • ctfmerge
    • ctfdump
19

CTF

Que?

DWARF

Toolchain

  • ctftools
    • ctfconvert ctfconv
    • ctfmerge
    • ctfdump
    • ctfstrip
20

CTF (ctfdump)

# ctfdump -s /bsd
  [0] \0
  [1] unknown
  [9] void
  [14] wsemul_ops
  [25] name
  [30] cnattach
  [39] attach
  [46] output
  [53] translate
  [63] detach
[..]
21

CTF (ctfdump)

# ctfdump -t /bsd
[..]
  [12] INTEGER signed encoding=SIGNED offset=0 bits=32
  [13] POINTER (anon) refers to 14
  [14] CONST (anon) refers to 15
  [15] STRUCT wsdisplay_emulops (72 bytes)
        cursor type=16 off=0
        mapchar type=18 off=64
        putchar type=22 off=128
        copycols type=22 off=192
        erasecols type=22 off=256
        copyrows type=16 off=320
        eraserows type=16 off=384
        alloc_attr type=22 off=448
       unpack_attr type=27 off=512
[..]
22

CTF (ctfdump)

# ctfdump -f /bsd
[..]
  [4009] FUNC (db_numargs) returns: 12 args: (50126, 13063)
  [4010] FUNC (wdc_preata_drive) returns: 12 args: (23414, 12)
  [4011] FUNC (bwi_set_addr_filter) returns: 1 args: (19183, 278, 3261)
  [4012] FUNC (uslhcom_get_version) returns: 12 args: (18476, 18480)
  [4013] FUNC (rt2560_eeprom_read) returns: 278 args: (12456, 444)
  [4014] FUNC (mpi_detach) returns: 1 args: (21925)
[..]
# ctfdump -t /bsd
[..]
  [12] INTEGER signed encoding=SIGNED offset=0 bits=32
  [50126] POINTER (anon) refers to 50127
  [50127] STRUCT callframe (24 bytes)
  [13063] POINTER (anon) refers to 13064
  [13064] TYPEDEF Elf64_Sym refers to 13065
  [13065] STRUCT (anon) (24 bytes)
[..]
23

CTF

Que?

DWARF

DDB

  • initial consumer of CTF
  • allows in-kernel introspection of its CTF
  • trace / pprint / show struct
24

DDB

Without CTF:

ddb> trace
db_enter() at db_enter+0x5
ddb_sysctl() at ddb_sysctl+0x141
sys_sysctl() at sys_sysctl+0x162
syscall() at syscall+0x1fb
ddb>

With CTF:

ddb> trace
db_enter() at db_enter+0x5
ddb_sysctl(ffff80000e524570,ffff8000ffffb410,0,de41ece8dade919e,0,3dade919e,fff
f8000ffffb410) at ddb_sysctl+0x141
sys_sysctl(ffff80000e524600,ca0,ffff8000ffffb410) at sys_sysctl+0x162
syscall() at syscall+0x1fb
ddb>
25

DDB

ddb> pprint ncpus
ncpus:  1
ddb>
ddb> pprint db_ctf
  db_ctf: {cth = (struct ctf_header *)0xffffffff81b5e2c0, rawctf = (const char *)
0xffffffff81b5e2c0, rawctflen = 659662, data = (const char *)0xffff8000001a9000,
dlen = 2097908, strtab = (char *)0xffffffff81ca9ff8, ctf_found = 1}
ddb>
ddb> show struct ctf_header 0xffffffff81b5e2c0
struct ctf_header at 0xffffffff81b5e2c0 (36 bytes) {cth_magic = 16961521,
cth_version = 258, cth_flags = 1, cth_parlabel = 0, cth_parname = 0, cth_lbloff = 0,
cth_objtoff = 8, cth_funcoff = 10902, cth_typeoff = 202952, cth_stroff = 1635504,
cth_strlen = 462404}
ddb>
26

Current state

and the road ahead

27

Status quo

  • g2k16:
    • dynamic profiling committed
    • illumos ctftools imported in ports
    • initial db_ctf.c committed
  • g2k17:
    • ISC-licensed toolchain imported in base
    • CTF-enabled kernels built by default
  • september 2017:
    • DDB_STRUCT option removed; show struct uses CTF
28

Future

  • DWARFv[3:5]
  • more involved compiler/linker?
  • portability to non-(amd64|i386)
    • dynamic profiling
    • DDB trace CTF usage
29

Future

  • DWARFv[3:5]
  • more involved compiler/linker?
  • portability to non-(amd64|i386)
    • dynamic profiling
    • DDB trace CTF usage
  • /usr/sbin/dtrace
29

Conclusion

30

Conclusion

Recap

  • ISC-licensed toolchain
  • CTF integrated and introspectable
  • Long ways to go...
  •                                  ...but making good progress
  • You can help!
31

Conclusion

Recap

Thanks

  • Martin Pieuchot (mpi@)
  • EuroBSDcon Foundation
  • Snow B.V.
32

Ze end.

Questions?

33

Agenda

  1. Introduction
  2. DTrace
  3. CTF
  4. Current state
  5. Conclusion
2
Paused

Help

Keyboard shortcuts

↑, ←, Pg Up, k Go to previous slide
↓, →, Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow