• Daniel Borkmann's avatar
    bpf, arm64: implement jiting of BPF_XADD · 85f68fe8
    Daniel Borkmann authored
    This work adds BPF_XADD for BPF_W/BPF_DW to the arm64 JIT and therefore
    completes JITing of all BPF instructions, meaning we can thus also remove
    the 'notyet' label and do not need to fall back to the interpreter when
    BPF_XADD is used in a program!
    
    This now also brings arm64 JIT in line with x86_64, s390x, ppc64, sparc64,
    where all current eBPF features are supported.
    
    BPF_W example from test_bpf:
    
      .u.insns_int = {
        BPF_ALU32_IMM(BPF_MOV, R0, 0x12),
        BPF_ST_MEM(BPF_W, R10, -40, 0x10),
        BPF_STX_XADD(BPF_W, R10, R0, -40),
        BPF_LDX_MEM(BPF_W, R0, R10, -40),
        BPF_EXIT_INSN(),
      },
    
      [...]
      00000020:  52800247  mov w7, #0x12 // #18
      00000024:  928004eb  mov x11, #0xffffffffffffffd8 // #-40
      00000028:  d280020a  mov x10, #0x10 // #16
      0000002c:  b82b6b2a  str w10, [x25,x11]
      // start of xadd mapping:
      00000030:  928004ea  mov x10, #0xffffffffffffffd8 // #-40
      00000034:  8b19014a  add x10, x10, x25
      00000038:  f9800151  prfm pstl1strm, [x10]
      0000003c:  885f7d4b  ldxr w11, [x10]
      00000040:  0b07016b  add w11, w11, w7
      00000044:  880b7d4b  stxr w11, w11, [x10]
      00000048:  35ffffab  cbnz w11, 0x0000003c
      // end of xadd mapping:
      [...]
    
    BPF_DW example from test_bpf:
    
      .u.insns_int = {
        BPF_ALU32_IMM(BPF_MOV, R0, 0x12),
        BPF_ST_MEM(BPF_DW, R10, -40, 0x10),
        BPF_STX_XADD(BPF_DW, R10, R0, -40),
        BPF_LDX_MEM(BPF_DW, R0, R10, -40),
        BPF_EXIT_INSN(),
      },
    
      [...]
      00000020:  52800247  mov w7,  #0x12 // #18
      00000024:  928004eb  mov x11, #0xffffffffffffffd8 // #-40
      00000028:  d280020a  mov x10, #0x10 // #16
      0000002c:  f82b6b2a  str x10, [x25,x11]
      // start of xadd mapping:
      00000030:  928004ea  mov x10, #0xffffffffffffffd8 // #-40
      00000034:  8b19014a  add x10, x10, x25
      00000038:  f9800151  prfm pstl1strm, [x10]
      0000003c:  c85f7d4b  ldxr x11, [x10]
      00000040:  8b07016b  add x11, x11, x7
      00000044:  c80b7d4b  stxr w11, x11, [x10]
      00000048:  35ffffab  cbnz w11, 0x0000003c
      // end of xadd mapping:
      [...]
    
    Tested on Cavium ThunderX ARMv8, test suite results after the patch:
    
      No JIT:   [ 3751.855362] test_bpf: Summary: 311 PASSED, 0 FAILED, [0/303 JIT'ed]
      With JIT: [ 3573.759527] test_bpf: Summary: 311 PASSED, 0 FAILED, [303/303 JIT'ed]
    Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
    Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    85f68fe8
Name
Last commit
Last update
..
842 Loading commit data...
fonts Loading commit data...
lz4 Loading commit data...
lzo Loading commit data...
mpi Loading commit data...
raid6 Loading commit data...
reed_solomon Loading commit data...
xz Loading commit data...
zlib_deflate Loading commit data...
zlib_inflate Loading commit data...
.gitignore Loading commit data...
Kconfig Loading commit data...
Kconfig.debug Loading commit data...
Kconfig.kasan Loading commit data...
Kconfig.kgdb Loading commit data...
Kconfig.kmemcheck Loading commit data...
Kconfig.ubsan Loading commit data...
Makefile Loading commit data...
argv_split.c Loading commit data...
asn1_decoder.c Loading commit data...
assoc_array.c Loading commit data...
atomic64.c Loading commit data...
atomic64_test.c Loading commit data...
audit.c Loading commit data...
bcd.c Loading commit data...
bch.c Loading commit data...
bitmap.c Loading commit data...
bitrev.c Loading commit data...
bsearch.c Loading commit data...
btree.c Loading commit data...
bug.c Loading commit data...
build_OID_registry Loading commit data...
bust_spinlocks.c Loading commit data...
chacha20.c Loading commit data...
check_signature.c Loading commit data...
checksum.c Loading commit data...
clz_ctz.c Loading commit data...
clz_tab.c Loading commit data...
cmdline.c Loading commit data...
compat_audit.c Loading commit data...
cordic.c Loading commit data...
cpu_rmap.c Loading commit data...
cpumask.c Loading commit data...
crc-ccitt.c Loading commit data...
crc-itu-t.c Loading commit data...
crc-t10dif.c Loading commit data...
crc16.c Loading commit data...
crc32.c Loading commit data...
crc32defs.h Loading commit data...
crc32test.c Loading commit data...
crc7.c Loading commit data...
crc8.c Loading commit data...
ctype.c Loading commit data...
debug_info.c Loading commit data...
debug_locks.c Loading commit data...
debugobjects.c Loading commit data...
dec_and_lock.c Loading commit data...
decompress.c Loading commit data...
decompress_bunzip2.c Loading commit data...
decompress_inflate.c Loading commit data...
decompress_unlz4.c Loading commit data...
decompress_unlzma.c Loading commit data...
decompress_unlzo.c Loading commit data...
decompress_unxz.c Loading commit data...
devres.c Loading commit data...
digsig.c Loading commit data...
div64.c Loading commit data...
dma-debug.c Loading commit data...
dma-noop.c Loading commit data...
dma-virt.c Loading commit data...
dump_stack.c Loading commit data...
dynamic_debug.c Loading commit data...
dynamic_queue_limits.c Loading commit data...
earlycpio.c Loading commit data...
extable.c Loading commit data...
fault-inject.c Loading commit data...
fdt.c Loading commit data...
fdt_empty_tree.c Loading commit data...
fdt_ro.c Loading commit data...
fdt_rw.c Loading commit data...
fdt_strerror.c Loading commit data...
fdt_sw.c Loading commit data...
fdt_wip.c Loading commit data...
find_bit.c Loading commit data...
flex_array.c Loading commit data...
flex_proportions.c Loading commit data...
gcd.c Loading commit data...
gen_crc32table.c Loading commit data...
genalloc.c Loading commit data...
glob.c Loading commit data...
globtest.c Loading commit data...
hexdump.c Loading commit data...
hweight.c Loading commit data...
idr.c Loading commit data...
inflate.c Loading commit data...
int_sqrt.c Loading commit data...
interval_tree.c Loading commit data...
interval_tree_test.c Loading commit data...
iomap.c Loading commit data...
iomap_copy.c Loading commit data...
iommu-common.c Loading commit data...
iommu-helper.c Loading commit data...
ioremap.c Loading commit data...
iov_iter.c Loading commit data...
irq_poll.c Loading commit data...
irq_regs.c Loading commit data...
is_single_threaded.c Loading commit data...
jedec_ddr_data.c Loading commit data...
kasprintf.c Loading commit data...
kfifo.c Loading commit data...
klist.c Loading commit data...
kobject.c Loading commit data...
kobject_uevent.c Loading commit data...
kstrtox.c Loading commit data...
kstrtox.h Loading commit data...
lcm.c Loading commit data...
libcrc32c.c Loading commit data...
list_debug.c Loading commit data...
list_sort.c Loading commit data...
llist.c Loading commit data...
locking-selftest-hardirq.h Loading commit data...
locking-selftest-mutex.h Loading commit data...
locking-selftest-rlock-hardirq.h Loading commit data...
locking-selftest-rlock-softirq.h Loading commit data...
locking-selftest-rlock.h Loading commit data...
locking-selftest-rsem.h Loading commit data...
locking-selftest-softirq.h Loading commit data...
locking-selftest-spin-hardirq.h Loading commit data...
locking-selftest-spin-softirq.h Loading commit data...
locking-selftest-spin.h Loading commit data...
locking-selftest-wlock-hardirq.h Loading commit data...
locking-selftest-wlock-softirq.h Loading commit data...
locking-selftest-wlock.h Loading commit data...
locking-selftest-wsem.h Loading commit data...
locking-selftest.c Loading commit data...
lockref.c Loading commit data...
lru_cache.c Loading commit data...
md5.c Loading commit data...
memory-notifier-error-inject.c Loading commit data...
memweight.c Loading commit data...
net_utils.c Loading commit data...
netdev-notifier-error-inject.c Loading commit data...
nlattr.c Loading commit data...
nmi_backtrace.c Loading commit data...
nodemask.c Loading commit data...
notifier-error-inject.c Loading commit data...
notifier-error-inject.h Loading commit data...
of-reconfig-notifier-error-inject.c Loading commit data...
oid_registry.c Loading commit data...
once.c Loading commit data...
parman.c Loading commit data...
parser.c Loading commit data...
pci_iomap.c Loading commit data...
percpu-refcount.c Loading commit data...
percpu_counter.c Loading commit data...
percpu_ida.c Loading commit data...
percpu_test.c Loading commit data...
plist.c Loading commit data...
pm-notifier-error-inject.c Loading commit data...
prime_numbers.c Loading commit data...
radix-tree.c Loading commit data...
random32.c Loading commit data...
ratelimit.c Loading commit data...
rational.c Loading commit data...
rbtree.c Loading commit data...
rbtree_test.c Loading commit data...
reciprocal_div.c Loading commit data...
refcount.c Loading commit data...
rhashtable.c Loading commit data...
sbitmap.c Loading commit data...
scatterlist.c Loading commit data...
seq_buf.c Loading commit data...
sg_pool.c Loading commit data...
sg_split.c Loading commit data...
sha1.c Loading commit data...
show_mem.c Loading commit data...
siphash.c Loading commit data...
smp_processor_id.c Loading commit data...
sort.c Loading commit data...
stackdepot.c Loading commit data...
stmp_device.c Loading commit data...
string.c Loading commit data...
string_helpers.c Loading commit data...
strncpy_from_user.c Loading commit data...
strnlen_user.c Loading commit data...
swiotlb.c Loading commit data...
syscall.c Loading commit data...
test-kstrtox.c Loading commit data...
test-string_helpers.c Loading commit data...
test_bitmap.c Loading commit data...
test_bpf.c Loading commit data...
test_firmware.c Loading commit data...
test_hash.c Loading commit data...
test_hexdump.c Loading commit data...
test_kasan.c Loading commit data...
test_module.c Loading commit data...
test_parman.c Loading commit data...
test_printf.c Loading commit data...
test_rhashtable.c Loading commit data...
test_siphash.c Loading commit data...
test_sort.c Loading commit data...
test_static_key_base.c Loading commit data...
test_static_keys.c Loading commit data...
test_user_copy.c Loading commit data...
test_uuid.c Loading commit data...
textsearch.c Loading commit data...
timerqueue.c Loading commit data...
ts_bm.c Loading commit data...
ts_fsm.c Loading commit data...
ts_kmp.c Loading commit data...
ubsan.c Loading commit data...
ubsan.h Loading commit data...
ucs2_string.c Loading commit data...
uuid.c Loading commit data...
vsprintf.c Loading commit data...
win_minmax.c Loading commit data...