• Linus Torvalds's avatar
    Merge branch 'zstd-minimal' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs · e7cdb60f
    Linus Torvalds authored
    Pull zstd support from Chris Mason:
     "Nick Terrell's patch series to add zstd support to the kernel has been
      floating around for a while. After talking with Dave Sterba, Herbert
      and Phillip, we decided to send the whole thing in as one pull
      request.
    
      zstd is a big win in speed over zlib and in compression ratio over
      lzo, and the compression team here at FB has gotten great results
      using it in production. Nick will continue to update the kernel side
      with new improvements from the open source zstd userland code.
    
      Nick has a number of benchmarks for the main zstd code in his lib/zstd
      commit:
    
          I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB
          of RAM. The VM is running on a MacBook Pro with a 3.1 GHz Intel
          Core i7 processor, 16 GB of RAM, and a SSD. I benchmarked using
          `silesia.tar` [3], which is 211,988,480 B large. Run the following
          commands for the benchmark:
    
            sudo modprobe zstd_compress_test
            sudo mknod zstd_compress_test c 245 0
            sudo cp silesia.tar zstd_compress_test
    
          The time is reported by the time of the userland `cp`.
          The MB/s is computed with
    
            1,536,217,008 B / time(buffer size, hash)
    
          which includes the time to copy from userland.
          The Adjusted MB/s is computed with
    
            1,536,217,088 B / (time(buffer size, hash) - time(buffer size, none)).
    
          The memory reported is the amount of memory the compressor
          requests.
    
            | Method   | Size (B) | Time (s) | Ratio | MB/s    | Adj MB/s | Mem (MB) |
            |----------|----------|----------|-------|---------|----------|----------|
            | none     | 11988480 |    0.100 |     1 | 2119.88 |        - |        - |
            | zstd -1  | 73645762 |    1.044 | 2.878 |  203.05 |   224.56 |     1.23 |
            | zstd -3  | 66988878 |    1.761 | 3.165 |  120.38 |   127.63 |     2.47 |
            | zstd -5  | 65001259 |    2.563 | 3.261 |   82.71 |    86.07 |     2.86 |
            | zstd -10 | 60165346 |   13.242 | 3.523 |   16.01 |    16.13 |    13.22 |
            | zstd -15 | 58009756 |   47.601 | 3.654 |    4.45 |     4.46 |    21.61 |
            | zstd -19 | 54014593 |  102.835 | 3.925 |    2.06 |     2.06 |    60.15 |
            | zlib -1  | 77260026 |    2.895 | 2.744 |   73.23 |    75.85 |     0.27 |
            | zlib -3  | 72972206 |    4.116 | 2.905 |   51.50 |    52.79 |     0.27 |
            | zlib -6  | 68190360 |    9.633 | 3.109 |   22.01 |    22.24 |     0.27 |
            | zlib -9  | 67613382 |   22.554 | 3.135 |    9.40 |     9.44 |     0.27 |
    
          I benchmarked zstd decompression using the same method on the same
          machine. The benchmark file is located in the upstream zstd repo
          under `contrib/linux-kernel/zstd_decompress_test.c` [4]. The
          memory reported is the amount of memory required to decompress
          data compressed with the given compression level. If you know the
          maximum size of your input, you can reduce the memory usage of
          decompression irrespective of the compression level.
    
            | Method   | Time (s) | MB/s    | Adjusted MB/s | Memory (MB) |
            |----------|----------|---------|---------------|-------------|
            | none     |    0.025 | 8479.54 |             - |           - |
            | zstd -1  |    0.358 |  592.15 |        636.60 |        0.84 |
            | zstd -3  |    0.396 |  535.32 |        571.40 |        1.46 |
            | zstd -5  |    0.396 |  535.32 |        571.40 |        1.46 |
            | zstd -10 |    0.374 |  566.81 |        607.42 |        2.51 |
            | zstd -15 |    0.379 |  559.34 |        598.84 |        4.61 |
            | zstd -19 |    0.412 |  514.54 |        547.77 |        8.80 |
            | zlib -1  |    0.940 |  225.52 |        231.68 |        0.04 |
            | zlib -3  |    0.883 |  240.08 |        247.07 |        0.04 |
            | zlib -6  |    0.844 |  251.17 |        258.84 |        0.04 |
            | zlib -9  |    0.837 |  253.27 |        287.64 |        0.04 |
    
      I ran a long series of tests and benchmarks on the btrfs side and the
      gains are very similar to the core benchmarks Nick ran"
    
    * 'zstd-minimal' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs:
      squashfs: Add zstd support
      btrfs: Add zstd support
      lib: Add zstd modules
      lib: Add xxhash module
    e7cdb60f
Name
Last commit
Last update
..
842 Loading commit data...
fonts Loading commit data...
lz4 Loading commit data...
lzo Loading commit data...
mpi Loading commit data...
raid6 Loading commit data...
reed_solomon Loading commit data...
xz Loading commit data...
zlib_deflate Loading commit data...
zlib_inflate Loading commit data...
zstd Loading commit data...
.gitignore Loading commit data...
Kconfig Loading commit data...
Kconfig.debug Loading commit data...
Kconfig.kasan Loading commit data...
Kconfig.kgdb Loading commit data...
Kconfig.kmemcheck Loading commit data...
Kconfig.ubsan Loading commit data...
Makefile Loading commit data...
argv_split.c Loading commit data...
asn1_decoder.c Loading commit data...
assoc_array.c Loading commit data...
atomic64.c Loading commit data...
atomic64_test.c Loading commit data...
audit.c Loading commit data...
bcd.c Loading commit data...
bch.c Loading commit data...
bitmap.c Loading commit data...
bitrev.c Loading commit data...
bsearch.c Loading commit data...
btree.c Loading commit data...
bug.c Loading commit data...
build_OID_registry Loading commit data...
bust_spinlocks.c Loading commit data...
chacha20.c Loading commit data...
check_signature.c Loading commit data...
checksum.c Loading commit data...
clz_ctz.c Loading commit data...
clz_tab.c Loading commit data...
cmdline.c Loading commit data...
compat_audit.c Loading commit data...
cordic.c Loading commit data...
cpu_rmap.c Loading commit data...
cpumask.c Loading commit data...
crc-ccitt.c Loading commit data...
crc-itu-t.c Loading commit data...
crc-t10dif.c Loading commit data...
crc16.c Loading commit data...
crc32.c Loading commit data...
crc32defs.h Loading commit data...
crc32test.c Loading commit data...
crc4.c Loading commit data...
crc7.c Loading commit data...
crc8.c Loading commit data...
ctype.c Loading commit data...
debug_info.c Loading commit data...
debug_locks.c Loading commit data...
debugobjects.c Loading commit data...
dec_and_lock.c Loading commit data...
decompress.c Loading commit data...
decompress_bunzip2.c Loading commit data...
decompress_inflate.c Loading commit data...
decompress_unlz4.c Loading commit data...
decompress_unlzma.c Loading commit data...
decompress_unlzo.c Loading commit data...
decompress_unxz.c Loading commit data...
devres.c Loading commit data...
digsig.c Loading commit data...
div64.c Loading commit data...
dma-debug.c Loading commit data...
dma-noop.c Loading commit data...
dma-virt.c Loading commit data...
dump_stack.c Loading commit data...
dynamic_debug.c Loading commit data...
dynamic_queue_limits.c Loading commit data...
earlycpio.c Loading commit data...
errseq.c Loading commit data...
extable.c Loading commit data...
fault-inject.c Loading commit data...
fdt.c Loading commit data...
fdt_empty_tree.c Loading commit data...
fdt_ro.c Loading commit data...
fdt_rw.c Loading commit data...
fdt_strerror.c Loading commit data...
fdt_sw.c Loading commit data...
fdt_wip.c Loading commit data...
find_bit.c Loading commit data...
flex_array.c Loading commit data...
flex_proportions.c Loading commit data...
gcd.c Loading commit data...
gen_crc32table.c Loading commit data...
genalloc.c Loading commit data...
glob.c Loading commit data...
globtest.c Loading commit data...
hexdump.c Loading commit data...
hweight.c Loading commit data...
idr.c Loading commit data...
inflate.c Loading commit data...
int_sqrt.c Loading commit data...
interval_tree.c Loading commit data...
interval_tree_test.c Loading commit data...
iomap.c Loading commit data...
iomap_copy.c Loading commit data...
iommu-common.c Loading commit data...
iommu-helper.c Loading commit data...
ioremap.c Loading commit data...
iov_iter.c Loading commit data...
irq_poll.c Loading commit data...
irq_regs.c Loading commit data...
is_single_threaded.c Loading commit data...
jedec_ddr_data.c Loading commit data...
kasprintf.c Loading commit data...
kfifo.c Loading commit data...
klist.c Loading commit data...
kobject.c Loading commit data...
kobject_uevent.c Loading commit data...
kstrtox.c Loading commit data...
kstrtox.h Loading commit data...
lcm.c Loading commit data...
libcrc32c.c Loading commit data...
list_debug.c Loading commit data...
list_sort.c Loading commit data...
llist.c Loading commit data...
locking-selftest-hardirq.h Loading commit data...
locking-selftest-mutex.h Loading commit data...
locking-selftest-rlock-hardirq.h Loading commit data...
locking-selftest-rlock-softirq.h Loading commit data...
locking-selftest-rlock.h Loading commit data...
locking-selftest-rsem.h Loading commit data...
locking-selftest-rtmutex.h Loading commit data...
locking-selftest-softirq.h Loading commit data...
locking-selftest-spin-hardirq.h Loading commit data...
locking-selftest-spin-softirq.h Loading commit data...
locking-selftest-spin.h Loading commit data...
locking-selftest-wlock-hardirq.h Loading commit data...
locking-selftest-wlock-softirq.h Loading commit data...
locking-selftest-wlock.h Loading commit data...
locking-selftest-wsem.h Loading commit data...
locking-selftest.c Loading commit data...
lockref.c Loading commit data...
lru_cache.c Loading commit data...
memory-notifier-error-inject.c Loading commit data...
memweight.c Loading commit data...
net_utils.c Loading commit data...
netdev-notifier-error-inject.c Loading commit data...
nlattr.c Loading commit data...
nmi_backtrace.c Loading commit data...
nodemask.c Loading commit data...
notifier-error-inject.c Loading commit data...
notifier-error-inject.h Loading commit data...
of-reconfig-notifier-error-inject.c Loading commit data...
oid_registry.c Loading commit data...
once.c Loading commit data...
parman.c Loading commit data...
parser.c Loading commit data...
pci_iomap.c Loading commit data...
percpu-refcount.c Loading commit data...
percpu_counter.c Loading commit data...
percpu_ida.c Loading commit data...
percpu_test.c Loading commit data...
plist.c Loading commit data...
pm-notifier-error-inject.c Loading commit data...
prime_numbers.c Loading commit data...
radix-tree.c Loading commit data...
random32.c Loading commit data...
ratelimit.c Loading commit data...
rational.c Loading commit data...
rbtree.c Loading commit data...
rbtree_test.c Loading commit data...
reciprocal_div.c Loading commit data...
refcount.c Loading commit data...
rhashtable.c Loading commit data...
sbitmap.c Loading commit data...
scatterlist.c Loading commit data...
seq_buf.c Loading commit data...
sg_pool.c Loading commit data...
sg_split.c Loading commit data...
sha1.c Loading commit data...
show_mem.c Loading commit data...
siphash.c Loading commit data...
smp_processor_id.c Loading commit data...
sort.c Loading commit data...
stackdepot.c Loading commit data...
stmp_device.c Loading commit data...
string.c Loading commit data...
string_helpers.c Loading commit data...
strncpy_from_user.c Loading commit data...
strnlen_user.c Loading commit data...
swiotlb.c Loading commit data...
syscall.c Loading commit data...
test-kstrtox.c Loading commit data...
test-string_helpers.c Loading commit data...
test_bitmap.c Loading commit data...
test_bpf.c Loading commit data...
test_debug_virtual.c Loading commit data...
test_firmware.c Loading commit data...
test_hash.c Loading commit data...
test_hexdump.c Loading commit data...
test_kasan.c Loading commit data...
test_kmod.c Loading commit data...
test_list_sort.c Loading commit data...
test_module.c Loading commit data...
test_parman.c Loading commit data...
test_printf.c Loading commit data...
test_rhashtable.c Loading commit data...
test_siphash.c Loading commit data...
test_sort.c Loading commit data...
test_static_key_base.c Loading commit data...
test_static_keys.c Loading commit data...
test_sysctl.c Loading commit data...
test_user_copy.c Loading commit data...
test_uuid.c Loading commit data...
textsearch.c Loading commit data...
timerqueue.c Loading commit data...
ts_bm.c Loading commit data...
ts_fsm.c Loading commit data...
ts_kmp.c Loading commit data...
ubsan.c Loading commit data...
ubsan.h Loading commit data...
ucs2_string.c Loading commit data...
usercopy.c Loading commit data...
uuid.c Loading commit data...
vsprintf.c Loading commit data...
win_minmax.c Loading commit data...
xxhash.c Loading commit data...