Closes: #1008486

This adds support for .deb ZStandard compression and decompression.
The main reason for this addition is due to Ubuntu having forked the
.deb ecosystem when they added support for this unilaterally, so now
there are many .deb in the wild using this compression format, which
cannot be handled by the upstream dpkg-deb tool.

Although at least now the ZStandard format is widely used on many
projects, has been specified within the IETF as RFC8878, so backwards
compatibility and format stability are no longer a concern, and it has
good trade offs between size and speed.

This has been implemented from scratch, based on the initial prototype
code used during the early request evaluation. Using the new advanced
API, with support for multi-threading, and using an I/O loop resembling
the one used with liblzma, as the plan is to eventually switch all
compressors to use a single I/O loop implementation.

Closes: #892664

Use a new enum dpkg_stream_status to track the stream status, instead of
using the liblzma codes directly in the I/O loop, which will make it
possible in the future to generalize and use a single loop for all
compressors.

Stop handling the xz/lzma actions using the liblzma specific actions,
and use a generic enum, renamed from the old dpkg_stream_status, which
was in fact partially handling the action. This will make it easier in
the future to use the same action for all compressors.

Tracking the compression filter operation and its status merges two
distinct values into the same variable making things more difficult,
as we were using bits when these should really be mutually exclusive
values.

Instead of passing the dpkg_stream_action enum, pass the io_lzma struct,
so that we can use other members if needed.

The compression level max bound check is specific to each compressor,
and we cannot simply use a single max bound for every compressor. Move
the check into the compressor_check_params() function where we can check
whether it is coherent with the compressor requested.

The dpkg_options_parse_arg_int() already makes sure we get a sanitized
value that is 0 or larger but not larger than INT_MAX.

We were testing showing or listing the contents, but not explicitly
building the compressed packages nor extracting them.

Include the ChangeLog.old file which is big, and will better exercise
the compressors.

The regex was not correctly matching at the beginning of the line with
leading spaces, which could be left by a previous replacement from
the same s/// operator. Instead switch to split the flag value and
filter based on a hash, which means we do not need to care about such
space issues anymore.

This change should not change the semantics for space separated options
in the flags, as the code was already splitting the passed values on
spaces, and then remapping on the entire flag value.

Improve strip unit tests to cover all these cases.

Closes: #1028044

Signed-off-by: Johannes Schauer Marin Rodrigues <josch@debian.org>
Signed-off-by: Guillem Jover <guillem@debian.org>

It does not make much sense to ship the upstream «git log» changelog
which contains lots of fluff and metadata. At the same time the recent
debhelper trimming is not ideal as there's no easy way to fetch old
entries anymore. So stop shipping the upstream changelog in favor of
not trimming the debian/changelog one, which acts as a proper user
readable changelog.

Closes: #1027716

When bootstrapping a cross-compiler, it might not have some of the
required parts available, such as a libc. Catch the error, emit it as a
warning and return undef, like with the other conditions where we skip
the test. Because in the end this is just a tainting flag, and it is not
worth it failing the entire build due to that.

Closes: #1027966

Changelog: internal

The $file argument should be part of the error() arguments not the g_()
arguments.

The new version fixes the remaining issue that allows the code to use it
as a backend. Add it as a preferred implementation if available.

The handling for deb 0.x formats that relocates files around once
extracted was using a buffer with a hardcoded size, not taking into
account the length of the directory which would overflow it.

Switch to use a dynamically allocated buffer to handle any destination
directory length.

Reported-by: Georgy Yakovlev <gyakovlev@gentoo.org>

When xz does not have enough memory given the requested threads, it
might end up adjusting the memory usage and number of threads, while
emitting a warning, and then exiting with non-zero. Both of which we
do not want from the library.

If xz cannot meet the memory usage limit on multi-threaded mode, then
it can end up falling back to single-threaded mode, which would then
generate different output, making it non-reproducible. This has the
drawback that it can then error out.

When xz does not have enough memory given the requested threads, it
might end up adjusting the memory usage and number of threads, while
emitting a warning, and then exiting with non-zero. Both of which we
do not want from the library.

These are causing test suite breakage on other projects, which will
block migration for dpkg. Silence them for now.

We are distinguishing the feature being undef to cover the builtin case,
and use_feature() coerces it to a bool values which makes it lose the
undef and causes the flags to include PIE flags.

Reported-by: Helmut Grohne <helmut@subdivi.de>

This is relevant with builtin features where use_feature() will coerce
the returned value into true or false, and where undef is a valid ternary
state we use to track the default the compiler might have.

Update to 1178t.

The unit tests did not make it clear that we are testing how the system
patch(1) works, and not local code. This has caused bugs reported in the
past on systems with unsafe patch(1) implementations.