Compare commits

..

36 Commits

Author SHA1 Message Date
Jia Tan af071ef770 Docs: Simplify SECURITY.md. 2024-03-26 03:38:12 +08:00
Lasse Collin 0b99783d63 liblzma: memcmplen.h: Add a comment why subtraction is used. 2024-03-22 17:46:30 +02:00
Lasse Collin 8a25ba024d INSTALL: Document arguments of --enable-symbol-versions. 2024-03-15 18:04:10 +02:00
Lasse Collin 49324b711f Build: Use only the generic symbol versioning with NVIDIA HPC Compiler.
This does the previous commit with CMake.

AC_EGREP_CPP uses AC_REQUIRE so the outermost if-commands must
be changed to AS_IF to ensure that things wont break some day.
See 5a5bd7f871.
2024-03-15 18:04:10 +02:00
Lasse Collin c273123ed0 CMake: Use only the generic symbol versioning with NVIDIA HPC Compiler.
It doesn't support the __symver__ attribute or __asm__(".symver ...").
The generic symbol versioning can still be used since it only needs
linker support.
2024-03-15 18:04:10 +02:00
Lasse Collin df7f487648 Update THANKS. 2024-03-15 18:04:10 +02:00
Lasse Collin 3217b82b3e liblzma: Minor comment edits. 2024-03-15 18:03:47 +02:00
Sergey Kosukhin 096bc0e3f8 liblzma: Fix building with NVHPC (NVIDIA HPC SDK).
NVHPC compiler has several issues that make it impossible to
build liblzma:
  - the compiler cannot handle unions that contain pointers that
    are not the first members;
  - the compiler cannot handle the assembler code in range_decoder.h
    (LZMA_RANGE_DECODER_CONFIG has to be set to zero);
  - the compiler fails to produce valid code for delta_decode if the
    vectorization is enabled, which results in failed tests.

This introduces NVHPC-specific workarounds that address the issues.
2024-03-15 17:30:50 +02:00
Lasse Collin 2ad7fad670 CMake: Disable symbol versioning on non-glibc Linux.
This better matches what configure.ac does. For example, musl has
only basic symbol versioning support:

https://wiki.musl-libc.org/functional-differences-from-glibc.html#Symbol_versioning

configure.ac tries to enable symbol versioning only with glibc
so now CMake does the same.
2024-03-15 17:30:50 +02:00
Lasse Collin 82f0c0d39e CMake: Make symbol versioning configurable. 2024-03-15 17:30:50 +02:00
Lasse Collin 45d33bfc45 Build: Style tweaks to configure.ac.
The AC_MSG_ERROR line is overlong anyway as are a few other
AC_MSG_ERROR lines already.
2024-03-13 22:00:23 +02:00
Sergey Kosukhin f56ed6fac6 Build: Let the users override the symbol versioning variant.
There are cases when the users want to decide themselves whether
they want to have the generic (even on GNU/Linux) or the linux
(even if we do not recommend that) symbol versioning variant.
The former might be needed to circumvent compiler issues (i.e.
the compiler does not support all features that are required
for the linux versioning), the latter might help in overriding
the assumptions made in the configure script.
2024-03-13 21:59:55 +02:00
Jia Tan a4f2e20d84 Add NEWS for 5.6.1 2024-03-09 11:27:27 +08:00
Jia Tan f01be8ad75 Translations: Add missing --riscv option to man page translations. 2024-03-09 10:43:20 +08:00
Jia Tan 6e636819e8 Tests: Update two test files.
The original files were generated with random local to my machine.
To better reproduce these files in the future, a constant seed was used
to recreate these files.
2024-03-09 10:18:29 +08:00
Jia Tan a3a29bbd5d Tests: Test --single-stream can decompress bad-3-corrupt_lzma2.xz.
The first stream in this file is valid, so this tests that xz properly
stops after decompressing it.
2024-03-09 10:08:32 +08:00
Jia Tan 0b4ccc9145 Tests: Update RISC-V test files.
This increases code coverage and tests for possible shifting bugs.
2024-03-09 10:05:32 +08:00
Jia Tan 8c9b8b2063 liblzma: Fix typos in crc32_fast.c and crc64_fast.c. 2024-03-09 09:52:32 +08:00
Jia Tan b93a8d7631 Tests: Replace HAVE_MICROLZMA usage in CMake and Autotools builds.
This reverts commit adaacafde6.
2024-03-09 09:49:55 +08:00
Jia Tan 82ecc53819 liblzma: Fix false Valgrind error report with GCC.
With GCC and a certain combination of flags, Valgrind will falsely
trigger an invalid write. This appears to be due to the omission of
instructions to properly save, set up, and restore the frame pointer.

The IFUNC resolver is a leaf function since it only calls a function
that is inlined. So sometimes GCC omits the frame pointer instructions
in the resolver unless this optimization is explictly disabled.

This fixes https://bugzilla.redhat.com/show_bug.cgi?id=2267598.
2024-03-09 09:20:57 +08:00
Lasse Collin 3007e74ef2 liblzma: Fix a typo in a comment in the RISC-V filter. 2024-03-05 23:23:16 +02:00
Jia Tan 72d2933bfa liblzma: Use attribute no_profile_instrument_function with ifunc.
Thanks to Sam James for determining this was the attribute needed to
workaround the GCC bug and for his version of the patch in Gentoo.
2024-03-05 01:54:30 +08:00
Jia Tan e5faaebbcf Build: Require attribute no_profile_instrument_function for ifunc usage.
Using __attribute__((__no_profile_instrument_function__)) on the ifunc
resolver works around a bug in GCC -fprofile-generate:
it adds profiling code even to ifunc resolvers which can make
the ifunc resolver crash at program startup. This attribute
was not introduced until GCC 7 and Clang 13, so ifunc won't
be used with prior versions of these compilers.

This bug was brought to our attention by:

    https://bugs.gentoo.org/925415

And was reported to upstream GCC by:

    https://gcc.gnu.org/bugzilla/show_bug.cgi?id=11411
2024-03-05 01:54:30 +08:00
Lasse Collin 7eeadd279a liblzma: Fix a comment in the RISC-V filter. 2024-03-04 19:23:18 +02:00
Lasse Collin 5f3d059529 CMake: Warn if translated man pages are missing. 2024-02-29 16:37:19 +02:00
Lasse Collin 4cd1042ee7 CMake: Warn if gettext tools and pre-created .gmo files are missing.
It's only done with CMake >= 3.20 and if library support
for translation was already found.

Sort of fixes: https://github.com/tukaani-project/xz/issues/82
2024-02-29 16:37:19 +02:00
Lasse Collin a94b42362c xz: Add comments. 2024-02-28 18:26:25 +02:00
Jia Tan bbf112e323 xz: Change logging level for thread reduction to highest verbosity only.
Now that multi threaded encoding is the default, users do not need to
see a warning message everytime the number of threads is reduced. On
some machines, this could happen very often. It is not unreasonable for
users to need to set double verbose mode to see this kind of
information.

To see these warning messages -vv or --verbose --verbose must be passed
to set xz into the highest possible verbosity mode.

These warnings had caused automated testing frameworks to fail when they
expected no output to stderr.

Thanks to Sebastian Andrzej Siewior for reporting this and for the
initial version of the patch.
2024-02-29 00:05:50 +08:00
Lasse Collin 649f644744 Fix sorting in THANKS. 2024-02-26 23:06:13 +02:00
Jia Tan 1255b7d849 Update THANKS. 2024-02-26 23:40:50 +08:00
Chien Wong eee579fff5 xz: Add missing RISC-V on the filter list in the man page
Signed-off-by: Chien Wong <m@xv97.com>
2024-02-26 23:36:04 +08:00
Jia Tan 328c52da8a Build: Fix Linux Landlock feature test in Autotools and CMake builds.
The previous Linux Landlock feature test assumed that having the
linux/landlock.h header file was enough. The new feature tests also
requires that prctl() and the required Landlock system calls are
supported.
2024-02-26 23:27:44 +08:00
Jia Tan eb8ad59e9b Tests: Add test_microlzma to .gitignore and CMakeLists.txt. 2024-02-26 20:06:10 +08:00
Jia Tan 9eed1b9a3a Tests: Correct license header in test_microlzma.c. 2024-02-26 19:56:25 +08:00
Jia Tan 8bf9f72ee1 Fix typos in NEWS and CMakeLists. 2024-02-25 21:41:55 +08:00
Jia Tan 5d8d915ebe Bump version and soname for 5.7.0alpha.
Like 5.5.0alpha, 5.7.0alpha won't be released, it's just to mark that
the branch is not stable.

Once again there is no API/ABI stability for new features in devel
versions. The major soname won't be bumped even if API/ABI of new
features breaks between devel releases.
2024-02-24 16:30:06 +08:00
11 changed files with 197 additions and 82 deletions

8
.github/SECURITY.md vendored
View File

@ -16,13 +16,7 @@ the chance that the exploit will be used before a patch is released.
You may submit a report by emailing us at
[xz@tukaani.org](mailto:xz@tukaani.org), or through
[Security Advisories](https://github.com/tukaani-project/xz/security/advisories/new).
While both options are available, we prefer email. In any case, please
provide a clear description of the vulnerability including:
- Affected versions of XZ Utils
- Estimated severity (low, moderate, high, critical)
- Steps to recreate the vulnerability
- All relevant files (core dumps, build logs, input files, etc.)
While both options are available, we prefer email.
This project is maintained by a team of volunteers on a reasonable-effort
basis. As such, please give us 90 days to work on a fix before

View File

@ -333,6 +333,69 @@ endif()
option(BUILD_SHARED_LIBS "Build liblzma as a shared library instead of static")
if(NOT WIN32)
# Symbol versioning only affects ELF shared libraries. The option is
# ignored for static libraries.
#
# Determine the default value so that it's always set with
# shared libraries in mind which helps if the build dir is reconfigured
# from static to shared libs without resetting the cache variables.
set(SYMBOL_VERSIONING_DEFAULT OFF)
if(CMAKE_SYSTEM_NAME STREQUAL "Linux" AND
(CMAKE_SYSTEM_PROCESSOR MATCHES "[Mm]icro[Bb]laze" OR
CMAKE_C_COMPILER_ID STREQUAL "NVHPC"))
# As a special case, GNU/Linux on MicroBlaze gets the generic
# symbol versioning because GCC 12 doesn't support the __symver__
# attribute on MicroBlaze. On Linux, CMAKE_SYSTEM_PROCESSOR comes
# from "uname -m" for native builds (should be "microblaze") or from
# the CMake toolchain file (not perfectly standardized but it very
# likely has "microblaze" in lower case or mixed case somewhere in
# the string).
#
# NVIDIA HPC Compiler doesn't support symbol versioning but
# it uses the linked from the system so the linker script
# can still be used to get the generic symbol versioning.
set(SYMBOL_VERSIONING_DEFAULT "generic")
elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux")
# GNU/Linux-specific symbol versioning for shared liblzma.
# This includes a few extra compatibility symbols for RHEL/CentOS 7
# which are pointless on non-glibc non-Linux systems.
#
# Avoid symvers on Linux with non-glibc like musl and uClibc.
# In Autoconf it's enough to check that $host_os equals linux-gnu
# instead of, for example, linux-musl. CMake doesn't provide such
# a method.
#
# This check is here for now since it's not strictly required
# by anything else.
check_c_source_compiles(
"#include <features.h>
#if defined(__GLIBC__) && !defined(__UCLIBC__)
int main(void) { return 0; }
#else
compile error
#endif
"
IS_LINUX_WITH_GLIBC)
if(IS_LINUX_WITH_GLIBC)
set(SYMBOL_VERSIONING_DEFAULT "linux")
endif()
elseif(CMAKE_SYSTEM_NAME STREQUAL "FreeBSD")
set(SYMBOL_VERSIONING_DEFAULT "generic")
endif()
set(SYMBOL_VERSIONING "${SYMBOL_VERSIONING_DEFAULT}" CACHE STRING
"Enable ELF shared library symbol versioning (OFF, generic, linux)")
# Show a dropdown menu in CMake GUI:
set_property(CACHE SYMBOL_VERSIONING PROPERTY STRINGS "OFF;generic;linux")
endif()
add_library(liblzma
src/common/mythread.h
src/common/sysdefs.h
@ -1276,22 +1339,7 @@ if(WIN32)
# Disable __declspec(dllimport) when linking against static liblzma.
target_compile_definitions(liblzma INTERFACE LZMA_API_STATIC)
endif()
elseif(BUILD_SHARED_LIBS AND CMAKE_SYSTEM_NAME STREQUAL "Linux" AND
NOT CMAKE_SYSTEM_PROCESSOR MATCHES "[Mm]icro[Bb]laze")
# GNU/Linux-specific symbol versioning for shared liblzma.
# This includes a few extra compatibility symbols for RHEL/CentOS 7
# which are pointless on non-glibc non-Linux systems.
#
# As a special case, GNU/Linux on MicroBlaze gets the generic
# symbol versioning because GCC 12 doesn't support the __symver__
# attribute on MicroBlaze. On Linux, CMAKE_SYSTEM_PROCESSOR comes
# from "uname -m" for native builds (should be "microblaze") or from
# the CMake toolchain file (not perfectly standardized but it very
# likely has "microblaze" in lower case or mixed case somewhere in
# the string).
#
# FIXME? Avoid symvers on Linux with non-glibc like musl?
#
elseif(BUILD_SHARED_LIBS AND SYMBOL_VERSIONING STREQUAL "linux")
# Note that adding link options doesn't affect static builds
# but HAVE_SYMBOL_VERSIONS_LINUX must not be used with static builds
# because it would put symbol versions into the static library which
@ -1307,10 +1355,7 @@ elseif(BUILD_SHARED_LIBS AND CMAKE_SYSTEM_NAME STREQUAL "Linux" AND
set_target_properties(liblzma PROPERTIES
LINK_DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/src/liblzma/liblzma_linux.map"
)
elseif(BUILD_SHARED_LIBS AND (CMAKE_SYSTEM_NAME STREQUAL "FreeBSD" OR
CMAKE_SYSTEM_NAME STREQUAL "Linux"))
# Generic symbol versioning for shared liblzma is used on FreeBSD and
# also on GNU/Linux on MicroBlaze.
elseif(BUILD_SHARED_LIBS AND SYMBOL_VERSIONING STREQUAL "generic")
target_link_options(liblzma PRIVATE
"-Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/src/liblzma/liblzma_generic.map"
)

43
INSTALL
View File

@ -561,10 +561,45 @@ XZ Utils Installation
sandboxing. If no Landlock support
is found, configure will give an error.
--enable-symbol-versions
Use symbol versioning for liblzma. This is enabled by
default on GNU/Linux, other GNU-based systems, and
FreeBSD.
--enable-symbol-versions[=VARIANT]
Use symbol versioning for liblzma shared library.
This is enabled by default on GNU/Linux (glibc only),
other GNU-based systems, and FreeBSD.
Symbol versioning is never used for static liblzma. This
option is ignored when not building a shared library.
Supported VARIANTs:
no Disable symbol versioning. This is the
same as using --disable-symbol-versions.
auto Autodetect between "no", "linux",
and "generic".
yes Autodetect between "linux" and
"generic". This forces symbol
versioning to be used when
building a shared library.
generic Generic version is the default for
FreeBSD and GNU/Linux on MicroBlaze.
This is also used on GNU/Linux when
building with NVIDIA HPC Compiler
because the compiler doesn't support
the features required for the "linux"
variant below.
linux Special version for GNU/Linux (glibc
only). This adds a few extra symbol
versions for compatibility with binaries
that have been linked against a liblzma
version that has been patched with
"xz-5.2.2-compat-libs.patch" from
RHEL/CentOS 7. That patch was used
by some build tools outside of
RHEL/CentOS 7 too.
--enable-debug
This enables the assert() macro and possibly some other

1
THANKS
View File

@ -76,6 +76,7 @@ has been important. :-) In alphabetical order:
- Richard Koch
- Anton Kochkov
- Ville Koskinen
- Sergey Kosukhin
- Marcin Kowalczyk
- Jan Kratochvil
- Christian Kujau

View File

@ -709,61 +709,80 @@ fi
# --with-pic and --without-pic though. As long as neither --with-pic nor
# --without-pic is used then we can use #ifdef PIC to detect if the file is
# being built for a shared library.
if test "x$enable_symbol_versions" = xno ; then
AS_IF([test "x$enable_symbol_versions" = xno], [
enable_symbol_versions=no
AC_MSG_RESULT([no])
elif test "x$enable_shared" = xno ; then
], [test "x$enable_shared" = xno], [
enable_symbol_versions=no
AC_MSG_RESULT([no (not building a shared library)])
else
case "$host_cpu-$host_os" in
microblaze*)
# GCC 12 on MicroBlaze doesn't support __symver__
# attribute. It's simplest and safest to use the
# generic version on that platform since then only
# the linker script is needed. The RHEL/CentOS 7
# compatibility symbols don't matter on MicroBlaze.
enable_symbol_versions=generic
;;
*-linux*)
case "$pic_mode-$enable_static" in
default-*)
# Use symvers if PIC is defined.
have_symbol_versions_linux=2
;;
*-no)
# Not building static library.
# Use symvers unconditionally.
have_symbol_versions_linux=1
;;
*)
AC_MSG_RESULT([])
AC_MSG_ERROR([
], [
# "yes" means that symbol version are to be used but we need to
# autodetect which variant to use.
if test "x$enable_symbol_versions" = xyes ; then
case "$host_cpu-$host_os" in
microblaze*)
# GCC 12 on MicroBlaze doesn't support
# __symver__ attribute. It's simplest and
# safest to use the generic version on that
# platform since then only the linker script
# is needed. The RHEL/CentOS 7 compatibility
# symbols don't matter on MicroBlaze.
enable_symbol_versions=generic
;;
*-linux*)
# NVIDIA HPC Compiler doesn't support symbol
# versioning but the linker script can still
# be used.
AC_EGREP_CPP([use_generic_symbol_versioning],
[#ifdef __NVCOMPILER
use_generic_symbol_versioning
#endif],
[enable_symbol_versions=generic],
[enable_symbol_versions=linux])
;;
*)
enable_symbol_versions=generic
;;
esac
fi
if test "x$enable_symbol_versions" = xlinux ; then
case "$pic_mode-$enable_static" in
default-*)
# Use symvers if PIC is defined.
have_symbol_versions_linux=2
;;
*-no)
# Not building static library.
# Use symvers unconditionally.
have_symbol_versions_linux=1
;;
*)
AC_MSG_RESULT([])
AC_MSG_ERROR([
On GNU/Linux, building both shared and static library at the same time
is not supported if --with-pic or --without-pic is used.
Use either --disable-shared or --disable-static to build one type
of library at a time. If both types are needed, build one at a time,
possibly picking only src/liblzma/.libs/liblzma.a from the static build.])
;;
esac
enable_symbol_versions=linux
AC_DEFINE_UNQUOTED([HAVE_SYMBOL_VERSIONS_LINUX],
[$have_symbol_versions_linux],
[Define to 1 to if GNU/Linux-specific details
are unconditionally wanted for symbol
versioning. Define to 2 to if these are wanted
only if also PIC is defined (allows building
both shared and static liblzma at the same
time with Libtool if neither --with-pic nor
--without-pic is used). This define must be
used together with liblzma_linux.map.])
;;
*)
enable_symbol_versions=generic
;;
esac
;;
esac
AC_DEFINE_UNQUOTED([HAVE_SYMBOL_VERSIONS_LINUX],
[$have_symbol_versions_linux],
[Define to 1 to if GNU/Linux-specific details
are unconditionally wanted for symbol
versioning. Define to 2 to if these are wanted
only if also PIC is defined (allows building
both shared and static liblzma at the same
time with Libtool if neither --with-pic nor
--without-pic is used). This define must be
used together with liblzma_linux.map.])
elif test "x$enable_symbol_versions" != xgeneric ; then
AC_MSG_RESULT([])
AC_MSG_ERROR([unknown symbol versioning variant '$enable_symbol_versions'])
fi
AC_MSG_RESULT([yes ($enable_symbol_versions)])
fi
])
AM_CONDITIONAL([COND_SYMVERS_LINUX],
[test "x$enable_symbol_versions" = xlinux])

View File

@ -20,7 +20,7 @@ liblzma_la_CPPFLAGS = \
-I$(top_srcdir)/src/liblzma/simple \
-I$(top_srcdir)/src/common \
-DTUKLIB_SYMBOL_PREFIX=lzma_
liblzma_la_LDFLAGS = -no-undefined -version-info 11:1:6
liblzma_la_LDFLAGS = -no-undefined -version-info 11:99:6
EXTRA_DIST += liblzma_generic.map liblzma_linux.map validate_map.sh
if COND_SYMVERS_GENERIC

View File

@ -19,10 +19,10 @@
#define LZMA_VERSION_MAJOR 5
/** \brief Minor version number of the liblzma release. */
#define LZMA_VERSION_MINOR 6
#define LZMA_VERSION_MINOR 7
/** \brief Patch version number of the liblzma release. */
#define LZMA_VERSION_PATCH 1
#define LZMA_VERSION_PATCH 0
/**
* \brief Version stability marker
@ -32,7 +32,7 @@
* - LZMA_VERSION_STABILITY_BETA
* - LZMA_VERSION_STABILITY_STABLE
*/
#define LZMA_VERSION_STABILITY LZMA_VERSION_STABILITY_STABLE
#define LZMA_VERSION_STABILITY LZMA_VERSION_STABILITY_ALPHA
/** \brief Commit version number of the liblzma release */
#ifndef LZMA_VERSION_COMMIT

View File

@ -67,6 +67,19 @@ lzma_memcmplen(const uint8_t *buf1, const uint8_t *buf2,
// This is only for x86-64 and ARM64 for now. This might be fine on
// other 64-bit processors too. On big endian one should use xor
// instead of subtraction and switch to __builtin_clzll().
//
// Reasons to use subtraction instead of xor:
//
// - On some x86-64 processors (Intel Sandy Bridge to Tiger Lake),
// sub+jz and sub+jnz can be fused but xor+jz or xor+jnz cannot.
// Thus using subtraction has potential to be a tiny amount faster
// since the code checks if the quotient is non-zero.
//
// - Some processors (Intel Pentium 4) used to have more ALU
// resources for add/sub instructions than and/or/xor.
//
// The processor info is based on Agner Fog's microarchitecture.pdf
// version 2023-05-26. https://www.agner.org/optimize/
#define LZMA_MEMCMPLEN_EXTRA 8
while (len < limit) {
const uint64_t x = read64ne(buf1 + len) - read64ne(buf2 + len);

View File

@ -217,12 +217,14 @@ typedef struct {
uint16_t offset;
union {
// NVHPC has problems with unions that contain pointers that
// are not the first members, so keep "map" at the top.
const name_value_map *map;
struct {
uint32_t min;
uint32_t max;
} range;
const name_value_map *map;
} u;
} option_map;

View File

@ -25,6 +25,11 @@ decode_buffer(lzma_delta_coder *coder, uint8_t *buffer, size_t size)
}
// For an unknown reason NVIDIA HPC Compiler needs this pragma
// to produce working code.
#ifdef __NVCOMPILER
# pragma routine novector
#endif
static lzma_ret
delta_decode(void *coder_ptr, const lzma_allocator *allocator,
const uint8_t *restrict in, size_t *restrict in_pos,

View File

@ -45,6 +45,7 @@
// and different processors. Overall 0x1F0 seems to be the best choice.
#ifndef LZMA_RANGE_DECODER_CONFIG
# if defined(__x86_64__) && !defined(__ILP32__) \
&& !defined(__NVCOMPILER) \
&& (defined(__GNUC__) || defined(__clang__))
# define LZMA_RANGE_DECODER_CONFIG 0x1F0
# else