This would be reported on reading as an 8-byte gap at the position where
a 32-bit DD for the entry would end.
The problem is that if flag bit 3 is set, and the uncompressed size
exceeds 2^32, writeDataDescriptor() will write 8-byte sizes, but the
Zip64 extension header is missing - the latter should also write 0 even
though the APPNOTE doens't appear to require it.
4.3.9.2 [...] When extracting, if
the zip64 extended information extra field is present for
the file the compressed and uncompressed sizes will be 8
byte values.
Also make sure that the ZipEntry nSize member actually contains the
uncompressed size, which is somewhat tricky, but is required in
writeLOC() to check if Zip64 is needed. (Assume that the DEFLATEd size
is never larger than uncompressed, which is true for every XML file).
Change-Id: I611e6ffda12d8e6953e9306dc4a4407be8e725a0
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/191185
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
Reviewed-by: Michael Stahl <michael.stahl@collabora.com>
Integrate build system support for the Zstandard (zstd) library and
implement the corresponding C++ decompression stream classes, enabling
future use of zstd-compressed streams in ZIP packages.
Build System (Part 2a):
- Add `--with-system-zstd=[yes|no|auto]` configure option.
- Implement system library checks (`zstd.h`, `libzstd`) in configure.ac.
- Define rules (`external/zstd/*`) for downloading and building the
internal zstd static library (decompression components only).
- Set up necessary CFLAGS/LIBS via configure.ac and config_host.mk.in.
- Configure linking via RepositoryExternal.mk (`gb_LinkTarget__use_zstd`).
- Update auxiliary files (download.lst, distro-configs, etc.).
C++ Implementation (Part 2b):
- Implement `InflateZstd` (in `include/package`) and `InflaterBytesZstd`
(in `package/inc`) inheriting from `Inflater`/`InflaterBytes`.
- Use the zstd streaming API (`ZSTD_decompressStream`) following the
existing single-call pattern established by the zlib implementation.
- Manage context (`ZSTD_DCtx`) and input buffer (`ZSTD_inBuffer`) state.
- Handle errors, stream completion (`ret == 0`), and input truncation.
- Add new classes to `package/Library_package2.mk` build.
This completes the infrastructure for zstd decompression support.
The next step (Part 2c) involves adding logic to select the
appropriate Inflater based on the ZIP entry's compression method ID.
Change-Id: Ia673f3f19b6a751ba5225394fcf8c3f145c4ae63
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/184243
Tested-by: Jenkins
Reviewed-by: Caolán McNamara <caolan.mcnamara@collabora.com>
Refactor ZIP decompression classes (Inflater, InflaterBytes) to support multiple decompression algorithms, enabling future Zstandard
support.
Make Inflater and InflaterBytes abstract base classes defining a common interface. Move the original zlib/Deflate logic into new derived classes InflateZlib and InflaterBytesZlib.
Update usage sites to instantiate the *Zlib variants via unique_ptr to the base class, preserving existing functionality. Move header files to standard locations (include/package, package/inc) and update includes and build files.
This pure refactoring prepares the codebase for adding Zstandard (zstd) decompression. Correct a minor logical bug in the ZipFile::getSizeAndCRC loop condition.
Change-Id: I9be3c02946e29eb2dae09f500eb47d8a4261af22
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/183504
Reviewed-by: Devansh Varshney <varshney.devansh614@gmail.com>
Tested-by: Caolán McNamara <caolan.mcnamara@collabora.com>
Reviewed-by: Michael Meeks <michael.meeks@collabora.com>
Reviewed-by: Caolán McNamara <caolan.mcnamara@collabora.com>
Tested-by: Jenkins
The PK78 signature is optional, and if the sizes in the local file
header are 0 then without the DD the recovery won't do anything useful.
If the latest local file header has the DD bit set in flags, then try to
find the DD before the next thing that has a PK?? signature.
Change-Id: I149a87805f7bcb6216fc909c20ece600ce77956d
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/183722
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
Tested-by: Jenkins
... duplicates; the directory names of AutoCorrect entries are
user-editable, so this needs to be supported.
AutoCorrect uses an ODF package because the ODF document loading code
requires the ODF document to be in an ODF storage with a MediaType
property.
AutoCorrect writes an empty mimetype file, and if such is present in an
.odt file that is being loaded, existing checks will detect it as
corrupted, so we can use this to check that the file is an AutoCorrect
file and turn off the case-insensitive check.
(regression from commit 4833f13124)
Change-Id: I43887f7dad0c8cbb465b4c0f1c38bcc3244a7675
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/173477
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
Tested-by: Jenkins
The problem is that ZipPackageStream::GetEncryptionData() doesn't handle
the checksum correctly; what is required here is *no checksum* but the
check of m_oImportedChecksumAlgorithm results in calling
m_rZipPackage.GetChecksumAlgID() instead, so it ends up in invalid
situation and assert:
package/source/zippackage/ZipPackageStream.cxx:656: virtual bool ZipPackageStream::saveChild(): Assertion `xEncData->m_nEncAlg != xml::crypto::CipherID::AES_GCM_W3C' failed.
Refactor this so all the imported algorithm identifiers are in a struct
in a std::optional member.
(regression from commit 09f23a3dc5)
Change-Id: I4b705520cd9bc800ce3c8611f8ad01a1e1008929
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/173342
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
Tested-by: Jenkins
The problem is that there is a ZipEntry ZipPackageStream::aEntry which
does not necessarily correspond to the values read from the Zip central
directory, because of a call to ZipPackageStream::setSize() in
ZipPackageStream::successfullyWritten() to override nSize and nMethod
for encrypted entries.
This triggers the consistency checks in readLOC(), where this aEntry is
passed.
Refactor things so that ZipPackageStream::aEntry and thus hopefully all
ZipEntry instances correspond to the Zip file's central directory.
For encrypted entries, store the decompressed size (which is read from
manifest.xml) in m_nOwnStreamOrigSize and pass it as a separate
parameter to the ZipFile functions that need it, and ultimately to
XUnbufferedStream, resolving multiple TODO comments.
(regression from commit efae4fc42d)
Change-Id: Ib6ea32595c6027b98da9196a2e2bd0a99b62a983
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/173273
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
Tested-by: Jenkins
... data descriptor; only allow it for encrypted ODF entries, which
requires reading the manifest first.
Change-Id: If36d31a4cb93e7af78f48be3ed899ad9d9bb28f0
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/171911
Tested-by: Jenkins
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
Check it contains same as central directory header, also check data
descriptor if available. Also check there are no gaps or overlaps.
This causes 3 fuzzer generated test documents to fail to load; adapt
tests.
Change-Id: If5813652f3bd03e90fdf95eb97e1e1523455b2b8
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/170571
Tested-by: Jenkins
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
This is required for OOXML, but not for ODF.
Unclear if there are use cases for this with ODF, can add some
conditions if it turns out to be a problem.
Change-Id: I3810da5c2273574135d133b4a9bbad98dc97af44
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/170223
Tested-by: Jenkins
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
Seeing as since:
commit e9531b792d
Date: Tue Feb 6 14:39:47 2024 +0100
sal: rtlRandomPool: require OS random device, abort if not present
Both rtl_random_createPool() and rtl_random_getBytes() first try to get
random data from the OS, via /dev/urandom or rand_s() (documented to
call RtlGenRandom(), see [1]).
we don't use the initial arg to rtl_random_getBytes anymore, drop the
requirement to have one. Then simplify our usages of that, and
addtionally deprecate rtl_random_createPool and rtl_random_destroyPool.
Change-Id: I13dcc067714a8a741a4e8f2bfcf2006373f832c4
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/167067
Tested-by: Jenkins
Reviewed-by: Caolán McNamara <caolan.mcnamara@collabora.com>
1. ManifestImport should attach KeyInfo to the first file-entry, not to
a "/" entry which may not exist or may be preceded by another
file-entry which requires the KeyInfo to decrypt
(regression from commit 91f35f22f0)
2. manifest:key-size on manifest:key-derivation is unfortunately
specified to have a default value of 16, but with AES256 the actual
size is 32, and for GPG the ManifestExport forgot to export it
Change-Id: I05dd2c9e219ba7f20ba33bf0b35d47a9e3df74cc
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/161072
Tested-by: Jenkins
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
https://www.rfc-editor.org/rfc/rfc9106.html
* add css::xml::crypto::KDFID constant group
* add "KeyDerivationFunction" to setEncryptionAlgorithms sequence
* Argon2 is used by default for wholesome ODF encryption, but
$LO_ARGON2_DISABLE can be set to use PBKDF2
* extend various structs in package
* use 3 new ODF attributes "loext:argon2-iterations" "loext:argon2-memory"
"loext:argon2-lanes" to store the arguments
* use this URL for now:
"urn:org:documentfoundation:names:experimental:office:manifest:argon2id"
* use default arguments according to second recommendation from "7.4.
Recommendations" of RFC9106; 64 MiB RAM should hopefully not be too
much even for 32 bit builds
Change-Id: I683118cc5e0706bd6544db6fb909096768ac9920
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/161009
Tested-by: Jenkins
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
... ODF wholesome encrypted package.
There can only be "mimetype", "encrypted-package", and files in
"META-INF".
Change-Id: I5eb46ba29a1a62e25af09e189e0a075a871c71c4
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/160718
Tested-by: Jenkins
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
AEAD provides the verification of the password automatically, by reading
the entire stream the tag at the end will be verified.
The existing attributes manifest:checksum-type/manifest:checksum leak
information about the plain text.
This was mitigated with the addChaffWhenEncryptedStorage() functions
(see commit f57baefbd3) but a better
solution that also works for non-XML streams is to simply omit the
attributes; authenticated encryption provides better verification
without any leak.
* "ChecksumAlgorithm" property can be set to void now to remove the
checksum
* change a bunch of members in EncryptionData, ZipPackage,
ZipPackageStream to optional
* change ZipFile::checkValidPassword() to open the stream and return it
Change-Id: Id95288d0c238c4f9940fc5a185df814e8edcbad3
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/160711
Tested-by: Jenkins
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
... and use it in the new experimental ODF encryption mode.
https://www.w3.org/TR/xmlenc-core1/#sec-AES-GCM
Unfortunately it turned out that NSS PK11_CipherOp() does not work with
CKM_AES_GCM because it is initialized with "context->multi = PR_FALSE"
in sftk_CryptInit(), so the one-step functions PK11_Encrypt() and
PK11_Decrypt() have to be used.
NSS 3.52 also changed a parameter struct definition - see
https://fedoraproject.org/wiki/Changes/NssGCMParams - which is not a
problem for RHEL or SUSE system NSS since those are rebased, but it
is likely a problem for less well maintained Ubuntu LTS, so use
the old struct definition which evidently still works with NSS 3.94.
NSS 3.52 also added a new PK11_AEADOp() API but it looks like this
doesn't support incremental encryption either.
Change-Id: Ibd4a672db74b65b1218926ba35ff8d2f70444c7e
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/160505
Tested-by: Jenkins
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
...in include files. This is a mix of automatic rewriting in include files and
manual fixups (mostly addressing loplugin:redundantfcast) in source files that
include those.
Change-Id: I1f3cc1e67b9cabd2e9d61a4d9e9a01e587ea35cc
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/158337
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
Implemented import + export for "Zip64 Extended Information Extra Field",
(in "Local file header" and "Central directory file header")
and for Data descriptor.
Focused only to be able to handle files with over 4GB uncompressed size,
in the zip archive.
The 64k filecount, and the 4GB compressed size limit is probably still present
Tried to follow pkware .ZIP File Format Specification,
Some cases were not clear to me and/or some zip compressing tool may not
perfectly follow the standard, like 'extra field' should be 28 bytes long,
but its reader now can read shorter (or longer) 'extra field'.
Replaced some 32bit codes with 64bit codes, in stream handling, in deflater.
Tested with an ods file that contained a content.xml that bigger then 4BG+
(import + export + reimport) on windows.
I think 4GB+ files import/export would be too slow fot unittest.
So, for unit test, used the small but zip64 format files,
that was attached to the bugzilla tickets
Note: It helps with Bug 128244 too (1 of the unittest tests it),
but that ods file missing manifest.xml, so LO won't be able to import it.
Change-Id: Idfeb90594388fd34ae719677f5d268ca9a484fb1
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/147306
Tested-by: Jenkins
Reviewed-by: Michael Meeks <michael.meeks@collabora.com>
to use in places where we know we can use the TempFileFast mechanism.
Speeds up export to EPUB from 43s to 17s on my windows machine.
This is essentially a copy of OTempFileService, but it does not
implement XTempFile (because these temp files have no name on Windows).
Also remove a couple of calls to Flush() (which is not necessary now),
and TellEnd(), which is a little slow on Windows.
Change-Id: Iced300bf8121eae08c7a011d62ec68b93330d84c
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/141598
Tested-by: Noel Grandin <noel.grandin@collabora.co.uk>
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
...so that its TOOLS_WARN_EXCEPTION can be used in
comphelper/source/misc/logging.cxx in a follow-up commit. (And while at it,
rename from diangose_ex.h to the more appropriate diagnose_ex.hxx. The
comphelper module is sufficiently low-level for this immediate use case, so use
that at least for now; o3tl might be even more suitable but doesn't have a
Library until now. Also, for the immediate use case it would have sufficed to
only break DbgGetCaughtException, exceptionToString, TOOLS_WARN_EXCEPTION,
TOOLS_WARN_EXCEPTION_IF, and TOOLS_INFO_EXCEPTION out of
include/tools/diagnose_ex.h into an additional new
include/comphelper/diagnose_ex.hxx, but its probably easier overall to just move
the complete include file as is.)
Change-Id: I9f3222d4ccf1a9ac29d7eb9ba1530d53e2affaee
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/138451
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
All call sites already effectively asked for an unsigned return type, including:
* The ThreadPool ctor took an nWorkers argument of type sal_Int32, but
internally stores that as std::size_t mnMaxWorkers.
* ZipOutputStream::reduceScheduledThreadTasksToGivenNumberOrLess apparently
benefits from an unsigned nThreadTasks parameter, getting rid of various casts
in its implementation that were necessary to silence signed vs. unsigned
comparison warnings.
The only drawback is that
comphelper::ThreadPool::getPreferredConcurrency() * 4
in package/source/zippackage/ZipPackageStream.cxx would now silently wrap around
instead of causing UB on overflow (which could be detected with appropriate
tools). Ideally, it would use some o3tl::saturating_mul if we had that, so add
a TODO comment for now.
While std::thread::hardware_concurrency returns unsigned, it looked more natural
to go with std::size_t here, as some call sites already used that (see above),
so the implementation of ThreadPool::getPreferredConcurrency is a natural place
to hide clamping std::thread::hardware_concurrency() to std::size_t (in the
unlikely case that std::size_t is of smaller rank than unsigned).
This required addition of o3tl::clamp_to_unsigned in o3tl/safeint.hxx.
Change-Id: I0a04a8b32e63ebfeb39f924c4b38520455a6fb38
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/135309
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
Gabor Kelemen