ZipPackageStream::saveChild() already had one threaded compression,
but that still uses only one thread for one stream. Many documents
contain many streams (where this is useful), but large documents
often contain one huge content.xml, which then would be compressed
using just one thread.
But it is in fact possible to do deflate in parallel on the same data,
at the cost of somewhat increased CPU usage (spread over threads).
This is handled separately from the background thread path, as
integrating these two approaches would probably be needlessly complex
(since they both internally use ThreadPool, the tasks should often
intermix and parallelize anyway).
On my 4-core (8 HT threads) machine this reduces the compression time
of tdf#113042 from 3s to 1s.
Change-Id: Ifbc889a27966f97eb1ce2ce01c5fb0b151a1bdf8
Reviewed-on: https://gerrit.libreoffice.org/73032
Tested-by: Jenkins
Reviewed-by: Luboš Luňák <l.lunak@collabora.com>
It can be easily separated out, it looked like hacked in. And
I will need to do more refactoring of the class, so this shouldn't
be more complex than necessary.
Change-Id: I302da55409e9195274907ca4939c37fbb2427b18
Reviewed-on: https://gerrit.libreoffice.org/73031
Tested-by: Jenkins
Reviewed-by: Luboš Luňák <l.lunak@collabora.com>
E.g. with the bugdoc from tdf#93553, most of the streams to save
are actually small, they just do not provide XSeekable. Since it
seems all the streams are already fully available by the time
the zip packaging is done, fallback to XInputStream::available()
to find out if the stream is small.
With this, the zipping part of saving tdf#93553 is now down
from ~1.5s to ~0.5s.
This presumably also makes the hack for tdf#93553 unnecessary.
Change-Id: Id9bb7d835400d6d8f147f5c11ade684a549aba53
Reviewed-on: https://gerrit.libreoffice.org/73030
Tested-by: Jenkins
Reviewed-by: Luboš Luňák <l.lunak@collabora.com>
A threadpool controls a number of threads that execute a number
of thread *tasks* from a queue. The API even says they are tasks.
So it's damn confusing when ZipPackageStream::saveChild()
claims to limit the number of threads to 4x the maximum number
of threads. It limits the number of queued thread tasks.
Change-Id: I317497f27a82d92a7c8566b14aaeae73a4ffef1f
Reviewed-on: https://gerrit.libreoffice.org/72677
Tested-by: Jenkins
Reviewed-by: Luboš Luňák <l.lunak@collabora.com>
Similar to clang-tidy readability-container-size-empty
Change-Id: Idd67f332b04857a39df26bad1733aae21236f105
Reviewed-on: https://gerrit.libreoffice.org/71764
Tested-by: Jenkins
Reviewed-by: Michael Stahl <Michael.Stahl@cib.de>
if we're doing a find/insert on a set or a map, it is better to just do
a conditional insert/emplace operation than triggering two lookups.
Change-Id: I80da5097f5a89fe30fa348ce5b6e747c34287a8d
Reviewed-on: https://gerrit.libreoffice.org/70937
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
...which has no effect on the result, but silences benign Clang
-fsanitize=implicit-signed-integer-truncation warnings
Change-Id: I0953914a35f2a8c4caa6f75d4918e3b3366d07e8
Reviewed-on: https://gerrit.libreoffice.org/68628
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
Since it is now possible to use C++14, it's time to replace
the temporary solution with the standard one
Change-Id: I8bee1344f7df82536f31bc5e4ec4fd379cac1d04
Reviewed-on: https://gerrit.libreoffice.org/66704
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
...when e.g. FIPS mode makes ZipFile::StaticGetCipher fail by throwing an
exception which would be caught by ZipPackageStream::saveChild (in
package/source/zippackage/ZipPackageStream.cxx) alright (and translated into
bSuccess = false), if ZipFile::StaticGetCipher didn't unhelpfully swallow and
ignore all exceptions in an outer try-catch.
Change-Id: I14376128515df1dd4bdac921edd8ab94cc9b7617
Reviewed-on: https://gerrit.libreoffice.org/59514
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
for those cases where we are doing relatively simple catching and rethrowing
e.g. catch in one thread and throw in main thread.
Change-Id: I6192017c4ec99dd671a9582f7b004096b0fc4525
Reviewed-on: https://gerrit.libreoffice.org/58588
Tested-by: Jenkins
Reviewed-by: Caolán McNamara <caolanm@redhat.com>
Tested-by: Caolán McNamara <caolanm@redhat.com>
under google oss-fuzz, asan + clang version 6.0.0 (trunk 315613)
when using our own exception thrower, but builtin C++11 support works
also, ofz#9266, ofz#9591, ofz#9597, ofz#9622, etc
Change-Id: I29c692da880b49268284de2f61be07fa94f33bc6
Reviewed-on: https://gerrit.libreoffice.org/58223
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
rtl/string.hxx and rtl/ustring.hxx both unnecessarily #include <sal/log.hxx> (and don't make use of it themselves), but many other files happen to depend on it.
This is a continuation of commit 6ff2d84ade to be able to remove those unneeded includes.
This commit adds missing headers to every file found by:
grep -FwL sal/log.hxx $(git grep -Elw 'SAL_INFO|SAL_INFO_IF|SAL_WARN|SAL_WARN_IF|SAL_DETAIL_LOG_STREAM|SAL_WHERE|SAL_STREAM|SAL_DEBUG')
to directories from l10ntools to reportdesign
Change-Id: Ia2dc93dd848c2dc0b6a8cb6e19849c614ec55198
Reviewed-on: https://gerrit.libreoffice.org/58205
Tested-by: Jenkins
Reviewed-by: Miklos Vajna <vmiklos@collabora.co.uk>
And fix leak in XclExpRowBuffer::Finalize, was not freeing the
synchronous task it creates
Change-Id: Id1e9ddb5d968e6b95d9d2b5ca0c9e50774580182
Reviewed-on: https://gerrit.libreoffice.org/56874
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
a zip file within a structured storage stream. underlying storage stream
claims larger size that it can satisfy on read
Change-Id: I8516b12df33ad78b2525192826f5e3aef2622eb8
Reviewed-on: https://gerrit.libreoffice.org/50238
Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Caolán McNamara <caolanm@redhat.com>
Tested-by: Caolán McNamara <caolanm@redhat.com>
regression from...
commit 7d8e94444d
Date: Thu Jan 11 08:47:15 2018 +0200
convert a<b?a:b to std::min(a,b)
cause old promotion was to the largest type, so with a large 64bit nDiff and
any 32bit nRequestedBytes the nRequestedBytes was promoted to 64bit, found to
be smaller than nDiff, then would fit in 32bit. newer code cast both to 32bit,
resulting in a negative nDiff which was smaller than nRequestedBytes, but going
on to cause havoc.
Change-Id: I9eb5c61302b3dccf5333c4dfbf9ac92ce49585b5
Reviewed-on: https://gerrit.libreoffice.org/48155
Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Caolán McNamara <caolanm@redhat.com>
Tested-by: Caolán McNamara <caolanm@redhat.com>
No need to derive password-based key, simply skip rtl_digest_PBKDF2
for the gpg4libre case.
Also pass down PBKDF2 iteration count from ZipPackage, which knows
about GPG encryption, instead of just always setting it in package
stream.
We otherwise needlessly iterate session key also for gpg encrypted
storages.
Change-Id: Ic96b2193f8541bbd109795fb9c0212a0a10c7344
Reviewed-on: https://gerrit.libreoffice.org/47783
Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Thorsten Behrens <Thorsten.Behrens@CIB.de>
Otherwise package code would _only_ try to decrypt for salt and
iteration count present, which does not make sense for gpg.
Change-Id: Ie366f086a3c14d6b54b91b4edee8cfef1a42c44b
Reviewed-on: https://gerrit.libreoffice.org/47782
Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Thorsten Behrens <Thorsten.Behrens@CIB.de>
... when importing ODF documents.
In CreatePackageEncryptionData(), add a 3rd SHA1 password hash,
PackageSHA1CorrectEncryptionKey, to EncryptionData.
Use it in ZipPackageStream::getDataStream(), which has 3 fall-backs
for SHA1 bugs now.
Also add a CorrectSHA1DigestContext, to be used together with
PackageSHA1CorrectEncryptionKey, and rename the existing one to
StarOfficeSHA1DigestContext, to be used together with the existing
2 PackageSHA1{UTF8,MS1252}EncryptionKey.
The fallback won't be used very often anyway: for the password SHA1
to be wrong, you need a password between 52 and 55 bytes long,
and for the SHA1/1K checksum to be wrong, you need a file
smaller than 1K with compressed size mod 64 between 52 and 55;
all XML files have enough random "chaff" added to be too large.
Test that we can read both correct SHA1 and StarOffice SHA1.
Change-Id: I988fa489b5e40c7657f404f18538f637d54d28f1
Caolán McNamara