ZipOutputEntryBase::processDeflated() was calling isDeflaterFinished()
to find out whether to finalize and dispose m_xCipherContext.
With ZipOutputEntryParallel this was always true, so the context
was disposed after the first call. That was originally correct, because
originally processDeflated() was called just once at the very end, but
353d4528b8 changed to calling it several
times.
Change-Id: I1e4a0c887e4ba7ac7b20fa315169eaa911335fde
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/98625
Tested-by: Jenkins
Reviewed-by: Luboš Luňák <l.lunak@collabora.com>
In this approach the input stream is read one batch (of constant size)
at a time and each batch is compressed by ThreadedDeflater. After
we are done with a batch, the deflated buffer is processed straightaway
(directed to file backed storage).
Change-Id: I2d42f86cf5898e4d746836d94bf6009a8d3b0230
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/86596
Tested-by: Jenkins
Reviewed-by: Luboš Luňák <l.lunak@collabora.com>
I end up with something that fails in the fuzzing setup but which
cannot be used as a testcase in a normal build
Change-Id: I20c73aa009fdc289d95d34a8adf8ddd5882f6694
Reviewed-on: https://gerrit.libreoffice.org/84013
Tested-by: Jenkins
Reviewed-by: Caolán McNamara <caolanm@redhat.com>
Tested-by: Caolán McNamara <caolanm@redhat.com>
and improve the pahole script so I can just run it once over the whole
codebase
Change-Id: I7e1775974a3a61f8c0e40646158f01163ace60cc
Reviewed-on: https://gerrit.libreoffice.org/76122
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
ZipPackageStream::saveChild() already had one threaded compression,
but that still uses only one thread for one stream. Many documents
contain many streams (where this is useful), but large documents
often contain one huge content.xml, which then would be compressed
using just one thread.
But it is in fact possible to do deflate in parallel on the same data,
at the cost of somewhat increased CPU usage (spread over threads).
This is handled separately from the background thread path, as
integrating these two approaches would probably be needlessly complex
(since they both internally use ThreadPool, the tasks should often
intermix and parallelize anyway).
On my 4-core (8 HT threads) machine this reduces the compression time
of tdf#113042 from 3s to 1s.
Change-Id: Ifbc889a27966f97eb1ce2ce01c5fb0b151a1bdf8
Reviewed-on: https://gerrit.libreoffice.org/73032
Tested-by: Jenkins
Reviewed-by: Luboš Luňák <l.lunak@collabora.com>
It can be easily separated out, it looked like hacked in. And
I will need to do more refactoring of the class, so this shouldn't
be more complex than necessary.
Change-Id: I302da55409e9195274907ca4939c37fbb2427b18
Reviewed-on: https://gerrit.libreoffice.org/73031
Tested-by: Jenkins
Reviewed-by: Luboš Luňák <l.lunak@collabora.com>
E.g. with the bugdoc from tdf#93553, most of the streams to save
are actually small, they just do not provide XSeekable. Since it
seems all the streams are already fully available by the time
the zip packaging is done, fallback to XInputStream::available()
to find out if the stream is small.
With this, the zipping part of saving tdf#93553 is now down
from ~1.5s to ~0.5s.
This presumably also makes the hack for tdf#93553 unnecessary.
Change-Id: Id9bb7d835400d6d8f147f5c11ade684a549aba53
Reviewed-on: https://gerrit.libreoffice.org/73030
Tested-by: Jenkins
Reviewed-by: Luboš Luňák <l.lunak@collabora.com>
A threadpool controls a number of threads that execute a number
of thread *tasks* from a queue. The API even says they are tasks.
So it's damn confusing when ZipPackageStream::saveChild()
claims to limit the number of threads to 4x the maximum number
of threads. It limits the number of queued thread tasks.
Change-Id: I317497f27a82d92a7c8566b14aaeae73a4ffef1f
Reviewed-on: https://gerrit.libreoffice.org/72677
Tested-by: Jenkins
Reviewed-by: Luboš Luňák <l.lunak@collabora.com>
Similar to clang-tidy readability-container-size-empty
Change-Id: Idd67f332b04857a39df26bad1733aae21236f105
Reviewed-on: https://gerrit.libreoffice.org/71764
Tested-by: Jenkins
Reviewed-by: Michael Stahl <Michael.Stahl@cib.de>
if we're doing a find/insert on a set or a map, it is better to just do
a conditional insert/emplace operation than triggering two lookups.
Change-Id: I80da5097f5a89fe30fa348ce5b6e747c34287a8d
Reviewed-on: https://gerrit.libreoffice.org/70937
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
...which has no effect on the result, but silences benign Clang
-fsanitize=implicit-signed-integer-truncation warnings
Change-Id: I0953914a35f2a8c4caa6f75d4918e3b3366d07e8
Reviewed-on: https://gerrit.libreoffice.org/68628
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
Since it is now possible to use C++14, it's time to replace
the temporary solution with the standard one
Change-Id: I8bee1344f7df82536f31bc5e4ec4fd379cac1d04
Reviewed-on: https://gerrit.libreoffice.org/66704
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
...when e.g. FIPS mode makes ZipFile::StaticGetCipher fail by throwing an
exception which would be caught by ZipPackageStream::saveChild (in
package/source/zippackage/ZipPackageStream.cxx) alright (and translated into
bSuccess = false), if ZipFile::StaticGetCipher didn't unhelpfully swallow and
ignore all exceptions in an outer try-catch.
Change-Id: I14376128515df1dd4bdac921edd8ab94cc9b7617
Reviewed-on: https://gerrit.libreoffice.org/59514
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
for those cases where we are doing relatively simple catching and rethrowing
e.g. catch in one thread and throw in main thread.
Change-Id: I6192017c4ec99dd671a9582f7b004096b0fc4525
Reviewed-on: https://gerrit.libreoffice.org/58588
Tested-by: Jenkins
Reviewed-by: Caolán McNamara <caolanm@redhat.com>
Tested-by: Caolán McNamara <caolanm@redhat.com>
under google oss-fuzz, asan + clang version 6.0.0 (trunk 315613)
when using our own exception thrower, but builtin C++11 support works
also, ofz#9266, ofz#9591, ofz#9597, ofz#9622, etc
Change-Id: I29c692da880b49268284de2f61be07fa94f33bc6
Reviewed-on: https://gerrit.libreoffice.org/58223
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
rtl/string.hxx and rtl/ustring.hxx both unnecessarily #include <sal/log.hxx> (and don't make use of it themselves), but many other files happen to depend on it.
This is a continuation of commit 6ff2d84ade to be able to remove those unneeded includes.
This commit adds missing headers to every file found by:
grep -FwL sal/log.hxx $(git grep -Elw 'SAL_INFO|SAL_INFO_IF|SAL_WARN|SAL_WARN_IF|SAL_DETAIL_LOG_STREAM|SAL_WHERE|SAL_STREAM|SAL_DEBUG')
to directories from l10ntools to reportdesign
Change-Id: Ia2dc93dd848c2dc0b6a8cb6e19849c614ec55198
Reviewed-on: https://gerrit.libreoffice.org/58205
Tested-by: Jenkins
Reviewed-by: Miklos Vajna <vmiklos@collabora.co.uk>
And fix leak in XclExpRowBuffer::Finalize, was not freeing the
synchronous task it creates
Change-Id: Id1e9ddb5d968e6b95d9d2b5ca0c9e50774580182
Reviewed-on: https://gerrit.libreoffice.org/56874
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
David Blatter