commit b14583ba37 initially added the
parquet filter with saying
Note that this change makes it possible to load Apache Parquet files
if and only if orcus has been built with the parquet import filter
enabled. Using orcus without the parquet import filter enabled will
not break the build or run-time behavior; you just can't load parquet
files.
This is bad since silently failing like this doesn't tell people what
the problem is. Not having the option at all is clearer and more deterministic.
And the internal liborcus in a "standard" LibreOffice build didn't build with parquet
enabled at all anyway, thus leaving LibreOffice offering the option without even able to
load the files.
This patch adds
a) an explicit --with-parquet-filter configure flag on which the config
is built or not
b) based on that flag also gives --with-parquet-filter to liborcus'
configure. (which then of course will make that liborcus linked
against libparquet. But if people explicitely specify it..)
c) fixes the build of liborcus against arrow 23 (see
https://gitlab.com/orcus/orcus/-/commit/bae0e6e659cf4df2590288f566004eef7096dee4 -
that incidentially also fixed the liborcus build against 22 for me
This is a generalized version of a local patch I carry in Debian - which
I added some time after I made liborcus in Debian *do*
--with-parquet-filter :)
Change-Id: I72b15893fe4fa8e08a556f27b2d6ab942c9858cc
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/204685
Tested-by: Jenkins
Reviewed-by: René Engelhard <rene@debian.org>
This includes one bug fix where orcus_xml did not call
import_factory's finalize() interface method per interface
contract. With it properly fixed in liborcus itself, it
is no longer necessary to manually call it at the end of
ScOrcusXMLContextImpl::importXML().
Also moving up the type detection order for the generic
HTML type to be before the generic XML type since something
like
<html><body>some text</body></html>
can be detected as a valid XML if it is checked first.
A (well-formed) HTML can be considered a speciazlied XML,
so it should come before XML.
Change-Id: I20f24706421104181ad3dc8250d93f78d6b611e9
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/190042
Tested-by: Jenkins
Reviewed-by: Kohei Yoshida <kohei@libreoffice.org>
For Windows build, explicitly define HAVE_FILESYSTEM which forces
orcus to use std::filesystem for all filesystem related operations.
Also, pass to the ctor of orcus::file_content a filepath as
std::u16string_view on Windows to hopefully correctly handle unicode
filepaths.
Change-Id: Ie6f5403212e73c81f1a36db0413133d26a3a67f2
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/189892
Reviewed-by: Kohei Yoshida <kohei@libreoffice.org>
Tested-by: Jenkins
Hopefully this is a more reliable fix for loading a file that contains
a unicode segment on Windows.
Change-Id: Iabe3ea849858290e782123840d8756ed07159dec
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/187762
Reviewed-by: Kohei Yoshida <kohei@libreoffice.org>
Tested-by: Jenkins
there was already parts of that in the makefile that would require
running "make gb_GCOV=YES" - but that alone was not enough, you were
also kinda expected to set LDFLAGS on the commandline (or at least it
had bitrotted to this state) - so just make it into a proper configure
switch.
Building with lto fails at least on baseline with
/opt/rh/gcc-toolset-12/root/usr/libexec/gcc/x86_64-redhat-linux/12/ld: /tmp/ccaE1KD4.ltrans31.ltrans.o:(.data.rel.ro.local+0x3befe0)
: undefined reference to `.L1'
/opt/rh/gcc-toolset-12/root/usr/libexec/gcc/x86_64-redhat-linux/12/ld: /tmp/ccaE1KD4.ltrans31.ltrans.o:(.data.rel.ro.local+0x3befe8)
: undefined reference to `.L1'
/opt/rh/gcc-toolset-12/root/usr/libexec/gcc/x86_64-redhat-linux/12/ld: /tmp/ccaE1KD4.ltrans31.ltrans.o:(.data.rel.ro.local+0x3bf018)
: undefined reference to `.L2'
/opt/rh/gcc-toolset-12/root/usr/libexec/gcc/x86_64-redhat-linux/12/ld: /tmp/ccaE1KD4.ltrans31.ltrans.o:(.data.rel.ro.local+0x3bf020)
: undefined reference to `.L2'
[…many more of those with incrementing references…]
So for now warn if both are used.
Change-Id: I2c3427b48174de77a6894293a5e784df8ee24bd1
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/186821
Tested-by: Jenkins
Reviewed-by: Christian Lohmaier <lohmaier+LibreOffice@googlemail.com>
Without this, the document model side would not know what grammer
to use to parse formula expressions.
Change-Id: I7e921f9c0f9b088fb98fe126ad4a84db006c8747
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/157413
Tested-by: Jenkins
Reviewed-by: Kohei Yoshida <kohei@libreoffice.org>
It causes a problem when building with Emscripten and
--enable-wasm-exceptions. Instead of fixing that problem, just don't
bother building executables we won't run anyway. (Ideally liborcus
should have some --disable-benchmark configure options, ping Kohei.)
Change-Id: Ic8c1d51513793bba85dc2833cc8fd9ec5b97ce76
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/145645
Tested-by: Jenkins
Reviewed-by: Tor Lillqvist <tml@collabora.com>
Adding new config option: --with-main-module=<writer/calc>
With this switcher we can build only WASM writer module or
calc module. The default value is 'writer'.
This commit also reverts: 26603bc9ef
(WASM fix orcus native exception handling (NEH))
Because Calc import is using liborcus for loading styles.xml.
Change-Id: I4c330ef8eea7d08214bf357531ee0bf7383ab788
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/137946
Reviewed-by: Thorsten Behrens <thorsten.behrens@allotropia.de>
Tested-by: Jenkins
This adds UTF8->UTF16 conversion of the paths passed to ctor of
file_content on Windows, since both boost::filesystem::file_size
and boost::interprocess::file_mapping take UTF-16 strings on this
platform. So the assumption is that 8-bit path strings passed to
orcus are UTF-8-encoded.
This partially reverts commit 75252e58d9
(it keeps use of osl_getThreadTextEncoding for platforms other than
Windows).
Change-Id: Ie467f71a65945f4f07ff432136ea06b811c3f794
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/137759
Tested-by: Jenkins
Reviewed-by: Mike Kaganski <mike.kaganski@collabora.com>
...that was present since 173214d920 "Fix liborcus
build for Android" (but doesn't seem to have caused issues all those years)
Change-Id: I70161f20a94655e8f485cd49bec9ce461c7507f7
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/136563
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
...as seen during CppunitTest_vcl_pdfexport:
> ==573913==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62b0001dba0e at pc 0x560576627186 bp 0x7ffeab9fa730 sp 0x7ffeab9f9ef0
> READ of size 26624 at 0x62b0001dba0e thread T0
> #0 in StrtolFixAndCheck(void*, char const*, char**, char*, int) at ~/github.com/llvm/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:3629:3
> #1 in strtol at ~/github.com/llvm/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:485:3
> #2 in orcus::sax_token_handler_wrapper_base::attribute(std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>) at workdir/UnpackedTarball/liborcus/src/parser/sax_token_parser.cpp:344:22
> #3 in orcus::sax_ns_parser<orcus::sax_token_parser<orcus::xml_stream_handler>::handler_wrapper>::handler_wrapper::attribute(orcus::sax::parser_attribute const&) at workdir/UnpackedTarball/liborcus/src/liborcus/../../include/orcus/sax_ns_parser.hpp:212:27
> #4 in orcus::sax_parser<orcus::sax_ns_parser<orcus::sax_token_parser<orcus::xml_stream_handler>::handler_wrapper>::handler_wrapper, orcus::sax_parser_default_config>::attribute() at workdir/UnpackedTarball/liborcus/src/liborcus/../../include/orcus/sax_parser.hpp:570:15
> #5 in orcus::sax_parser<orcus::sax_ns_parser<orcus::sax_token_parser<orcus::xml_stream_handler>::handler_wrapper>::handler_wrapper, orcus::sax_parser_default_config>::declaration(char const*) at workdir/UnpackedTarball/liborcus/src/liborcus/../../include/orcus/sax_parser.hpp:389:9
> #6 in orcus::sax_parser<orcus::sax_ns_parser<orcus::sax_token_parser<orcus::xml_stream_handler>::handler_wrapper>::handler_wrapper, orcus::sax_parser_default_config>::element() at workdir/UnpackedTarball/liborcus/src/liborcus/../../include/orcus/sax_parser.hpp:242:13
> #7 in orcus::sax_parser<orcus::sax_ns_parser<orcus::sax_token_parser<orcus::xml_stream_handler>::handler_wrapper>::handler_wrapper, orcus::sax_parser_default_config>::body() at workdir/UnpackedTarball/liborcus/src/liborcus/../../include/orcus/sax_parser.hpp:214:13
> #8 in orcus::sax_parser<orcus::sax_ns_parser<orcus::sax_token_parser<orcus::xml_stream_handler>::handler_wrapper>::handler_wrapper, orcus::sax_parser_default_config>::parse() at workdir/UnpackedTarball/liborcus/src/liborcus/../../include/orcus/sax_parser.hpp:182:5
> #9 in orcus::sax_ns_parser<orcus::sax_token_parser<orcus::xml_stream_handler>::handler_wrapper>::parse() at workdir/UnpackedTarball/liborcus/src/liborcus/../../include/orcus/sax_ns_parser.hpp:277:14
> #10 in orcus::sax_token_parser<orcus::xml_stream_handler>::parse() at workdir/UnpackedTarball/liborcus/src/liborcus/../../include/orcus/sax_token_parser.hpp:215:14
> #11 in orcus::xml_stream_parser::parse() at workdir/UnpackedTarball/liborcus/src/liborcus/xml_stream_parser.cpp:68:9
> #12 in orcus::orcus_xls_xml::detect(unsigned char const*, unsigned long) at workdir/UnpackedTarball/liborcus/src/liborcus/orcus_xls_xml.cpp:94:16
> #13 in orcus::detect(unsigned char const*, unsigned long) at workdir/UnpackedTarball/liborcus/src/liborcus/format_detection.cpp:68:9
> #14 in (anonymous namespace)::OrcusFormatDetect::detect(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue>&) at sc/source/filter/orcus/filterdetect.cxx:83:31
> 0x62b0001dba0e is located 0 bytes to the right of 26638-byte region [0x62b0001d5200,0x62b0001dba0e)
> allocated by thread T0 here:
> #0 in operator new[](unsigned long) at ~/github.com/llvm/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:98:3
> #1 in SvMemoryStream::AllocateMemory(unsigned long) at tools/source/stream/stream.cxx:1698:12
> #2 in SvMemoryStream::SvMemoryStream(unsigned long, unsigned long) at tools/source/stream/stream.cxx:1544:9
> #3 in (anonymous namespace)::OrcusFormatDetect::detect(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue>&) at sc/source/filter/orcus/filterdetect.cxx:71:20
This started to occur now after a95c585433 "Pump
XInputStream into an SvMemoryStream rather than an OStringBuffer" no longer
guarantees that the memory range passed into
orcus::detect(const unsigned char* buffer, size_t length)
is followed by a null byte at buffer[length]. (There appears to be no
documentation for that function, but it looks unreasonable to me that it should
require callers to provide a buffer thus terminated, and I rather assume that
what is observed here is an orcus bug.)
The problematic calls of std::strtol were used in code apparently meant to parse
strings matching the XML VersionNum grammar production, and then store the two
dot-separated numbers each as uint8_t. The new code using a local readUint8
accepts a different set of strings now than the original code using std::strtol,
but the new set is arguably closer to what the actual XML VersionNum grammar
production accepts (which is '1.' [0-9]+ for XML 1.0 and '1.1' for XML 1.1), so
this change should be OK.
Change-Id: I1668542c96ced64667cb9f251e79126e1a54ac30
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/136405
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
This is basically ea68de2968 but
for LDFLAGS. A number of external libs cannot use this because
their libtool mishandles -fuse-ld.
Change-Id: Idee379eb0a3afb475b536519ee3de064b4e218f4
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/133639
Tested-by: Jenkins
Reviewed-by: Luboš Luňák <l.lunak@collabora.com>
While this fixes the build with --enable-wasm-exceptions, the
resulting binary generates an "indirect call signature mismatch",
which doesn't happen with the non-NEH build.
The Chrome DWARF backtrace points to CallbackTaskScheduling().
Further debugging reveals, it's actually the UpdateMinPeriod
call for the "desktop::Desktop m_firstRunTimer" Timer.
Disturbingly the debug dynamic_cast<Timer> at the start of the
job loop fails, and the fallback generic job output is
chosen, AKA:
info:vcl.schedule: 6516 0x3871618 i: 0 a: 1 p: 1 \
desktop::Desktop m_firstRunTimer
m_firstRunTimer is a Timer member in the Desktop class, so this
looks like some memory corruption or toolchain problem.
The size difference is more then 10% and it's supposed to be
faster too. FWIW the optimized link time is still high and needs
32GB+ memory compared to the 13GB non-optimized memory usage.
Change-Id: I06d37ecece09000fd3b72a73e7bf40f0b0f61457
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/130216
Tested-by: Jenkins
Reviewed-by: Jan-Marek Glogowski <glogow@fbihome.de>
No idea why we just provided the platform flags when cross-
compiling. In the curious case, where the host platform is
detected as x86_64-pc-mingw32 per default and we actually
want to override it with x86_64-pc-cygwin, we don't do a
cross compile, but must override the host platform.
But there is additional special handling needed for the omitted
cross-platform build in the special case of --host=i686-pc-cygwin
and --build=x86_64-pc-cygwin, where we deliberatly ignore cross
building; Windows is already a slow build, so try to keep this
optimization (AMD64 can execute x86 binaries).
There is the theoretical case, where the externals config.guess
would have detected something else and that "magically" even
worked, while the LO detected triplet would fail, but this
should be fixed in the external in any way.
Change-Id: Ib7a9719e0e406fe90334b7611dc3f01b51692bfa
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/129153
Tested-by: Jenkins
Reviewed-by: Jan-Marek Glogowski <glogow@fbihome.de>
(this was meant as a prerequisite for enabling its -D_LIBCPP_DEBUG=1 counterpart
when building against libc++ on macOS, but which got stalled for now after
running into the issue described at
<https://lists.llvm.org/pipermail/libcxx-dev/2021-October/001222.html>
"[libcxx-dev] Building a program with -D_LIBCPP_DEBUG=1 against a libc++ that is
not itself built with that define")
Change-Id: If466dce595a9311b2afbae41d5ddcaecc6f3c57b
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/124678
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
...as seen with recent GCC 11 trunk libstdc++:
> orcus_xlsx.cpp: In function ‘size_t orcus::{anonymous}::get_schema_rank(orcus::schema_t)’:
> orcus_xlsx.cpp:313:59: error: incomplete type ‘std::numeric_limits<long unsigned int>’ used in nested name specifier
> 313 | return it == rank_map.end() ? numeric_limits<size_t>::max() : it->second;
> | ^~~
etc.
Change-Id: If92cfb565ed9344b2ec1403793d7aeff8bd019ad
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/104074
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
Rene Engelhard