Instead of selecting the first encoding returned by ucsdet_detectAll
(except the 5 CJK when they have confidence of 10), recalculate
confidence by removing 10 only when one of the 5 CJK encodings then
check each encoding returned by ucsdet_detectAll and select the one with
the max confidence with this new rule.
I also tested this again on tdf#172905, no regression.
Change-Id: I6a8e2b30e8467fefd53c02296484e48ca467c662
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/209143
Tested-by: Jenkins
Reviewed-by: Julien Nabet <serval2412@yahoo.fr>
whereas 0 is the starting point (eg for iso-8859-*)
So don't retain the CJK character sets if they're just at 10
and there's no other character sets
+ add some debug traces to track charsets detected by libicu
and their associated confidence
Change-Id: I7d89d06ffb963143276aff749e7d1a6f7a531cca
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/208899
Tested-by: Jenkins
Reviewed-by: Julien Nabet <serval2412@yahoo.fr>
Starts a process of converting the call-sites of SvFileStream
to just write data, relying on the stream being set
into the correct encoding somewhere during initialisation.
In this commit, I change WriteByteStringLine to do that.
For most of the call sites, they either:
(a) use the stream encoding explicitly
(b) already set the stream to the encoding being used at the call-site
Only in buildFontMenuName did I need to make the stream explicitly
set the stream encoding.
Also add a utility constructor to SvFileStream to make it easy
to set the stream in the correct mode at open-time.
Ideally a stream should be set into the correct encoding
at open time and never changed.
Change-Id: I57afa74807e15320a176a05d43c94c377dad4561
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/200289
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
Use the correct term. E.g., Unicode charset has different encodings
like UTF-8 and UTF-16, and we mean encoding here.
Change-Id: I65e90b31cda6e39fdb1584b492e6e0b9821865a5
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/200285
Tested-by: Jenkins
Reviewed-by: Mike Kaganski <mike.kaganski@collabora.com>
Seems like recent font/pdf related changes have exposed a latent
bug(?) in the SvMemoryStream class. Specifically, writing to
SvMemoryStream using a source pointer that points inside the SvMemoryStream
data buffer causes trouble when the buffer is re-allocated.
Make SvMemoryStream tolerant of this, although it is a rather
dodgy use-case.
READ of size 177 at 0x531000155fe7 thread T0
SvMemoryStream::PutData(void const*, unsigned long) /tools/source/stream/stream.cxx:1578:5
SvStream::WriteBytes(void const*, unsigned long) /tools/source/stream/stream.cxx:1195:26
vcl::filter::PDFDocument::WriteCatalogObject(int, vcl::filter::PDFReferenceElement*&) /vcl/source/filter/ipdf/pdfdocument.cxx:577:27
freed by thread T0 here:
in SvMemoryStream::FreeMemory() /tools/source/stream/stream.cxx:1694:9
in SvMemoryStream::ReAllocateMemory(long) /tools/source/stream/stream.cxx:1671:9
in SvMemoryStream::PutData(void const*, unsigned long) /tools/source/stream/stream.cxx:1569:22
in SvStream::WriteBytes(void const*, unsigned long) /tools/source/stream/stream.cxx:1195:26
in vcl::filter::PDFDocument::WriteCatalogObject(int, vcl::filter::PDFReferenceElement*&) /vcl/source/filter/ipdf/pdfdocument.cxx:577:27
Change-Id: I2613ed66cec600b7f44e575a56f6a97c0ff3053d
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/199959
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
...which helps avoid some supposedly spurious -Werror=free-nonheap-object seen
at least with recent GCC 16 trunk,
> In destructor ‘virtual SvLockBytes::~SvLockBytes()’,
> inlined from ‘void SvRefBase::ReleaseRef()’ at include/tools/ref.hxx:160:36,
> inlined from ‘tools::SvRef<T>::~SvRef() [with T = writerfilter::rtftok::RTFTokenizer]’ at include/tools/ref.hxx:56:46,
> inlined from ‘writerfilter::rtftok::RTFLookahead::~RTFLookahead()’ at sw/source/writerfilter/rtftok/rtflookahead.cxx:37:1:
> include/tools/stream.hxx:115:48: error: ‘void operator delete(void*, std::size_t)’ called on pointer ‘((tools::SvRef<writerfilter::rtftok::RTFTokenizer>*)this)[1].tools::SvRef<writerfilter::rtftok::RTFTokenizer>::pObj’ with nonzero offset 56 [-Werror=free-nonheap-object]
> 115 | virtual ~SvLockBytes() override { close(); }
> | ^
etc.
Change-Id: I26a599f0e3258e6bf6a4d6e63e559b41029862db
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/197842
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <stephan.bergmann@collabora.com>
and
cid#1659699 Overflowed return value
cid#1659759 Overflowed return value
cid#1659922 Overflowed return value
cid#1660190 Overflowed return value
cid#1660286 Overflowed return value
cid#1660457 Overflowed return value
Change-Id: Ie9fffe4994b10ed3d4feb14eaf7950e3f01714ef
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/189927
Reviewed-by: Caolán McNamara <caolan.mcnamara@collabora.com>
Tested-by: Jenkins
Fix several SetError implementations, to make sure that original
errors are not rewritten, but OTOH warnings could be replaced by
errors.
Change-Id: Ic5e71b791ff3426ff1144778a610ab77fcbf24d1
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/160530
Tested-by: Jenkins
Reviewed-by: Mike Kaganski <mike.kaganski@collabora.com>
Implemented import + export for "Zip64 Extended Information Extra Field",
(in "Local file header" and "Central directory file header")
and for Data descriptor.
Focused only to be able to handle files with over 4GB uncompressed size,
in the zip archive.
The 64k filecount, and the 4GB compressed size limit is probably still present
Tried to follow pkware .ZIP File Format Specification,
Some cases were not clear to me and/or some zip compressing tool may not
perfectly follow the standard, like 'extra field' should be 28 bytes long,
but its reader now can read shorter (or longer) 'extra field'.
Replaced some 32bit codes with 64bit codes, in stream handling, in deflater.
Tested with an ods file that contained a content.xml that bigger then 4BG+
(import + export + reimport) on windows.
I think 4GB+ files import/export would be too slow fot unittest.
So, for unit test, used the small but zip64 format files,
that was attached to the bugzilla tickets
Note: It helps with Bug 128244 too (1 of the unittest tests it),
but that ods file missing manifest.xml, so LO won't be able to import it.
Change-Id: Idfeb90594388fd34ae719677f5d268ca9a484fb1
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/147306
Tested-by: Jenkins
Reviewed-by: Michael Meeks <michael.meeks@collabora.com>
As discussed in the mailing list thread starting at
<https://lists.freedesktop.org/archives/libreoffice/2023-January/089808.html>
"Plan to remove dead C++ UNO bridge implementations (bridges/source/cpp_uno/*)",
the bridge implementation at bridges/source/cpp_uno/gcc3_aix_powerpc is
apparently dead and should thus be removed. However, that was the only bridge
implementation for AIX, which implies that support for the AIX platform as a
whole is dead and should thus be removed.
Change-Id: I96de3f7f97d4fd770ff78256f0ea435383688be9
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/146057
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
(The "clang-format off" in tools/source/misc/json_writer.cxx is necessary
because otherwise the code between the SAL_WNODEPRECATED_DECLARATIONS_PUSH/POP
macros would be ill-formatted in a way that would trigger loplugin:indentation.)
Change-Id: Ic96787865d4c96be07c41f4939893420dfa04046
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/142339
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
Its comment in include/tools/stream.hxx tells:
Switch to no endian swapping and write 0xfeff
It was introduced in commit 3c2105e07d
Author Eike Rathke <er@openoffice.org>
Date Fri Dec 22 00:19:05 2000 +0000
new: read/write Unicode or Bytecode
and included a call to SetEndianSwap( FALSE ). That call was dropped
in commit 8130714148
Author Noel Grandin <noel@peralex.com>
Date Mon Jan 05 08:47:31 2015 +0200
fdo#84938: replace NUMBERFORMAT_INT_ constants with 'enum class'
(likely by accident).
To simplify the fix, drop redundant m_nEndian: m_isSwap is enough.
Change-Id: Ia9a0fe2d55563e7ba21bd4cf17c4ca999c6feaf7
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/141521
Tested-by: Jenkins
Reviewed-by: Mike Kaganski <mike.kaganski@collabora.com>
EmbeddedObjectRef::GetGraphicStream() creates a writable
SvMemoryStream, read the graphics data into it and then returns
the stream, which will be afterwards used to decode the graphics.
But if the data is broken, incorrect seeking may cause a seek
way past the buffer, and since the stream is writable, it would
be done and cause problems such as running out of memory.
The VersionCompatRead class is one such place that reads size
from the stream and then seeks based on the read data.
Add SvMemoryStream::MakeReadOnly() that will change the stream
to be read-only after the initial read of the data into it.
Change-Id: I1d44aabaf73071a691adafa489e65e4f5e3f021d
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/137002
Tested-by: Jenkins
Reviewed-by: Luboš Luňák <l.lunak@collabora.com>
Caolán McNamara