Previously, the CJK BreakIterator used custom dictionaries for Chinese
and Japanese. This change removes these custom dictionaries in favor of
the upstream ICU implementation, which uses an externally-maintained
frequency dictionary for these languages.
This change also removes support code for dictionary-based break
iterators, as it is no longer used.
Change-Id: I55c4ce9c842d1751997309fd7446e0a6917915dc
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/166136
Reviewed-by: Caolán McNamara <caolan.mcnamara@collabora.com>
Tested-by: Jenkins
Tested-by: Caolán McNamara <caolan.mcnamara@collabora.com>
Previously, a custom BreakIterator was used for Thai grapheme clusters.
This change deletes the custom BreakIterator, in favor of the ICU
implementation.
Change-Id: Icec94c73a5734c2059786dfbba085f487c488d7c
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/166156
Tested-by: Jenkins
Reviewed-by: Eike Rathke <erack@redhat.com>
This change completes the review of BreakIterator rule customizations,
and adds unit tests for relevant customizations.
Change-Id: I06678fcccfc48d020aac64dd9f58ff36a763af30
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/166017
Tested-by: Jenkins
Reviewed-by: Eike Rathke <erack@redhat.com>
...now that warning about O[U]String vars that could be O[U]StringLiteral is no
longer useful
Change-Id: I389e72038171f28482049b41f6224257dd11f452
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/157992
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
We were requiring ICU 4.6 which was released in 2011, and ifdef'ing our
way through newer ICU versions. ICU is a core dependency and it makes no
sense to build LibreOffice with such ancient versions of it.
This change requires ICU 66 (released in 2020), and removes all the
ifdefs for older versions. There are more cleanups to do, but these will
be done separately.
Change-Id: I2e4f7608a08f4d531b0a4c74bbfdf91a451f833f
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/153387
Tested-by: Jenkins
Reviewed-by: خالد حسني <khaled@libreoffice.org>
Unicode 14, 5 new scripts, 12 new Unicode blocks.
In i18npool/qa/cppunit/test_breakiterator.cxx
TestBreakIterator::testLao() had to be disabled/adapted.
Needs to be investigated, see comments there.
As is, Lao script word break has regressions.
Correct UBLOCK_TANGUT_SUPPLEMENT Unicode range endpoint to
0x18D7F, see
https://www.unicode.org/versions/Unicode14.0.0/erratafixed.html
for which ublock_getCode(0x18D8F) now returned UBLOCK_NO_BLOCK and
thus luckily the assert in svx/source/dialog/charmap.cxx hit.
Change-Id: I4bad16ecfab3f44be365b8f884c57f34af68218e
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/125322
Reviewed-by: Eike Rathke <erack@redhat.com>
Tested-by: Jenkins
...from aa2064c5c5 "Improve
loplugin:stringliteralvar", similar to 8b32a3edad
"Fix stack-use-after-scope" (though this case doesn't appear to have caused any
actual issues).
(After manual inspection, there appear to be no further missing `static` at
least in aa2064c5c5 "Improve
loplugin:stringliteralvar".)
Change-Id: I2b3d0d8d2af1d65f0c5bef8a858107020a620974
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/124137
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
Issue the "instead of O[U]String, pass [u16]string_view" diagnostic also for
operator call arguments. (The "rather than copy, pass subView()" diagnostic is
already part of handleSubExprThatCouldBeView, so no need to repeat it explicitly
for operator call arguments.)
(And many call sites don't even require an explicit [u16]string_view, esp. with
the recent ad48b2b02f "Optimized OString operator
+= overloads". Just some test code in sal/qa/ that explicitly tests the
O[U]String functionality had to be excluded.)
Change-Id: I8d55ba5a7fa16a563f5ffe43d245125c88c793bc
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/115589
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
...by re-enabling the code temporarily #if'ed-out in
a528392e71 "Fixed/improved
loplugin:cppunitassertequals" (and which then triggers lots of other
lopglugin:cppunitassertequal CPPUNIT_ASSERT -> CPPUNIT_ASSERT_EQUAL warnings).
For two css::uno::Reference equality comparisons in cppu/qa/test_any.cxx, it
was more straightforward to rewrite them with an explicit call to operator ==
(which silences loplugin:cppunitassertequal) than to adapt them to
CPPUNIT_ASSERT_EQUAL's requirement for arguments of identical types.
In sc/qa/unit/ucalc_pivottable.cxx, ScDPItemData needs toString, which has been
implemented trivially for now, but might want to combine that with the
DEBUG_PIVOT_TABLE-only ScDPItemData::Dump.
Change-Id: Iae6d09cf69bd4e52fe4411bba9e50c48e696291c
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/110546
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
similar to...
commit 8578a1c9d1
Date: Thu Nov 26 15:47:26 2020 +0200
tdf#138481: Trust the built-in break iterator character data in ICU
Don't use our own char.txt.
the char_in.txt hasn't really changed since 2008 and is woefully out of
date at this point.
we have cppunit tests for the only documented bug that touched
char_in.txt, #i111152# and tdf#40292, for tdf#40292 change the test
to test what was actually reported as a bug
Change-Id: I8e35b102b0a46d2c63e47e055e472892f65022ac
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/106763
Tested-by: Jenkins
Reviewed-by: Caolán McNamara <caolanm@redhat.com>
...introduced with 5d8f0fad50 "Adapt
CPPUNIT_ASSERT to C++20 deleted ostream << for sal_Unicode (aka char16_t)" (see
there for details) but erroneously removed with
877f40ac3f "tdf#42949 Fix new IWYU warnings in
directories [h-r]*"
Change-Id: Id22a4c0fdfe1471e2455ec3316f2c6c93cc00b22
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/105549
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
<http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1423r3.html> "char8_t
backward compatibility remediation", as implemented now by <https://gcc.gnu.org/
git/?p=gcc.git;a=commit;h=0c5b35933e5b150df0ab487efb2f11ef5685f713> "libstdc++:
P1423R3 char8_t remediation (2/4)" for -std=c++2a, deletes operator << overloads
that would print an integer rather than a (presumably expected) character.
But for simplicity (and to avoid issues with non-printing characters), keep
printing an integer here.
Change-Id: I751b99ee32d418eb488131ffa130d6f7d6d38dc7
Reviewed-on: https://gerrit.libreoffice.org/84348
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
It started out as a wrapper around character literals, but has by now become a
wrapper around arbitrary single characters. Besides updating the documentation,
this change is a mechanical
for i in $(git grep -Fl OUStringLiteral1); do sed -i -e s/OUStringLiteral1/OUStringChar/g "$i"; done
Change-Id: I1b9eaa4b3fbc9025ce4a4bffea3db1c16188b76f
Reviewed-on: https://gerrit.libreoffice.org/80892
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
Found with bin/find-unneeded-includes
Only removal proposals are dealt with here.
Change-Id: Ic331c845f0a8f06c4a8f8f79b6f87e26ca7c3a7d
Reviewed-on: https://gerrit.libreoffice.org/72972
Tested-by: Jenkins
Reviewed-by: Michael Stahl <Michael.Stahl@cib.de>
BreakIterator_CTL in the non CharacterIteratorMode::SKIPCELL mode did
not handle UTF-16 surrogate pairs at all, causing backspace to delete
lone surrogates which is really bad. Just copied the corresponding code
from BreakIterator_Unicode.
Additionally, BreakIterator_th was not correctly skipping non-Thai text
and always treating one character as Thai.
Change-Id: Ia379327e042ff602fc19a485c4cbd1a3683f9230
Reviewed-on: https://gerrit.libreoffice.org/54631
Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Eike Rathke <erack@redhat.com>
Korean words are composed of Hangul and are separated
by space or newline. This patch improves line breaking
function in CJK break iterator so that it does not
break Korean words in the middle. It now breaks at the
first character of the last Korean word.
Change-Id: I91b20733c0c5ec4755bf68eb0d7c14c42c1f3556
Reviewed-on: https://gerrit.libreoffice.org/42987
Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Eike Rathke <erack@redhat.com>
We are classifying characters in the “Combining Diacritical Marks”
Unicode block with ScriptType::LATIN, but these are combining marks and
can combine with any script and should have been ScriptType::WEAK. Just
removing them from the range in scriptList does the trick as we will
fallback to getting the script classification based on the Unicode
script property.
Change-Id: I3577f4b03360a1c8e094a207f01b6bbb6abbaf30
Reviewed-on: https://gerrit.libreoffice.org/35811
Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Caolán McNamara <caolanm@redhat.com>
Tested-by: Caolán McNamara <caolanm@redhat.com>
...which had already been conditional on U_ICU_VERSION_MAJOR_NUM > 51, so
452a06b705 "Activate unit test for Lao
breakiterator support" should merely have extended that #if to the function
declaration/definition in response to loplugin:unreffun, not unconditionally
enabled it.
Change-Id: Ia1ce4e99b4a960b1e70cae0a61b0d526b0353d9a
In f24fa8efad added unit test wasn't
activated. Activating it now reveals that the test is failing. Add
TODOs for the failing asserts.
Change-Id: I6c9f36be3aa84a5d5092d2a3f576af39a15b935e
Noel Grandin