Use OUString::iterateCodePoints() to get the next code point. Just
incrementing the index does not work with surrogate pairs.
This is an existing bug that was exposed by properly handling surrogate
pairs while case folding in:
commit 8922952429
Author: Khaled Hosny <khaled@libreoffice.org>
Date: Mon Jul 24 18:45:38 2023 +0300
i18nutil: Fix ICU case folding of surrogate pairs
Change-Id: I51328ae6d17ef935d5be00f7327949a26724b9ab
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/207750
Tested-by: Jenkins
Reviewed-by: Khaled Hosny <khaled@libreoffice.org>
Since 5a03d511f4 commit,
ICU line breaking rules are now honored and Asian typography options
for paragraph formats are now ignored.
Respect cases where users apply a list that is different from the
default forbidden character list, or cases where no forbidden character
list is applied.
Change-Id: I7d021c1dcb1e9722eb5f7ae6295b949641dcfe01
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/198442
Tested-by: Jenkins
Reviewed-by: Jonathan Clark <jonathan@libreoffice.org>
If the spelling dictionary contains them with non-ASCII apostrophe.
Previously the words were checked only with apostrophes converted
to their ASCII version, resulting continuous false alarms, despite
the correct orthography in the document and in the spelling
dictionary.
Now the words are checked both with ASCII conversion and with
the original non-ASCII apostrophes (because still there are
UTF-8 dictionaries with ASCII apostrophes, moreover, dictionaries
containing mixed apostrophes in the same dic file).
Example words for the Hungarian dictionary: d’Arc, d’Alembert,
McDonald’s (and their several recognized suffixed forms: d’Arcért,
d’Arckal, d’Arcként, d’Arcnak, d’Arcot etc. etc.)
Add unit tests for 1) break iterator and 2) spell checking with
non-ASCII apostrophes.
Change-Id: I24a570df41fa5aba2e7b67dde0db33377717dc2a
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/196248
Reviewed-by: László Németh <nemeth@numbertext.org>
Tested-by: Jenkins
Restores Hungarian break iterator by recognizing ordinal
numbers with affixes, e.g. "50.-et" etc. (to spell checking
bad forms, like "50.-at").
Regression from commit 44699b3de3
"tdf#49885 BreakIterator rule upgrades".
Follow-up to commit f4fe6df6aa
"tdf#162514 i18npool: Handle abbreviations in dictionary breakiterator"
and commit 167bbe31c0
"tdf#162514 i18npool: apply fix for Hungarian abbreviations, too".
Change-Id: I6b6ea363a509a9d8af8330330f78cc23c210d107
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/196225
Tested-by: Jenkins
Reviewed-by: László Németh <nemeth@numbertext.org>
Upstream ICU includes non-standard rule extensions for Chinese and
Japanese, which add line breaking opportunities between U+201C/U+201D
double quotation marks and leading/trailing CJK ideographs. This change
backports those rule customizations, for correct layout of Simplified
Chinese.
Regression from:
commit 5a03d511f4
Author: Jonathan Clark <jonathan@libreoffice.org>
Date: Mon Apr 7 21:37:26 2025 -0600
tdf#130592 i18npool: Fix incorrect line breaking in mixed CJK+Latin
Change-Id: I3cf6c0cd23e38f9a89c98fdd6c6f665f175b676c
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/190904
Tested-by: Jenkins
Reviewed-by: Jonathan Clark <jonathan@libreoffice.org>
This change updates the CJK BreakIterator to run ICU line breaking rules
after applying user forbidden rules. This fixes several identified CJK
GUI text rendering issues, most significantly that embedded English runs
were previously broken using CJK break rules, rather than using the
correct English rules.
By implementing this fallback to the ICU rules, the CJK BreakIterator
now also correctly handles non-breaking characters. This change
therefore also fixes tdf#117554.
Change-Id: I3bacc7e2b559729df2c37c010974d180c8eef1f5
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/183825
Tested-by: Jenkins
Reviewed-by: Jonathan Clark <jonathan@libreoffice.org>
Regression from commit 14c6cde779:
"tdf#49885 Updated CJK BreakIterator to use ICU"
Previously, languages requiring dictionary-based break iterators were
handled by instantiating a stock ICU break iterator as a special case.
tdf#49885 upgraded our custom rules to support passthrough for
dictionary-based breaking, so this special case is no longer necessary.
Change-Id: Iebb06de82eb511946e5b220e5dc414440838b03c
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/177713
Tested-by: Jenkins
Reviewed-by: Jonathan Clark <jonathan@libreoffice.org>
Previously, Danish words ending with hyphens were not handled correctly
during spell checking. This change fixes the issue by modifying the
da_DK locale to reference the existing breakiterator rules for languages
with word-final hyphens.
Change-Id: I9b08fb5246b5b846814dc3b4b9ba249b29fc7a84
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/177504
Tested-by: Jenkins
Reviewed-by: Jonathan Clark <jonathan@libreoffice.org>
Previously, Writer counted characters for all CJK languages, rather than
words. This is the correct behavior for Chinese and Japanese, which make
extensive use of ideographs. However, it is not correct for Korean.
This change adjusts the Writer word count algorithm to count Korean
words, rather than Korean characters.
Change-Id: I6e77136867baca1a7b51248886ee5fd7073ad364
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/170621
Tested-by: Jenkins
Reviewed-by: Jonathan Clark <jonathan@libreoffice.org>
Per Unicode rules, ZWJ shouldn't be treated as a line break opportunity.
This change adds regression tests to verify lines do not break at ZWJ.
tdf#114160 was a regression introduced during a prior ICU upgrade. This
was fixed by commit 44699b3de3, but no
tests were added for ZWJ at that time.
Change-Id: Ieca919daea11dc161ae35fb6ffe5bd366cf4a6f0
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/170534
Tested-by: Jenkins
Reviewed-by: Jonathan Clark <jonathan@libreoffice.org>
Hebrew text may use the character RIGHT DOUBLE QUOTATION MARK as a
substitute for HEBREW PUNCTUATION GERSHAYIM. This change customizes the
ICU word BreakIterator rules to that end.
Change-Id: I03a48729de103505a2f68f9a1635c0f0cd7d126a
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/170536
Reviewed-by: Jonathan Clark <jonathan@libreoffice.org>
Tested-by: Jenkins
Fix word breaking rules also for editing. Previously
the word was selected with the following narrow no-break
space, e.g. at French words before exclamation and question
marks (where narrow no-break space allows to get correct
typography, if the OpenType/Graphite font doesn't have
this feature).
Add this and the previous fixes for Hungarian, which
handled by extra word-breaking rule files.
Follow-up to commit 6e002da161
"tdf#161737 i18npool: fix fake spelling alarms with NNBSP".
Change-Id: I7230bd356e5f0360172b652e615a61d96131d336
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/169624
Tested-by: Jenkins
Reviewed-by: László Németh <nemeth@numbertext.org>
Fix word break by excluding narrow no-break space at the
end of the words for spell checking.
This was a problem e.g. for French, where (automatically? or
manually) inserted narrow no-break space is used to get correct
typography before exclamation and question marks, also after and
before guillemets, if the OpenType/Graphite font doesn't have this
feature).
Regression from commit 44699b3de3
"tdf#49885 BreakIterator rule upgrades".
Note: this fixes also the problem, when digits separated
by NNBSP thousand separator weren't handled by spell checking,
alarming fake spelling mistakes, when "Check words with numbers"
was enabled in Tools->Options->Languages and Locales->Writing Aids.
(TODO: at the case of thousand separators, remove NBSP by the
linguistic module or by the spell checking dictionaries to allow
to check numbers with thousand separators and with correct suffix.)
Change-Id: I36e10add7e0ba840f207a375ccc8668dbfef9572
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/169618
Tested-by: Jenkins
Reviewed-by: László Németh <nemeth@numbertext.org>
Similar to the straight (typewriter or ASCII) apostrophe, straight
double quotation mark (") matches its typographic variants now,
like other word processors do.
Note: regex search doesn't use this matching, similar to the apostrophe
search.
Follow-up to commit d40f2d02df
"tdf#117643 Writer: fix apostrophe search regression".
Change-Id: If6a3ee00750828583cd0cfc4aa7f7b656ea9bd1e
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/169605
Reviewed-by: László Németh <nemeth@numbertext.org>
Tested-by: Jenkins
Previously, the CJK BreakIterator used custom dictionaries for Chinese
and Japanese. This change removes these custom dictionaries in favor of
the upstream ICU implementation, which uses an externally-maintained
frequency dictionary for these languages.
This change also removes support code for dictionary-based break
iterators, as it is no longer used.
Change-Id: I55c4ce9c842d1751997309fd7446e0a6917915dc
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/166136
Reviewed-by: Caolán McNamara <caolan.mcnamara@collabora.com>
Tested-by: Jenkins
Tested-by: Caolán McNamara <caolan.mcnamara@collabora.com>
Previously, a custom BreakIterator was used for Thai grapheme clusters.
This change deletes the custom BreakIterator, in favor of the ICU
implementation.
Change-Id: Icec94c73a5734c2059786dfbba085f487c488d7c
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/166156
Tested-by: Jenkins
Reviewed-by: Eike Rathke <erack@redhat.com>
This change completes the review of BreakIterator rule customizations,
and adds unit tests for relevant customizations.
Change-Id: I06678fcccfc48d020aac64dd9f58ff36a763af30
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/166017
Tested-by: Jenkins
Reviewed-by: Eike Rathke <erack@redhat.com>
...now that warning about O[U]String vars that could be O[U]StringLiteral is no
longer useful
Change-Id: I389e72038171f28482049b41f6224257dd11f452
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/157992
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
It was mapped to uppercase sigma (U+03A3) while it should be mapped to
uppercase lunate sigma (U+03F9). Fix by letting this slot fallback to
ICU case folding.
Change-Id: I14ffa0151c740779b67af14be8c7af8c51c3a1e0
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/154845
Tested-by: Jenkins
Reviewed-by: خالد حسني <khaled@libreoffice.org>
If we are requested to case map a character not present in our case
mapping data, fallback to ICU case mapping functions.
We should switch completely to ICU at some point, but we need to
evaluate our case mapping data and see if it differs from ICU and if
there is a reason for it.
Does not handle the case of U+03F2 turning into Sigma from tdf#97152.
Change-Id: Icf13ac7aab6d07b2a90fc0ff5ef1c4f50c7a7f8c
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/154803
Tested-by: Jenkins
Reviewed-by: خالد حسني <khaled@libreoffice.org>
We were requiring ICU 4.6 which was released in 2011, and ifdef'ing our
way through newer ICU versions. ICU is a core dependency and it makes no
sense to build LibreOffice with such ancient versions of it.
This change requires ICU 66 (released in 2020), and removes all the
ifdefs for older versions. There are more cleanups to do, but these will
be done separately.
Change-Id: I2e4f7608a08f4d531b0a4c74bbfdf91a451f833f
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/153387
Tested-by: Jenkins
Reviewed-by: خالد حسني <khaled@libreoffice.org>
...as seen with the additions to CppunitTest_i18npool_transliteration made here,
> i18npool/source/transliteration/textToPronounce_zh.cxx:175:17: runtime error: call to function get_zh_zhuyin through pointer to incorrect function type 'unsigned short **(*)()'
> workdir/CustomTarget/i18npool/indexentry/zh_zhuyin.cxx:1512: note: get_zh_zhuyin defined here
> #0 in i18npool::TextToPronounce_zh::TextToPronounce_zh(char const*) at i18npool/source/transliteration/textToPronounce_zh.cxx:175:17
> #1 in i18npool::TextToChuyin_zh_TW::TextToChuyin_zh_TW() at i18npool/source/transliteration/textToPronounce_zh.cxx:149:5
> #2 in TextToChuyin_zh_TW_CreateInstance(com::sun::star::uno::Reference<com::sun::star::lang::XMultiServiceFactory> const&) at i18npool/source/registerservices/registerservices.cxx:236:1
> #3 in cppu::(anonymous namespace)::OFactoryComponentHelper::createInstanceEveryTime(com::sun::star::uno::Reference<com::sun::star::uno::XComponentContext> const&) at cppuhelper/source/factory.cxx:173:24
> #4 in cppu::(anonymous namespace)::OFactoryComponentHelper::createInstanceWithContext(com::sun::star::uno::Reference<com::sun::star::uno::XComponentContext> const&) at cppuhelper/source/factory.cxx:230:12
> #5 in non-virtual thunk to cppu::(anonymous namespace)::OFactoryComponentHelper::createInstanceWithContext(com::sun::star::uno::Reference<com::sun::star::uno::XComponentContext> const&) at cppuhelper/source/factory.cxx
> #6 in cppuhelper::ServiceManager::Data::Implementation::doCreateInstance(com::sun::star::uno::Reference<com::sun::star::uno::XComponentContext> const&) at cppuhelper/source/servicemanager.cxx:709:26
> #7 in cppuhelper::ServiceManager::Data::Implementation::createInstance(com::sun::star::uno::Reference<com::sun::star::uno::XComponentContext> const&, bool) at cppuhelper/source/servicemanager.cxx:675:16
> #8 in cppuhelper::ServiceManager::createInstanceWithContext(rtl::OUString const&, com::sun::star::uno::Reference<com::sun::star::uno::XComponentContext> const&) at cppuhelper/source/servicemanager.cxx:1006:36
> #9 in non-virtual thunk to cppuhelper::ServiceManager::createInstanceWithContext(rtl::OUString const&, com::sun::star::uno::Reference<com::sun::star::uno::XComponentContext> const&) at cppuhelper/source/servicemanager.cxx
> #10 in i18npool::TransliterationImpl::loadBody(rtl::OUString const&, com::sun::star::uno::Reference<com::sun::star::i18n::XExtendedTransliteration>&) at i18npool/source/transliteration/transliterationImpl.cxx:619:45
> #11 in i18npool::TransliterationImpl::loadModuleByName(std::basic_string_view<char16_t, std::char_traits<char16_t>>, com::sun::star::uno::Reference<com::sun::star::i18n::XExtendedTransliteration>&, com::sun::star::lang::Locale const&) at i18npool/source/transliteration/transliterationImpl.cxx:630:5
> #12 in i18npool::TransliterationImpl::loadModuleByImplName(rtl::OUString const&, com::sun::star::lang::Locale const&) at i18npool/source/transliteration/transliterationImpl.cxx:278:9
> #13 in (anonymous namespace)::Transliteration::testTextToChuyin_zh_TW() at i18npool/qa/cppunit/transliteration.cxx:120:27
For one, there had always been a mismatch between the return type `const
sal_uInt16**` generated in i18npool/source/indexentry/genindex_data.cxx since
d319c4611e "INTEGRATION: CWS i18n24 (1.1.2); FILE
ADDED" vs. `sal_uInt16**` used in
i18npool/source/transliteration/textToPronounce_zh.cxx since
f4705bf0a3 "INTEGRATION: CWS i18n24 (1.8.36); FILE
MERGED".
And for another (and more severe, as it caused random writes), there had also
been a mismatch between the parameters `(sal_Int16 &max_index)` newly generated
in i18npool/source/indexentry/genindex_data.cxx since
7696cd3902 "INTEGRATION: CWS i18n31 (1.4.60); FILE
MERGED" (and correctly used in
i18npool/source/indexentry/indexentrysupplier_asian.cxx since
58dcf0ffaf "INTEGRATION: CWS i18n31 (1.9.60); FILE
MERGED") vs. the original `()` used in
i18npool/source/transliteration/textToPronounce_zh.cxx ever since
f4705bf0a3 "INTEGRATION: CWS i18n24 (1.8.36); FILE
MERGED".
For DISABLE_DYNLOADING, the second (missing max_index parameter) issue appears
to have been broken even further with 9db03b879b
"i18npool: fix wrong static function symbols", replacing the wrong
sal_uInt16** get_zh_zhuyin();
sal_uInt16** get_zh_pinyin();
declarations in i18npool/source/transliteration/textToPronounce_zh.cxx with the
even worse declarations
sal_uInt16** get_collator_data_zh_zhuyin();
sal_uInt16** get_collator_data_zh_pinyin();
corresponding to function definitions generated by
i18npool/source/collator/gencoll_rule.cxx (which also happen to have zero
parameters, but non-matching return types, and apparently completely different
collation vs. transliteration semantics).
Change-Id: Id91b17eeb7fcdd0c711d52a624375356dc47fc32
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/150302
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
The typos all originated with d2140a6320
"i18npool: create instances with uno constructors", causing three intended
constructor attributes to rather be plain character data. Which apparently went
unnoticed until recently (see the TODOs resolved here that had been introduced
with 456a146b9e "tdf#151971: Fix used
implementation names of transliteration services"), in part because the Parser
class in cppuhelper/source/servicemanager.cxx silently ignores any unexpected
character data via xmlreader::XmlReader::Text::NONE.)
Change-Id: Ia8fdbc09c67d10530b4d86dbbbde2b6b84038e66
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/150021
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
...after 04af4e4f55 "[API CHANGE] Fix all bad
UNOIDL identifiers across offapi" had changed the spelling (character case) of
some of the css.i18n.TransliterationModules[New] enum values involved here, so
that the TmItem1 macro generated broken TMList::implName values now. (Which in
turn caused TransliterationImpl::loadBody to throw "unsatisfied query for
interface of type com.sun.star.i18n.XExtendedTransliteration!"
css::uno::RuntimeExceptions, which remained uncaught.)
Also add a test verifying that loading all those transliteration services no
longer fails throwing exceptions. Which lead to two open TODOs: For one, the
value of maxCascade in i18npool/inc/transliterationImpl.hxx might come from a
time when there were fewer TransliterationModules[New] enum values and might no
longer be appropriate. This would need some further investigation. But for
another, there are two transliteration services that cannot currently be
instantiated. That looks like a regression that should be fixed in a follow-up
commit.
Change-Id: Icfca3e841360d4b471013e2c96d6868a75a21a1c
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/150018
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
Don’t skip search results that are in the middle of a grapheme cluster
(AKA cell in LO speak).
It is not clear why it was done like this, as these checks are present
all the way back to the first commit of this file:
commit 36eb193f48
Author: Rüdiger Timm <rt@openoffice.org>
Date: Tue Apr 8 15:01:00 2003 +0000
INTEGRATION: CWS calc06 (1.1.2); FILE ADDED
2003/03/26 15:54:42 er 1.1.2.1: #i3393# moved from i18n module, cleaned out tools module usage, and added support for regexp
But ignoring such results and only for so-called “complex” scripts seems
arbitrary, and as the linked issue shows, people want to be able to
search for combining marks. Furthermore, it prevents searching for a
base character followed by a combining mark, unless ignoring diacritics
is enabled.
Change-Id: I530788d928861ddfa18dd7b813d0a13f53c0b77b
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/138410
Tested-by: Jenkins
Reviewed-by: خالد حسني <khaled@aliftype.com>
30.6001 shows month days without Jan and Feb.
According to the below link, it is calcuated as (365-31-28)/10 = 30.6
but because of a floating point bug, it was used as 30.6001 as a
workaround.
"30.6001, 25 year old hack?"
https://www.hpmuseum.org/cgi-sys/cgiwrap/hpmuseum/archv011.cgi?read=31650
The value 30.6 is used as i18nutil::monthDaysWithoutJanFeb here
instead of 30.6001. The new value is ~30.60000038 which is > 30.6, so
the calculations should be correct. In order to make sure, a unit test
is added, and part of the values are checked against the values
calculated by this website:
Julian Day and Civil Date Calculator
https://core2.gsfc.nasa.gov/time/julian.html
Change-Id: I8cc7e046514dc3de652a1c37399e351cb2b614dc
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/125813
Tested-by: Jenkins
Reviewed-by: Eike Rathke <erack@redhat.com>
Unicode 14, 5 new scripts, 12 new Unicode blocks.
In i18npool/qa/cppunit/test_breakiterator.cxx
TestBreakIterator::testLao() had to be disabled/adapted.
Needs to be investigated, see comments there.
As is, Lao script word break has regressions.
Correct UBLOCK_TANGUT_SUPPLEMENT Unicode range endpoint to
0x18D7F, see
https://www.unicode.org/versions/Unicode14.0.0/erratafixed.html
for which ublock_getCode(0x18D8F) now returned UBLOCK_NO_BLOCK and
thus luckily the assert in svx/source/dialog/charmap.cxx hit.
Change-Id: I4bad16ecfab3f44be365b8f884c57f34af68218e
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/125322
Reviewed-by: Eike Rathke <erack@redhat.com>
Tested-by: Jenkins
...from aa2064c5c5 "Improve
loplugin:stringliteralvar", similar to 8b32a3edad
"Fix stack-use-after-scope" (though this case doesn't appear to have caused any
actual issues).
(After manual inspection, there appear to be no further missing `static` at
least in aa2064c5c5 "Improve
loplugin:stringliteralvar".)
Change-Id: I2b3d0d8d2af1d65f0c5bef8a858107020a620974
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/124137
Tested-by: Jenkins
Reviewed-by: Stephan Bergmann <sbergman@redhat.com>
The scenarios are:
1. Calling sequence's begin() and end() in pairs to pass to algorithms
(both calls use getArray(), which does the COW checks)
2. In addition to #1, calling end() again when checking result of find
algorithms, and/or begin() to calculate result's distance
3. Using non-const sequences in range-based for loops, which internally
do #1
4. Assigning sequence to another sequence variable, and then modifying
one of them
In many cases, the sequences could be made const, or treated as const
for the purposes of the algorithms (using std::as_const, std::cbegin,
and std::cend). Where algorithm modifies the sequence, it was changed
to only call getArray() once. For that, css::uno::toNonConstRange was
introduced, which returns a struct (sublclass of std::pair) with two
iterators [begin, end], that are calculated using one call to begin()
and one call to getLength().
To handle #4, css::uno::Sequence::swap was introduced, that swaps the
internal pointer to uno_Sequence. So when a local Sequence variable
should be assigned to another variable, and the latter will be modified
further, it's now possible to use swap instead, so the two sequences
are kept independent.
The modified places were found by temporarily removing non-const end().
Change-Id: I8fe2787f200eecb70744e8b77fbdf7a49653f628
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/123542
Tested-by: Jenkins
Reviewed-by: Mike Kaganski <mike.kaganski@collabora.com>
Khaled Hosny