Following tdf#173579 by removing broken dynamic loading of collators,
I focused on Kurdish Latn collation.
Root cause and fix provided by GPT-5.6 Luna
Cause:
when "kmr-Latn-TR", LO uses Language "qlt", Country "TR" (Turkey), Variant "kmr-Latn-TR"
but "kmr-Latn-TR" wasn't used in LOCAL_RULE_LANGS (a variable generated in i18npool/CustomTarget_collator.mk)
and used in i18npool/source/collator/collator_unicode.cxx, method Collator_Unicode::loadCollatorAlgorithm
so "func" and "funclen" weren't initialized with appropriate collator functions.
Fix:
rename file and functions containing "ku" by using "kmr_Latn"
(I suppose the right move since it mimics Eike Rathke's commit 45d4b9b5e3
"renamed ku* to kmr-Latn*, fdo#63460")
Add a specific use case bKmrLatn to load "kmr_Latn" collator functions
to deal with "qlt" language (yes it's a bit kludgy...)
We must remind that "kmr_Latn" can be used for "kmr_Latn_TR" but also with "kmr_Latn_SY"
that's why I avoided using "kmr_Latn_TR"
Also, to avoid the bKmrLatn specific case,
I thought about changing "LangID" tag of i18npool/source/localedata/data/kmr_Latn_TR.xml
from "qlt" into "kmr-Latn" but I'm not sure of the impact.
Change-Id: Ifc608da19b4bd2d05667121e3c6f2455f5d45ca9
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/210977
Tested-by: Jenkins
Reviewed-by: Julien Nabet <serval2412@yahoo.fr>
After de43ed1088
"merge index_data into i18npool
(*) it is a small library
(*) this avoids another dynamic symbol lookup
(*) this unblocks more --enable-mergedlibs optimisation
"
SAL_DLLPUBLIC_EXPORT has been removed so collators weren't available for
dynamic loading use. So there was only 1 kind of sort possible in Japanese
(see description in the bugreport).
To make Japanese (but not only them) collators work again,
I kept on the same way of the quoted commit by removing
DISABLE_DYNLOADING blocks and by adding declarations of functions
in i18npool/inc/collator_unicode.hxx
Change-Id: I36529c12ae67fe7eb0562c6798ef55fdbea02f04
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/211111
Reviewed-by: Julien Nabet <serval2412@yahoo.fr>
Tested-by: Jenkins
In Find and Replace, you can search using regular expressions.
Some regex's don't match to a character, only a position.
OOo restricted the ability to find these zero-length matches.
LO allowed it only for the very end position (for matching '$')
in 4.0.3 with commit ccc349d3ab
Author: Eike Rathke on Sat Mar 9 18:50:33 2013 +0100
regex: don't loop 10000 identical matches to find a single $ anchor
I don't see any justification for what OOo did,
and I don't see any reason why not to accept every match/position.
AFAICS this only affects three modules:
-Calc: no significant downsides noticed (unless already true for '$').
Replace gets stuck. Need to replace/find next/replace/find next...
ReplaceAll, Find, and FindAll work nicely.
Find-Backwards doesn't work - a general regex problem.
-macro editor: gets 'stuck' finding same position every time
(but that also happened already with '$').
There is no 'Find All' (so nothing bad can happen there)
and 'Replace All' works nicely. No practical downside.
-Writer: Always reports bZeroMatch as 'false'
except for special code handling '$'.
So for Writer it has no effect at all
(except for returning more unused bZeroMatch).
So in the end this patch primarily benefits Calc
(although it opens the door for Writer to try to handle bZeroMatch
and it does solve the macro-editor complaint in tdf#132870).
Calc can now find the start of all cells ('^')
or employ look-ahead queries like '(?<=ab).
Change-Id: I2d69ca2032682f17bdff3412a8d9efd113b366e9
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/209080
Reviewed-by: Justin Luth <jluth@mail.com>
Tested-by: Jenkins
Use OUString::iterateCodePoints() to get the next code point. Just
incrementing the index does not work with surrogate pairs.
This is an existing bug that was exposed by properly handling surrogate
pairs while case folding in:
commit 8922952429
Author: Khaled Hosny <khaled@libreoffice.org>
Date: Mon Jul 24 18:45:38 2023 +0300
i18nutil: Fix ICU case folding of surrogate pairs
Change-Id: I51328ae6d17ef935d5be00f7327949a26724b9ab
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/207750
Tested-by: Jenkins
Reviewed-by: Khaled Hosny <khaled@libreoffice.org>
In order to represent the (codepoint of a) Unicode character
in UTF-16, multiple UTF-16 code units may be needed.
So far, the Korean Hangul/Hanja character conversion
was assuming that each Unicode character was only
consisting of a single UTF-16 code unit, and splitting
into single sal_Unicode characters accordingly in
TextConversion_ko::getCharConversions.
This breaks for characters represented by multiple
code units, like "𡽜" (CJK Unified Ideograph [1],
unicode codepoint: 139100/U+21F5C) which is represented
by 2 UTF-16 code units.
As a consequence, an invalid OUString/UTF-16 string would
be passed to HangulHanjaConversionDialog::SetCurrentString
for some of the suggestions.
Fix this by no longer splitting into single sal_Unicode,
but use o3tl::iterateCodePoints to split into actual
code points.
Example: When using "Tools" -> "Language" ->
"Hangul/Hanja Conversion..." for a Korean text of
"한글맞춤법", the string of all characters for potential
conversions in TextConversion_ko::getCharConversions is:
"韓漢寒限閑恨旱汗翰邯罕悍澣閒瀚桿捍閈暵鼾犴僩嫺熯嫻駻鷳扞狠僴覵瞷猂邗骭撊鷴垾憪橌㸁䛞䦘闲鹇娴汉焊闬韩仠佷傼兯哻娨嫨屽晘橺浫澖皔蓒蔊蛝螒豻貋釬銲雗鶾鷼麲㒏㛠㡾㢨㪋㯗㲦㵄䁂䂅䈨䉯䍐䍑䎯䏷䓍䓳䕿䗙䥜䦥䮧𡽜忓"
Previously, the last 3 of them ("䮧𡽜忓") would incorrectly be
split into (stdout output, showing something is wrong):
"䮧", " " (something wrong, but a valid UTF-16 string), "?" (wrong, and invalid UTF-16 string), "忓"
With the fix in place, they remain intact:
"䮧", "𡽜", "忓"
Also adjust HangulHanjaConversionDialog::OnSuggestionModified
accordingly to compare the number of code points the
string consists of, not the number of UTF-16 code units.
Otherwise, the "Replace"/"Always Replace" buttons in the
"Hangul/Hanja Conversion" dialog would remain greyed
out for the above example when selecting the "𡽜" character
(that now correctly gets displayed with the fix in
TextConversion_ko::getCharConversions in place).
Thanks to Jonathan Clark for pointing out that additional
aspect in review!
Add a corresponding test case.
While the fix is in i18npool, the new test is implemented
in editeng, where the HangulHanjaConversion class uses
the i18npool logic via a UNO service.
The test is implemented there because the component context
availabe in i18npool doesn't supply the required service.
The same test implemented there would fail like this:
An uncaught UNO exception
- com.sun.star.uno.DeploymentException: {Message: "component context fails to supply service com.sun.star.linguistic2.ConversionDictionaryList of type com.sun.star.linguistic2.XConversionDictionaryList at /home/michi/development/git/libreoffice/workdir/UnoApiHeadersTarget/offapi/normal/com/sun/star/linguistic2/ConversionDictionaryList.hpp:45", Context: @0x7fff17acecf0}
[1] https://codepoints.net/U+21F5C
Change-Id: I1ab202cc0d3b5a6ff5d220f4b8e90c578f591158
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/207017
Reviewed-by: Jonathan Clark <jonathan@libreoffice.org>
Tested-by: Jenkins
The index for Esperanto was marked as using Latin-1 whereas the special
Esperanto characters are actually in Latin Extended-A so they were being
ignored.
Change-Id: Iadda7f75feefac753e95ae4a8933dcd62899e4f1
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/204864
Reviewed-by: Neil Roberts <nroberts@libreoffice.org>
Tested-by: Jenkins
Replace memcpy() with std::copy() and std::copy_n() in
calendar_gregorian.cxx and cclass_unicode_parser.cxx to improve
type safety by eliminating manual sizeof() calculations.
Change-Id: I112d5b9528d9384b62eb2449283bf0e0d1f20f03
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/204358
Reviewed-by: Julien Nabet <serval2412@yahoo.fr>
Tested-by: Jenkins
Since 5a03d511f4 commit,
ICU line breaking rules are now honored and Asian typography options
for paragraph formats are now ignored.
Respect cases where users apply a list that is different from the
default forbidden character list, or cases where no forbidden character
list is applied.
Change-Id: I7d021c1dcb1e9722eb5f7ae6295b949641dcfe01
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/198442
Tested-by: Jenkins
Reviewed-by: Jonathan Clark <jonathan@libreoffice.org>
since we only use one break-iterator these days,
expose a additional C++ interface to that, as well as the existing UNO interface,
and add a method which allows us to avoid allocating an OUString when calling endOfScript().
Shaves off 20% of the temporary allocations.
Change-Id: I7092affc1820555353e8bf6498e11c9eff42cfad
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/198276
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
we allocate an awful lot of temporary OUString while looking up translation strings.
Add some API to LanguageTag to reduce this by using a common OStringBuffer.
Reduces peak memory usage from 420M to 283M
Change-Id: Ie55267631f2178e3a6590eebf5876c963a4cda69
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/198197
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
If the spelling dictionary contains them with non-ASCII apostrophe.
Previously the words were checked only with apostrophes converted
to their ASCII version, resulting continuous false alarms, despite
the correct orthography in the document and in the spelling
dictionary.
Now the words are checked both with ASCII conversion and with
the original non-ASCII apostrophes (because still there are
UTF-8 dictionaries with ASCII apostrophes, moreover, dictionaries
containing mixed apostrophes in the same dic file).
Example words for the Hungarian dictionary: d’Arc, d’Alembert,
McDonald’s (and their several recognized suffixed forms: d’Arcért,
d’Arckal, d’Arcként, d’Arcnak, d’Arcot etc. etc.)
Add unit tests for 1) break iterator and 2) spell checking with
non-ASCII apostrophes.
Change-Id: I24a570df41fa5aba2e7b67dde0db33377717dc2a
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/196248
Reviewed-by: László Németh <nemeth@numbertext.org>
Tested-by: Jenkins
Restores Hungarian break iterator by recognizing ordinal
numbers with affixes, e.g. "50.-et" etc. (to spell checking
bad forms, like "50.-at").
Regression from commit 44699b3de3
"tdf#49885 BreakIterator rule upgrades".
Follow-up to commit f4fe6df6aa
"tdf#162514 i18npool: Handle abbreviations in dictionary breakiterator"
and commit 167bbe31c0
"tdf#162514 i18npool: apply fix for Hungarian abbreviations, too".
Change-Id: I6b6ea363a509a9d8af8330330f78cc23c210d107
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/196225
Tested-by: Jenkins
Reviewed-by: László Németh <nemeth@numbertext.org>
Replace by operator[], begin() and end(), which don't have overhead
since commit 2484de6728 (Remove non-
const Sequence::begin()/end() in internal code, 2021-10-15)
and commit fb3c04bd19 (Drop non-const
Sequence::operator[] in internal code, 2021-11-05).
Change-Id: I221470b3439bb0dc6e7705b0d99deb3cd2063484
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/195176
Tested-by: Jenkins
Reviewed-by: Mike Kaganski <mike.kaganski@collabora.com>
This has led to
commit 62a9df7473
"find-unneded-includes: avoid replacing includes with forward-declarations"
now the replacement of unicode/regex.h with a versioned namespace is not suggested
Change-Id: Ic30954561739591bd483382ed75a56f5901ac993
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/192428
Tested-by: Jenkins
Reviewed-by: Miklos Vajna <vmiklos@collabora.com>
Upstream ICU includes non-standard rule extensions for Chinese and
Japanese, which add line breaking opportunities between U+201C/U+201D
double quotation marks and leading/trailing CJK ideographs. This change
backports those rule customizations, for correct layout of Simplified
Chinese.
Regression from:
commit 5a03d511f4
Author: Jonathan Clark <jonathan@libreoffice.org>
Date: Mon Apr 7 21:37:26 2025 -0600
tdf#130592 i18npool: Fix incorrect line breaking in mixed CJK+Latin
Change-Id: I3cf6c0cd23e38f9a89c98fdd6c6f665f175b676c
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/190904
Tested-by: Jenkins
Reviewed-by: Jonathan Clark <jonathan@libreoffice.org>
Most of the information has been extracted from
https://www.localeplanet.com/icu/sq-AL/index.html
Jona Azizaj also helped to translate some strings.
Thanks for that.
Finally, many thanks to Eike Rathke for his valuable
clues and reviews in gerrit on how to add a new locale
Change-Id: Ibf762e7983f9e4aaf81ad8965f31d69fae0fd7ee
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/186666
Reviewed-by: Eike Rathke <erack@redhat.com>
Tested-by: Jenkins
Otherwise roundtripped formulas in XLSX containing them
can end up with extra parenthesis, eg:
- Expected: OR(D1=0,D1<>““)
- Actual : OR(D1=0,D1<>““))
Change-Id: I6d296687cc2467ca0cdee329f9b04f6c956d2de0
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/183987
Reviewed-by: Aron Budea <aron.budea@collabora.com>
Tested-by: Jenkins
This change updates the CJK BreakIterator to run ICU line breaking rules
after applying user forbidden rules. This fixes several identified CJK
GUI text rendering issues, most significantly that embedded English runs
were previously broken using CJK break rules, rather than using the
correct English rules.
By implementing this fallback to the ICU rules, the CJK BreakIterator
now also correctly handles non-breaking characters. This change
therefore also fixes tdf#117554.
Change-Id: I3bacc7e2b559729df2c37c010974d180c8eef1f5
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/183825
Tested-by: Jenkins
Reviewed-by: Jonathan Clark <jonathan@libreoffice.org>
Julien Nabet