Add new paragraph justification option "Split compound words" for
custom compound-based "smart" hyphenation. Add compound-based "smart"
hyphenation support for Danish, Dutch, Esperanto, Estonian, German,
Norwegian, Swedish (and potentially, Icelandic, Finnish and
Latvian) languages by fixing Hunspell morphological analysis and the
associated lingucomponent code. (Previously compound-based "smart"
hyphenation was enabled only for Hungarian.)
With the new option it's possible to keep MSO/MS-DOCX interoperability,
also introducing a better hyphenation algorithm for several languages.
Details:
* Add check box to Alignment in paragraph formatting dialog window;
* Store properties in paragraph model:
css::style::ParagraphProperties::ParaIsCompoundBased
* Implement it by checking the new option in the previously
added commit f9f694f5d9f39c5801bc5fec45867252c293bbcb;
* Add ODF import/export (loext:compound-based);
* Add ODF unit tests by the new tdf173389_smart_hyphenation_nl.odt
and tdf173389_smart_hyphenation_hu.odt. Now use the previous
tdf119908_smart_hyphenation_hu for disabled compound-based "smart"
hyphenation, also the new tdf119908_smart_hyphenation_nl.odt.
Note: the Dutch test document results better compound-based
hyphenation in two positions.
* Add Hunspell patch to fix Danish, Dutch, Esperanto, Estonian,
German, Norwegian, Swedish (and future Icelandic, Latvian,
Finnish) morphological analysis for compound word constituents;
* hyphenimp.cxx hyphenate(): fix tokenization for consecutive
pa: elements, like resulted in the fixed Hunspell dictionaries.
Follow-up to commit f9f694f5d9
"tdf#173389 sw: add compound-based “smart” hyphenation for readability".
Change-Id: I4414bd588646ecf6dc86b77642833ac9c7820a98
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/211341
Tested-by: Jenkins
Reviewed-by: László Németh <nemeth@numbertext.org>
As a natural extension of the "smart" hyphenation, following
traditional typography, prefer readability of the hyphenated
compound words and better word spacing in the following way:
* when it's possible to hyphenate the same compound word within
the custom word spacing range, choose compound splitting or
hyphenate just before the suffix or inside the suffix, not
inside a compound constituent.
* limit plain, i.e. non-compound-based smart hyphenation, when
it can results in better word spacing without losing readability:
do not disable hyphenation within the custom word spacing range, if
the hyphenation splits the compound word between its constituents;
* if also shrinking results in a non-compound splitting hyphenation
inside the second or third etc. constituent of a compound word,
always shrink the line to show bigger part of that non-first
constituent (within the custom word spacing range).
For these, extend lingucomponent HYPH_COMPOUND_MIN_LEADING and
HYPH_COMPOUND_MIN_TRAILING API to support disabling hyphenation of
non-compound words, also disabling hyphenation inside the first
constituent (but not inside the suffix or just before the suffix).
This allows to recognize compound words during line breaking.
Details: Hyphenator::hyphenate().
The initial development enables compound-based "smart" hyphenation
only for the Hungarian language, yet. The patch contains minor
fixes of the compound-based Hungarian hyphenation, resulting
better compound-based "smart" hyphenation with recognition of
verb prefixes and leg-/legesleg- superlative prefixes. (Note:
this fixed also COMPOUND_MIN_LEADING/TRAILING settings for the
hyphenation of the compound words with dictionary-based hyphenation
(only hy:, without pa: Hunspell dictionary fields, i.e. without
compound splitting heuristics of Hunspell).
Change-Id: I26f31329cdea20f03dbde7742587089eb5df5ae2
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/210423
Reviewed-by: László Németh <nemeth@numbertext.org>
Tested-by: Jenkins
Add new minimum character limit paragraph property to limit
hyphenation left to the compound constituent boundaries:
com::sun::star::text::ParaHyphenationCompoundMinTrailingChars
– With the previous option for the hyphenation right to the
constituent boundary, this new option allow to limit or disable
hyphenation completely inside the compound constituents, or
hyphenate only at constituent boundaries, with or without
inside the suffix of the word or constituents;
– xmloff: add "no-limit" to loext:hyphenation-compound-push-char-count
and loext:hyphenation-compound-push-char-count;
– "no-limit" values of this and the *LeadingChars option mean
a maximum value. When both *LeadingChars and *TrailingChars
got "no-limit", hyphenation before and inside the suffixes are
forbidden, i.e. it is allowed only at constituent boundaries;
– textflowpage.ui: set max character count to 99, in both options;
– fix ParaHyphenationCompoundMinLeadingChars implementation for
compound words with multiple constituents, when the middle word(s)
are shorter, than the limit (e.g. Hungarian "tűz|kő|pattintás"
wasn't hyphenated as "tűzkő-pattintás", only as "tűz-kőpattintás").
Follow-up to commit 3a332d9f1c
"tdf#158885 cui offapi sw xmloff: fix hyphenation at stem boundary"
and commit c899d3608d
"tdf#158885 sw: don't hyphenate right after a stem boundary".
Change-Id: I0b71562727332217364f9ace85e8ac6ea3abc8b1
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/196345
Reviewed-by: László Németh <nemeth@numbertext.org>
Tested-by: Jenkins
If the spelling dictionary contains them with non-ASCII apostrophe.
Previously the words were checked only with apostrophes converted
to their ASCII version, resulting continuous false alarms, despite
the correct orthography in the document and in the spelling
dictionary.
Now the words are checked both with ASCII conversion and with
the original non-ASCII apostrophes (because still there are
UTF-8 dictionaries with ASCII apostrophes, moreover, dictionaries
containing mixed apostrophes in the same dic file).
Example words for the Hungarian dictionary: d’Arc, d’Alembert,
McDonald’s (and their several recognized suffixed forms: d’Arcért,
d’Arckal, d’Arcként, d’Arcnak, d’Arcot etc. etc.)
Add unit tests for 1) break iterator and 2) spell checking with
non-ASCII apostrophes.
Change-Id: I24a570df41fa5aba2e7b67dde0db33377717dc2a
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/196248
Reviewed-by: László Németh <nemeth@numbertext.org>
Tested-by: Jenkins
Replace by operator[], begin() and end(), which don't have overhead
since commit 2484de6728 (Remove non-
const Sequence::begin()/end() in internal code, 2021-10-15)
and commit fb3c04bd19 (Drop non-const
Sequence::operator[] in internal code, 2021-11-05).
Change-Id: I221470b3439bb0dc6e7705b0d99deb3cd2063484
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/195176
Tested-by: Jenkins
Reviewed-by: Mike Kaganski <mike.kaganski@collabora.com>
and
cid#1660225 Variable copied when it could be moved
cid#1660211 Variable copied when it could be moved
cid#1660185 Variable copied when it could be moved
cid#1660173 Variable copied when it could be moved
cid#1660168 Variable copied when it could be moved
cid#1660166 Variable copied when it could be moved
cid#1660165 Variable copied when it could be moved
cid#1660163 Variable copied when it could be moved
cid#1660152 Variable copied when it could be moved
cid#1660149 Variable copied when it could be moved
cid#1660147 Variable copied when it could be moved
cid#1660145 Variable copied when it could be moved
cid#1660142 Variable copied when it could be moved
cid#1660140 Variable copied when it could be moved
cid#1660137 Variable copied when it could be moved
cid#1660136 Variable copied when it could be moved
cid#1660135 Variable copied when it could be moved
cid#1660128 Variable copied when it could be moved
cid#1660124 Variable copied when it could be moved
cid#1660122 Variable copied when it could be moved
cid#1660117 Variable copied when it could be moved
cid#1660114 Variable copied when it could be moved
cid#1660110 Variable copied when it could be moved
cid#1660109 Variable copied when it could be moved
cid#1660105 Variable copied when it could be moved
cid#1660104 Variable copied when it could be moved
cid#1660102 Variable copied when it could be moved
cid#1660100 Variable copied when it could be moved
cid#1660098 Variable copied when it could be moved
cid#1660080 Variable copied when it could be moved
cid#1660079 Variable copied when it could be moved
cid#1660078 Variable copied when it could be moved
cid#1660075 Variable copied when it could be moved
cid#1660071 Variable copied when it could be moved
cid#1660070 Variable copied when it could be moved
cid#1660064 Variable copied when it could be moved
cid#1660063 Variable copied when it could be moved
cid#1660060 Variable copied when it could be moved
cid#1660058 Variable copied when it could be moved
cid#1660053 Variable copied when it could be moved
cid#1660050 Variable copied when it could be moved
cid#1660049 Variable copied when it could be moved
cid#1660046 Variable copied when it could be moved
cid#1660042 Variable copied when it could be moved
cid#1660028 Variable copied when it could be moved
cid#1660023 Variable copied when it could be moved
cid#1660022 Variable copied when it could be moved
cid#1660021 Variable copied when it could be moved
cid#1660020 Variable copied when it could be moved
cid#1660017 Variable copied when it could be moved
cid#1660013 Variable copied when it could be moved
cid#1660005 Variable copied when it could be moved
cid#1660003 Variable copied when it could be moved
cid#1659999 Variable copied when it could be moved
cid#1659998 Variable copied when it could be moved
cid#1659995 Variable copied when it could be moved
cid#1659993 Variable copied when it could be moved
cid#1659989 Variable copied when it could be moved
cid#1659988 Variable copied when it could be moved
Change-Id: I2a30c8a2ac8785050d49b98635b0b406535b185a
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/189056
Reviewed-by: Caolán McNamara <caolan.mcnamara@collabora.com>
Tested-by: Jenkins
and
cid#1660283 Variable copied when it could be moved
cid#1660278 Variable copied when it could be moved
cid#1660277 Variable copied when it could be moved
cid#1660272 Variable copied when it could be moved
cid#1660271 Variable copied when it could be moved
cid#1660269 Variable copied when it could be moved
cid#1660257 Variable copied when it could be moved
cid#1660251 Variable copied when it could be moved
Change-Id: I6684eaf2162dd3550e2eab3f72cccbaa09448f43
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/188925
Tested-by: Jenkins
Reviewed-by: Caolán McNamara <caolan.mcnamara@collabora.com>
The encoding of the string passed to Hunspell/hyphen service depends on the
encoding of the dictionary itself. When the usual UTF-8 encoding is used,
the resulting octet string may be longer than the original UTF-16 code unit
count. In that case, the length of the buffer receiving the positions will
be longer, respectively. But on return, the buffer will only contain data
in positions corresponding to the characters, not code units (it is unclear
if we even need to pass buffer that large). So just as the following loop
only iterates up to nWord length, the calculation of hyphen count must use
its length, too, not the length of encWord.
I suspect that the use of UTF-16 code units as hyphen positions is wrong;
it will break in SMP surrogate pairs. The proper would be to iterate code
points. However, I don't have data to test, so let it be TODO/LATER.
Change-Id: Ieed5e696e03cb22e3b48fabc14537372bbe74363
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/177077
Reviewed-by: Mike Kaganski <mike.kaganski@collabora.com>
Tested-by: Jenkins
we can just take a "const &".
(found by running clang-tidy with the
performance-unnecessary-copy-initialization warning)
Change-Id: I20fd208c65303da78170b1ac06c638fdf3aa094b
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/176267
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
Tested-by: Jenkins
When there is a grammar check error that it has no suggestions for, Duden
may return a JSON without suggestions, but still informing about the type
of the error, like
...
{
"offset": 468,
"length": 8,
"errorcode": 21,
"type": "orth",
"proposals": []
},
...
Before the change, we underlined the error, but right-clicking it would
only bring the normal Writer's context menu, as if there were no error.
This was made in commit 7697ef9d8f (Proof
reading suggestions:, 2023-11-28); before that change, the context menu
was shown, but since it had no suggestions, and no explanation text, it
was considered better to just not show the grammar check results. But
no menu for an underlined word is no better; so let's instead show the
explanation, based on the type of the error reported by Duden - that is
stored in SingleProofreadingError::aShortComment. Also set aFullComment,
which is used in the grammar check dialog.
Incidentally, it even improves popups with suggestions, because it now
has the explanation as well, where previously only was an icon without
any text.
It may be further improved in a follow-up, because there is "errorcode"
in the message, which we may use to add more details to the explanation.
tdf#160127 should be fixed with this, too; but if there are cases when
there are neither suggestions, nor explanation available, it will need
further work.
Change-Id: I62cde204e0142d6226c9c9486f1cdb5ffc20dca8
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/175017
Reviewed-by: Mike Kaganski <mike.kaganski@collabora.com>
Tested-by: Jenkins
Add new hyphenation option "Compound characters at line end",
equivalent of libhyphen's COMPOUNDLEFTHYPHENMIN, to limit bad
pattern based hyphenation of compound words using morphological
analysis of Hunspell.
* Add checkbox to Text Flow in paragraph formatting dialog window
* Store property in paragraph model:
css::style::ParagraphProperties::ParaHyphenationCompoundMinLeadingChars
* Add ODF import/export (loext:hyphenation-compound-remain-char-count)
* Add ODF unit tests
Note: slower Hunspell based hyphenation is used only if
ParaHyphenationCompoundMinLeadingChars >= 3 (we assume that
libhyphen hyphenation patterns cover the smaller distances
correctly). Hunpell based hyphenation doesn't introduce
new hyphenation breaks, only detects the stem boundaries
from the libhyphen based hyphenation breaks.
Follow-up to commit c899d3608d
"tdf#158885 sw: don't hyphenate right after a stem boundary",
replacing hyphenation zone dependence with the new "Compound
characters at line end".
Note: preset COMPOUNDLEFTHYPHENMIN values aren't loaded yet
from hyphenation dictionaries.
Note: the suffix of the last stem of the compound is always
hyphenated, i.e. the distance limits only hyphenation
inside the stem, not inside its suffix or at the end of the
stem before the suffix.
Change-Id: I46a0288929a66f7453e3ff97fbc5a0c6a01f038f
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/164983
Tested-by: László Németh <nemeth@numbertext.org>
Reviewed-by: László Németh <nemeth@numbertext.org>
deprecated in 2008 (and seemingly never even implemented)
Replace the use of this property in sample code and
in Java unit tests with another property (I arbitrarily
selected "IsIgnoreControlCharacters")
Change-Id: I2cb474f895ac221952b59e98bfdbc01012095519
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/164355
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>
in compound words to get better typography or orthography
with more readable text, if hyphenation zone is enabled.
If there are multiple possible break points in the word
according to the libhyphen based hyphenation, keep only
the best ones using Hunspell morphological data
based on compound word decomposition of non-dictionary
words (pa: fields), and extra morphological data of dictionary
words (hy: fields) or their combination.
For readability and tradition, orthography and typography
prefer or only allow hyphenation between stems in compound
words in several languages, like Danish, Dutch, German,
Hungarian, Norwegian and Sweden.
Hyphenation zone is to avoid of too much or bad hyphenation.
Preferring stem boundaries for hyphenation within the hyphenation
zone is a natural extension of it, i.e. skip hyphenation within
stems, if there is stem boundary within the hyphenation zone.
Now skip break points after skip boundaries, if their
distance is 3 or less characters (COMPOUNDLEFTHYPHENMIN = 4).
Skip also break points on stem boundaries, if there is a
weighted stem boundary before them within 3 characters.
Weighted stem boundaries are there between the pa: fields,
(stems resulted by the compound word decomposition),
or in hy: field, boundaries marked by double || instead of
a single |.
More information: man (5) hunspell, and option -m hunspell.
Note: break points skipped only in the last stems for languages
with fogemorphemes, yet, because of their incomplete Hunspell
output for morphological analysis.
Change-Id: I739908716d11a9c2db0c9d36fba8657ba6f53bee
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/161498
Tested-by: Jenkins
Reviewed-by: László Németh <nemeth@numbertext.org>
The problem is that curl 8.3.0 removed the NSS backend, so we now
have no other choice than to use the bundled OpenSSL on Linux.
Currently any curl https connection fails with:
CurlSession.cxx:963: curl_easy_perform failed: (60) SSL certificate problem: unable to get local issuer certificate
Apparently this requires manually telling curl which CA certificates to
trust; there is a configure flag --with-ca-bundle but that is useless as
it tries to load the file relative to whatever is the current working
directory, and also did i mention that there are at least 3 different
locations where a Linux system may store its system trusted CA
certificates because ALL ABOUT CHOICE.
So add a new header with an init function to try out various file
locations listed in this nice blog article and call it from way too many
places that independently use curl.
https://www.happyassassin.net/posts/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/
TODO: perhaps bundle a cacert.pem as a fallback in case the system chose
to innovate by putting its certificates in yet another unexpected place
(regression from commit c2930ebff8)
Change-Id: Ibf1cc0069bc2ae011ecead9a4c2b455e94b01241
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/158915
Tested-by: Jenkins
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>
Reason for revert: See discussion in https://gerrit.libreoffice.org/c/core/+/154302
- it's causing CppunitTest_sw_layoutwriter2 to fail on Windows.
- it's causing CppunitTest_sw_layoutwriter to fail on Linux.
- Probably other recent CI failures are also related.
- A similar commit was reverted one year ago for similar reasons -> https://gerrit.libreoffice.org/c/core/+/135859
This commit contains the following commits:
Revert "lok: remove old hack for LanguageTool locales"
This reverts commit 9a5329a266.
Revert "lok: LanguageTool provides list of languages"
This reverts commit 21d0489a5e.
Revert "lok: remove duplicated locales for LanguageTool"
This reverts commit a0865169ab.
Revert "Load the locales from config file for languagetool"
This reverts commit 81b0d9a951.
Change-Id: I3dc48097615f510e33e233e868b6b10704d81e67
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/154342
Tested-by: Jenkins
Reviewed-by: Xisco Fauli <xiscofauli@libreoffice.org>
László Németh