ODFstudio/odfstudio
2
0
Fork 0
Code Issues Packages Activity

vcl: PDFEncryptorR6: fix subtle off-by-one in computeHashR6

Browse Source 
The end condition `E.back() > (nRound - 32)` is using the wrong value of
`nRound` and so the loop can end too early.

This has caused a few CI builds to fail after pushing the padding fix,
as the input to `computeHashR6` includes a random IV it happened rarely.

Fix it by counting the iterations of the loop that have been done.

Signed-off-by: Michael Stahl <michael.stahl@collabora.com>
Change-Id: I8bbdc85439789db3753942b1292ba48d39b76541
Reviewed-on: https://gerrit.collaboraoffice.com/c/online/+/8625
Reviewed-by: Tomaž Vajngerl <tomaz.vajngerl@collabora.com>
(cherry picked from commit 082e5a49669e2d9058c2b3675c829a495b2044f3)
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/209277
Tested-by: Jenkins
Reviewed-by: Xisco Fauli <xiscofauli@libreoffice.org>
This commit is contained in:
Michael Stahl
2026-08-19 00:20:39 +02:00
committed by Xisco Fauli
parent aac2961770
commit 01de4c0213
2 changed files with 30 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vcl/qa/cppunit/pdfexport/PDFEncryptionTest.cxx
+26
View File
@@ -150,6 +150,32 @@ CPPUNIT_TEST_FIXTURE(PDFEncryptionTest, testComputeHashForR6)
}
}
CPPUNIT_TEST_FIXTURE(PDFEncryptionTest, testComputeHashForR6LoopEndCondition)
{
sal_uInt8 const pPassword[] = { 'T', 'e', 's', 't' };
// This should be 80 rounds.
std::vector<sal_uInt8> aValidationSalt = parseHex("6A5260B509E42E4F");
CPPUNIT_ASSERT_EQUAL(
std::string("058d0adf871eaedf549963a8a60499af48c647abacf32605bf69518d4366dfe9"),
comphelper::hashToString(vcl::pdf::computeHashR6(pPassword, 4, aValidationSalt)));
// This should be 69 rounds.
aValidationSalt = parseHex("B8B897AF2CB52F00");
CPPUNIT_ASSERT_EQUAL(
std::string("e9f540fad0c2531781e7d9671da52527ab281497cb0d51bf0e986dffa0e9e323"),
comphelper::hashToString(vcl::pdf::computeHashR6(pPassword, 4, aValidationSalt)));
// Test the owner key generation, taking additional U argument.
// This should be 69 rounds.
std::vector<sal_uInt8> U = parseHex("7BD210807A0277FECC52C261C442F02E1AD62C1A23553348B8F8AF7320"
"DC9978FAB7E65E1BF4CA76F4BE5E6D2AA8C7D5");
aValidationSalt = parseHex("35A0F8DF2F495D23");
CPPUNIT_ASSERT_EQUAL(
std::string("a8a6e16751faebf5f8aef5d02eae8ddd8885fa3fe9d29e132ff9abe86bd4d0c8"),
comphelper::hashToString(vcl::pdf::computeHashR6(pPassword, 4, aValidationSalt, U)));
}
CPPUNIT_TEST_FIXTURE(PDFEncryptionTest, testGenerateUandUE)
{
// Checks we calculate U and UE correctly
vcl/source/pdf/PDFEncryptorR6.cxx
+4 -2
View File
@@ -192,7 +192,9 @@ std::vector<sal_uInt8> computeHashR6(const sal_uInt8* pPassword, size_t nPasswor
std::vector<sal_uInt8> E;
sal_Int32 nRound = 1; // round 0 is done already
// Counts the repetitions of steps a) to d) done so far.
// K is the input to the first repetition and is not counted.
sal_Int32 nRound = 0;
do
{
// Step a)
@@ -237,7 +239,7 @@ std::vector<sal_uInt8> computeHashR6(const sal_uInt8* pPassword, size_t nPasswor
}
// Step e) and f)
// We stop iteration if we do at least 64 rounds and (the last element of E <= round number - 32)
while (nRound <= 64 || E.back() > (nRound - 32));
while (nRound < 64 || E.back() > (nRound - 32));
// Output - first 32 bytes
return std::vector<sal_uInt8>(K.begin(), K.begin() + 32);