Bug 39749 - calc unit test writing
Summary: calc unit test writing
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
(earliest affected)
Hardware: Other All
: medium normal
Assignee: Not Assigned
Keywords: difficultyMedium, easyHack, skillCpp, skillScript
Depends on:
Blocks: Dev-related
  Show dependency treegraph
Reported: 2011-08-02 02:03 UTC by Björn Michaelsen
Modified: 2021-08-09 05:23 UTC (History)
12 users (show)

See Also:
Crash report or crash signature:
Regression By:

example of test document to be used with testcode (43.98 KB, application/vnd.oasis.opendocument.spreadsheet)
2015-01-29 16:54 UTC, Winfried Donkers
proposal for function testing (3.24 KB, text/plain)
2015-01-29 17:01 UTC, Winfried Donkers

Note You need to log in before you can comment on or make changes to this bug.
Description Björn Michaelsen 2011-08-02 02:03:18 UTC
==== calc unit test writing ====

'''Background:''' We have a new prototype unit test framework in LibreOffice. The first set of tests is in sc/qa/unit/ and needs your love and attention. We should add new unit tests for each significant C++ class, verifying functionality. Some ideas would be: testing formula parsing (with awkward corner cases etc.), creating large spreadsheets eg. chains of single-cell references, or chains of small range references, and ensuring re-computation dependencies work. It would also be good to have some data-pilot regression tests, and so on.

'''Skills:''' simple C++ programming
Comment 1 Markus Mohrhard 2011-11-01 19:53:24 UTC
adding some more information to reflect our new test desgin in calc:


We have now additionally the possibility to load documents and test the sheet content through a csv file or some code. Several test documents are already created but still need a lot of love. They can be found at: http://cgit.freedesktop.org/libreoffice/core/tree/sc/qa/unit/data/

Skills for this task: knowledge about calc, maybe reading OpenFormula Spec and OpenOffice/LibreOffice documentation/help
Comment 2 Florian Reisinger 2012-05-18 09:10:21 UTC
Deteted "Easyhack" from summary
Comment 3 Björn Michaelsen 2013-10-04 18:47:22 UTC
adding LibreOffice developer list as CC to unresolved EasyHacks for better visibility.

see e.g. http://nabble.documentfoundation.org/minutes-of-ESC-call-td4076214.html for details
Comment 4 Ravi 2013-12-24 01:30:18 UTC Comment hidden (obsolete)
Comment 5 Markus Mohrhard 2013-12-24 02:46:58 UTC
(In reply to comment #4)
> Hi 
> I am a student at clemson university, and I would really really like to
> contribute on this one.This is my first interaction with any open source
> project. I have worked on personal C++ projects, and have good grip on
> language.  The following link mentioned in first comment -
> http://comments.gmane.org/gmane.comp.documentfoundation.libreoffice.qa/42 
> says "No such message". 
> So far:
> 1. I have downloaded and build libreoffice.
> 2. I have looked into "sc/qa/unit/" folders and tried to understand already 
> built unit tests.  
> 3. I understand that the "helper" folder contains the handler files which
> loads csv test data (from "data" folder) and run thesetests.

Great. These are already the most important steps.

> 4. However, I am still not sure, how exactly these unit tests are run, where
> is the main function,how to load the data?. A little help or a link to
> document where this is explained would be of awesome help. My question is
> "How exactly do we run these unit tests?".

They are controlled by the makefiles in sc/. sc/Module_sc.mk contains which tests are actually included and the corresponding makefiles in sc/ how to build them.

The main function is in sal/ inside the cppunittester applciation but we don't touch that part of the code when we write tests. The cppunittester is included into the test through the CPPUNIT_PLUGIN_IMPLEMENT macro and the remaining cppunit macros register the tests in cppunit. More information abotu cppunit can be found at http://people.freedesktop.org/~mmohrhard/cppunit/index.html

So most of the file loading is already abstracted and hidden behind some layers in our testing framework. You need to follow the different class hierarchy that goes into test/ and unotest/. Depending on the test we either use direct calls to the Libreoffice code or use the public UNO API.

Taking sc/qa/unit/filters-test.cxx as an example and picking testRangeNameODS which tests the import of range names from ODS. Test method is declared in line 63, added to the test suite in line 84, and definition starts in line 224. The first call in that method loads the file and returns the corresponding ScDocShell (an internal representation of a calc document).

Sadly there is not much documentation except for https://wiki.documentfoundation.org/Development/Calc_Import_Unit_Tests which I wrote when i first implemented the csv based document comparison. Most of this page should still be correct.

> I really intent to contribute and learn. I am a quick learner, and I really
> find it interesting. I would really appreciate some pointers. Like where
> exactly the class files located, for which we are building test cases?, some
> example test cases for any other module we have.

So these are not real unit tests and we don't test single classes. You can call it regression testing or integration testing. Basically we normally try to test the correct interaction of larger pieces of the code. Testing single classes is nearly never possible in our code base. There are some exceptions for lower levels where we have real unit tests (also the sc_rangelst_test does something similar). So normally we test parts of sc/source/core or sc/source/filter or both together.
Comment 6 Winfried Donkers 2015-01-29 16:54:22 UTC
Created attachment 112922 [details]
example of test document to be used with testcode

Test document is not finished yet.
Comment 7 Winfried Donkers 2015-01-29 17:01:30 UTC
Created attachment 112923 [details]
proposal for function testing

For some reason or other I can't submit to gerrit right now (error: unpack failed: error Missing unknown a9da257bf9f57fdbfd0f400b30080abc4d582ec1), so I use this old method.

Aim of the test code is to minimise changes to the C++ testing code and put the emphasis on the test documents (ods, xlsx, etc.).
Adding a test document only requires one line in the test code.

The attached test code handles the function-tests itself and import tests. For export tests I think of opening and ods document, save as e.g. xlsx, open xlsx and use the attached test code.

Raal worked on the test document (which does not yet include all argument types), I based the test code on Kohei's test code for new Excel 2010 functions.
Comment 8 Robinson Tryon (qubit) 2015-12-14 06:50:47 UTC Comment hidden (obsolete)
Comment 9 jani 2016-02-17 07:28:00 UTC Comment hidden (obsolete)
Comment 10 Ekansh Jha 2017-08-04 06:02:35 UTC Comment hidden (obsolete)
Comment 11 Ekansh Jha 2017-08-04 06:02:58 UTC Comment hidden (obsolete)
Comment 12 Dennis Francis 2017-08-07 07:07:14 UTC
(In reply to Ekansh Jha from comment #11)
> I am new to LO. Just want to ask if this issue is still open, and how to get
> started to solve this bug.

Hey Ekansh,

The aim of this ticket is to create tests for all of the Calc spreadsheet functions. As of now, the tests for each Calc spreadsheet function is put in a fods file which is created with Calc itself(no C++ coding involved).
For examples of how to create these files, see the fods documents in sc/qa/unit/data/functions/<category>/ which asserts the results of spreadsheet function for different test inputs. For example, sc/qa/unit/data/functions/statistical/var.fods tests the spreadsheet function VAR().

Each of these fods document consists of at least two sheets. 

1. Sheet2 contains a column of function evaluation at test points, a column of corresponding expected values, another column(with boolean values) that checks if computed and expected values are close enough.

2. Sheet1 contains a summary of results in Sheet2, with a single boolean value in cell "B3" that shows whether _any_ of the tests in Sheet2 failed or not. The unit testing framework for checking the fods files will check the value of Sheet1.B3 and complain if not TRUE.

If you have any questions, you can either ask here or in the irc channel #libreoffice-dev. 

Thanks !
Comment 13 Shinnok 2017-08-25 13:32:12 UTC Comment hidden (obsolete)
Comment 14 Ajay Mahato 2019-01-02 17:02:50 UTC Comment hidden (obsolete)
Comment 15 Winfried Donkers 2019-01-03 09:02:08 UTC
(In reply to Ajay Mahato from comment #14)
> I would like to work on this bug. For starters which unit test should I
> start writing?

Please wait till Markus Mohrhard responds, he is the Calc unit test expert and this bug report is somewhat out of date.
You can also put your question(s) to the developers mailing list or IRC.
Comment 16 Xisco Faulí 2019-02-09 03:42:56 UTC Comment hidden (obsolete)
Comment 17 Ajay Mahato 2019-02-09 14:50:11 UTC
I am working on this issue but multiple people can work on this bug.
Comment 18 Xisco Faulí 2019-03-12 03:36:05 UTC Comment hidden (obsolete)
Comment 19 Xisco Faulí 2019-06-11 08:16:23 UTC Comment hidden (obsolete)
Comment 20 guillaume.monzerol 2019-09-25 01:11:55 UTC
This is the first LibreOffice bug I am working on.
I will start looking into the source code for new possible unit tests. If a more experienced contributor could point me to relevant parts of the code to test and/or give me unit tests idea that would be appreciated!
Comment 21 Bashar Jaan Khan 2020-02-26 19:14:05 UTC
I am planning to work on this issue. I am not sure I understand the awkward corner case part. I also don't understand data-pilot regressions tests.
Any help is appreciated!
Comment 22 Xisco Faulí 2020-08-03 15:37:19 UTC
You can find a list of missing calc unittests in https://wiki.documentfoundation.org/MissingUnitTests#import