Bug reproduced with both libreoffice-still and libreoffice-fresh on two different Archlinux computers. STR: In Calc, try to Insert > Char… and click on one of the buttons to select cells range in "Data Series" or "Data Range" in the wizard. You will notice that when you hold your click and select cells, the rectangle of your selection is acting pretty randomly, the limit often not stopping on the cell under your cursor. A second part to the problem is that even when you think you selected the correct cells with what you see on your screen, when releasing your mouse button it can happen that completed range text in the wizard window doesn't match what was selected in the sheet. When trying the same thing on a Xorg session, this bug doesn't occur, reflecting this is only affecting Wayland.
I am seeing the same behaviour with Wayland, but while entering formulae like =sum(<range>) Selecting the cell range gives the random results while moving the mouse and a different final result as described. It works 100% under Xorg. Interestingly simply selecing cells works just fine; it seems only related to expressions etc, where the cell is colour-highlighted The problem has been around for quite some time.
Set version. On 2016-12-18 LibreOffice Still was 5.1.6.
It doesn't seem to reproduce anymore on 5.4.2 Therefore, closing as WFM.