Bug 147326 - UI: "augmented reality" hovering mode (immediate display of underlying formulas etc.)
Summary: UI: "augmented reality" hovering mode (immediate display of underlying formul...
Status: RESOLVED INSUFFICIENTDATA
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
unspecified
Hardware: All All
: medium enhancement
Assignee: Not Assigned
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-02-09 20:00 UTC by Robert Lacroix
Modified: 2023-11-25 03:15 UTC (History)
2 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Lacroix 2022-02-09 20:00:16 UTC
Some imagination required.

Augmented reality tools overlay images of the real world with information, without modifying or touching the real world. The analogy in the spreadsheet world is ability to see things in your spreadsheet without touching it.

The mechanism of seeing without touching is pointer hovering. I suggest that this is a browsing/authoring activity but I will not dictate how users decide to use this mode, if at all, once they get their hands on it. It's a UI experiment, one which is feasible if it can be hacked together with a modest coding effort. As an experiment, this specification can be changed to suite the implementer's preference. I present it as 4 modifications to be done in order, but modification 3 could be done anytime before modification 4.

Modification 1: Implement hovering mode. For the purpose of UI evaluation this need not be a user-selectable UI option. This change set is sufficient for illustrating basic operation of hovering mode.

Let the selected cell range before hovering mode be called the primary selection, and the single cell under the pointer in hovering mode be called the secondary selection.

Entering hovering mode. A toolbar item or menu item activates hovering mode. Hovering mode is not available while in cell edit mode.
Effects:
- store the current worksheet "savability" state to a temporary variable.
- disable saving the worksheet; hovering mode is not a persistent worksheet state.
- store the current worksheet protection state to a temporary variable.
- turn on worksheet protection without password if not already active, bypassing the protection dialog. This prevents inadvertent changes while in hovering mode.
- store the primary selection to a temporary variable.

Leaving hovering mode. Any mouse click or keypress leaves hovering mode.
Effects: undo the operations of entering hovering mode.
- restore the primary selection from the temporary variable.
- restore worksheet protection state from the temporary variable.
- restore the "savability" state of the worksheet from the temporary variable.

During hovering mode.
The secondary becomes the current cell selection. Only one cell is selected, not a larger range. The cell selection tracks the pointer movement. The formula of whichever cell is under the pointer shows up in the input line. Do not display the formula in the cell itself.

Modification 2: Left-clicking a cell stores the secondary selection into the temporary variable, replacing the stored primary selection, before processing the leaving code of modification 1. The net effect of modification 1 and modification 2 is the same as clicking on another cell while not in cell edit mode and not in hover mode, but letting you see cell formulas as the pointer passes over cells. Leaving by another method results in the effects of modification 1 (no net change). Alternatively, right-click could be processed as a left-click followed immediately by adding the right click event to the head of the event queue. This allows the right click to immediately raise a context menu on the cell under the pointer while leaving hover mode.

Modification 3: Colour formulas in the input line the same as you see in a cell while in F2 cell formula editing mode (whether or not you are in cell editing mode). This change alone is worthy of an independent enhancement request, which this request then depends upon for modification 4.

Modification 4: In hovering mode, display all the ranges comprising the formula in the input line, the same way as you see ranges outlined in colours when you are in F2 cell editing mode. But do not not display the formula in the cell under the pointer. The effect adding modification 4 to hovering mode is a rapid way scan to around a spreadsheet and see the layout of the formulas.

And it would just look awesome as you move the mouse around in hovering mode on a worksheet with complicated formulas, like computer blinkenlights.
Comment 1 Buovjaga 2023-03-03 11:41:38 UTC
Interesting proposal. Might be better as an extension, if possible. Let's ask UX.
Comment 2 Heiko Tietze 2023-03-07 12:33:49 UTC
I suspect the idea has some flaws, but first things first. Let's start with a use case. Something beyond "I'd enjoy blinkenlight" :-). Maybe "I enter a complex formula in cell A1 and want to see how it responds if one of its arguments changes to ensure the correct insertion." - ideally with some basic data. Do you have such an example?
Comment 3 Robert Lacroix 2023-04-17 21:34:20 UTC
The visual cortex is insanely faster at recognizing patterns than the conscious mind. To the user it feels like a video game. Blinkenlights empowers the user to see the spreadsheet more efficiently.

> Let's start with a use case.

A user wants to verify that a pattern of cell references occurs in formulas consistently across a group of cells, and that certain formulas in the group differ in a specific way. This is useful where frequent copying/pasting of cells can introduce mistakes, and when transposing may produce unexpected results.

This is somewhat trivial for small spreadsheets, but is time-consuming and error-prone for larger spreadsheets. For very large sheets, it's better to unconditionally copy and paste a desired repeating formula pattern into the range, because that's faster and more reliable than visual inspection. Best of course would be to use protected cells, but that's not always desirable or feasible. Range formulas can be another solution, but they're not the easiest objects to work with and in my experience most people never use them.

For a simple example, a column contains relative references forming a running calculation like a cumulative sum, and the first row contains a different formula pattern or a constant to initialize the calculation. The user wants to ensure that the relative formula in every rows is the same except the first row.

With status quo, the user must go to each cell to see its formula. This displays a black-and-white cell formula in the input line, which the user must read and interpret. More visual feedback is provided by pressing F2 which adds coloured outlines to cell references and colours the corresponding references of the in-cell formula (but not the input line), at the cost of adding two more keystrokes for each cell examined. This is better but imperfect, however, since some outlines may be hidden by the in-cell editor, particularly when the in-cell formula covers multiple cells, and it adds the risk of inadvertently changing a formula. Simply colouring cell references in the input line the same as the in-cell editor would improve regular F2 edit mode, and that's essentially modification 3.

The new hovering mode allows the user to scan the pointer down the column and watch as the pattern of coloured cell outlines follows the pointer down, remaining the same or momentarily jumping to a different shape or colour layout. This is extremely fast by virtue of relying on the user's inate ability to see disturbances in a pattern, much faster than using the higher-level brain functions of reading and understanding formulas. This basically automates the keystrokes of the manual F2 process but also protects the cells. The user validates the first formula in a group by inspection, then swipes the pointer through the range to see that other formulas use the same pattern.

The fine motor control required of the user to swipe the mouse over the cells of interest can even be eliminated, if LO highlights the block of adjacent cells that share the same relative formula as the cell under the pointer.

I realize as well that this will be fast, the user may want the page to scroll automatically when approaching the visible edge of the cells currently displayed. However, that's not part of this proof-of-concept.
Comment 4 QA Administrators 2023-04-18 03:23:38 UTC Comment hidden (obsolete)
Comment 5 Heiko Tietze 2023-04-18 06:34:01 UTC
In a nutshell: you want to "copy", actually just remember the content of a cell and temporarily "paste" somewhere else to see how the function works. Correct?
Comment 6 Robert Lacroix 2023-04-18 13:57:32 UTC
(In reply to Heiko Tietze from comment #5)
> In a nutshell: you want to "copy", actually just remember the content of a
> cell and temporarily "paste" somewhere else to see how the function works.
> Correct?

No, not at all. My description of the implementation may be clouding your understanding.

This mode is a very fast way for the user to review the pattern of cell references in a lot cells. The action is very fast. A user can swipe across a group of cells in a fraction of a second, following the pointer and seeing the pattern of coloured cell reference outlines moving along with the pointer, so that the user can decide whether a cell reference outline looks out of place.

As the user slides the pointer around in hover mode, the input line displays the formula from the cell under the pointer. The cell itself is not showing its formula in the cell, as this may hide some of the coloured cell reference outlines and it visually clutters the sheet under the pointer.

The input line is not the user's primary focus unless the user decides to linger over a cell and look at the formula in the input line. Instead the user visually follows the pointer and relies on a wider field of vision to see the pattern of cell references as the pointer moves.

If the user chooses to focus at the input line while moving the pointer, the user sees references changing rapidly, but there is little certainty that the user is passing over the desired cells because the user is not following the exact position of the pointer. The user may be monitoring the changing cell address to the left of the input line while moving the pointer and looking at the input line, but in principle a user can focus on only one thing at a time - unless the user is a chameleon.
Comment 7 Cor Nouws 2023-04-26 18:38:11 UTC Comment hidden (obsolete)
Comment 8 Cor Nouws 2023-04-26 18:40:49 UTC
So short: hovering shows cell content in formula bar.
Interesting.
If, then I would say moving the mouse is enough to end the visibility?
Comment 9 Heiko Tietze 2023-04-27 08:56:26 UTC
We discussed the topic in the design meeting.

The use case is still not perfectly clear, but it seems not to be a frequent anyway. Which means to rather solve per extension. The interaction sounds clumsy and would be very slow with the tooltip delay.

But first of all, View > Value Highlighting highlights the content in different colors making the formula easy to spot. And the newly introduced option under Calc > View >
Formula indicator and hint does the same plus a tooltip with the cell content, the formula.

Is this sufficient for you?
Comment 10 QA Administrators 2023-10-25 03:14:21 UTC Comment hidden (obsolete)
Comment 11 QA Administrators 2023-11-25 03:15:32 UTC
Dear Robert Lacroix,

Please read this message in its entirety before proceeding.

Your bug report is being closed as INSUFFICIENTDATA due to inactivity and
a lack of information which is needed in order to accurately
reproduce and confirm the problem. We encourage you to retest
your bug against the latest release. If the issue is still
present in the latest stable release, we need the following
information (please ignore any that you've already provided):

a) Provide details of your system including your operating
   system and the latest version of LibreOffice that you have
   confirmed the bug to be present

b) Provide easy to reproduce steps – the simpler the better

c) Provide any test case(s) which will help us confirm the problem

d) Provide screenshots of the problem if you think it might help

e) Read all comments and provide any requested information

Once all of this is done, please set the bug back to UNCONFIRMED
and we will attempt to reproduce the issue. Please do not:

a) respond via email 

b) update the version field in the bug or any of the other details
   on the top section of our bug tracker

Warm Regards,
QA Team

MassPing-NeedInfo-FollowUp