The hint "The function is always case insensitive, independent..." at the beginning of https://help.libreoffice.org/master/en-US/text/scalc/01/func_unique.html is not appropriate for the UNIQUE function because it does not have a criterion as a parameter. From my tests I would say that the function is case-sensitive. "g" and "G" are considered to be different, for example.
And there is no way to follow the instruction about the mode modifier of regular expressions.
In regard to case-sensitive or case-insensitive see bug 162601.