Download it now!
Bug 121469 - FIREBIRD: Migration for table names with special characters fails
Summary: FIREBIRD: Migration for table names with special characters fails
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Base (show other bugs)
Version:
(earliest affected)
6.2.0.0.alpha1+
Hardware: All All
: high major
Assignee: Not Assigned
URL:
Whiteboard: target:6.3.0 target:6.2.1
Keywords:
Depends on:
Blocks: Database-Firebird-Migration
  Show dependency treegraph
 
Reported: 2018-11-16 18:19 UTC by Robert Großkopf
Modified: 2019-02-05 09:37 UTC (History)
9 users (show)

See Also:
Crash report or crash signature:


Attachments
example HSQL base file (4.40 KB, application/vnd.oasis.opendocument.database)
2018-11-16 19:42 UTC, Drew Jensen
Details
Pictures are worth lots of words (1.87 MB, application/vnd.oasis.opendocument.presentation)
2018-11-18 02:13 UTC, Drew Jensen
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Großkopf 2018-11-16 18:19:36 UTC
Have just tried to migrate a database with
Version: 6.2.0.0.alpha1
Build ID: ff46ad24d1d3cbcea45895520483ed1fd4ff488b
CPU threads: 6; OS: Linux 4.12; UI render: default; VCL: kde5; 
Locale: de-DE (de_DE.UTF-8); Calc: threaded

There is a table "Straße" in the HSQLDB.
Migration made "Stra\u00dfe" in Firebird.

So table names aren't set right if there are special characters.

There are bugs reported for fields in the table, which seems to be fixed (field in table "Stra\u00dfe" is named "Straße" - works), but the setting of table names isn't saved.
Comment 1 Drew Jensen 2018-11-16 19:42:20 UTC
Created attachment 146698 [details]
example HSQL base file

Confirmed with Ubuntu 18.04 
LibreOffice Version: 6.2.0.0.alpha1+
Build ID: 85dba18a3d3644a4ac49bd32ea106a4d69159fb4
CPU threads: 4; OS: Linux 4.15; UI render: default; VCL: gtk3; 
TinderBox: Linux-rpm_deb-x86_64@86-TDF, Branch:master, Time: 2018-11-13_23:14:30


Attached test file (hsql) with both table name and a column as "Straße"
Comment 2 Drew Jensen 2018-11-16 19:44:18 UTC
Added issue 121069 to see also. There problem, during migration, when special characters in column names and the column is used in a foreign key.
Comment 4 Drew Jensen 2018-11-17 13:18:28 UTC
(In reply to m.a.riosv from comment #3)
> Maybe because non 7-bit ASCII are not allowed as regular identifiers, they
> must be delimited.
> http://www.firebirdtest.com/file/documentation/reference_manuals/fblangref25-
> en/html/fblangref25-structure-identifiers.html
> 
> It has no change on FB 3 see page 22
> https://www.google.com/
> url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=2ahUKEwjY7JXXt9veAhUqNOwKHXSnDWc
> QFjAAegQIARAC&url=https%3A%2F%2Ffirebirdsql.
> org%2Ffile%2Fdocumentation%2Frelease_notes%2FFirebird-3.0.0-ReleaseNotes.
> pdf&usg=AOvVaw2PXMthc9Ii-VAJjhxHKy_j

No, if you creat that table or the relationship using Base w/emb firebird database there is no problem with these characters. It is the migration assistant code having the problem.
Comment 5 m.a.riosv 2018-11-17 17:35:14 UTC
If I'm not wrong LibreOffice base always use delimiters.
Doing a test with direct SQL with a sentence like:
CREATE TABLE TESTTABLE (CUST_NO INTEGER NOT NULL, CUSTOMER VARCHAR(25) NOT NULL, PRIMARY KEY (CUST_NO) )
works, but
CREATE TABLE Straße (CUST_NO INTEGER NOT NULL, CUSTOMER VARCHAR(25) NOT NULL, PRIMARY KEY (CUST_NO) )
givers an error:
1: firebird_sdbc error:*Dynamic SQL Error *SQL error code = -104 *Token unknown - line 1, column 18.
Just in the position of ß.
With CREATE TABLE "Straße" seems work.

Perhaps the issue it's the migration process not using delimiters.
Comment 6 Robert Großkopf 2018-11-17 18:12:50 UTC
(In reply to m.a.riosv from comment #5)
> CREATE TABLE Straße (CUST_NO INTEGER NOT NULL, CUSTOMER VARCHAR(25) NOT
> NULL, PRIMARY KEY (CUST_NO) )

This is wrong code for creating a table with lower characters. "ß" is a special lower character. With 
CREATE TABLE Strasse
the table will be named STRASSE.
With the right delimiters 
CREATE TABLE "Straße" 
it will work well as observed by you.

There will be used delimiters during the migration-process, because the tables where shown with lower and upper case characters. So this isn't the problem of the migration-process.
Comment 7 Drew Jensen 2018-11-18 02:13:27 UTC
Created attachment 146716 [details]
Pictures are worth lots of words

Here is what I find using a recent build of LO 6.2Alpha1 on Linux.

I found no problem using unicode characters with Base GUI controls with ODB files using HSQL or Firebird, this included Table Designer, Query Designer, Form Designer and Report Builder.

Created the table Straße with a column Straße in HSQL and used Drag-n-Drop to create a copy of it in an empty Firebird ODB, with no problem. 

Created the table Straße with a column Straße in Firebird and used Drag-n-Drop to create a copy of it in an empty HSQL ODB, with no problem. 

For the rest of the checks, I put them in pictures - an Impress file attached.
Comment 8 Julien Nabet 2019-01-30 08:55:35 UTC
*** Bug 123020 has been marked as a duplicate of this bug. ***
Comment 9 Stephan Bergmann 2019-01-30 14:32:54 UTC
See bug 117115 comment 14: "I assume the issue with table names has not yet been addressed (cf. bug 121469)?"
Comment 10 Julien Nabet 2019-01-31 09:25:42 UTC
Tamas:
I created a local patch consisting to:
- copy lcl_ConvertToUTF8 and lcl_IndexOfUnicode from createparser.cxx to parseschema.cxx
- in parserschema.cxx, SchemaParser::parseSchema method replace:
OUString sSql = xTextInput->readLine();
by 
OUString sSql = lcl_ConvertToUTF8(OUStringToOString(xTextInput->readLine(), RTL_TEXTENCODING_UTF8));

Of course, it'd better to centralize ConvertToUTF8 mechanism because rColumnName in parseColumnPart method must need it, but what do you think about the principle?
Comment 11 Commit Notification 2019-01-31 13:42:15 UTC
Tamas Bunth committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/2ec13411b7e7b18d1ea28faee9ad93e0b7b21dde%5E%21

tdf#121069, tdf#121469 migrate special characters

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 12 Commit Notification 2019-02-05 09:37:47 UTC
Tamas Bunth committed a patch related to this issue.
It has been pushed to "libreoffice-6-2":

https://git.libreoffice.org/core/+/16710fab5248b21c1c2cb8b6a4eac3a7a2d64792%5E%21

tdf#121069, tdf#121469 migrate special characters

It will be available in 6.2.1.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.