Bug 114721 - Special char: find the char by drawing it
Summary: Special char: find the char by drawing it
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: LibreOffice (show other bugs)
(earliest affected)
Hardware: All All
: medium enhancement
Assignee: Not Assigned
Depends on:
Blocks: Special-Character
  Show dependency treegraph
Reported: 2017-12-27 20:39 UTC by LE GARREC Vincent
Modified: 2018-11-16 11:03 UTC (History)
10 users (show)

See Also:
Crash report or crash signature:

ocr_symbol.tar.xz (8.57 KB, application/x-xz)
2017-12-27 20:39 UTC, LE GARREC Vincent

Note You need to log in before you can comment on or make changes to this bug.
Description LE GARREC Vincent 2017-12-27 20:39:22 UTC
Created attachment 138698 [details]

It would be really nice to be able to find char by drawing it.

I started working on it. Please find enclosed few patches. If you rather github : https://github.com/bansan85/core/tree/ocr. I didn't used gerrit because patches are actually really unstable.

What does patches do and how does it work ?

1) Add a check button to show a drawing area that allow drawing.
2) Draw the char. Left click to add black, right click to rub out.
3) Uncheck the draw check button and let the "almost" magic do:
4) Ĝet list of all "special character" and convert it in 15*15px bitmap.
5) Use lib fann to create and learn the neuronal network. Be careful, this step is taking about 30-45 minutes. After the network created, the result is saved in /tmp/fann.net so it can be reused after. The idea is to make a (online?) database with generated network for each font to avoid this step.
6) Search the drawn char in the network and sort result by probability for each char (output of the network).
7) Show sorted char in the window.

Theses steps is what it *should* do. But I'm having lots of trouble :

After my modifications, opening the special Character often crashes. I almost done nothing during this step, so I don't know how to solve this problem. Maybe I did something wrong when I added the check box or the drawing area.

Is it a problem to add a dependency with fann ? If it is, I can create my own library, I already done it 10 years ago. By additional heavy work, the library should use blas / opencl.

Part 6 looks to be buggy. I can't open the network generated using fann under Libreoffice but I can open it by a simple "hello world fann". Maybe the instability of the opening the window is the reason ???

Where should I put the code about fann and ocr ? In another class ? In the cui lib ? In another lib ?

So actually, I need the help of someone that already done some development for LibreOffice. I worked on the base but I'm stuck for writing the final code with enough QA.

Any help will be appreciated.

Best regards,
Comment 1 V Stuart Foote 2017-12-28 19:12:21 UTC
Hi Vincent, thank you for posting--you've obviously put a lot of work into it. Would think arranging a branch on LO project Gerret is the way to go--you can fully test with assistance of other project devs, and then migrate when stable.

Not my area of expertise, but I would think using an Artificial Neural Network (NN) based recognition of a search glyph trace input against a training library of prepared NN for existing on system fonts might work for doing our handwritten "drawing" of a glyph. Is there a better data model to hold the NN training of the fonts and match the user input against? Tesseract-ocr or Caffe... [1][2]

Otherwise assume the FANN project's libfann 2.2.0 + [3][4] is license compatible and can be compiled cross platform Windows, macOS, Linux, Android.

I notice you've set GUI to include a 80x150 px character input pad--but that might be made a bit wider--and would square 150x150px facilitate resampling of the node? 

And since then key in implementing the NN search is the FANN "resampling" of the input to an NN node for search. Will working with 15x15px nodes and NN trainings of the fonts provide sufficient resolution to differentiate glyphs from more complex Unicode blocks/scripts (think of CJK "traditional" glyphs).

I'd think using an NN based on 32x32, or possibly 64x64 px matrix might be required.  

Of course the more complex NN would require a larger cache to match against--and building dynamically is out of the question. Also, some of the fonts that would benefit from handwritten "drawing" search are going to hold tens-of-thousands of glyphs, would a FANN based search scale that large?

Another issue would be UI--selecting and training the NN of each target font in cache on system would be required.  The project could not host the NN for the fonts (bandwidth and storage).  So, project could deploy a few of the FANN based NN trainings with installation packaging, but LibreOffice GUI would have to guide the user's selection of additional fonts to parse and hold locally--and include some estimate of the size of the NN cache.

Finally, returns of NN search must remain keyed to Unicode point of the source font as that remains significant--not clear the FANN would keeps the Unicode details of the matching glyphs.

[1] https://github.com/tesseract-ocr/
[2] https://github.com/BVLC/caffe

[3] https://github.com/libfann/fann
[4] http://leenissen.dk/fann/html/files2/advancedusage-txt.html
Comment 2 LE GARREC Vincent 2017-12-29 08:55:36 UTC
Thanks for your advice. Actually, my goal is to make a proof of concept (PoC) to see 1) if this function is useful and 2) under what constraints.

I finally found what was my problem. A .cxx was not automatically recompiled after modification of a header. make clean and make do the trick.

I will try to create a branch on gerrit once all will work. It should take a week.

I decided to use NN because I know how to use it to do recognize of character. The PoC and feedback will tell if it's a good idea.

I decided to use 15*15px to reduce size of the NN. Actually, the NN generated by fann is a 300Mo plain text file. Increasing the size must be testing. And improving the size of internal layers of the NN is a need but it's out of my skill.

Actually, the number of characters is about 2200. But I found there's a way to use all of them. It will increase by a lot the number of characters.

Theoretically, a NN can handle any number of character. The NN need X input (number of pixels of the image to recognize) and Y output (number different character). But this will increase, again..., a lot the cache.

I will continue to work on it now I'm not stuck anymore and will keep you in touch when the PoC will be finished.
Comment 3 LE GARREC Vincent 2017-12-31 03:09:05 UTC
I just pushed to gerrit (https://gerrit.libreoffice.org/#/c/47204/1 to https://gerrit.libreoffice.org/#/c/47213/1). Let's see what will append. I'm not very comfortable with submitting patches that big to a community so wait and see reaction.

Here a demo : https://www.youtube.com/watch?v=PgDCU_NN1ho
Comment 4 Buovjaga 2018-01-26 16:53:04 UTC
Let's put status to NEW as this is accepted to its own branch etc.: https://cgit.freedesktop.org/libreoffice/core/log/?h=private/bansan/chardraw
Comment 5 Heiko Tietze 2018-11-12 09:08:27 UTC
What a pity that you don't continue, apparently. The drawing capability was also part of the original idea at https://design.blog.documentfoundation.org/2015/03/26/libreoffice-design-session-special-character/
Comment 6 LE GARREC Vincent 2018-11-16 10:51:16 UTC
I understand. I'm also sad. But doing this job took me 2 weeks of vacation.

I stopped because I had trouble to remove an unwanted dependency (external library).

It also took me lots of time to understand how to change the UI. Just few months after, the "weld" migration broke lots of my work (and my little motivation).

Maybe it could be a good GSOC (GSoC 2019 has been announced few days ago) and I could be a mentor. But for now, I don't have enough free time for coding.