Created attachment 133204 [details]
Improper rendering of inline formulas

Created attachment 133205 [details]
LibreOffice file with rendering errrors

Created attachment 133206 [details]
LibreOffice settings

Anti-aliasing and OpenGL formatting turned on but this did not have any effect.

Most of the formulas do not use the frame style "Formula" but the frame style "OLE". The frame style "OLE" includes an "anchor to paragraph", frame style "Formula" has an "anchor as char". To get an "anchor as char" the formulas have got a direct formating via object properties. But not only the anchor was set, but the padding (=spacing to content) was set to a non-zero value in tab border.
In 5.2 that did not matter, because the objects have no border. But in the meantime LibreOffice honors a padding without border, as it should be. The padding does not increase the box of the formula (similar to the green handles), but the padding reduces the place for the content. So you see a tiny content.

You have to repair the document. Set all formulas to style "Formula" and set the "spacing to content" to zero for each formula.

Your document seems to be very large, so you need a way to do it automatically. You can write a macro for this task or you can edit the styles manually in the file source. If you need assistance, please ask on mailing list or forum.

Another question is, why the formulas have got the wrong style. Do you remember, how these formulas were generated?

Regina,

Thank you for diagnosing this. I'm glad that not all of the document is like this. The appendix material is a remnant from an MS Word document I converted several years ago to OpenOffice/LibreOffice. The format change corrects the issues. Setting the "spacing to content" to zero also makes it look better. However, when I add space in the text before and after the formula, I sometimes see a squiggly line appear (see attachment). What does this indicate?

In the whole document, I have 0.1mm front and back space to the formulas added. Manual removal will be too much of an effort. I will ask on the mailing list but any suggestions from you would be useful.

Created attachment 133225 [details]
squiggly line surrounding formula in text

The "squiggly line" belongs to the grammar check. The grammar check ignores formulas and therefore sees two spaces, which would be not correct.

We do not want support topics in Bugzilla, therefore please use mailing list or forum. Nevertheless, the document ContainsFormulaTools.odt in https://ask.libreoffice.org/en/question/80285/easy-way-to-change-fonts-of-all-mathematical-formulas-ole/ might help you to see, how macros on formulas basically work. 

"from an MS Word document I converted several years ago". That is likely the reason for the problem. The import filter from MS Word have been improved notable in the last years, and in a current version such problem should no longer occur. In case you get problems in converting from MS Word with a current version of LibreOffice, please file a new bug report and attach the source document.

I will close this issue as "Notabug", because the problems are errors in the document itself and the situation, which produces the error in the document, can no longer be reproduced.

Regina,

One last question about this issue. I'd had help form someone who programmatically removed the spacing around the formulas. When I open the file in Writer, the formula is still rendered with the 0.1mm Left and Right spacing. After I open the formula and close it, the 0mm spacing is rendered correctly. Is there a global option to correctly render all formulas together in the entire document instead of having to open-close them individually?

If you will handle really _all_ formulas in the document, then try (on a copy of your document!) the following macro. The line oXCOEO.update() refreshes the view of the formula in the Writer document. 

sub SetBorderMarginToZero
dim oCurrentController as variant: oCurrentController = ThisComponent.getCurrentController()
dim oDoc as variant: oDoc=ThisComponent
if not(oCurrentController.supportsService("com.sun.star.text.TextDocumentView")) then
	msgbox("Macro works only in text documents.")
	exit sub
end if
dim oModelTextDocument as variant: oModelTextDocument = oCurrentController.Model
dim oEmbeddedObjects as variant: oEmbeddedObjects = oModelTextDocument.EmbeddedObjects
dim nIndex as long
dim nEndIndex as long: nEndIndex = oEmbeddedObjects.Count-1
dim oEmbeddedObject as variant: rem like green handle status
dim oModel as variant: rem like edit status

dim oXCOEO as variant: rem oExtendedControlOverEmbeddedObject
for nIndex=0 to nEndIndex
	oEmbeddedObject = oEmbeddedObjects.getByIndex(nIndex)
	oModel = oEmbeddedObject.Model: rem might be empty; css.comp.math.FormulaDocument
	if Not(isEmpty(oModel)) then
		if oModel.supportsService("com.sun.star.formula.FormulaProperties") then
			rem It is a formula object.
			oModel.LeftMargin = 0
			oModel.RightMargin = 0
			oModel.TopMargin = 0
			oModel.BottomMargin = 0
			oXCOEO = oEmbeddedObject.ExtendedControlOverEmbeddedObject
			oXCOEO.update()
		end if	
	end if
next nIndex
end sub