Created attachment 152444 [details]
reproducer

The attached test.docx is a heavily reduced testcase distilled from the crashtestdocuments file cloudon/File_726.docx, where the latter appears to be a genuine document from the wild, not a maliciously crafted one.

Loading test.docx at least in recent master LO built with UBSan causes

> sw/source/core/layout/frmtool.cxx:3543:129: runtime error: signed integer overflow: 2305843009213689813 * 2305843009213689813 cannot be represented in type 'long'
>  #0 in GetFrameOfModify(SwRootFrame const*, SwModify const&, SwFrameType, SwPosition const*, std::pair<Point, bool> const*) at sw/source/core/layout/frmtool.cxx:3543:129 (instdir/program/../program/libswlo.so +0xdb83ad2)
>  #1 in SwContentNode::getLayoutFrame(SwRootFrame const*, SwPosition const*, std::pair<Point, bool> const*) const at sw/source/core/docnode/node.cxx:1154:42 (instdir/program/../program/libswlo.so +0xcc942c4)
>  #2 in SwCursorShell::GetCurrFrame(bool) const at sw/source/core/crsr/crsrsh.cxx:2369:25 (instdir/program/../program/libswlo.so +0xb23f5d7)
>  #3 in SwViewShell::SizeChgNotify() at sw/source/core/view/viewsh.cxx:1054:70 (instdir/program/../program/libswlo.so +0x10336772)
>  #4 in AdjustSizeChgNotify(SwRootFrame*) at sw/source/core/layout/pagechg.cxx:824:21 (instdir/program/../program/libswlo.so +0xdda98dd)
>  #5 in SwRootFrame::CheckViewLayout(SwViewOption const*, SwRect const*) at sw/source/core/layout/pagechg.cxx:2377:9 (instdir/program/../program/libswlo.so +0xdda4d44)
>  #6 in SwPageFrame::Paste(SwFrame*, SwFrame*) at sw/source/core/layout/pagechg.cxx:941:21 (instdir/program/../program/libswlo.so +0xddadf51)
>  #7 in (anonymous namespace)::doInsertPage(SwRootFrame*, SwPageFrame**, SwFrameFormat*, SwPageDesc*, bool, SwPageFrame**) at sw/source/core/layout/pagechg.cxx:1263:16 (instdir/program/../program/libswlo.so +0xddc0899)
>  #8 in SwFrame::InsertPage(SwPageFrame*, bool) at sw/source/core/layout/pagechg.cxx:1336:9 (instdir/program/../program/libswlo.so +0xddbe517)
>  #9 in SwFrame::GetNextSctLeaf(MakePageType) at sw/source/core/layout/sectfrm.cxx:1741:13 (instdir/program/../program/libswlo.so +0xdfa7423)
>  #10 in SwFrame::GetLeaf(MakePageType, bool) at sw/source/core/layout/flowfrm.cxx:819:23 (instdir/program/../program/libswlo.so +0xd9d8702)
>  #11 in SwFlowFrame::MoveFwd(bool, bool, bool) at sw/source/core/layout/flowfrm.cxx:1885:21 (instdir/program/../program/libswlo.so +0xd9fdad4)
>  #12 in SwContentFrame::MakeAll(OutputDevice*) at sw/source/core/layout/calcmove.cxx:1314:9 (instdir/program/../program/libswlo.so +0xd921d48)
>  #13 in SwFrame::PrepareMake(OutputDevice*) at sw/source/core/layout/calcmove.cxx:281:21 (instdir/program/../program/libswlo.so +0xd8f4623)
>  #14 in SwFrame::Calc(OutputDevice*) const at sw/source/core/layout/trvlfrm.cxx:1791:37 (instdir/program/../program/libswlo.so +0xe131abe)
>  #15 in SwTextFrame::CalcFollow(o3tl::strong_int<int, Tag_TextFrameIndex>) at sw/source/core/text/frmform.cxx:282:32 (instdir/program/../program/libswlo.so +0xe61c910)
>  #16 in SwTextFrame::AdjustFollow_(SwTextFormatter&, o3tl::strong_int<int, Tag_TextFrameIndex>, o3tl::strong_int<int, Tag_TextFrameIndex>, unsigned char) at sw/source/core/text/frmform.cxx:615:14 (instdir/program/../program/libswlo.so +0xe62efcf)
>  #17 in SwTextFrame::FormatAdjust(SwTextFormatter&, WidowsAndOrphans&, o3tl::strong_int<int, Tag_TextFrameIndex>, bool) at sw/source/core/text/frmform.cxx:1170:9 (instdir/program/../program/libswlo.so +0xe63a8fc)
>  #18 in SwTextFrame::CalcPreps() at sw/source/core/text/frmform.cxx:941:25 (instdir/program/../program/libswlo.so +0xe62381d)
>  #19 in SwTextFrame::Format(OutputDevice*, SwBorderAttrs const*) at sw/source/core/text/frmform.cxx:1867:13 (instdir/program/../program/libswlo.so +0xe653f69)
>  #20 in SwContentFrame::MakeAll(OutputDevice*) at sw/source/core/layout/calcmove.cxx:1496:17 (instdir/program/../program/libswlo.so +0xd9284dd)
>  #21 in SwFrame::PrepareMake(OutputDevice*) at sw/source/core/layout/calcmove.cxx:364:5 (instdir/program/../program/libswlo.so +0xd8f72fc)
>  #22 in SwFrame::Calc(OutputDevice*) const at sw/source/core/layout/trvlfrm.cxx:1791:37 (instdir/program/../program/libswlo.so +0xe131abe)
>  #23 in CalcContent(SwLayoutFrame*, bool) at sw/source/core/layout/fly.cxx:1451:25 (instdir/program/../program/libswlo.so +0xda5fd64)
>  #24 in SwLayoutFrame::FormatWidthCols(SwBorderAttrs const&, long, long) at sw/source/core/layout/wsfrm.cxx:3766:13 (instdir/program/../program/libswlo.so +0xe1d5011)
>  #25 in SwSectionFrame::Format(OutputDevice*, SwBorderAttrs const*) at sw/source/core/layout/sectfrm.cxx:1439:21 (instdir/program/../program/libswlo.so +0xdf9a3fd)
>  #26 in SwLayoutFrame::MakeAll(OutputDevice*) at sw/source/core/layout/calcmove.cxx:1007:13 (instdir/program/../program/libswlo.so +0xd9165e3)
>  #27 in SwSectionFrame::MakeAll(OutputDevice*) at sw/source/core/layout/sectfrm.cxx:837:20 (instdir/program/../program/libswlo.so +0xdf88e47)
>  #28 in SwFrame::PrepareMake(OutputDevice*) at sw/source/core/layout/calcmove.cxx:364:5 (instdir/program/../program/libswlo.so +0xd8f72fc)
>  #29 in SwFrame::Calc(OutputDevice*) const at sw/source/core/layout/trvlfrm.cxx:1791:37 (instdir/program/../program/libswlo.so +0xe131abe)
>  #30 in SwLayAction::FormatLayout(OutputDevice*, SwLayoutFrame*, bool) at sw/source/core/layout/layact.cxx:1210:19 (instdir/program/../program/libswlo.so +0xdc5507f)
>  #31 in SwLayAction::FormatLayout(OutputDevice*, SwLayoutFrame*, bool) at sw/source/core/layout/layact.cxx:1378:29 (instdir/program/../program/libswlo.so +0xdc5b559)
>  #32 in SwLayAction::FormatLayout(OutputDevice*, SwLayoutFrame*, bool) at sw/source/core/layout/layact.cxx:1378:29 (instdir/program/../program/libswlo.so +0xdc5b559)
>  #33 in SwLayAction::InternalAction(OutputDevice*) at sw/source/core/layout/layact.cxx:552:25 (instdir/program/../program/libswlo.so +0xdc41f02)
>  #34 in SwLayAction::Action(OutputDevice*) at sw/source/core/layout/layact.cxx:349:5 (instdir/program/../program/libswlo.so +0xdc3b6f0)
>  #35 in SwViewShell::ImplEndAction(bool) at sw/source/core/view/viewsh.cxx:295:17 (instdir/program/../program/libswlo.so +0x10313c0f)
>  #36 in SwViewShell::EndAction(bool) at sw/inc/viewsh.hxx:600:9 (instdir/program/../program/libswlo.so +0xb2c47f8)
>  #37 in SwCursorShell::EndAction(bool, bool) at sw/source/core/crsr/crsrsh.cxx:254:18 (instdir/program/../program/libswlo.so +0xb233608)
>  #38 in SwView::OuterResizePixel(Point const&, Size const&) at sw/source/uibase/uiview/viewport.cxx:1125:22 (instdir/program/../program/libswlo.so +0x125334e6)
>  #39 in SfxViewFrame::DoAdjustPosSizePixel(SfxViewShell*, Point const&, Size const&, bool) at sfx2/source/view/viewfrm.cxx:1688:18 (instdir/program/libsfxlo.so +0x5558e13)
>  #40 in SfxViewFrame::Resize(bool) at sfx2/source/view/viewfrm.cxx:2476:13 (instdir/program/libsfxlo.so +0x5580821)
>  #41 in SfxFrameViewWindow_Impl::Resize() at sfx2/source/view/viewfrm2.cxx:75:17 (instdir/program/libsfxlo.so +0x55b5f22)
>  #42 in vcl::Window::ImplCallResize() at vcl/source/window/event.cxx:527:5 (instdir/program/libvcllo.so +0x6192b95)
>  #43 in vcl::Window::Show(bool, ShowFlags) at vcl/source/window/window.cxx:2257:13 (instdir/program/libvcllo.so +0x69dad3a)
>  #44 in SfxBaseController::ConnectSfxFrame_Impl(SfxBaseController::ConnectSfxFrame) at sfx2/source/view/sfxbasecontroller.cxx:1228:24 (instdir/program/libsfxlo.so +0x54d3d42)
>  #45 in SfxBaseController::attachFrame(com::sun::star::uno::Reference<com::sun::star::frame::XFrame> const&) at sfx2/source/view/sfxbasecontroller.cxx:532:9 (instdir/program/libsfxlo.so +0x54cfc45)
>  #46 in (anonymous namespace)::SfxFrameLoader_Impl::impl_createDocumentView(com::sun::star::uno::Reference<com::sun::star::frame::XModel2> const&, com::sun::star::uno::Reference<com::sun::star::frame::XFrame> const&, comphelper::NamedValueCollection const&, rtl::OUString const&) at sfx2/source/view/frmload.cxx:597:18 (instdir/program/libsfxlo.so +0x5450d8c)
>  #47 in (anonymous namespace)::SfxFrameLoader_Impl::load(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&, com::sun::star::uno::Reference<com::sun::star::frame::XFrame> const&) at sfx2/source/view/frmload.cxx:714:13 (instdir/program/libsfxlo.so +0x5449a82)
>  #48 in framework::LoadEnv::impl_loadContent() at framework/source/loadenv/loadenv.cxx:1152:37 (instdir/program/../program/libfwklo.so +0x1e73d76)
>  #49 in framework::LoadEnv::startLoading() at framework/source/loadenv/loadenv.cxx:385:20 (instdir/program/../program/libfwklo.so +0x1e6469f)
>  #50 in framework::LoadDispatcher::impl_dispatch(com::sun::star::util::URL const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&, com::sun::star::uno::Reference<com::sun::star::frame::XDispatchResultListener> const&) at framework/source/dispatch/loaddispatcher.cxx:106:19 (instdir/program/../program/libfwklo.so +0x1b6b668)
>  #51 in framework::LoadDispatcher::dispatchWithReturnValue(com::sun::star::util::URL const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) at framework/source/dispatch/loaddispatcher.cxx:58:12 (instdir/program/../program/libfwklo.so +0x1b6d6b5)
>  #52 in non-virtual thunk to framework::LoadDispatcher::dispatchWithReturnValue(com::sun::star::util::URL const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) at framework/source/dispatch/loaddispatcher.cxx (instdir/program/../program/libfwklo.so +0x1b6d804)
>  #53 in comphelper::SynchronousDispatch::dispatch(com::sun::star::uno::Reference<com::sun::star::uno::XInterface> const&, rtl::OUString const&, rtl::OUString const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) at comphelper/source/misc/synchronousdispatch.cxx:62:31 (instdir/program/libcomphelper.so +0x1570c14)
>  #54 in desktop::DispatchWatcher::executeDispatchRequests(std::__debug::vector<desktop::DispatchWatcher::DispatchRequest, std::allocator<desktop::DispatchWatcher::DispatchRequest> > const&, bool) at desktop/source/app/dispatchwatcher.cxx:502:26 (instdir/program/libsofficeapp.so +0x9024bc)
>  #55 in desktop::RequestHandler::ExecuteCmdLineRequests(desktop::ProcessDocumentsRequest&, bool) at desktop/source/app/officeipcthread.cxx:1360:38 (instdir/program/libsofficeapp.so +0x972cb0)
>  #56 in desktop::Desktop::OpenClients() at desktop/source/app/app.cxx:2148:14 (instdir/program/libsofficeapp.so +0x7e4739)
>  #57 in desktop::Desktop::OpenClients_Impl(void*) at desktop/source/app/app.cxx:1935:9 (instdir/program/libsofficeapp.so +0x7de8c1)
>  #58 in desktop::Desktop::LinkStubOpenClients_Impl(void*, void*) at desktop/source/app/app.cxx:1918:1 (instdir/program/libsofficeapp.so +0x7d9d7a)
>  #59 in Link<void*, void>::Call(void*) const at include/tools/link.hxx:112:45 (instdir/program/libvcllo.so +0x6acdbc1)
>  #60 in ImplHandleUserEvent(ImplSVEvent*) at vcl/source/window/winproc.cxx:1964:30 (instdir/program/libvcllo.so +0x6abb56f)
>  #61 in ImplWindowFrameProc(vcl::Window*, SalEvent, void const*) at vcl/source/window/winproc.cxx:2517:13 (instdir/program/libvcllo.so +0x6aa4645)
>  #62 in SalFrame::CallCallback(SalEvent, void const*) const at vcl/inc/salframe.hxx:299:29 (instdir/program/libvcllo.so +0x9bb7fa8)
>  #63 in SalGenericDisplay::ProcessEvent(SalUserEventList::SalUserEvent) at vcl/unx/generic/app/gendisp.cxx:67:22 (instdir/program/libvcllo.so +0x9c53201)
>  #64 in SalUserEventList::DispatchUserEvents(bool) at vcl/source/app/salusereventlist.cxx:109:17 (instdir/program/libvcllo.so +0x8e746e8)
>  #65 in SalGenericDisplay::DispatchInternalEvent(bool) at vcl/unx/generic/app/gendisp.cxx:52:12 (instdir/program/libvcllo.so +0x9c52c47)
>  #66 in call_userEventFn(void*) at vcl/unx/gtk3/gtk3gtkdata.cxx:853:27 (instdir/program/libvclplug_gtk3lo.so +0xce6dbe)
>  #67 in g_idle_dispatch at ../glib/gmain.c:5627:11 (/lib64/libglib-2.0.so.0 +0x4c7da)
>  #68 in g_main_dispatch at ../glib/gmain.c:3189:28 (/lib64/libglib-2.0.so.0 +0x4fedc)
>  #69 in g_main_context_dispatch at ../glib/gmain.c:3854:7 (/lib64/libglib-2.0.so.0 +0x4fedc)
>  #70 in g_main_context_iterate at ../glib/gmain.c:3927:5 (/lib64/libglib-2.0.so.0 +0x5026f)
>  #71 in g_main_context_iteration at ../glib/gmain.c:3988:12 (/lib64/libglib-2.0.so.0 +0x50312)
>  #72 in GtkSalData::Yield(bool, bool) at vcl/unx/gtk3/gtk3gtkdata.cxx:528:31 (instdir/program/libvclplug_gtk3lo.so +0xce17ec)
>  #73 in GtkInstance::DoYield(bool, bool) at vcl/unx/gtk3/../gtk/gtkinst.cxx:404:29 (instdir/program/libvclplug_gtk3lo.so +0xcf9a98)
>  #74 in ImplYield(bool, bool) at vcl/source/app/svapp.cxx:457:48 (instdir/program/libvcllo.so +0x91af8b3)
>  #75 in Application::Yield() at vcl/source/app/svapp.cxx:521:5 (instdir/program/libvcllo.so +0x91aee37)
>  #76 in Application::Execute() at vcl/source/app/svapp.cxx:438:9 (instdir/program/libvcllo.so +0x91aebac)
>  #77 in desktop::Desktop::Main() at desktop/source/app/app.cxx:1620:17 (instdir/program/libsofficeapp.so +0x7d3700)
>  #78 in ImplSVMain() at vcl/source/app/svmain.cxx:202:35 (instdir/program/libvcllo.so +0x92569bc)
>  #79 in SVMain() at vcl/source/app/svmain.cxx:236:12 (instdir/program/libvcllo.so +0x925ff30)
>  #80 in soffice_main at desktop/source/app/sofficemain.cxx:170:12 (instdir/program/libsofficeapp.so +0x9b47b1)
>  #81 in sal_main at desktop/source/app/main.c:48:15 (instdir/program/soffice.bin +0x323dcc)
>  #82 in main at desktop/source/app/main.c:47:1 (instdir/program/soffice.bin +0x323da6)
>  #83 in __libc_start_main at /usr/src/debug/glibc-2.29-24-g2ec0b166bf/csu/../csu/libc-start.c:308:16 (/lib64/libc.so.6 +0x23f32)
>  #84 in _start at <null> (instdir/program/soffice.bin +0x24e02d)
> 
> SUMMARY: UndefinedBehaviorSanitizer: signed-integer-overflow sw/source/core/layout/frmtool.cxx:3543:129 in 

The reason appears to be that

>                     SwTwips nTmp  = TWIPS_MAX/2 - (getFrameArea().Top()+10000);

in SwTextFrame::CalcPreps (sw/source/core/text/frmform.cxx) produces values of aDiff.getY() that are too large at

>                     const sal_uInt64 nCurrentDist = sal_Int64(aDiff.getX()) * sal_Int64(aDiff.getX()) + sal_Int64(aDiff.getY()) * sal_Int64(aDiff.getY()); // opt: no sqrt

in GetFrameOfModify (sw/source/core/layout/frmtool.cxx).  (Note that nTmp had already been reduced from LONG_MAX to TWIPS_MAX/2 in <https://gerrit.libreoffice.org/plugins/gitiles/core/+/f2e3655255db4032738849cd4b77ce67a6e2c984%5E!> "Avoid -fsanitize=signed-integer-overflow", and presumably needs to be reduced further.)