Bug 113974 - Ability to parse/filter JSON data and extract fields
Summary: Ability to parse/filter JSON data and extract fields
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
7.3.7.2 release
Hardware: x86-64 (AMD64) Linux (All)
: medium enhancement
Assignee: Not Assigned
URL:
Whiteboard: parsejson, with error at 12th request...
Keywords:
: 109073 (view as bug list)
Depends on:
Blocks: Calc-Function
  Show dependency treegraph
 
Reported: 2017-11-21 16:04 UTC by Dan Dascalescu
Modified: 2023-08-01 20:10 UTC (History)
15 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dan Dascalescu 2017-11-21 16:04:24 UTC
Description:
REST is far more popular than SOAP nowadays, and it's time to have a FILTERJSON function equivalent to FILTERXML.

Steps to Reproduce:
The vast majority of REST APIs return JSON data.

Actual Results:  
FILTERJSON (or "PARSEJSON") could use dotted notation to extract object fields.

Expected Results:
A1: =WEBSERVICE("https://api.github.com/repos/DmytroBazunov/LibreOfficeGetRestPlugin/issues/6")
B1: = PARSEJSON(A1, "user.login")  // "dandv"


Reproducible: Always


User Profile Reset: No



Additional Info:
There's a poorly maintained plugin that attempts to do this, https://github.com/DmytroBazunov/LibreOfficeGetRestPlugin/issues/6


User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36
Comment 1 Buovjaga 2017-11-23 10:19:23 UTC
Implementing this will be possible with the nearly-ready dataprovider: https://cgit.freedesktop.org/libreoffice/core/tree/sc/source/ui/dataprovider
Comment 2 Dan Dascalescu 2018-01-11 01:53:55 UTC
Nice, I see very recents commits.

As extra motivation, I was trying to extract Bitcoin prices from this API, 

=SUBSTITUTE(WEBSERVICE("https://min-api.cryptocompare.com/data/pricemulti?fsyms=BTC,ETH,LTC&tsyms=USD"), """:{""USD""", "")

The SUBSTITUTE above is just prep. After that, to get at a value, I came up with a formula looks like this:

=VALUE(MID(A1, SEARCH("(?<=BTC:)\d", A1), FIND("}", A1, SEARCH("(?<=BTC:)\d", A1)) - SEARCH("(?<=BTC:)\d", A1)))

A week later, that thing above looks more cryptic than elliptic curve cryptography :) I have no idea what it does at a glance and why it's so repetitive.
Comment 3 Kevin Suo 2021-10-21 15:46:37 UTC
LibreOffice uses liborcus as an mandatory external lib, and Orcus provides json support. Thus this filer may not be hard to implement.
Comment 4 Kevin Suo 2021-10-21 15:50:35 UTC
*** Bug 109073 has been marked as a duplicate of this bug. ***
Comment 5 Dan Dascalescu 2021-11-29 13:36:45 UTC Comment hidden (obsolete)
Comment 6 Buovjaga 2021-11-29 13:48:25 UTC
(In reply to Dan Dascalescu from comment #5)
> 4 years later, any progress?

Yes, data provider got a UI: https://libreoffice-dataproviders.blogspot.com/2021/08/data-providers.html

But: "The external data currently supported are CSV , HTML , XML and Base". So what Kevin said in comment 3 is still relevant.
Comment 7 Melroy 2022-07-23 23:48:29 UTC
Still really waiting on FILTERJSON() function. Please add it to LibreOffice! PLEASE GOD!
Comment 8 m_a_riosv 2023-01-21 00:20:53 UTC
As in many cases, web-pages to import are in JSON format, it will be really useful.
Comment 9 papou84 2023-08-01 20:10:31 UTC
I started to parse this JSON using this formula :
=PARSEJSON(A1, "outputs.monthly.fixed.get(1).E_m") . I get : 31.29
For month 2 I used : 
=PARSEJSON(A1, "outputs.monthly.fixed.get(2).E_m") . I get : 71.38
untill I get to month 12 :
=PARSEJSON(A1, "outputs.monthly.fixed.get(12).E_m") . and there I get no such node found.

It seems that after 11 it's bugging

JSON file :

{"inputs":{"location":{"latitude":50.0,"longitude":8.0,"elevation":104.0},"meteo_data":{"radiation_db":"PVGIS-SARAH","meteo_db":"ERA-Interim","year_min":2005,"year_max":2016,"use_horizon":true,"horizon_db":"DEM-calculated"},"mounting_system":{"fixed":{"slope":{"value":0,"optimal":false},"azimuth":{"value":0,"optimal":false},"type":"free-standing"}},"pv_module":{"technology":"c-Si","peak_power":1.0,"system_loss":14.0},"economic_data":{"system_cost":null,"interest":null,"lifetime":null}},"outputs":{"monthly":{"fixed":[{"month":1,"E_d":0.59,"E_m":18.36,"H(i)_d":0.8,"H(i)_m":24.67,"SD_m":2.13},{"month":2,"E_d":1.12,"E_m":31.29,"H(i)_d":1.41,"H(i)_m":39.36,"SD_m":6.45},{"month":3,"E_d":2.3,"E_m":71.38,"H(i)_d":2.82,"H(i)_m":87.54,"SD_m":11.19},{"month":4,"E_d":3.68,"E_m":110.26,"H(i)_d":4.57,"H(i)_m":137.06,"SD_m":15.67},{"month":5,"E_d":4.04,"E_m":125.13,"H(i)_d":5.08,"H(i)_m":157.51,"SD_m":17.94},{"month":6,"E_d":4.44,"E_m":133.16,"H(i)_d":5.71,"H(i)_m":171.22,"SD_m":13.81},{"month":7,"E_d":4.27,"E_m":132.5,"H(i)_d":5.57,"H(i)_m":172.78,"SD_m":11.89},{"month":8,"E_d":3.65,"E_m":113.22,"H(i)_d":4.72,"H(i)_m":146.37,"SD_m":7.96},{"month":9,"E_d":2.73,"E_m":81.93,"H(i)_d":3.49,"H(i)_m":104.71,"SD_m":7.8},{"month":10,"E_d":1.58,"E_m":48.97,"H(i)_d":2.02,"H(i)_m":62.65,"SD_m":6.44},{"month":11,"E_d":0.7,"E_m":21.12,"H(i)_d":0.95,"H(i)_m":28.45,"SD_m":2.53},{"month":12,"E_d":0.46,"E_m":14.27,"H(i)_d":0.65,"H(i)_m":20.04,"SD_m":1.24}]},"totals":{"fixed":{"E_d":2.47,"E_m":75.13,"E_y":901.57,"H(i)_d":3.16,"H(i)_m":96.03,"H(i)_y":1152.37,"SD_m":2.73,"SD_y":32.7,"l_aoi":-4.28,"l_spec":"1.45","l_tg":-6.31,"l_total":-21.76}}},"meta":{"inputs":{"location":{"description":"Selected location","variables":{"latitude":{"description":"Latitude","units":"decimal degree"},"longitude":{"description":"Longitude","units":"decimal degree"},"elevation":{"description":"Elevation","units":"m"}}},"meteo_data":{"description":"Sources of meteorological data","variables":{"radiation_db":{"description":"Solar radiation database"},"meteo_db":{"description":"Database used for meteorological variables other than solar radiation"},"year_min":{"description":"First year of the calculations"},"year_max":{"description":"Last year of the calculations"},"use_horizon":{"description":"Include horizon shadows"},"horizon_db":{"description":"Source of horizon data"}}},"mounting_system":{"description":"Mounting system","choices":"fixed, vertical_axis, inclined_axis, two_axis","fields":{"slope":{"description":"Inclination angle from the horizontal plane","units":"degree"},"azimuth":{"description":"Orientation (azimuth) angle of the (fixed) PV system (0 = S, 90 = W, -90 = E)","units":"degree"}}},"pv_module":{"description":"PV module parameters","variables":{"technology":{"description":"PV technology"},"peak_power":{"description":"Nominal (peak) power of the PV module","units":"kW"},"system_loss":{"description":"Sum of system losses","units":"%"}}},"economic_data":{"description":"Economic inputs","variables":{"system_cost":{"description":"Total cost of the PV system","units":"user-defined currency"},"interest":{"description":"Annual interest","units":"%/y"},"lifetime":{"description":"Expected lifetime of the PV system","units":"y"}}}},"outputs":{"monthly":{"type":"time series","timestamp":"monthly averages","variables":{"E_d":{"description":"Average daily energy production from the given system","units":"kWh/d"},"E_m":{"description":"Average monthly energy production from the given system","units":"kWh/mo"},"H(i)_d":{"description":"Average daily sum of global irradiation per square meter received by the modules of the given system","units":"kWh/m2/d"},"H(i)_m":{"description":"Average monthly sum of global irradiation per square meter received by the modules of the given system","units":"kWh/m2/mo"},"SD_m":{"description":"Standard deviation of the monthly energy production due to year-to-year variation","units":"kWh"}}},"totals":{"type":"time series totals","variables":{"E_d":{"description":"Average daily energy production from the given system","units":"kWh/d"},"E_m":{"description":"Average monthly energy production from the given system","units":"kWh/mo"},"E_y":{"description":"Average annual energy production from the given system","units":"kWh/y"},"H(i)_d":{"description":"Average daily sum of global irradiation per square meter received by the modules of the given system","units":"kWh/m2/d"},"H(i)_m":{"description":"Average monthly sum of global irradiation per square meter received by the modules of the given system","units":"kWh/m2/mo"},"H(i)_y":{"description":"Average annual sum of global irradiation per square meter received by the modules of the given system","units":"kWh/m2/y"},"SD_m":{"description":"Standard deviation of the monthly energy production due to year-to-year variation","units":"kWh"},"SD_y":{"description":"Standard deviation of the annual energy production due to year-to-year variation","units":"kWh"},"l_aoi":{"description":"Angle of incidence loss","units":"%"},"l_spec":{"description":"Spectral loss","units":"%"},"l_tg":{"description":"Temperature and irradiance loss","units":"%"},"l_total":{"description":"Total loss","units":"%"}}}}}}