Calculation Definitions

Introduction

Datamodel

• Current time series implementation:
• a time series (called item entity in the database) has a link to an itemDefinition which can refer to a FunctionModel if it is calculated

• Planned time series implementation:
• a time series has a property itemAction (indexSetCode) which defines how the time series is derived and has the following values:
• INGEST: the events of the time series are automatically ingested via an IOT device or REST API and persisted in the database
• STREAM: the events are available via Websockets (live data) but are not necessarily persisted
• CALCULATE: the time series events are derived and resolved from an expression or mathematical calculation model (eg pricing) which is defined in a visual modeler, using its relation (via ItemDefinition) to Calculation or CalculationDefinition
• via its ItemDefinition we can define how a time series with itemAction CALCULATE is derived via:
• a Calculation: this is a defined way of deriving a time serie (no parameters to be entered)
• a CalculationDefinition: a more flexible time serie which also requires additional parameters to be entered
• Calculations and CalculationDefinitions can be of different types:
• calculationType = EXPRESSION, by using a mathematical expression and scopes that define the inputs
• calculationType = STATE-EXPRESSION, identical as expression, but only uses the last event values (will only give one value at any time)
• calculationType = CALCULATION, by using a more complex mathematical calculation model managed in a visual modeler (previously this was called a FunctionModel)
 

Format of the Calculation Definition data block for calculationType = EXPRESSION

{
  "inputDefinitions": [],
  "calculation": { // what needs to be calculated and using what inputs
    "select": [], // optional nuQl query to identify time series which can use the inputDefinitions inputs
    "expression": "a + b", // the mathematical expression/ function to be used and its parameters, eg fn(a,b) = a + b;
    "expressionType": "TIMESERIES", // a single VALUE or TIMESERIES which requires more parameters
		"eventDateRange": {}, -> type eg entry, from, to
		"queryOptions": {} // overwrite of calculation queryOptions
    "removeBuckets": [], // optional field in case of type TIMESERIES to remove buckets at the beginning or end (negative)
                         // eg [0, -1] -> remove first bucket en last bucket in the expression result
    "scope": { // the parameters/inputs to be used in the expression
      "a": { // parameters the first character has to be a letter!
						// internal variables that can be used in an expression
							// time -> bucket start time in ms
							// from, to, interval -> range inputs in ms
							// now -> current time in ms
							// index -> the bucket index, starting at 0
							// previousValue -> bucket value minus 1: a previously calculated value
							// previousPreviousValue -> bucket value minus 2
							// firstValue -> value first bucket
        "type": "TIMESERIES", // DEFAULT, can be VALUE, TIMESERIES, REDUCE or another EXPRESSION
						// note if we use in an expression a parameter of type VALUE with parameters of type TIMESERIES
            // the value will be used in all buckets
				"type": "id" |
											"firstValue" | -> first value is kept for all buckets
											"lastValue" |
											"aggregation" -> to do an aggregation over multiple items make sure dataSeries.retrieveMultiple = true!
				"timeseries": { // in case of type TIMESERIES
						"selectVariable": "{{?variable}}", // OPTIONAL, the select variable to be used (can actually be one or more timeseries)
						"items": [{"id": <string>, "indexSetId": <string>}], // OPTIONAL 
                       // we can also give a list of timeseries without using a nuQl query
                       // the id/indexSet can also be selected in the inputDefinitions (item select)
            "lag": <number>, // OPTIONAL, only valid for timeseries; lag 1-> take previousValue, if not present this will be 0
            "bucketIndex": 0, // OPTIONAL, if the type is a TIMESERIES and we have a VALUE expression we need to define which value
                          // of the TIMESERIES to be used in the expression, eg 0 first, -1, last
						"metric": "avg", // in case of multiple items
                      // possible values: 'avg' | 'count' | 'min' | 'max' | 'sum' | 'multiply' | 'open' | 'close'
											// will calculate for every bucket de average value of the items
            "queryType": "EVENTS", // possible values:
                               // EVENTS,
															 // AGGREGATION (this is the normal timeline)
															 // SINGLEAGGREGATION (one timeline, item(s) are used as filters - uses the singleTimeline API) 
						"eventDateRange": {}, -> type eg entry, from, to
						"queryOptions": {} // overwrite of calculation queryOptions
				},
				"value": <number>, // OPTIONAL in case of type VALUE
											 // a single value
											 // the value can also be selected in the inputDefinitons
        "reduce": { // OPTIONAL in case of reduce -> the reduce type to be used (single value)
					"reduceType": "
                      // possible values: 'percentile10', 'sum', 'percentile50', 'avg', 'percentile25',
                      // 'percentile90', 'stddev', 'mad', 'median', 'count','max', 'min', 'percentile75'
          "eventDateRange": {}, -> type eg entry, from, to
				},                                 
				"expression": <string> // OPTIONAL in case of type EXPRESSION, the external ID of the EXPRESSION of which we will use the result as input
      }
    }
  }
}

Event Date Range

"eventDateRange": {
	"type": "timestamp", -> what datefield to be used, currently always timestamp, but can later be entry, exit, from, to
	"period": "This week",   // optional - default is this week
													 // possible values:
																'This hour',
																'Last hour',
																'Today',
																'Yesterday',
																'This week',
																'Last week',
																'7 days till now',
																'14 days till now',
																'This month',
																'Last month',
																'This year',
																'Last year',
																'Year till now',
																'Custom range'
"from": "2021-08-06T22:00:00.000Z", // in case of custom range isoString UTC format
"to": "2021-09-06T22:00:00.000Z" // in case of custom range
"timeZone": "Europe/Brussels" // DEFAULT
		// from or to can also be set to:
				{{now}} - we will use the current datetime as from or to
				{{today}} - we will use the start of the day as from or to	
}

Query Options

"queryOptions": 
   { 
        "pagination": {
					 "limit": 2, -> number of events to be retrieved, in case of a status card the limit will determine the % of trend indicator, comparing the first with the last value
					 "offset": 0
				},
				"sorting": {
						"order": ['timestamp-']
				},
				"aggregation": {
						"metric": "avg", -> possible values: 'avg' | 'count' | 'min' | 'max' | 'sum' | 'diff' | 'cardinality' | 'open' | 'close'
						"interval": "Hourly", -> possible values: '5 minute','10 minute','15 minute','20 minute','30 minute','Hourly','Daily','Weekly','Monthly','Quarterly','Yearly'
						"gapStrategy": {
								"type": "meet", -> possible values: 'meet', 'interpolate', 'push', 'pull'
								"config": {
										"meet": 0.5 // in case of type meet, meet percentage (0.5 = 50%) 
										"algorithm": "linear" // in case of type interpolate --> possible values: 'linear' | 'spline' | 'loess' | 'neville' | 'akimaSpline' | 'dividedDifference'
								}
						},
				},
				"indexSet": "<guid>", // required for a singletimelinequery, singleAggregation
		    "filter": { // new from version 1.0.24
				    "tenantCodes": 
						[
							{
					      "key": "segment"
					      "operator": "or", -> possible values: 'or', 'and', 'not'
					      "values": ["B2B", "B2C"]scale
					    }
						],
				    "keyValues":
						[
							{
					      "key": "someKey",
					      "operator": "or", -> possible values: 'or', 'and', 'not'
					      "values": ["bla"]
					    }
						],
				    "entityReferences":
						[
							{
						    "type": "Party", -> can be any entity type
					      "ids": ["4f3180f0-002f-11ec-be06-0759d5e31a59"],
					      "roles": ["partyType#CUSTOMER"], -> an IndexSetCode
								"operator": "or", -> possible values: 'or', 'and', 'not'
					    }
						],
				    "numberValues": // use from and/or to to specify a range
						[
							{
					      "from": 10,
								"fromInclusive": true,
								"to": 20,
								"toInclusive": false,
								"invert": false // default, if true, invert the filter
					    }
						],
						"metaDates": 
						[
							{
					      "type": "entry", // entry, exit, from, to 
								"from": "2021-08-06T22:00:00.000Z", // one has to be given
						    "to": "2021-09-06T22:00:00.000Z",
								"fromInclusive": true, // (optional) default true
			          "toInclusive": false,   // (optional) default false
								"invert": false // default, if true, invert the filter
					    }
						],
						"tags": {
							"operator": "or", -> possible values: 'or', 'and', 'not'
							"values": ["bla"]
						}
						"sourceIds": ["0.1.3"],
						"itemIds": ["<guid>"] // used for aggregated SingleTimeline queries where itemIds are used as filter
				}
		}