Rules and Redundancy

Contains tips for configurators working with Aware IM
Post Reply
PointsWell
Posts: 1457
Joined: Tue Jan 24, 2017 5:51 am
Location: 'Stralya

Rules and Redundancy

Post by PointsWell »

I've been thinking about this since Jaymer asked about a complex rule and have drawn on my prior life working for an advanced rules engine supplier.

Prior versions of (up to V7, I think) attempted to build a rules matrix for the business rules that you created. It very quickly became complicated, but it was good for showing you the route through your business rules. Unlike linear programming it appears that AIM checks records against all the permutations of the rule checking whether both the rule and the inverse are true (based on looking at what happens in the logger).

If you consider the rules engine as trying to filter a record through various options the aim should be to get that record to the action required as fast as possible. If you have any experience with writing SQL queries that check against multiple tables there is a fast way and a slow way to write a query to get the same result. A rules engine is the same.

Redundancy in rules is your enemy.

Code: Select all

IF BO.Att=1 AND BO.OthAtt='xyx' THEN Do Stuff ELSE
IF BO.Att=1 AND Bo.OthAtt='abc' THEN Do More Stuff ELSE
IF BO.Att=2 AND BO.OthAtt='pqr' THEN Something Different ELSE
IF BO.Att=2 AND BO.OthAtt='lmn' THEN The Other Different Thing
The above example creates a matrix of 4 rules and 2 conditions and in each of the rules the same thing is being asked multiple times. 4x2

In the above example by streaming out the rule based on BO.Att value you reduce the size of the matrix to two rules and one condition.

Code: Select all

IF BO.Att=1 THEN DoTheChecksOn1 ELSE
IF BO.Att=2 THEN DoTheChecksOn2
If you stream records where BO.Att=1 to another sub process you then reduce the matrix at this sub process level to 2 x 1, does Att= 1? Yes or No, does Att=2? Yes or No.

You then need to do the subsequent checking in a sub process that receives only the BO.Att=1 records so DoTheChecksOn1 looks like:

Code: Select all

IF BO.OthAtt='xyz' THEN Do stuff ELSE
IF BO.OthAtt='abc' THEN Do mores stuff
Another 2x1 matrix with simple choices.

You are also asking the questions once and will have less difficulty maintaining this; in the event that another test for BO.Att=1 and BO.OthAtt='efg' requires one line of code in one sub process making the matrices 2x1 + 3x1 instead of one larger process where the matrix grows from 4x2 to 5x2

The process maps for your processes will also become easier to read and a bit more useful.
Last edited by PointsWell on Fri Jan 31, 2020 3:26 am, edited 4 times in total.
Jaymer
Posts: 2430
Joined: Tue Jan 13, 2015 10:58 am
Location: Tampa, FL
Contact:

Re: Rules and Redundancy

Post by Jaymer »

k, I accept that.

The Limitation in Aware of having only 1 IF/THEN/ELSE per Rule is a pain.
... and the inability to "BREAK" out of a Rule doesn't help (this post).
In my complex Process, a BREAK would help once the TRUE "IF" branch was found, hopefully saving time by not having to execute further down the Process. When I simplified my Process to 3 separate Rules, the BREAK would exit and prevent further unnecessary tests once the condition was met. So It wouldn't always be all 3 rules executing. In fact, I'd put my "most common" condition first so the majority of the time Aware would never see the 2nd or 3rd Rule because the Break would [gracefully] end the Process.

And While I am used to having a Process with 1 or 2 lines in it (as you have described here) and then calling the ".1" or ".2" sub-process to do more Logic, one thing I've been wondering about is the Overhead of Aware writing Context as it spawns other Processes.
ANSWER: In simple tests, I have not seen it "write Context" except when the Process is FIRST started (from a Form button in my explanation below) - So there is no "overhead" of writing the current context IN A PROCESS before that PROCESS calls a subprocess (not that I have seen yet, I may be wrong). When we "pass" a BO by Specifying that BO as Input in a Called Process, Aware re-reads that Record anyway - its not passing current values in Memory.

???eh?

So If I passed in CUST as Input to this top level Process, and I follow your Logic (referring to PointsWell's main argument in this thread) to make a simple test, and then call Process_1 or Process_2, CUST gets also passed to those Processes.
---> I wonder if another internal Context record has to be written to disk to serve as input for those Processes - Aware could have no idea the complexity of the destination Process, and certainly couldn't know this is a tiny process only acting as a "sub-process" to save on Logic complexity. Of course, we'd need an answer from Support OR an analysis of SQL Profiler to see if we can see this context being written....

OK, 2 hours later I'm back with results.
It turns out that that I still really don't know whats in the binary code written into EXECUTION_CONTEXTS. But its not what I thought it was. (BTW, mine is 27,222 characters of ASCII when pasted into an Editor, so its half that in Bytes being written & read.) It probably is some Record IDs (Like the Main BO needed for input to the SubProcess - but its NOT actually the data thats in that Main BO record. We think, from other programming languages, that when we say we "pass" the Cust record to the Process we are passing values IN MEMORY (Sure, it can be call by name, call by reference, call by value - skip that for now) - But thats not what Aware is doing. (I assume this, because if it was the data of a record(s), then why re-read the Main BO from the DB, why not just unpack it from the binary?)

From the trace below, I can see this:
1) Whenever I'm on a Form and I click a Panel Operation to start a Process to send this Customer an Email, for example, Aware 1st re-reads that record (SELECT *) and reads ALL "ps" and "ob" reference tables (and a few more to resolve shortcuts).
*** Just because you "see it" on the form and you think its "current"... its NOT.
2) It then Writes a context record - but it appears it doesn't need to sometimes... it sniffs ahead (somehow) and IF I removed the Display Message action, then it doesn't write context (I only tested this about 10 times cause I couldn't figure out why test2 wasn't Writing/Reading EXECUTION CONTEXTS - I could be wrong, but thats what 30 mins of testing and hair pulling determined.) .
3) The 1st thing the subtask does: Read the CONTEXT record (but not every time, see #2 above)
4) Then it reads the MAIN BO (Select *)
5) Then it [again] reads all "ps" and "ob" reference tables (and a few more to resolve shortcuts).
--- This is where Test1 ends
6) Test2 (identical to test1 up to this point) calls Test2B (Following PointsWell's example/suggestion in this thread) and guess what???
7) The called Process: Read the CONTEXT record, Read the MAIN BO (Select *), Then it [again] ...... Get the idea?

So this begs the question...
DO I REALLY NEED TO WORRY IF AWARE DOES 4 COMPARISONS or 12 - which are in Memory,
OR Do I want the Rules Engine to work a little easier while, IN MY CASE, I'm going to do 9-15 more database reads by calling a SubProcess ?

(9-15?.... the Main BO, the 8 reference tables, plus various shortcuts)


A Simple Process:
Test1 - does nothing, only a Display Message
Test2 (and subtask Test2b) - Test2 just calls Test2b (the "sub-process", which also has BO as input), then Display Message

1 Button on a Form of the "Main BO". Button is Start Process. The "Main BO" is input of the Processes
Screen Shot 2020-01-30 at 10.06.21 PM.png
Screen Shot 2020-01-30 at 10.06.21 PM.png (31.62 KiB) Viewed 15927 times


Profiler Results:
--> My "Main BO" in these results is called RO (for Repair Order). I've omitted all the reference table reads. Sorry, this is hard to read. All my notes are in <brackets>. Most All else is direct from MSSQL Profiler.

TEST1 MSSQL Profiler

Code: Select all

<I am on a Form>
<I press a Panel Operation calling Test1, with RO as Input>
<it appears the server reads the record thats needed by called Process (ie. the Record on the Form)>
	exec sp_prepexec @p1 output,N'@P0 bigint',N'SELECT * FROM RO WHERE ID=@P0        ',12914 <-- this is the record ID
	<8 more reference tables are read>
<save context>:
	exec sp_prepexec @p1 output,N'@P0 bigint,@P1 varbinary(8000),@P2 varbinary(max),@P3 bigint,@P4 nvarchar(4000),@P5 bit',N'INSERT INTO EXECUTION_CONTEXTS VALUES(@P0,@P1,@P2,@P3,@P4,@P5)',8274,NULL,0x789CED7D07981C47957FCF748FB4DA55B6259B60DC5E39DBDA24C996A   <snip>
<calls Process>
------- Now the Process starts -----------
<Test1 starts by reading Context passed to it>
	exec sp_prepexec @p1 output,N'@P0 bigint',N'SELECT * FROM EXECUTION_CONTEXTS WHERE ID=@P0        ',8274  <-- record ID
<then it re-reads the main RO record that was passed to it>
	exec sp_prepexec @p1 output,N'@P0 bigint',N'SELECT * FROM RO WHERE ID=@P0        ',12914
<8 more reference tables are read>
	exec sp_prepexec @p1 output,N'@P0 bigint',N'DELETE FROM EXECUTION_CONTEXTS WHERE ID=@P0        ',8274
<nothing actually happens in this Process except DISPLAY MSG>
<end Test1>
-------- Leaves Process -----------
<it appears the main form reads Context back in>
<and main form refreshes - by reading main rec and the 8 reference tables>	
	exec sp_prepexec @p1 output,N'@P0 bigint',N'SELECT * FROM EXECUTION_CONTEXTS WHERE PGID=@P0        ',8274
	exec sp_prepexec @p1 output,N'@P0 bigint',N'SELECT * FROM RO WHERE ID=@P0        ',12914

Read the main BO only 1 time actually in the process
TEST2 MSSQL Profiler

Code: Select all

read & save
	exec sp_prepexec @p1 output,N'@P0 bigint',N'SELECT * FROM RO WHERE ID=@P0        ',12829
	exec sp_prepexec @p1 output,N'@P0 bigint,@P1 varbinary(8000),@P2 varbinary(max),@P3 bigint,@P4 nvarchar(4000),@P5 bit',N'INSERT INTO EXECUTION_CONTEXTS VALUES(@P0,@P1,@P2,@P3,@P4,@P5)',8546,NULL,0x789CED5D09981C4775EE99E991565A1D2BC9960DC4B8BDF2211FDA4B8 ... <snip>
---- process starts -----
read context, BO & reference tables
	exec sp_prepexec @p1 output,N'@P0 bigint',N'SELECT * FROM EXECUTION_CONTEXTS WHERE ID=@P0        ',8546
	exec sp_prepexec @p1 output,N'@P0 bigint',N'SELECT * FROM RO WHERE ID=@P0        ',12829
now a funky sequence where it Deletes context - then re-reads it (which doesn't make sense to me)
	exec sp_prepexec @p1 output,N'@P0 bigint',N'DELETE FROM EXECUTION_CONTEXTS WHERE ID=@P0        ',8546
----- So I guess the SubProcess starts here -----
and repeats the read context, BO & reference tables
	exec sp_prepexec @p1 output,N'@P0 bigint',N'SELECT * FROM EXECUTION_CONTEXTS WHERE PGID=@P0        ',8546
	exec sp_prepexec @p1 output,N'@P0 bigint',N'SELECT * FROM RO WHERE ID=@P0        ',12829
----- process ends -----

Read the main BO 2 times, 1 in main process, 1 in sub-process
Last edited by Jaymer on Fri Jan 31, 2020 6:21 am, edited 1 time in total.
Click Here to see a collection of my tips & hacks on this forum. Or search for "JaymerTip" in the search bar at the top.

Jaymer
Aware Programming & Consulting - Tampa FL
PointsWell
Posts: 1457
Joined: Tue Jan 24, 2017 5:51 am
Location: 'Stralya

Re: Rules and Redundancy

Post by PointsWell »

Jaymer wrote:... and the inability to "BREAK" out of a Rule doesn't help (this post)
In this example you don't have to break out of any rules. You've gone down the chain for the test being True. You don't have to go down the other branches for the test being False in this instance. For example, if BO.Att=1 you test that against the first rule IF BO.Att=1 or IF BO.Att=2, only one of which it will pass and therefore only one chain of subsequent processes to be handled. Once it has completed the branch for BO.Att=1 it continues to the next rule in the main process (or ends the process gracefully if there are no more rules).
Jaymer wrote:And While I am used to having a Process with 1 or 2 lines in it and then calling the ".1" or ".2" sub-process to do more Logic, one thing I've been wondering about is the Overhead of Aware writing Context as it spawns other Processes.
This is a good question and I don't know enough about the overheads of managing one complex rule versus multiple simple rules. Were you able to ascertain a reduction in time from the example you had raised yesterday (ignoring the incorrect result)?
Jaymer wrote:So If I passed in CUST as Input to this top level Process, and I follow your Logic to make a simple test, and then call Process_1 or Process_2, CUST gets also passed to those Processes.
Only to those processes that the test is passed for. IF BO.Att=1 then only the sub process for 1 gets called the test, BO.Att=2 test will be false and nothing in that sub process chain will be called but I don't know how things work under the hood. Again it is a question of the cost of a large complex rule v multiple small simple rules. If it uses twice the storage for half of the time then I guess the question is fast and greedy better than slow and lean, depends on how many other processes you have running.
Jaymer wrote:---> I wonder if another internal Context record has to be written to disk to serve as input for those Processes - Aware could have no idea the complexity of the destination Process, and certainly couldn't know this is a tiny process only acting as a "sub-process" to save on Logic complexity. Of course, we'd need an answer from Support OR an analysis of SQL Profiler to see if we can see this context being written....
-- k, hold on
Those are absolutely questions for Vlad. Let me know if your profiler shows a difference.
PointsWell
Posts: 1457
Joined: Tue Jan 24, 2017 5:51 am
Location: 'Stralya

Re: Rules and Redundancy

Post by PointsWell »

===Post your testing update===

I think a more representative test would be to use your complex query versus two simplified tests. The point of the two process method is simplification, which isn't being tested by doing two simple tests but with different processes. The reason I say that is because the SQL is only half of the workload, the missing bit is what AIM is doing.

For example it may require more database reads, but unless you are on a different network with latency issues or heavy traffic then these costs are not necessarily significant in the event that the AIM server is chewing up memory to formulate the complex query.

There is also the issue that the server is getting confused with too complex and repetitive business rule tests.
customaware
Posts: 2391
Joined: Mon Jul 02, 2012 12:24 am
Location: Ulaanbaatar, Mongolia

Re: Rules and Redundancy

Post by customaware »

Probably a little off topic but this is always a concern....

I have a Time Range between 22:00 and 06:00am the following day

People normally work 18:00 to 06:00 the following day.
I need to calc the number of hours they work between the Range Hours which is obviously normally 8 hours.

But need to consider the alternatives...for example
Start at 22:00 and Finish at 03:00 ..... 5 Hours
Start at 03:00 and Finish at 06:00 ..... 3 Hours
Start at 02:00 and Finish at 05:00 ----- 2 Hours

So....using 4 distinct Rules works...

Rule 1: WS <= RS AND WE >= RE
If
TestBO.WorkTimeStart>=TestBO.HourRangeStart AND
TestBO.WorkTimeStart<TestBO.HourRangeEnd AND
TestBO.WorkTimeEnd<=TestBO.HourRangeEnd AND
TestBO.WorkTimeEnd>TestBO.HourRangeStart
Then
TestBO.HoursInRange=TestBO.WorkTimeEnd-TestBO.WorkTimeStart

Rule 2: WS <= RS AND WE > RS AND WE <= RE
If
TestBO.WorkTimeStart<=TestBO.HourRangeStart AND
TestBO.WorkTimeEnd>TestBO.HourRangeStart AND
TestBO.WorkTimeEnd<=TestBO.HourRangeEnd
Then
TestBO.HoursInRange=TestBO.WorkTimeEnd-TestBO.HourRangeStart

Rule 3: WS >= RS AND WS < RE AND WE >= RE
If
TestBO.WorkTimeStart>=TestBO.HourRangeStart AND
TestBO.WorkTimeStart<TestBO.HourRangeEnd AND
TestBO.WorkTimeEnd>=TestBO.HourRangeEnd
Then
TestBO.HoursInRange=TestBO.HourRangeEnd-TestBO.WorkTimeStart

Rule 4: WS >= RS AND WS < RE AND WE <= RE AND WE > RS
If
TestBO.WorkTimeStart>=TestBO.HourRangeStart AND
TestBO.WorkTimeStart<TestBO.HourRangeEnd AND
TestBO.WorkTimeEnd<=TestBO.HourRangeEnd AND
TestBO.WorkTimeEnd>TestBO.HourRangeStart
Then
TestBO.HoursInRange=TestBO.WorkTimeEnd-TestBO.WorkTimeStart

Now... if I combine all into a single Rule........ the 3rd expression fails.


Rule: Combined

IF
TestBO.WorkTimeStart<=TestBO.HourRangeStart AND
TestBO.WorkTimeEnd>=TestBO.HourRangeEnd
THEN
TestBO.HoursInRange=TestBO.HourRangeEnd-TestBO.HourRangeStart

ELSE

If
TestBO.WorkTimeStart<=TestBO.HourRangeStart AND
TestBO.WorkTimeEnd>TestBO.HourRangeStart AND
TestBO.WorkTimeEnd<=TestBO.HourRangeEnd
THEN
TestBO.HoursInRange=TestBO.WorkTimeEnd-TestBO.HourRangeStart

ELSE

If
TestBO.WorkTimeStart>=TestBO.HourRangeStart AND
TestBO.WorkTimeStart<TestBO.HourRangeEnd AND
TestBO.WorkTimeEnd>=TestBO.HourRangeEnd
THEN
TestBO.HoursInRange=TestBO.HourRangeEnd-TestBO.WorkTimeStart

ELSE

If
TestBO.WorkTimeStart>=TestBO.HourRangeStart AND
TestBO.WorkTimeStart<TestBO.HourRangeEnd AND
TestBO.WorkTimeEnd<=TestBO.HourRangeEnd AND
TestBO.WorkTimeEnd>TestBO.HourRangeStart
THEN
TestBO.HoursInRange=TestBO.WorkTimeEnd-TestBO.WorkTimeStart

So, I am pretty sure I don't have an error in there (But it is a real possibility) but I have seen this behavior on the odd occasion before so tend to use separate rules now to be on the safe side.
Cheers,
Mark
_________________
AwareIM 6.0, 8.7, 8.8, 9.0 , MariaDB, Windows 10, Ubuntu Linux. Theme: Default, Browser: Arc
Upcloud, Obsidian....
Image
PointsWell
Posts: 1457
Joined: Tue Jan 24, 2017 5:51 am
Location: 'Stralya

Re: Rules and Redundancy

Post by PointsWell »

Mark's current rules (as above) look like this:
Current.jpg
Current.jpg (48.31 KiB) Viewed 15798 times
I copied and pasted this from the individual rules and it highlighted an error, as the same test is being carried out twice (the last column is identical to the first). I colour coded the steps to highlight where the same condition is being tested redundantly in multiple rules.

These can be broken down like this:
outcomes.png
outcomes.png (53.18 KiB) Viewed 15802 times
At the moment each record is being individually against all conditions, which means each record is being tested 4 times.
  1. Inside Range Start & Inside Range End
  2. Inside Range Start & Outside Range End
  3. Outside Range Start & Inside Range End
  4. Outside Range Start & Outside Range End
But this is highly redundant testing, testing Inside Range Start twice, and Outside Range Start twice. It also makes the tests more complex than they need to be, resulting in this case with 4 complex tests. The complexity becomes exponential the more possible options through the matrix.

If a decision tree approach is taken, streaming the second layer of the tree to one of two sub process, then only two tests are ever being run
tree.png
tree.png (109.36 KiB) Viewed 15802 times
The tests then become

Code: Select all

IF Inside Range Start Then CheckEndForInsideStart  ELSE
IF Outside Range Start Then CheckEndForOutsideStart
CheckEndForInsideStart:

Code: Select all

IF Inside Range End then WE-WS ELSE
IF Outside Range End then RE-WS
CheckEndForOutsideStart:

Code: Select all

IF Inside Range End then WE-RS ELSE
IF Outside Range End then RE-WS
Then the processes become simpler, and every test that is run is efficient, that is to say you are not redundantly testing records against unnecessary tests and all testing ends with an outcome. There's no need for "break" code because there is only one route through the decision tree and zero redundancy.

It will also make your log files easier to read.
tford
Posts: 4238
Joined: Sat Mar 10, 2007 6:44 pm

Re: Rules and Redundancy

Post by tford »

Excellent analysis & documentation of applying your framework to Mark's example, PointWell !!

I'm curious to hear your thoughts on this, Mark.
Tom - V8.8 build 3137 - MySql / PostGres
customaware
Posts: 2391
Joined: Mon Jul 02, 2012 12:24 am
Location: Ulaanbaatar, Mongolia

Re: Rules and Redundancy

Post by customaware »

Sean did an excellent job and certainly gave me something to think about, not so much for my exact circumstance but definitely for others in the future.

For example...and here is where I am most likely to be looking at using that thinking next.

Imagine you needed to apply a different rule for each day of a month... 1-31

IF D=1 .....
IF D=2......
IF D=3......
.
.
.
IF D=31....

Hence, 31 Rules which all have to be tested.

Sean's logic, although not precisely... a bit like a divide and conquer solution... Why test if you don't need to.

So D<=15 Go Subprocess ... then keep splitting while it makes sense
D>15 Fo Subprocess ... then keep splitting while it makes sense

Then Split again and if necessary again, whereby you reduce the number of tests down to maybe 5 rather than 31. Obviously there is a little overhead calling the subprocess but still faster than doing all the tests.

Additionally, you need to consider if the need for speed is worth the effort and if that place in your app warrants the effort.
Cheers,
Mark
_________________
AwareIM 6.0, 8.7, 8.8, 9.0 , MariaDB, Windows 10, Ubuntu Linux. Theme: Default, Browser: Arc
Upcloud, Obsidian....
Image
PointsWell
Posts: 1457
Joined: Tue Jan 24, 2017 5:51 am
Location: 'Stralya

Re: Rules and Redundancy

Post by PointsWell »

I'd probably not go down that route as D=1 .. D=31 are unique rules and low complexity.
Post Reply