 |
charneus
Wizard
Joined: 19 Jun 2005
Posts: 1876
Location: California
|
 Posted: Fri Jun 18, 2010 11:56 pm
[3.20]Extensive benchmark testing
|
Updated the script per replies.
So, I decided to do some benchmark testing for %additem, %delitem, %delnitem, #ADDITEM, #DELITEM, and #DELNITEM. The results are as follows:
For adding items (9 tests):
| Code: |
--------------------------------------------------
Benchmark test 1: Using %additem(item, variable)
Total items: 1000
Total time: 2087ms
--------------------------------------------------
Benchmark test 2: Same as above, adding more
Total items: 2000
Total time: 7304ms
--------------------------------------------------
Benchmark test 3: Adding even more
Total items: 3000
Total time: 13574ms
--------------------------------------------------
Benchmark test 4: Using %additem(item, @variable)
Total items: 1000
Total time: 4171ms
--------------------------------------------------
Benchmark test 5: Same as above, adding more
Total items: 2000
Total time: 14503ms
--------------------------------------------------
Benchmark test 6: Adding even more
Total items: 3000
Total time: 26693ms
--------------------------------------------------
Benchmark test 7: Using #ADDITEM
Total items: 1000
Total time: 32ms
--------------------------------------------------
Benchmark test 8: Same as above, adding more
Total items: 2000
Total time: 103ms
--------------------------------------------------
Benchmark test 9: Adding even more
Total items: 3000
Total time: 108ms
--------------------------------------------------
|
For deleting (18 tests, variable contains 3000):
| Code: |
--------------------------------------------------
Benchmark test 1: Using %delitem(item, variable)
Total items: 2000
Total time: 11322ms
--------------------------------------------------
Benchmark test 2: Same as above, deleting more
Total items: 1000
Total time: 6599ms
--------------------------------------------------
Benchmark test 3: Deleting the rest
Total items: 0
Total time: 2103ms
--------------------------------------------------
Benchmark test 4: Using %delitem(item, @variable)
Total items: 2000
Total time: 20917ms
--------------------------------------------------
Benchmark test 5: Same as above, deleting more
Total items: 1000
Total time: 12349ms
--------------------------------------------------
Benchmark test 6: Deleting the rest
Total items: 0
Total time: 3976ms
--------------------------------------------------
Benchmark test 7: Using #DELITEM
Total items: 2000
Total time: 8514ms
--------------------------------------------------
Benchmark test 8: Same as above, deleting more
Total items: 1000
Total time: 4987ms
--------------------------------------------------
Benchmark test 9: Deleting the rest
Total items: 0
Total time: 1641ms
--------------------------------------------------
Benchmark test 10: Using #DELNITEM
Total items: 2000
Total time: 10834ms
--------------------------------------------------
Benchmark test 11: Same as above, deleting more
Total items: 1000
Total time: 6151ms
--------------------------------------------------
Benchmark test 12: Deleting the rest
Total items: 0
Total time: 1915ms
--------------------------------------------------
Benchmark test 13: Using %delnitem(variable, number)
Total items: 2000
Total time: 20933ms
--------------------------------------------------
Benchmark test 14: Same as above, deleting more
Total items: 1000
Total time: 11898ms
--------------------------------------------------
Benchmark test 15: Deleting the rest
Total items: 0
Total time: 3630ms
--------------------------------------------------
Benchmark test 16: Using %delnitem(variable, number)
Total items: 2000
Total time: 30583ms
--------------------------------------------------
Benchmark test 17: Same as above, deleting more
Total items: 1000
Total time: 17704ms
--------------------------------------------------
Benchmark test 18: Deleting the rest
Total items: 0
Total time: 5411ms
-------------------------------------------------- |
As you can see, depending on what you use, you have considerable slowdown. It seems to be quicker using the (item, variable) format rather than (item, @variable), though.
No bug as previously mentioned, see explanation below:
The script I used to produce these results are as follows:
For adding:
| Code: |
<?xml version="1.0" encoding="ISO-8859-1" ?>
<cmud>
<alias name="bmarkadd" copy="yes">
<value>TestVar=%null
#SAY {Benchmark test 1: Using "%additem(item, variable)"}
$Secs=%secs
#1000 {TestVar=%additem(%random(1,9999), TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 2: Same as above, adding more}
$Secs=%secs
#1000 {TestVar=%additem(%random(1,9999), TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 3: Adding even more}
$Secs=%secs
#1000 {TestVar=%additem(%random(1,9999), TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
TestVar=%null
#SAY {Benchmark test 4: Using "%additem(item, @variable)"}
$Secs=%secs
#1000 {TestVar=%additem(%random(1,9999), @TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 5: Same as above, adding more}
$Secs=%secs
#1000 {TestVar=%additem(%random(1,9999), @TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 6: Adding even more}
$Secs=%secs
#1000 {TestVar=%additem(%random(1,9999), @TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
TestVar=%null
#SAY {Benchmark test 7: Using "#ADDITEM"}
$Secs=%secs
#1000 {#ADDITEM TestVar %i}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 8: Same as above, adding more}
$Secs=%secs
#1000 {#ADDITEM TestVar %eval(%i+1000)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 9: Adding even more}
$Secs=%secs
#1000 {#ADDITEM TestVar %eval(%i+2000)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}</value>
</alias>
</cmud> |
For deleting:
| Code: |
<?xml version="1.0" encoding="ISO-8859-1" ?>
<cmud>
<alias name="bmarkdel" copy="yes">
<value>TestVar=%null
#3000 {#ADDITEM TestVar %i}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 1: Using "%delitem(item, variable)"}
$Secs=%secs
#1000 {TestVar=%delitem(%i, TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 2: Same as above, deleting more}
$Secs=%secs
#1000 {TestVar=%delitem(%eval(%i+1000), TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 3: Deleting the rest}
$Secs=%secs
#1000 {TestVar=%delitem(%eval(%i+2000), TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#3000 {#ADDITEM TestVar %i}
#SAY {Benchmark test 4: Using "%delitem(item, @variable)"}
$Secs=%secs
#1000 {TestVar=%delitem(%i, @TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 5: Same as above, deleting more}
$Secs=%secs
#1000 {TestVar=%delitem(%eval(%i+1000), @TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 6: Deleting the rest}
$Secs=%secs
#1000 {TestVar=%delitem(%eval(%i+2000), @TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#3000 {#ADDITEM TestVar %i}
#SAY {Benchmark test 7: Using "#DELITEM"}
$Secs=%secs
#1000 {#DELITEM TestVar %i}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 8: Same as above, deleting more}
$Secs=%secs
#1000 {#DELITEM TestVar %eval(%i+1000)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 9: Deleting the rest}
$Secs=%secs
#1000 {#DELITEM TestVar %eval(%i+2000)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#3000 {#ADDITEM TestVar %i}
#SAY {Benchmark test 10: Using "#DELNITEM"}
$Secs=%secs
#1000 {#DELNITEM TestVar %numitems(@TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 11: Same as above, deleting more}
$Secs=%secs
#1000 {#DELNITEM TestVar %numitems(@TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 12: Deleting the rest}
$Secs=%secs
#1000 {#DELNITEM TestVar %numitems(@TestVar)}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#3000 {#ADDITEM TestVar %i}
#SAY {Benchmark test 13: Using "%delnitem(variable, number)"}
$Secs=%secs
#1000 {TestVar=%delnitem(TestVar,%numitems(@TestVar))}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 14: Same as above, deleting more}
$Secs=%secs
#1000 {TestVar=%delnitem(TestVar,%numitems(@TestVar))}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 15: Deleting the rest}
$Secs=%secs
#1000 {TestVar=%delnitem(TestVar,%numitems(@TestVar))}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#3000 {#ADDITEM TestVar %i}
#SAY {Benchmark test 16: Using "%delnitem(variable, number)"}
$Secs=%secs
#1000 {TestVar=%delnitem(@TestVar,%numitems(@TestVar))}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 17: Same as above, deleting more}
$Secs=%secs
#1000 {TestVar=%delnitem(@TestVar,%numitems(@TestVar))}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}
#SAY {Benchmark test 18: Deleting the rest}
$Secs=%secs
#1000 {TestVar=%delnitem(@TestVar,%numitems(@TestVar))}
#SAY {Total items: %numitems(@TestVar)}
#SAY {Total time: %eval(%secs-$Secs)ms}
#SAY {%repeat("-",50)}</value>
</alias>
</cmud>
|
Both were done in an untitled session.
Charneus |
|
Last edited by charneus on Sat Jun 19, 2010 3:23 am; edited 6 times in total |
 |
|
|
GeneralStonewall
Magician
Joined: 02 Feb 2004
Posts: 364
Location: USA
|
Posted: Sat Jun 19, 2010 12:12 am
|
The issue with the 500 variables left is this:
Imagine you have a list with three items: a|b|c
And you use the following script: #3 {var = %delnitem( @var, %i)}
This is what's going to happen:
var = %delnitem( "a|b|c", 1) = "b|c"
var = %delnitem( "b|c", 2) = "b"
var = %delnitem( "b", 3) = "b"
See the issue? |
|
|
|
|
charneus
Wizard
Joined: 19 Jun 2005
Posts: 1876
Location: California
|
Posted: Sat Jun 19, 2010 12:16 am
|
Ah, you're right - I hadn't thought of that. I should have done %numitems(@TestVar) instead of %i for %delnitem, then.
Charneus |
|
|
|
|
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
Posted: Wed Jun 23, 2010 9:01 pm
|
Some comments on this:
1) Yes, using %function(item,var) is always going to be faster than %function(item,@var). This is the purpose of the "var" by-reference feature that was added. When using @var CMUD needs to make a copy of the entire table before passing it to the function.
When doing
var = %function(item,var)
CMUD (3.21) is now smart enough to notice you are assigning to the same variable that it already has a reference pointer to, so this is very fast (same speed as #additem). However
var = %function(item,@var)
is slower because not only does CMUD make a copy of @var to pass to the %function, but then it also needs to make a copy of the function result to assign to the @var variable. CMUD cannot optimize this case.
2) When doing benchmarks, you should avoid using %eval in your script at all costs. In fact, we need to get rid of this tendency for people to be using %eval. The rule in CMUD is "if you think you need %eval, you probably dont". If you find a case where (expression) doesn't work the same as %eval(expression) then you should report it as a bug.
For example, in your first set of tests near the end where you have
#1000 {#ADDITEM TestVar %eval(%i+1000)}
the result is actually twice as slow as just doing:
#1000 {#ADDITEM TestVar (%i+1000)}
So stop using %eval :)
3) When deleting items from a list, CMUD needs to do more work than when adding items. When deleting from an array, CMUD needs to move down all of the items above the value being deleted. So it is slower to delete from the start of the list than deleting from the end of the list. With a table, the same thing actually applies because CMUD also maintains an internal array of the table keys so that it can do %ismember on the table. |
|
|
|
|
Yodous
Apprentice
Joined: 23 Jun 2004
Posts: 105
Location: Poland
|
Posted: Wed Jun 23, 2010 9:28 pm
|
| Zugg wrote: |
| If you find a case where (expression) doesn't work the same as %eval(expression) then you should report it as a bug. |
And what do You say about
| Code: |
| #SA %abs((100-18)/11)+1 |
It produce "7+1" instead of 8.
Even adding %int around the %abs doesn't fix it. Only putting there %eval() fix the problem. Is it a bug?
And does %eval(smth) and %int(smth) or %abs(smth) is a big difference? You said that %eval() is very slow. So what do You say about %abs and %int etc. Those function sometimes helpes, where the %eval() was 'required' |
|
|
|
|
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
Posted: Wed Jun 23, 2010 9:56 pm
|
You need to put () around the *entire* expression! So the correct line should be:
| Code: |
| #SA (%abs((100-18)/11)+1) |
which works correctly.
%eval is slow because it calls the compiler at runtime to convert the string into an executeable expression. Other functions such as %abs, %int, etc work directly on the stack results and are all fast. So it's just %eval, %exec, #EXEC that are slower because they all call the compiler. |
|
|
|
|
charneus
Wizard
Joined: 19 Jun 2005
Posts: 1876
Location: California
|
Posted: Wed Jun 23, 2010 10:21 pm [3.20]BUG: (expression) doesn't evaluate in following conditions
|
Whoops... thought I made a new post, not a topic reply... fixed.
Charneus |
|
|
|
|
Yodous
Apprentice
Joined: 23 Jun 2004
Posts: 105
Location: Poland
|
Posted: Thu Jun 24, 2010 4:05 am
|
| Zugg wrote: |
You need to put () around the *entire* expression! So the correct line should be:
| Code: |
| #SA (%abs((100-18)/11)+1) |
which works correctly.
%eval is slow because it calls the compiler at runtime to convert the string into an executeable expression. Other functions such as %abs, %int, etc work directly on the stack results and are all fast. So it's just %eval, %exec, #EXEC that are slower because they all call the compiler. |
Ok. So please allow me to have one more question :)
Create a function 'test':
| Code: |
| #RETURN %if($statSum<=17, 0, (%abs(($statSum-18)/11)+1)) |
And try to use it #SA @test(10).. without %eval() it is returning (7+1) even if I put there "$statSum = %int($statSum)" |
|
|
|
|
GeneralStonewall
Magician
Joined: 02 Feb 2004
Posts: 364
Location: USA
|
Posted: Thu Jun 24, 2010 4:27 am
|
While not quite addressing the issue, and a bit off-topic, why not use:
| Code: |
#IF ( $statSum <= 17) {
#RETURN 0
} {
#RETURN (%abs( ( $statSum - 18) / 11) + 1)
} |
|
|
|
|
|
charneus
Wizard
Joined: 19 Jun 2005
Posts: 1876
Location: California
|
Posted: Thu Jun 24, 2010 5:04 am
|
| Yodous wrote: |
| Zugg wrote: |
You need to put () around the *entire* expression! So the correct line should be:
| Code: |
| #SA (%abs((100-18)/11)+1) |
which works correctly.
%eval is slow because it calls the compiler at runtime to convert the string into an executeable expression. Other functions such as %abs, %int, etc work directly on the stack results and are all fast. So it's just %eval, %exec, #EXEC that are slower because they all call the compiler. |
Ok. So please allow me to have one more question :)
Create a function 'test':
| Code: |
| #RETURN %if($statSum<=17, 0, (%abs(($statSum-18)/11)+1)) |
And try to use it #SA @test(10).. without %eval() it is returning (7+1) even if I put there "$statSum = %int($statSum)" |
This is a bug I reported in a new post as indicated above. It seems that %ifs are evaluating it as a string, not an expression.
Charneus |
|
|
|
|
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
Posted: Thu Jun 24, 2010 5:27 pm
|
| Quote: |
| It seems that %ifs are evaluating it as a string, not an expression |
Yep, that's exactly correct. In fact you can see this by typing "%if" on the command line and then press F1 to get the quick Reference help. You'll notice that the second and third arguments for %if are "string" types and not expressions. Added to bug list. |
|
|
|
|
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
Posted: Thu Jun 24, 2010 8:49 pm
|
Back to benchmarking:
In 3.21 I have made some new performance improvements:
1) Doing something like:
list = %additem(item, @list)
no longer makes 2 copies of the list. It makes one copy when it encounters the @list argument, but then when assigning the result of %additem, CMUD is now smart enough to realize the the value on the stack is already a copy and doesn't need to be copied again.
2) Using %delitem and #DELITEM have better optimized code for updating it's internal pointer tables. About a factor of 2 speed improvement from this.
3) When deleting an item from the END of an array (either via "#DELNITEM list %numitems(list)" or just via #DELITEM where the item happens to be the last in the list) is now much faster (doesn't need to update the internal table). This is about a factor of 10 speed improvement.
Because of 3, the fastest way to delete all items in a list loop is now either of the following lines:
| Code: |
#FORALL @list {#DELNITEM list %numitems(list)}
#FORALL @list {#CALL %delnitem( list, %numitems(list))}
|
|
|
|
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
