Zugg Software :: View topic

Posted: Tue Aug 14, 2007 1:56 am

Over the weekend I realized that I didn't finish implementing several planned features. So today, instead of doing the Package Library bug fixes, I went back and finished some other features that were half-done. Today I finished the new Trigger Testing tab. Rather than creating a full Trigger "debugger" at this time, I mainly ported the zMUD Test page and then added some new features to it.

In the new "Test Pattern" tab of the Trigger editor, the "Sample text to test the trigger" is now a drop-down combobox field. When you click the drop-down arrow, the auto-generated sample pattern is shown, along with the last 20 lines from the MUD. When you select a line from the MUD to test, or when you enter your own test pattern, the test pattern will also be saved with that trigger. The next time you look at the Test tab for the trigger, you'll see the previous sample text that you tested.

The Test screen also handles the named trigger argument patterns. So if you have ($local:subpattern) in the pattern, then the matching argument box will show "$local : subpattern" instead of "%1 : subpattern". The syntax highlighter for the trigger pattern also handles the named subpatterns now.

Took a while to get this all working properly. While getting this to work, I found an obscure bug that was preventing some optional arguments from saving. These were options that used the XML "options" field in the database (which is where the test pattern is stored for each trigger). Not sure if this bug was causing any other problems or not.

I also found a problem with the Save flag and the UseDef flag for variables that was getting messed up in some cases. This took a while to fix because of how convoluted the code was. I finally changed the variables so that the UseDef property actually uses the Save flag directly, instead of trying to maintain two options that were used for the same thing. Then I fixed it so that when loading an existing package with UseDef, it will properly convert it to the new Save flag option. The difficulty was that the Save Flag is available for all settings via the database "flag" field, whereas the UseDef was using the "opt" field which is different for each setting type.

I also wasted several hours earlier today messing around with some attempted hacks on our web site. Seems that there is a script that attempts to hack any stores running xcart. They modify their session cookie values to attempt to inject bad SQL into the store, and it looks like they are hoping that I'm dumb enough to be running PHP with "register_globals" enabled. Fortunately it doesn't look like they did any damage, but they did cause a lot of SQL errors to get into the log file. So I had to clean out the logs and then modify the PHP scripts to detect this kind of hack and abort before generating the log file error. I also added a bunch of IP addresses to the global site ban list. I really *hate* wasting my time because of hackers. Not what I wanted to work on today.

The other new feature that isn't completed that I need to work on tomorrow is the Telnet Option triggers (ATCP support). I went through and cleaned up my to-do list and consolidated it with some lists that I had written on paper, so now I have a better and more complete list of what stuff still needs to be done before the release.

But I still feel like I need a full week of pure bug fixing, so now it's looking more like next week again for the release. Yeah, I know, it's always one more week. I still have *way* too many items in the bug list to fix for 2.0, so I'm just going to have to do the best I can to identify the highest priority issues to fix.

Posted: Tue Aug 14, 2007 2:19 am

Noooooooo.... I was looking forward to taking a crack at 2.0 this weekend. While it's disappointing, I'll continue to be patient. I know you're a perfectionist and want to make sure we get a quality product.

I'm looking forward to when it's finally released.

Posted: Tue Aug 14, 2007 2:26 am

It's very stressful, because "Yes", I *am* a perfectionist, but it will *still* probably be full of bugs. But I just can't release it with some obviously half-finished features, and I really want to fix at least *some* of the more major problems in 1.34. I really do understand that a lot of people are waiting for 2.0, and sales are certainly reflecting that as well. But I still really need to emphasize that 2.0 will be a BETA version and will have plenty of problems. Too much low-level stuff has changed and it's going to cause trouble no matter how many bugs I try to fix.

Magician Joined: 09 Oct 2003 Posts: 459 Location: USA

Sadly, I've been promoted to coder status on my mud and can't really do any real playing until I finish my first project and get promoted so I get a second char.

Besides that, SCA tournament season is gearing up. Twisted Evil

Posted: Wed Aug 15, 2007 12:09 am

Today I got the basic Telnet Trigger type added. This will be the start of ATCP implementation. But in v2.0, I'm just going to have telnet triggers and not any actual ATCP code. In other words, you'll be able to use telnet triggers to implement ATCP yourself. In a future version I'll add more actual ATCP implementation and recognize the various suboption text that is sent.

The way telnet triggers work in v2.0 is that you just create a trigger and in the Type box there is a new option called "Telnet". When this is selected, the parameter value field is displayed to enter the telnet option number that you want to intercept.

When CMUD receives a telnet option request from the server, it will check to see if you have a trigger assigned to this option number. If your trigger is enabled, then CMUD will send the IAC WILL command back to the server (saying that the client supports that option). Any telnet trigger assigned to that option number with a blank pattern will also be fired.

If CMUD receives telnet option sub text via IAC SB Option Text IAC SE from the MUD, then CMUD will fire any trigger assigned to the option number that matches the specific "Text" sent from the MUD in the suboption command.

A new command called #SENDSB has been added to allow you to send suboption text back to the MUD. The format of #SENDSB is:

Wizard Joined: 14 Aug 2004 Posts: 1269

Posted: Wed Aug 15, 2007 10:16 pm

Wizard Joined: 14 Aug 2004 Posts: 1269

Is there a way to log all the raw telnet stuff? Also, I doubt I'd use it, but is there a way to send IAC WON'T in a telnet trigger? It looks like you coded it so that it sends IAC WILL automatically in a telnet trigger. I don't know much about telnet options, so I don't know if it would be useful to be able to refuse a telnet option, but I imagine it would be, at least for testing MUD server code.

Wizard Joined: 14 Aug 2004 Posts: 1269

Posted: Thu Aug 16, 2007 12:00 am

Yes, it's the last 20 unique lines. Actually, whenever anything is added to the dropdown list, it only adds it if it doesn't already exist. So, if a MUD line also matches one of your 5 saved test patterns, then it won't be added again either.

If your trigger assigned to the telnet option is disabled, then CMUD will send an IAC WONT instead of the IAC WILL. So the Enabled property of your telnet trigger controls whether the option is sent to the server, which is how you'd mostly expect it to work.

If you use the #DEBUG command:

#DEBUG test.txt test.raw

then the "test.raw" file will contain a low-level log of all network activity to and from the MUD. You can use that to debug low-level telnet stuff (that's what I use myself).

Posted: Thu Aug 16, 2007 3:17 am

Keep up the great work Zugg!!! For all the fixes already posted CMUD will be awesome.

I can't believe I'm going to say this, but while we all want CMUD don't push yourself too hard. We need your fresh and alert for the "forum explosion."

Seriously though, go at a healthy steady pace, otherwise Fang or myself will be forced to drop Chiara a line. Twisted Evil

Posted: Thu Aug 16, 2007 10:07 am

From what she's posted before, I don't think she needs any help, Tech :P

Magician Joined: 25 Oct 2000 Posts: 461 Location: USA

All sounds very encouraging!

A question on the ATCP stuff... if we can just create the telnet triggers ourselves, couldn't someone just release a Package with whatever mud's options already triggered that folks could just customize on their own? I guess I'm just unclear on why what you have already in 2.0 isn't "enough".

Posted: Thu Aug 16, 2007 12:26 pm

I'm not sure how Zugg is intending to implement ATCP. He could have an MXP-like system that creates variables automatically based on ATCP values, but as you say, someone could just write a package to do that.

I think it also has something to do with ATCP requiring authentication (see this post).

Posted: Thu Aug 16, 2007 6:16 pm

Supplementary question about the Lua implementation in CMUD 2:

I know you've changed print to be something like #say or #echo, but have you changed io.read as well? The way it normally works is that it tries to get input from stdin, the prompt that you enter scripts at in interactive mode. Will this work on the command line in CMUD 2, waiting for the next text to be typed on the command line? If not, is there a way to make it so without using some complex oninput construction? Would oninput triggers even work in Lua mode? I doubt it since nothing is being sent to the MUD.

Additionally, how would CMUD's Lua implementation handle something like print("hello" on the command line? Lua in interactive mode will wait for the bracket to be closed before it runs the code. Will CMUD do that or will it run the code immediately?

Posted: Thu Aug 16, 2007 7:04 pm

Nope, I haven't done anything with io.read. You'll need to use something like zs.prompt to call the #PROMPT command. Accessing the command line directly, or trying to wait until the next command line is typed would be too complicated. OnInput triggers do not work in Lua mode, since the line is being sent to the Lua interpreter and not to the MUD. The Lua command line mode is more for debugging Lua and not for MUD playing.

Right now, if you enter print("hello on the Lua command line, you'll get a syntax error from Lua. CMUD doesn't handle the interactive mode right now.

Posted: Thu Aug 16, 2007 10:34 pm

OK, I'm about to throw damn Windows through my window. I've wasted my whole day on stupid Windows crap.

Remember when I found the problem of updating stuff like Buttons from the background thread? You can't update the UI in a background thread, so it adds a message to the queue of the main window telling the main window to refresh a setting when it can (like a button).

So what happens if the button gets deleted in the background thread before the update message is processed? The button gets deleted in the background thread, but the main UI still has a message in it's queue to refresh the button. When this message runs, it now references a memory location that is no longer value (the button object has been freed).

I thought about using the PeekMessage API routine to remove the update message from the queue, but STUPID WINDOWS only lets you remove messages in the queue of your own thread! So even though a thread can *Post* a message to another thread (the main UI thread), it cannot *remove* a message from another thread. This is stupid! How am I supposed to remove the update message from the main queue so that it doesn't reference the object that has been removed? I can't "synchronize" the background thread with the main thread every time I want to remove a settings object...that would make things horribly inefficient.

I have no idea why Microsoft has this stupid restriction on PeekMessage. If you can *add* a message to another queue, then there should be a way to *remove* a message from another queue.

This doesn't just apply to buttons, but applies to all settings. And the main problem is with temporary triggers that are created in CMUD to do stuff like the #IF (a =~ b) pattern testing that get deleted after they are needed.

I can't believe I'm wasting my entire day on this crap. But this crap is causing CMUD to crash all over the place at weird times, so I have to figure out how to fix it. But right now, I'm completely clueless. Grrrrrrr.

Newbie Joined: 27 Oct 2005 Posts: 7

I don't especially care for you, but a friend of mine who's more generous pointed out your problem and I can sympathize in a way, so I'll offer my advice.

If you're looking for a solution that's not likely to require major rewrites, you're probably trying to solve the wrong problem. Why worry about removing the update from the queue, when you can worry about deleting the object cleanly instead?

Post a delete message. I believe you can specify custom messages with something like WM_USER+some_offset. Then the situation goes something like this:

1) Post update message
2) Post delete message
3) Queue gets processed, update runs. You may also be able to set some sort of a flag on the object to prevent this update from actually doing anything at the same time as you post the delete message, if that's what you want.
4) Queue gets processed, delete runs, and object is destroyed.

Problem solved.

Don't they tell you in any CS class that you can only delete from the front of a queue? ;-)

Posted: Thu Aug 16, 2007 10:57 pm

Talk about a kludge. Listen to what I came up with to solve this problem:

OK, so I can *add* messages to the main queue, but not delete them, right? And the problem is that there is a message in the queue that references an object that I'm about to destroy. I can't remove this message. So instead, I add *another* message to the queue. This is a "ObjectDeleted" message. The message itself doesn't do anything at all when it is executed.

Now, in the message handler for the Invalidate message, I look at the message queue for any ObjectDeleted message. I pull out all of these messages and look to see if any match the object that I'm about to Invalidate. If I find a match, then I don't call the Invalidate method of the object (because it's been deleted). Then I put all of the ObjectDeleted messages back into the queue in case there are any other Invalidate messages that still reference it.

The reason for the silly business of pulling all of the messages out of the queue and putting them back in is that the Windows PeekMessage routine for looking for a message in the queue can only look for a message with a specific "message id" value (the ID of "ObjectDeleted" in this case). PeekMessage cannot search based upon the wParam and lParam arguments to the message. So I can't look for a specific ObjectDeleted message where lParam is the object pointer. All I can do is extract *all* of the ObjectDeleted messages, then look through them for one where the Msg.lParam matches the lParam of the Invalidate message that I'm processing. Then I put them all back into the queue.

What a mess. I can't believe I'm being forced to write code like this. All because of the stupid thread restriction of PeekMessage. Geez. At least it works now...after wasting 5 hours on this.

Posted: Thu Aug 16, 2007 11:02 pm

And Heretic, don't get me started. I've been programming for 30 years. You can't just post a "delete" message when you are removing the object from a background thread. Have you ever even done any thread programming? The main UI thread only runs when the background thread goes to sleep. The background thread needs to remove the object immediately from the internal database...not when the main UI thread gets around to processing it's messages.

And you can delete from any position in a queue that you want. The PeekMessage API does exactly that...it removes a message anywhere in the queue. So don't give me that kind of answer and assume that I'm an idiot.

Posted: Thu Aug 16, 2007 11:14 pm

[Edit] I was late to the party, so you can ignore my comments.

Does the UI thread know if the message was from another thread? Maybe you can just have it check the ones that were added externally to it ( and have possibly been removed. ) If the UI thread can look ahead in it's queue, then when you remove an object you can post a message to the background UI queue. Then it can lookahead in it's queue to see if theres are remove message and handle accordingly. If the API allows you to see the contents of the UI queue then your can make it a little bit more efficient by first checking to see if there's an update on the UI queue before posting the remove message.

Not sure if any of that is feasible since I don't know the API, but it's an idea.

Newbie Joined: 27 Oct 2005 Posts: 7

Yes, I have experience with thread programming, using my own GTK MUD client.

Without knowing your code base I'll admit I could be making some invalid assumptions, but in the general case I can't see any reason why any object needs to be deleted immediately. The worst I can imagine is that some trigger or some such may be able to run in the time between the posting and processing of the message. Even if this is the case, then my third point from before resolves it. Post a delete message and set a flag on the object that marks it as disabled, so it doesn't get used any further. The object will still be deleted -very- soon, and especially as it pertains to the user interface no one would ever know the difference anyway.

My comment about the queue is entirely true. Quoting Wikipedia:

"A queue (pronounced /kuː/) is a particular kind of collection in which the entities in the collection are kept in order and the principal (or only) operations on the collection are the addition of entities to the rear terminal position and removal of entities from the front terminal position."

Try deleting from the middle of an STL queue object in C++. The windows message queue is somewhat more generous than most queues, but as soon as you hear the term "queue" you shouldn't -expect- to have that ability. I'm not assuming you're an idiot, my gripes with you are in another form entirely, which it would do neither of us any good to get into here.

In any case, I concede that without being intimately familiar with your code I can't know all possible cases here. Glad you have a solution, even if you're unhappy with it.

Posted: Thu Aug 16, 2007 11:47 pm

I only called it a "queue" because that's what Microsoft calls it: the "message queue". I'm fully aware of the definition of queue...you don't need to be smart of quote the Wiki. My point was that Microsoft already provides the PeekMessage API to remove a message anywhere in the queue, and they should have a version for removing a message from a queue that doesn't belong to the current thread. They already provide PostMessageThread to put a message into a different thread, so they should have had an equivalent PeekMessageThread routine. We shouldn't have to play these kind of "message games".

There are several cases where you want an object deleted immediately. You could have the following code:

Posted: Fri Aug 17, 2007 12:53 am

Moving on to other more interesting stuff today:

I finally finished the implementation of the #WAITFOR command. As you might recall from various forum posts, the tricky part of #WAITFOR was figuring out how the script should access any subpatterns in the pattern that it was waiting for.

Normally, %1..%99 and "named" subpatterns are implemented as local stack variables. These are handled at compile-time. But in the case of #WAITFOR, the pattern is not computed at compile time. #WAITFOR takes an arbitrary string argument for the pattern, so it needs to be tested at run-time. Since the named subpatterns are not defined at compile-time, any reference like $subpattern will fail with "unknown local variable".

The solution was pretty easy once I also worked on the #IF (a =~ pattern) expression. It has the same problem with run-time vs compile-time (again, the pattern can be a run-time string). The solution that I used with #IF was to create a function called %pat that returns the nth subpattern. For example:

Posted: Sat Aug 18, 2007 4:17 am

More bug fixes today, and more thread bugs fixed. Several of the bugs were related to some of the new features. For example, I uncovered some problems with the optimized string list and database record variable hash tables. It wasn't updating the string-value cache in some obscure cases, and it wasn't converting old database variables (like those stored in the session.db file) into the new format.

I also fixed the problem with Prompt Triggers not firing on IAC EOR or IAC GA. So that should be helpful to a lot of people. I also fixed the #MENU command to work with the mapper, and uncovered a problem where closing the map window and then reopening it could prevent any of the mapper functions from working. This might explain those cases where the mapper stops following people or stops working in other ways. Hopefully this will fix it and make the mapper more reliable.

The thread bugs were trickier to track down and fix. When CMUD 2.0 executes a command, it creates a new thread (or uses an existing thread from it's internal pool), and then tells the thread to execute the command. It then waits until the thread finishes, or suspends itself.

This is just what you want normally. For example, if you have a script like this: