Zugg Software :: View topic - Temporary solution to the zMapper issue with CMUD 3.x map format

Posted: Fri Nov 13, 2009 6:10 pm

As we all know, the CMUD 3.x map database format is different from the format used in CMUD 2.x, zMUD, and zMapper. This prevents people using CMUD 3.x from exchanging their map data with other players, and also prevents zMapper users from customizing their CMUD 3.x map.

The *ultimate* solution to this problem is for me to write CMapper...an update to zMapper that uses the new SQLITE database format. However, that is a big project and isn't going to happen soon. The main reason is that, like zMUD, zMapper does not compile within the newer versions of Delphi that I'm using and also doesn't work properly on Vista, Win7, etc (again, like zMUD). So rewriting zMapper to get it to compile in the newer Delphi compiler, fixing the Vista/Win7 issues, dealing with 3rd party components that are no longer supported (used in the Shape Editor in zMapper), and fixing various other bugs becomes a rather daunting project.

I was going to just ignore this because my personal priority right now is getting the TeSSH client released and selling (getting the web site up, converting help files, etc). However, the more I thought about this, the more I realized that I just couldn't release a PUBLIC version of CMUD 3.x with this mapper incompatibility problem.

So, as a temporary solution to this problem, I am going to write a Conversion Program for the mapper. This program will be a standalone, free download that will allow you to convert TO AND FROM the old mapper MDB and the new mapper DBM database formats. This solution will work as long as no new database fields are added to the new mapper. Right now in v3.x, this isn't a problem. Once we get into Phase 2 and Phase 3 of the mapper rewrite, then this method might not work anymore, or you might lose features during the conversion. But for now it should work fine.

It's a bit inconvenient to convert a map from CMUD 3.x back to zMapper, then edit it with zMapper, and then convert it back to CMUD 3.x format. But at least it will make it possible to still use zMapper in some way.

Posted: Fri Nov 13, 2009 6:15 pm

I am running Vista, soon to be Windows 7, so I'll have to wait for CMapper no matter what happens but this is great for all the XP folks who are using zMapper. This is sort of an ancillary question but, is it true that customers who purchased CMud Pro will be able to use CMapper features without the additional purchase of CMapper when it does come out? I read something about this on the forums and was curious if this was still the case.

Enchanter Joined: 09 Sep 2007 Posts: 605

Posted: Fri Nov 13, 2009 7:49 pm

Actually, writing the conversion program will only take a week total. I already have the routines to convert from MDAC/ADO to SQLITE and the routines to convert in the reverse direction are very straightforward. It's a very simple program.

TeSSH Public is expected for January. CMUD Public maybe a week or so later. Then there is more work on the TeSSH web site and the conversion of the TeSSH web site to become the new Zuggsoft web site. Then probably the Phase 2 of the mapper, which is more involved with the mapper Configuration and getting the config stored into the Shared Package Library. So I'm not expecting any big changes to the map database format at that time. Then I'll be prioritizing Phase 3 of the mapper with the rewrite of the Database Module. It could easily be a YEAR from now before the conversion program would stop working.

Posted: Sat Nov 14, 2009 4:14 pm

You might want to also consider cleaning up the mapper data at the same time. I found that my mapper file had dozens to hundred of invalid references and orphaned items that could be cleaned up.

Posted: Sat Nov 14, 2009 7:27 pm

No, I probably won't be doing that Reed. That's the kind of stuff that can easily take the MONTH to debug that Arde was talking about. All I will be doing is a straight database format conversion (SQLite <--> ADO/MDAC). It will be based purely on the raw table data. There won't be any "smarts" about the actual data itself.

Cleaning up the map data like you suggest is extremely complex and can too easily cause more harm than good if it isn't done correctly. That's beyond this particular temporary solution for map compatibility.

My analogy for this would be like asking for the Windows upgrade to not just copy/convert your system Registry but for it to also go through and "Clean" the registry. No. With an upgrade you have two choices: Migrate the existing Registry, or start from scratch with a clean install. Cleaning the registry is a huge task that is way beyond what the upgrade process can handle. Dealing with cleaning a map database is similar (although not nearly as complex obviously)

Enchanter Joined: 09 Sep 2007 Posts: 605

May be during Phase 2 some, if not all, ideas from ReedN' and bothkill' awesome script could be implemented in CMUD?

Posted: Sun Nov 15, 2009 1:54 am

I'll admit I got it wrong the first few times, but once I got the algorithms ironed out it worked fairly well. I've been using my cleaned map for probably about 6 months now and I haven't run into any issues with it. But I see your point, it might be beyond the scope of what you're planning on doing.

I've liked the results immensely. I went through and reordered so that my zones are all contiguous ranges and I was able to reclaim all my lower range unused numbers (from deleted rooms) so I don't have to type a huge number when dealing with room numbers.

Posted: Mon Nov 16, 2009 5:27 pm

ReedN: Email me your script or a link to it so that I can put it into my issue-tracking database for possible implementation in the future.

Posted: Tue Nov 17, 2009 2:18 am

Sure, I sent it to 'sales@zuggsoft.com'.

Posted: Tue Nov 17, 2009 5:48 pm

Got it, thanks!

Wanderer Joined: 12 Jul 2002 Posts: 73

Zugg, has this program been released ?

Would luv to be able ti import my zMUD / zMAP maps I found in my backup in CMUD (see topic: http://forums.zuggsoft.com/forums/viewtopic.php?t=34388 )

BR,
Boris

Wizard Joined: 22 Mar 2007 Posts: 2320

Knuffel, nothing new has been released in a couple months. Also, nothing here is going to help you with the problem you described in the other forum. The problem you have is that your map is from Zmud pre-7.20. The problem Zugg is talking about here is that Zmapper will work on Cmud 2.37 maps, but not on Cmud 3.xx maps.

Wanderer Joined: 12 Jul 2002 Posts: 73

Rahab, OK Thanks, lets continue in the other thread then :-(

Posted: Mon Dec 21, 2009 5:25 pm

The next version along with the map converter will be released towards the end of January. I didn't want to release anything before the holiday break when I wouldn't be able to support it.

Posted: Thu Jan 21, 2010 12:59 am

Started working on this today. While it's a bit harder than I initially thought, I'll still be done by the end of the week with the basic program.

Man, I really wish SQL had standardized the data format names! When creating tables with the SQL CREATE TABLE command, it's a real pain that the data type names are different between SQLite and ADO (and any other database driver). TEXT vs MEMO, BLOB vs BINARY, BOOLEAN vs BIT, etc. It makes writing a general-purpose database conversion a royal PITA! And I won't even go into the difficulty of extracting the Schema data to determine what the Indexes for the database are and how to convert that information into proper "CREATE INDEX" SQL commands. Blah!

As I mentioned, the first version of this MapConversion program will be very simple and will just convert To/From the old ADO *.MDB format and the new SQLite *.dbm format.

In the future, I've got a lot of plans for this little program. I plan to do a Pro version (yes, the Pro version will cost a small amount) which will let you control exactly which Zones to copy, and also let you Overwrite existing zones, or "Merge" zones from two different files together. The Merge operation is rather complex, so I'm not doing this in the first version. But the Merge operation will eventually allow multiple people mapping the same zone of a MUD to merge their results. Gets tricky when merging data that has different Room Types or Exit Types or Shapes/Icons, but it's a solveable problem once I get some time to work on it. I'll also be adding various tools for correcting corrupted data as per Reed's script.

The Pro version will be free for any zMapper user but will probably cost $9.95 for other users. Remember that there will always be a FREE version that will perform the basic map conversion, converting entire map files.

That's the latest. Should be done with this tool on Friday. I'll be spending next week on other bug fixes and hope for a Beta release a week from Friday.

Posted: Thu Jan 21, 2010 3:19 pm

YEY!

Atleast your only converting SQLITE and ADO, no mysql mssql or any others, yet.... still would have thought that that your Delphi DB wrapper would automatically convert corresponding DB types to the same Delphi type.. but that doesn't help with your CREATE TABLE commands :(

Your future plans sound reallly cool!

Magician Joined: 17 Apr 2005 Posts: 496

Sweet. :)

Posted: Thu Jan 21, 2010 5:56 pm

Posted: Thu Jan 21, 2010 6:34 pm

hmm as a thought zugg... why do you have to re-create a NEW map DB file every time, ship the program with a BLANK template, when the user goes to export it COPY & RENAME and then fill the file with info.

My comment was aimed not so much at the mapper but at the DB rewrite, that will be fun cause you won't have hardcoded DB to generate, but a dynamic db that users will be changing :(.

And i don't really know how much longer fixing map files will take, fixing the map files is CPU intensive while reading, creating, and updating is all disk intensive. I did create a LUA script to copy zones, read info into a table, generated a oldmapid >> newmapid adjusted id's and inserted. With a little tweaking i bet you could get it pretty fast.

Posted: Thu Jan 21, 2010 7:14 pm

In the perl script I created it took about 4 hours the first time I ran it, then once I turned off some of the safety mechanisms that were slowing it down it took about 30 seconds.

Posted: Thu Jan 21, 2010 7:22 pm

Hmm as an after thought...

I see NO reason to renumber/order rooms, or zones for that matter other than ERROR CORRECTION, removing all the blank spaces where you deleted rooms imho is a waste of programing time and cpu time. A computer will add, think, allocate memory, delete memory ect for the number 1 just as fast as 2,123,456,789, they're BOTH 32 bit signed INTs.

Zones are a little different matter since there is the ZONE tab displays them in order by ZONEID (reverse order). That's not necessarily a bad thing... if your always using the zone ZOO you don't want it at the bottom, maybe an option in the cmud mapper to display zones in alphabetical order?

But going back to this nice new app... would re numbered roomnum's be a bad thing probably not, there are no really really good reason to use roomnum over a roomid.. besides i'm too lazy or i_can't_come_up_with_a_nice_unique_name_for_this_room_without_it_becoming_so_dam_long.

anyway... my 2 cents of sidetracking thought.

Posted: Thu Jan 21, 2010 10:50 pm

No, it's an issue with the internal database structure. Each room has an ID value (Room Key) which is a unique auto-increment field. Each time you add a room to a map, this field is incremented. The Exit Link between rooms has a RoomTo and RoomFrom fields that point to the ID of the room records.

Now imaging you are loading a zone from an existing map file into a blank new map. In a blank new map, the first room created with have a Room ID of 1. But in the existing map, the first room in the zone you are copying has some other ID value (call it X). When the exit records for that room are copied into the new map file, all references to room X need to be replaced with the new room ID of 1.

Or, think of it another way. Imagine loading that same zone into a file that already has thousands of rooms in it. Let's say the ID of the room in the existing map (X) has a value of 100. There is already a room 100 in the existing big map that you are merging into. Again, you have to patch all references to room 100 with the new room ID value in the new map (which might be 1000).

In any case, all objects in the map database are like this. They all reference internal ID values that might conflict with existing IDs. Same is true for the Meta data, such as Room Type, Exit Type, Icons, and Shapes. All of those links have to be recreated when just copying a single zone into an existing map file.

Remember the distinction between the Room ID (key), which is the internal database key, and the Room "vNum" which is a user-defined or MUD-defined number that usually relates back to the real MUD map, but might only be unique within the current area/zone. I am talking about the database ID here, not the vNum.

Posted: Thu Jan 21, 2010 11:02 pm

Oh, and today I learned why the ADO conversion is SO SLOW in some cases. Most of the fields in the old ADO MDB database are "compressed", which means they are ASCII strings and not Unicode strings. However, there are a couple of MEMO/TEXT fields that were not marked as "compressed", so they can store Unicode. The SCRIPT field is a good example of this kind of field.

When Zeoslib comes across an ADO field like this, it gets converted internally into a Delphi WideString field, which is a Unicode string of any length. So far, so good. But the Delphi DataSet component for handling the database has a "MaxSize" value associated with each database field. Since the Unicode WideString is not considered a "Blob" or "Memo" field by Delphi, this MaxSize is returned as 536870910. When Delphi tries to read this field from the database, it first creates a buffer of size "MaxSize".

Now, most fields have null or very short string values. But Delphi is allocating an internal buffer of MaxSize each time it attempts to read the value. This is a HUGE overhead. Imagine allocating 536870910 bytes of buffer just to read a NULL string value!!

While I appreciate the idea of ZeosLib to treat TEXT/MEMO fields as normal strings instead of forcing us to deal with reading Blobs, this overhead is just too huge. So I need to find a way to work around it. I can detect a NULL field value without causing the buffer to be allocated, but for records that have a short string value, Delphi still allocates the huge buffer. So I need to figure out how to override the Delphi behavior and when I see a huge Field.MaxSize value like this, do my own buffer allocations.

I don't think the existing ADO->Sqlite map conversion routine in CMUD has this problem because it is using the specific ADODataset objects instead of the more general-purpose ZeosLib dataset objects for the ADO MDB database. I switched to the Zeoslib routines in the new MapConvert program so that it could handle either ADO or SQLite in the same conversion routines.

Seems like nasty problems like this are always happening.

Posted: Thu Jan 21, 2010 11:56 pm

Wyrm: I guess the benefit of renumbering is in the eye of the beholder. For me I often scry someone and then need to walk to their room. For me it is much more convenient to type #walk 235 instead of #walk 393894. I also reordered the numbering so that all the rooms within a zone are in the same range. That way when I work with other rooms in the zone I know they will be close in number to the room I'm working with. Additionally since Cmud doesn't (or didn't) keep the zone sorting preference, I was able to sort them myself into the order I wanted and then I didn't have to worry about doing it every time I started Cmud.