Zugg Software :: View topic

Posted: Thu Apr 03, 2008 9:02 pm

OK, since we have been talking about this in a couple of threads, I thought I'd consolidate the discussion of this item. I don't really have time to work on it right now, but the need for it has been coming up more and more lately. So this will be the place to discuss how such a system might work...in case I suddenly get the urge to write it.

First, here is what I'm using right now. Right now bugs are reported via a) Forum posts, b) Email, c) crash dumps from CMUD itself, and d) bugs that I find during my own programming and testing. I have a Windows program called "MyLife Organized", which is basically a simple project management task organizer. It has a TreeView on the left showing various tasks (organized within projects/categories). You can drag/drop these tasks to rearrange their order easily, or drag them into a different category. Each task has a checkbox next to it that you can click when the task is completed (which then draws a strikeout line through the task). On the right side is a detailed Notes panel where I can put a description of the task. You can post URL links and they can be clicked on to load a URL in a browser.

So, when a bug is reported in the forums, I first either confirm that I can reproduce it, or wait for one of the Gurus to confirm it. I then click on the task list, press the Insert key to insert a new task, and enter a short title for the "bug". Then I copy/paste the procedure to reproduce the bug from the forum post into the task Notes field, and then I copy/paste the URL for the forum topic itself.

All of our Support email and the crash dumps from CMUD go into an online helpdesk system called HelpSpot. When I get a bug report via email, I do something similar to the above procedure for a forum post, except the URL that I paste into the Notes field points to the HelpSpot request.

Bugs that I discover myself during testing are entered the same way, except that there isn't any URL to paste (unless I create my own topic on the forum to discuss the issue).

To be honest, bugs that are reported via the CMUD Crash Dump system are currently mostly ignored. Mainly because there are too many of them for me to process easily. What the crash dump database in HelpSpot *does* do for it is to group the crashes by the CRC of the crash ID. So if several people submit a report for the same crash, those reports get grouped together. This allows me to see if there is a particular crash that is happening to a lot of people. But mostly that doesn't happen much anymore (it was great in the early betas of CMUD, but less useful now).

If someone submits a crash dump specifically related to an existing bug or forum post, then I add the link for that crash to the Notes field in the task list so that I can easily display the crash dump to see the specific stack traceback of the bug.

The main problem with crash dumps is that the vast majority of them a) do not contain any information on how the crash was generated, and b) are not easily reproduced. Also, since the crash dumps reference the stack traceback of the CMUD code, once a new version of CMUD is released, the dumps from previous versions become much less useful. In most cases, the crash dumps already relate to known bugs. But there might definitely be some cases where I am ignoring crash dumps from bugs that haven't been reported in the forums.

To give some basic statistics, I have 344 items in my task list, of which typically 30 are in the "fix for next version" category, and about 200 are in the "cannot confirm" or "needs to be confirmed" category. For example, any tasks that were in the list from the 1.34 public version or earlier are in the "needs to be confirmed" category since there have been so many changes since then. I need to spend a couple of days at some point going through this entire list and getting rid of stuff that are no longer bugs or problems.

In the crash dump system, there are thousands of entries. These never get deleted or changed when a bug is fixed. So most of these entries are worthless. In fact, even now and then I just go in and delete crashes from versions a long time ago. For example, there isn't anything in the database from before v1.34. There is just too much data and no good way to keep it organized.

The key is to come up with a way to identify a particular bug. In the crash dump system we get a unique CRC value for each bug. A specific bug "topic" might be associated with multiple CRC values. There is a tremendous amount of work involved in receiving the automated crash dump and then filing them into the proper bug "topic" in some other database (or my task manager for example). This is currently one of the big problems with the existing system.

The other problem with the current system is that I don't get any archive of past bugs. I use the checkbox in the task manager to mark a bug as "fixed for the next version". When I release a new version, I copy each of these topics into the Version History for the new version. I often modify the topic subjects at this time to make the version history a bit more readable. When I have completed the manual editing of the version history, I then delete the tasks in the task list that are marked as fixed. This cleans up the task list so that it's ready for work on the next version. But the downside is that there is no archiving of topics. Once a bug is deleted, it is gone from the task list. If the bug wasn't really fixed, then I need to recreate it for the next version. It helps if there was a forum post containing the history of the bug that I can link to. Otherwise, any Notes that I made on the previous version of the bug get lost. This can be annoying sometimes.

The current bug list is not available to the public (because it's sitting in this local MyLife Organized program). So Taz (one of our Gurus) has taken it upon himself to create a thread in the CMUD Beta Forum to track a list of known bugs. He has a link to each thread that discusses a particular bug. Each time I release a new version, he tries to go through and determine which entries have been fixed. When someone posts a new bug in the forum, he tries to determine if it's the same as an existing thread, or if it's a new issue to add to his list. This is a *very* manual process. It doesn't necessarily reflect my own bug list (although I try to ensure that any issue on his list *is* in my own list). But it's the only way we currently have to give people a list of existing bugs.

On the positive side, I like the MyLife Organized task list because it is always running on my computer (minimizes to the system tray) so it is always just a single-click away. This makes it very quick and easy to copy a forum post into the bug list, or if I think of something while in the shower I can easily come in and add it to the bug/feature list. The interface is simple and easy. I just press the Insert key to add a new task, then type the subject line. I don't have to bring it up in a web browser or log into a web page, or wait for web pages to reload, or any of that. It's just fast, simple, and easy. And that is what keeps me using it.

That's a description of the current procedure I use for tracking bugs and some of the problems associated with it. Since this post is already long, I will create a followup to this which will describe the desired features of a new bug tracking system.

Posted: Thu Apr 03, 2008 9:17 pm

Zugg,

There's a bug tracker called FogBugz that I use. It is made by the company owned by the annoying fella, Joel Spolsky. It comes in 3 versions, free hosted single user, paid hosted, and server install (Windows and *nix). I know that you can integrate it with Subversion, so that it automatically synchronizes tasks with checkins. I've heard, but not confirmed, that there are integration paths between HelpSpot and FogBugz. The owner of HelpSpot hangs out at Joel's forums. I see him posting on "Business Of Software" every once in a while. I see you there sometimes. Here's the link to that forum:

http://discuss.joelonsoftware.com/default.asp?pg=pgDiscussTopics&ixDiscussGroup=5

There url for FogBugz:

http://fogcreek.com/FogBugz/

Posted: Thu Apr 03, 2008 9:22 pm

Ahhh, I see that you already looked at FogBugz on the other thread. There's another bug tracker that I used called Axosoft OnTime. It's heavier than FogBugz, but it has a lot more features.

http://www.axosoft.com/

I'll be interested in what other people have to say about this.

Posted: Thu Apr 03, 2008 9:24 pm

Now that I have described the current system that I use to track bugs, let's start talking about how a new crash dump system could potentially improve this situation. First, here are the basic requirements:

1) Must be quick and easy to create new bugs in the database (especially for me). Any web-based interface would need to be very fast and probably AJAX enabled to minimize excessive page loading.

2) Needs to have an easy way to mark a bug as "fixed". Just a checkbox would be nice. If there was a global field that indicated the current version that I was working on, that version could be automatically added to the bug to indicate which version the bug was fixed in. I really do *not* want to manually enter the version number that a bug was fixed in for each bug that I fix.

3) Must have a way to categorize bugs. For example "critical bug", "easy to fix for the next version", "feature request", "cannot be confirmed", "needs to be confirmed", etc. Some of these concepts are pretty nebulous, and a single bug might have multiple "tags". In addition, it might be nice to have additional category "tags" such as "mapper", "parser", "settings editor", "docking", etc to indicate which module the bug relates to (my current system doesn't have that).

4) Need to be able to associate multiple crash dump CRC values with a particular bug report. I will need a way to add links to the bug so that clicking on a link takes me to that specific crash dump (whether it's in HelpSpot, or in a separate table of the bug tracking database). When a new crash dump is received, it needs to be added to any existing bug topic that matches the CRC. Some sort of "new" indicator needs to be shown to indicate that a new crash dump has been received for that bug report.

5) Needs to have a way to easily add a Forum post to the bug list. Preferably there would be a button in the forum topic display like "Add this to bug database". Once a topic has been added to the bug database, the button would change to a link to indicate that the post has already been added to the bug list. In fact, it seems like the forum itself could serve as the "comment" database for the bug tracker. Similar to how we use the forums for the "comments" in the Knowledge Base. If there are multiple places to comment on a bug, it just gets confusing for users and hard to consolidate the information.

6) Obviously there needs to be a way to view an overall list of bugs. Ideally I could add information to each bug about how important it is, and how easy I think it is to fix. Then the list could dynamically prioritize the bugs by showing the most important and easiest to fix bugs first on the list. I also need to be able to "hide" certain categories (such as "needs to be confirmed") so that I don't need to stare at them all of the time. It's important to filter the large amount of information so that I don't get overwhelmed.

7) When a bug is marked as fixed and the version is saved, then it should be possible to automatically create the Version History. In order to maintain the current history of bug fixes, it would be nice to have a way to process the current Version History so that it automatically creates bug tracking topics for these old bugs, and marks them as fixed in the appropriate version. Then the entire Version History page could be made completely automatted and database driven. This would decrease the work needed to release a new version, and also give users a "realtime" look at what bugs are already fixed for the next version. The next version could even have an "ETA" field that indicates my best guess at when it will be released. Then when people ask "when will the next version be released" you can just point to the version history page.

8) There needs to be a way to consolidate multiple bug reports into a single entry. It's common for more than one person to report the same bug. And once that is determined, the issues need to be merged into a single bug report.

9) Ideally each bug would have an automatic "status", such as "new bug, unconfirmed", and "new bug, confirmed". When a new user posts a bug report, it is new and unconfirmed. Either myself or one of the Gurus can confirm the bug (and indicate which version it was confirmed in). Then it is marked as "confirmed". Then, myself or a Guru determines if this is a duplicate bug report or a new bug. If it is a duplicate, then it is merged with an existing bug topic. In either case, the bug becomes "active". When I fix a bug it is marked as complete. We might even have a status of "confirmed fix" which can be used post-release to indicate that the bug was really fixed as intended. With this kind of system, I could focus my attention on the "active" bugs and allow the Gurus to help with confirmation of bugs and dealing with potential duplicates.

10) The system needs user access levels. An overall Admin (me), along with Guru access (read,write,delete,move,merge,etc). Individual users should have read access and the ability to add a new bug post, and to add comments about their bug report (but not edit or delete it).

11) The web-based or database part of this tracker needs to run on our existing Linux server, and be written either in PHP or Ruby on Rails. It obviously needs to work in both Internet Explorer and Firefox (which is what I use). It needs to work with the MySQL database server.

Feel free to post your ideas or suggestions. If I think of additional requirements, I'll also post them. But I think that about covers it.

Posted: Thu Apr 03, 2008 9:27 pm

I have already looked at FogBugZ in the past. And like most systems, I found it overly complicated. If you look at some of the features that I am interested in, you can see that a fair amount of customization would be needed. And I just didn't find it quick and easy to use (at least in the limited demos that I did). Because of some of my specific requirements such a forum integration, I'm not sure I'll find any existing commercial solution for this. And in most cases the commercial system have a lot of other "bloat" and stuff that I'd never use that just clutters the system. Maybe I'm wrong and I need to take another look, but that is what I remember from when I was evaluating FogBugZ vs HelpSpot.

Posted: Thu Apr 03, 2008 9:43 pm

My comment are Axosoft OnTime are:

1) They make good demo movies.
2) It is deceivingly expensive

For example, they say that a "single-user" is free. But when you look at the web customer portal, "free" only applies to FIVE registered users. I need to be able to associate feedback with *any* forum user on this site. I don't want them all to be "anonymous". Then start looking at the prices when the number of users goes up...geez.

They don't have any implementation/system requirements information that I could find. While I like the idea of a local Windows app and a separate "Web portal" for customers that access the same database, they don't give the requirements on the web side. What database does it use? It looks an awful lot like Microsoft SQL Server to me. Needs to be MySQL. Does the web stuff run on Linux? Or is it all written in ASP.NET?

So yes, it might fit some of my needs. But it would probably take as long to customize their system as it would for me to write my own system that interfaces with the forums since I don't need a huge fraction of their feature set. And I really can't imagine that it's going to handle my needs for "free" and once you get out of their "free" box, it gets *very* expensive.

edited: Finally found the system requirements on their Download page. Confirmed that it requires Microsoft SQL Server and requires IIS. Slicertool: you must know that we use MySQL and Apache/Linux here, right? There is no way I am going to run an IIS server no matter how good the software is. Been there...done that...don't ever want to go back. Sorry.

Posted: Thu Apr 03, 2008 10:02 pm

OK, I also have to say that I went back over to Joel's FogBugZ site just to see the progress on that and Joel does the *BEST* demo movies. You'll have to go run his demo at http://media.fogcreek.com/fogcreek.com/FogBugz/60movie/60movie.html to see what I mean, but it's hilarious. And I'll have to spend a bit more time there because it has come a *long* way since I last saw it.

Edited: Now I remember why I didn't want to use FogBugZ either. Again, it's a cost issue. Yeah, the demo looks great...bunch of great features. It even has a Wiki! But again, look at the user pricing details. Let's say that I wanted to give Guru's write access to the bug database and Wiki. Even if it's just 4 Gurus, please me (admin free) plus Chiara. That's 5 users, and that's $1000 (not counting the $200 a year for maintenance) Still too expensive for me to handle.

Also, I sure wish he'd rewrite his Forum code. I've always hated how his forums look. Not useful at all for a support forum where people need to post code snippets, etc. So I'd still be needing to use our own forums and then figure out how to integrate them.

SubAdmin Joined: 18 Nov 2001 Posts: 5182

I have the sneaking suspicion that this will likely be something you would have to write yourself. I am sure there are other devlopers out there that have similar needs, but I doubt there are any that also has your breadth of languages. I think it is reasonably possible that if your design and write a system that integrates with the software currently being used, you might be able sell it to enough other programmers to offset your time.

Essentially it is something you have to view as how much time you would spend writing it, and how much time it would actually save you on bug tracking per day. Then you can figure how many days before it reaches a break even. If you were to sell it, even to just 1 other developer, then you can turn whatever dollar amount into a pay per day figure and calculate a new break even cost.

Overall, I believe it would make sense for you to write an excellent system that meets all of your needs and desires. In the long term it profits you; in the short term the time to write it is extremely expensive. I am basing that short term view on what you have stated most consistently brings in customers, new versions of CMud. I do think it is possible to sell any bug tracking system you write; I would just expect the sales to be very small. No matter what I guess the actual numbers to be I really can't see this as something more then an amusement project until it reaches a near production quality state.

Posted: Fri Apr 04, 2008 6:56 am

Hmm, something I need to look at in more detail is RedMine which is a free issue-tracking system written in Ruby on Rails. Looks like it has a lot of nice features, including a decent forum and wiki. I try to look beyond some of the "style/color" issues, since it should be easy to change the templates for it to anything I wanted (something that is really easy in Rails).

I started looking at some of the detailed bug reports of their own system, and it looks like they have a rich markup language for code, and also handle file attachments (which would also be a nice feature). Lots of ways to make related topics too. Anyway, it's late tonight, so maybe I'll look it over more this weekend. Too bad their local demo is currently down.

Posted: Fri Apr 04, 2008 10:28 am

There are so many bug trackers, Viji, I wonder if there'd be an appreciable market for any such software, especially given that it's outside Zugg's market. Supporting it commercially would probably take up a chunk of time, too.

Posted: Fri Apr 04, 2008 4:33 pm

Yeah, I doubt I'd actively sell anything like this. Selling it means supporting it, and that's just too distracting. That's the path I went down with zApp (build an application system for a new product (emobius) and then sell the development system). Sure, if someone liked what I did and wanted to give me money for the PHP/Ruby/Whatever source code and didn't want any support in return, then I'd be happy to make the trade. But that's not likely to happen.

And as Fang mentioned, and I confirmed in my research last night, there are a *lot* of bug/issue tracking systems. Probably a lot of junk, but it would take time to go through it, so even marketing such a system would take time and money that I don't have.

Honestly, I'd be happy just to get something that helped me personally and wouldn't worry about trying to make it "commercial quality". I'm also happy taking something free like RedMine and then modifying it to suit my needs. While I haven't looked at the RedMine code specifically yet, I am finding that Ruby/Rails code tends to be very easy for me to figure out, even with little documentation. Maybe it's just because I already think in an OOP.

Also, another realization that I had last night when working on some of the MyMuds design was that I don't really have to completely choose between PHP and RoR. As long as I write the RoR stuff to use our existing database, then both can still coexist on the same server. For example, the current forums can continue as they are, but the MyMuds forum interface can still be written in RoR. Then, over time as the RoR interface to the forums improve, maybe the Zuggsoft side will get changed over too. But both systems should be able to display the same forums/topics/posts by just accessing the existing database.

On the bug tracking side, that means that the main bug tracking interface could be written in RoR, but I could still interface to that via PHP in the existing forums, and even write a standalone Delphi program for my own fast access to the bug tracking database. I really have a lot of different options with this.

SubAdmin Joined: 18 Nov 2001 Posts: 5182

Posted: Sat Apr 05, 2008 12:38 am

Posted: Mon Apr 21, 2008 5:48 pm

If you like the system that you've got, and you want to expand it, why not look into the API/synching tools for MyLife Organized? It appears they have at least some of those. Then you could keep your interface, and try and bolt on one of the many web-based systems out there?

BTW for more suggestions: Bugzilla (bugzilla.org), RT (bestpractical.com).

C.