mattmecham.com

Persistent Topic Marking

Persisent topic marking has been something that's often been requested across various forums. Until now, most systems relied on cookies to mimic a reliable system. To a point, this is fine but the usual problems with cookies plague the system; unreliable storage, easy corruption and them not being shared across computers.

IPB 2.0.0 introduced a semi-DB method where topics that you've read were added to a database but this didn't do anything for forum markers which still relied on a mixture of cookies and your last visit date (which itself is problematic).

IPB 2.1 has a system where both the read topics and forum marker information is stored in the database.

Now, before you run for the hills shouting "bloat, inefficient, it'll make IPB too slow!" - you might want to understand how the system works.

The system is based around Remco's wonderful system for the Invision Blog software. It's hard to convert the code into English, so I won't bother. Suffice to say that it doesn't take up much more resources than IPB 2.0.0s system and in fact, it's probably a touch more efficient. Infrequently it needs to determine how many topics are unread in the forum and it runs a COUNT(*) query on the topics table. This is all indexed so full table scans, temporary tables and filesorts are all avoided. It doesn't do this on every page view and I estimate that it will only do it for 10% of the topics you'll read. It avoids the need to query the database for topics you've read which avoids a rather nasty 'range' query. I've also removed the filesort operation when in the forum view, so overall IPB 2.1 should be more efficient than IPB 2.0.3 even with this new marking system.

This system also means that you don't have to view a forum first to get your unread topics synchronised. If there was only one new post/topic in the forum and you went straight to it from the board index and went back to the board index, the forum marker will be marked as read.

Naturally, for those who wish to squeeze every ounce of performance out of their board, this system can be switched off and it'll go back to using cookies.

In other news, a less exciting use of Ajax has been implemented in the ACP. It's rather basic at the moment, but it's a nice little touch to avoid large page refreshes. I've included a little movie of this feature in action. I might make the final implementation much more fancy with each row written with a cross or other icon and convert it to a tick when it's complete.

I think that there's a fair bit of bandwidth to be saved using this method. Oh, and to that crazy guy over on the vB forums who is determined to prove that anything with Ajax in it is massively bloated and likely to crash your browser. Wise up, dude. It's just a few lines of javascript, it's not loading a java sized program into your memory.

I've also added a little movie of IPB 2.1's wonderful new inline editing and moderation. It just doesn't get easier than this to manage your forums.

ACP Ajax refresh (Quicktime .mov file, 1mb)
In-line moderation and editing (Quicktime .mov file, 4.8mb)
Working in the ACP (Quicktime .mov file, 5.1mb)

Well, that was a difficult day

I have decided (again) that web development can be a huge time waster. So much time is wasted chasing your tail on difficult to reproduce bugs or on bugs which are hard to debug. It gets quickly frustrating.

I had two 'biggies' on my schedule for today.

The first was a very simple report: "When I load the rich text editor and then hit 'back' in my browser, IE 6 crashes".

That's the kind of bug which will either be fixed in a few seconds, or will take forever. Unfortunately for me, it was the latter. Firstly, I googled on 'ie6 crash wysiwyg' which brought up a few interesting links about a recent security update. I fired up my PC laptop and attempted to recreate the problem but couldn't.
I ran Windows Update and when the PC restarted, I was able to recreate the problem every single time. A few wild goosechases on security reports went nowhere. I started commenting out blocks of code in the rich text editor to try and narrow the problem down and eventually traced it to the rather innocent looking rtewindow.document.designMode = "on". Remove that  and  there is no crash when you hit 'back' but also there is no working RTE. I did a bit of fiddling and noticed that document.contentEditable = true; didn't suffer from this problem. Unfortunately it would mean rewriting a large portion of the IE/RTE code. Something that I didn't want to do.
Armed with the cause of the problem, I did some more google research and found an interesting article on IE6 and DOCTYPES; especially when loading different pages into iframes.
Getting a bit warmer now. I had another look at the HTML it produces and found the culprit. A single blank HTML file which is loaded into the RTE to prime the colour picker had a dodgy DOCTYPE. I fixed that and the crash was gone!

The second bug was a little more complex. Mozilla (Firefox, etc) fails to honour the "meta" redirect now and again. Google came through again with a link to a Mozilla bug which found the problem to be with the gecko engine and nothing to do with IPB. When 'midas' is loaded, subsequent meta-redirects fail.
I decided to add a bit of javascript to the redirect page to force a setTimeout function if the user is using an affected version of gecko browser.

The moral of the story? Google is the best way to fix bugs.

(?:Dis)?Organi(?:z|s)ed Chaos

So, it's the day after the afternoon before. In that time, we upgraded our corporate forums, suffered a DDoS attack, send out 12,000 emails, fixed 50 odd bugs and even found time to write in a legacy mode to enable the blog and gallery to work without modification.

The upgrade to 2.1 seems to have gone fairly well. It was a bumpy ride for the first few hours and the odd (hundred) bugs here and there did their bit to add to the overall chaos. Ever wondered what it's like to upgrade a relatively busy forum to a largely untested release? It goes something like this:

1:00pm: Turn company forums offline. Add in a redirect to a basic index.html file.
1:30pm: New file uploads complete
1:50pm: MySQL updates done surprisingly quickly. Note to self: MySQL 4 is so much better than MySQL 3.
2:00pm: Hack index.php to allow access for myself and the US office IP address.
2:10pm: Various staff members AIM me with bugs they've found.
2:14pm: It's apparently turned into a competition.
2:22pm: Fixed the first round of bugs (parse errors, the biggies)
2:34pm: AIM is lit up like a christmas tree
2:38pm: Fixed some more bugs. Start roughing out an announcement
2:43pm: AIM is lit up like the Empire State Building
2:44pm: Few more bugs are fixed.
3:00pm: Finished announcement. Load Safari to post it (saves logging out of Mozilla)
3:01pm: Notice the standard editor doesn't work at all with Safari.
3:20pm: It does now.
3:22pm: MySQL error when posting from Safari
3:30pm: Get a message from CTerry via Stewart 'Stop slacking'
3:31pm: Motivated, I submit the announcement again
3:32pm: Remove IP address filter and turn board back online
3:33pm: Charles starts sending emails to our subscribers
3:45pm: Can't load a forum. This can't be good. ssh [email protected]....
3:50pm: MySQL locks up, 156 locked processes. Lindy?
3:51pm: MySQL restarted.
4:10pm: Ah, that topic table index? Yeah. I'll go add that now.
4:15pm: MySQL locks up again.
4:20pm: Oh yes, LOW PRIORITY was removed from the topic counter update
4:25pm: Server restarted
4:26pm: Things are much improved. Nearly 400 users active and load @ 0.3 per CPU
4:30pm: Initial feedback comes rolling in. Bug reports are posted everywhere.
4:30pm - 6:00pm: Crazy bug fixing period.

Or something like that, anyway.

I've spent the better part of today bug fixing and fiddling with bits of javascript that didn't work on Opera. Overall feedback has been good. Apparently some people have taken offence to the topic rating images and the rich text editor buttons. I was thinking that the blue posting form would have caused the most upset.

I was here!

Aren't semi-palindromic consecutive blog titles fun?

So, IPB 2.1 continues and this last few weeks I've been concentrating on the ACP. There are a few reasons that the ACP has needed a darn good sorting out which I touched on during my last blog entry.

In its simplest form, the back-end code needed sorting out. It's mainly a syntax / tidy thing than a refactoring. Obviously the new IPS kernel stuff has to be implemented but for the most part, BBEdit's regex/grep search and replace takes care of the donkey work. As I'm going through the ACP I'm starting to pull the hard coded HTML out of the source files and into ACP skin files. I won't have the time to complete this process - that can wait until IPB 2.2 (or more likely IPB 3.0) but it does allow some more imaginative ACP layouts.

The biggest front-end change has been the introducting of top level 'tabs'. These tabs allow the admin easy access to 5 main sections (the screenshot here only has four. Another "admin" tab has since been introduced to remove some of the weight from "Tools & Settings"). These sections are "Content Management", "Look & Feel", "Tools & Settings" and "Admin". Each section has its own menu set and it makes it a little more logical to navigate. It's less intimidating for those new to IPB too.

It also makes the "per admin" permissions easier to manage. You can opt to lock an admin out of certain tabs completely or only allow partial access. Another plus (and the motivation for doing this) is that it ties in with IP.Dynamic and our other project which is under development giving the IPS applications a fairly universal feel.

I've also cleaned up some other areas. A bit of DHTML has helped combine the "manage forums" and "manage moderators" into one screen. (Screenshot)

A new section almost completely copied from IP.Dynamic (that's how good the new IPS kernel is, I can virtually share ACP files between IPB and IPD) is the 'components' framework. This allows one to administer their plug-ins and add-ons from the ACP. A picture is worth a thousand words, so here's a thousand words.
An API is going to be written to allow developers a quick way to manipulate the component DB data making updating and installing new tools a little easier. Naturally, this will be documented and made available to everyone.

I think that about covers it. I'm going to make some new screen-shots up for an official announcement later this week which will also include the front-end stuff that's changed.

Where was I?

PHP is boring. After all the fun and laughter of javascript (a.k.a pulling hair), I'm back coding in PHP on much less interesting and fun parts of IPB 2.1.

I've had to seriously evaluate the devlopment situation and I've had to make some hard choices about what IPB 2.1 is going to be. I had outlined it as a mere point release which fixes a few bugs and introduces a few tweaks and a few new features but I also wanted to rewrite some seriously outdated code and make some pretty large changes. This pretty much had to happen anyway so that IPB can use the new IPS kernel methods and classes.

The ACP also need(s|ed) a damned good sorting out as this has some of the oldest code of the entire system. The HTML is either hard-coded or built from cookie-cutter functions to build tables and the language is all hard-coded. Also, it has some hang-overs from IPB 1 still in there.

I also want to have a good go at consolidating some of the board's templates and introducing much more DHTML/Ajax to make it more fun(ctional).

There are also some very ugly design decisions left over from day 1 which needs to be sorted out and some odd DB schematics and phrasing which kills me everytime I see it.

However, the amount of time it's going to take to do everything I really want to do is much longer than the few months I'm allowing myself and the project would be too much for a point release. Not mentioning the fact that changing all the templates / template names and CSS will cause a revolt in the community.

So, I've been ruthless and stripped back my plans some-what for IPB 2.1. It's now more of a stepping stone for IPB 3.0 which is already being planned. IPB 2.1 will use the new IPS kernel (as developed in IP.Dynamic), will include several new features (WYSIWYG, etc) and have some Ajax tweaking (but not as much as I'd eventually like to see) but will fall a little short of the total rennovation it was turning out to be. The ACP is part 2.1 and part 3.0 in that the ACP HTML is being moved into cached PHP files as-and-when I add/change an ACP section. I won't move all the HTML over at this stage as I can't justify spending a few months on something that won't look much (if any) different at the end of the development stage.

This gives us the best of both worlds: a nice intermediate release which adds stability, enhances efficiency and includes several new features and also builds the foundations for our next major release.