« I was here! | Main | Well, that was a difficult day »

Friday, April 22, 2005

(?:Dis)?Organi(?:z|s)ed Chaos

So, it's the day after the afternoon before. In that time, we upgraded our corporate forums, suffered a DDoS attack, send out 12,000 emails, fixed 50 odd bugs and even found time to write in a legacy mode to enable the blog and gallery to work without modification.

The upgrade to 2.1 seems to have gone fairly well. It was a bumpy ride for the first few hours and the odd (hundred) bugs here and there did their bit to add to the overall chaos. Ever wondered what it's like to upgrade a relatively busy forum to a largely untested release? It goes something like this:

1:00pm: Turn company forums offline. Add in a redirect to a basic index.html file.
1:30pm: New file uploads complete
1:50pm: MySQL updates done surprisingly quickly. Note to self: MySQL 4 is so much better than MySQL 3.
2:00pm: Hack index.php to allow access for myself and the US office IP address.
2:10pm: Various staff members AIM me with bugs they've found.
2:14pm: It's apparently turned into a competition.
2:22pm: Fixed the first round of bugs (parse errors, the biggies)
2:34pm: AIM is lit up like a christmas tree
2:38pm: Fixed some more bugs. Start roughing out an announcement
2:43pm: AIM is lit up like the Empire State Building
2:44pm: Few more bugs are fixed.
3:00pm: Finished announcement. Load Safari to post it (saves logging out of Mozilla)
3:01pm: Notice the standard editor doesn't work at all with Safari.
3:20pm: It does now.
3:22pm: MySQL error when posting from Safari
3:30pm: Get a message from CTerry via Stewart 'Stop slacking'
3:31pm: Motivated, I submit the announcement again
3:32pm: Remove IP address filter and turn board back online
3:33pm: Charles starts sending emails to our subscribers
3:45pm: Can't load a forum. This can't be good. ssh [email protected]
3:50pm: MySQL locks up, 156 locked processes. Lindy?
3:51pm: MySQL restarted.
4:10pm: Ah, that topic table index? Yeah. I'll go add that now.
4:15pm: MySQL locks up again.
4:20pm: Oh yes, LOW PRIORITY was removed from the topic counter update
4:25pm: Server restarted
4:26pm: Things are much improved. Nearly 400 users active and load @ 0.3 per CPU
4:30pm: Initial feedback comes rolling in. Bug reports are posted everywhere.
4:30pm - 6:00pm: Crazy bug fixing period.

Or something like that, anyway.

I've spent the better part of today bug fixing and fiddling with bits of javascript that didn't work on Opera. Overall feedback has been good. Apparently some people have taken offence to the topic rating images and the rich text editor buttons. I was thinking that the blue posting form would have caused the most upset.

April 22, 2005 | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference (?:Dis)?Organi(?:z|s)ed Chaos:


Way to go matt....!

Posted by: Zain | Apr 22, 2005 9:22:04 PM

Now, we're just going to have to do that with 5x the active users over at Neowin. At least by then the source will be a bit more stable :D

Posted by: Tim Dorr | Apr 22, 2005 9:36:32 PM

It seemed that when I clicked "New Posts" the first couple of times it would show perfectly, then it would show the entire IPS Forum page in that little bit, then it was later fixed. Was this one of the bug fixes?

Posted by: Don Wilson | Apr 22, 2005 10:20:23 PM


IT's good to see all the long-awaited features faling into place, the WYSIWYG editor has to be me favourite 'new thing'. I'm not too sure on the topic rating, but that's just me :).

Posted by: Oblivion. | Apr 22, 2005 10:34:59 PM


Got rid of that ugly memberlist query so you might be able to get 5 x the members on the site. ;)

Posted by: Matt | Apr 22, 2005 10:40:57 PM

Sounds like fun!

Posted by: Surferdude | Apr 22, 2005 10:47:56 PM

Mega thumbsup Matt! You ultimatley surprised me with the update soo quickly... "We plan to update the company forums in the coming weeks..." That was last weekend!
Btw... the blue posting screen could take some getting used to, but maybe make it match the Editor... which I hope is skinable O:)

LOL! Aim like a christmas tree :)
Try a Chat next time... that way each other knows who's reported what... not [D] :)

Posted by: Cybertimber2005 and Cooldude7273 | Apr 22, 2005 11:51:07 PM

Poor Matt,

With al the feedback in the customer forums (be it contructive or other) you guys did an OUTSTANDING job. There are bugs, there will be them but you guys stepped up and worked very fast on them. I was rather impressed to see what you have done. 2.1 is looking to be a very good upgrade! KEEP UP THE WORK!

We love you Matt (well when you give us blogs and those wonderfull pics of ID LOL).

Posted by: Louis | Apr 23, 2005 12:45:02 AM

Wait... the ugly memberlist query?

The one that grabs EVERY single user? and crashes mySQL and we've had to sdiable at IZE?


Er - good job - I love the blue posating form (adds a bit more cxlass really) and well - while the topic rating images may not be the best choice of image, I'm not going to take offence (just change them :P)

Anyways :D.... er... nice job :D cant wait to try it out myself *itches to see the new ACP*

Posted by: | Apr 23, 2005 1:24:26 AM

sure you had some fun matt ^_^

Posted by: akurashy | Apr 23, 2005 3:59:53 AM

ditch IPB and get vBulletin :P jk IPB is a good piece of software... just too many bugs atm... Matt you could've done a better job at getting those bugs fixed before you upgraded the comp. forums... blogs dont work, chat dont work, posting text area resize thing dont work in FF and users cannot change profile settings.... WAY too many bugs... Well I guess you just gotta learn from your mistakes.... keep up the good work matt and same goes to the rest of the IPS staff!

Posted by: JPT62089 | Apr 23, 2005 8:25:43 AM

What do you expect from an alpha release? it's not as if it's been released to the world and every upgraded IPB is suffering from these bugs!

Can't wait for the final, i know it'll be a while yet...

Posted by: Bob | Apr 23, 2005 10:39:38 AM

Read the blogs and comments here. Read the posts in Invision's forums. They all say pretty much the same thing "When are we getting it?" We are like abunch of rabid dogs staring at food that is just 2 inches in front of our faces. I am sure that our pressure had some cause on this.

Relax and enjoy the process.

Posted by: Louis | Apr 23, 2005 11:18:36 AM

"blogs dont work, chat dont work, posting text area resize thing dont work in FF and users cannot change profile settings.... WAY too many bugs..."

All those bugs were fixed in under 20 hours of the board going live. Most of them were fixed within a few hours.

Posted by: Matt | Apr 23, 2005 11:31:03 AM

LOL, i laughed a few times reading that entry

Good job on IPB v2.1, its shaping up to be a excellent new release

Posted by: | Apr 23, 2005 3:17:49 PM

Hey Matt,

It's working great! (I won't start about the bugz and that the RTE crashed FF103)

Goodluck and enjoy your weekend!

Posted by: Franklin | Apr 23, 2005 3:39:34 PM

Sounds cool!

I should start reading this more often..

Posted by: Jaak | Apr 23, 2005 9:29:54 PM

Sorry that so many of those christmas tree lights were mine. :-"

Wonderful thing, 2.1 - Great stuff Matt! :)

Posted by: William | Apr 23, 2005 10:14:34 PM

Looks pretty nice, I would suspect that if you change things in such a huge PHP coding enviroment you'd get at least 100 bugs.
I noticed for some reason The download for the IPB board is down.

Posted by: Dave Sims | Apr 24, 2005 2:59:25 AM

nice work Matt well worth the trouble

Posted by: Will | Apr 24, 2005 9:44:22 AM

But Matt you could've fixed those bugs before you upgraded the support forums (The blog and stuff I understand but the edit profile, no working +/- things in the post screen, etc.) But now that you fixed em very nice release! just gotta change that UGLY mew PM notification.... very nice though lol

Posted by: JPT62089 | Apr 25, 2005 5:04:24 AM

You have to remember that most bugs don't show themselves until you test the board with 400-500 members online in a live environment.

Posted by: Wilko | Apr 25, 2005 10:37:10 AM

I haven't had any PMs to see the new notificatio nin action :(

Posted by: Madob | Apr 25, 2005 3:10:58 PM

I haven't had any PMs to see the new notification in action :(

Posted by: Madbob | Apr 25, 2005 3:17:03 PM

I haven't had any PMs to see the new notificatio nin action :(

Posted by: Madbob | Apr 25, 2005 3:18:16 PM

The comments to this entry are closed.