I hold the possibly generous belief that if fixing the lag were easy, Turbine would have already done so. I further hypothesize this means it must be a subtle, evil bug. So what I want to do here is to provide as much detail as I can about an experience that ended up sucking because of lag, in hopes that this is useful in ultimately tracking down and fixing the bug. Perhaps others can do the same either here or in their own threads. Naturally I can't stop mindless "fix the freakin' lag you lazy developers" posts here or elsewhere, but I'll just remind you that's been done and doesn't seem to have helped.
I was running as Vrol on Thelanis from roughly 9p ET to 10:40p ET last night (3/8/09), doing Wizard King (I hope that's the name -- I'm a n00b in higher level content because I constantly start new characters!). Group of 6, mostly from the guild Remember Honor. Although there was random ~1 sec lag here and there, the first hour and a half was at least playable. Then pretty suddenly we all began to experience massive lag.
[Comment: This seems to eliminate the "client side" hypothesis that once or twice I've seen thrown around. The probability of 6 geographically distributed players hitting msasive lag at the same time on their local clients is negligible.]
The lag quickly ramped up to completely unplayable -- taking a step once every several seconds, somewhere around 10:30p ET. It remained like this for about 10 minutes, after which there was a slight improvement: still unplayable, but now able to take several steps at a time before lagging out.
At roughly 10:45p ET, we gave up. We then all recalled out. Once out of the instance, the lag disappeared and we could run around normally in the Sands.
[Comment: This seems incredibly important from a diagnostic standpoint, so I'm especially curious if this has been the standard experience. It seems to prove that the issue isn't hardware related, because I can't see why we would have changed hardware when we recalled out. It seems to suggest to me that something went wrong in the instance -- i.e. some kind of software bug.]
Unfortunately, since we weren't running a controlled experiment, we didn't try going back in, so I can't say if the "saved" instance remained hosed until it reset. Perhaps others have done that and can provide that information.
Good luck Devs! Godspeed to find and correct this horrible lag!