That small group of people also has a percentage of people that are willing to lead most raids after they get the raid down as pugs. The truely casual does not lead or post many lfms. That means as you lose this group of endgame raiders from them having nothing to do, you get a declining lfm pool that casuals can join. Now each casual is not going to join every lfm but since the pool of casuals is larger in your theory that means that they all get to raid or "hard" mode quest once in a great while as the mood strikes them. Fast forward to now when many if not most of the raiders have left and lfm's especially for endgame are down and even the casuals have less to do. This puts the game in more and more of a downward spiral as even some casuals start to leave thinking it's a dying game.