Page 1 of 17 1234511 ... LastLast
Results 1 to 20 of 330
  1. #1
    Community Member Wanesa's Avatar
    Join Date
    Sep 2012
    Location
    Prague
    Posts
    527

    Default How to make the game loading faster - DDOPreload official

    Hi guys.
    (sorry for all mistakes and typos in the article. English is not my natural language)

    From time to time i can see articles about troubles during loading the game for the first time. After character selection, loading phase can take up to 5 minutes. During loading, the game client doesn't respond to the server messages so kicking out from the game server is the most result of this situation.

    You can probably see that:

    (keep eye on time at left bottom corner)
    (can you see red icon on top of the image?)

    After hours of the investigation, me and my friend have developed a small tool that can speed up this loading phase and definitively solve problem with red icon in the first enter to the game. DDOPreloader. This program reads all DDO's necessary files into Windows cache which is apparently much faster than any classical HDD. DDOClient can benefit from the windows cache and starts faster. Program can be downloaded here:

    new version (v2): https://dl.dropboxusercontent.com/u/...DOPreload2.zip
    old version (v1): https://dl.dropboxusercontent.com/u/...DDOPreload.zip

    There is a screen-shot:


    Following links shows how faster is entering to the game (after clean boot)

    (keep eye on clock)

    The program is distributed free with the source code under GPL licence. You need at least 2GB of the physical memory installed at your board. It will not work if there is less memory - don't use on 1GB memory board.
    Note that windows cache works as magic with its own will. Sometimes it can remove cached content before it is really need. You should also stop all (mostly unwanted) memory optimizers (actually they still not work) and applications that can manipulate with the windows cache. Stopping th Superfetch (sysmain) service may also help.

    Enjoy and please report all bugs to me.
    Last edited by Wanesa; 07-23-2014 at 09:24 AM. Reason: New version

    Thelanis: Shewind the Airbender (Sorc20/Epic5 -> Bard20/Epic8 -> Rog20/Epic8/Epic2 -> Harper_FvS20/Epic4 -> Art19), Azaxe (Rog18/Wiz2 -> Sorc20/Epic6)

  2. #2
    Community Member
    Join Date
    Sep 2009
    Posts
    225

    Default thanks

    I was already using the previous (compiled with /SUBSYSTEM:CONSOLE) version.

    It really helps until last release. Now loading takes between 40 seconds and 10 minutes... with or without preloader... I've to figure out what's going on.

    Anyway, thanks a lot for sharing this preloader, it has already saved me several hours

  3. #3
    Community Member Thrudh's Avatar
    Join Date
    May 2006
    Posts
    17,004

    Default

    I recently upgraded to a SSD and 12 GB of RAM, and re-installed Windows 7 from scratch.

    Now it takes about 10 seconds from when I click on a character to when the game is fully loaded.

    Not sure how much of that is the SSD, and how much is the fresh clean install.
    Quote Originally Posted by Teh_Troll View Post
    We are no more d000m'd then we were a week ago.
    Quote Originally Posted by Eth View Post
    When you stop caring about xp/min this game becomes really fun. Trust me.
    Quote Originally Posted by Gary_Gygax
    No single character has all the skills and resources needed to guarantee success in all endeavors; favorable results can usually only be achieved through group effort. No single player character wins, in the sense that he or she defeats all other player characters; the goal of the forces of good can only be attained through cooperation, so that victory is a group achievement rather than an individual one.

  4. #4
    Hero AZgreentea's Avatar
    Join Date
    Dec 2009
    Location
    Lost in the Swamp
    Posts
    3,638

    Thumbs up

    Thank you for continuing to support the preloader for those of use who dont have the scratch to buy a new hard drive and RAM right now just to play DDO.
    The problem is never how to get new, innovative thoughts into your mind, but how to get old ones out. Every mind is a building filled with archaic furniture. Clean out a corner of your mind and creativity will instantly fill it.
    Dee Hock

  5. #5

    Default

    Quote Originally Posted by Thrudh View Post
    I recently upgraded to a SSD and 12 GB of RAM, and re-installed Windows 7 from scratch.

    Now it takes about 10 seconds from when I click on a character to when the game is fully loaded.

    Not sure how much of that is the SSD, and how much is the fresh clean install.
    That's pretty much all SSD.

    I don't know what Turbine is doing when game assets are being loaded, but is has got to use the most random read pattern ever devised. They really need to address the file structures or loading scheme to minimize disk thrash. It gets worse with every update. A complete uninstall/reinstall cycle (to internally defrag the "unfraggable" data files) no longer has a significant impact on load times. There are simply too many resources to load all at once.

    And many thanks to Wanesa for taking the time to code the preloader. I don't use it personally, but it seems to have helped a great many users. So, HUZZAH!
    The newest computer can merely compound, at speed, the oldest problem in the relations between human beings, and in the end the communicator will be confronted with the old problem, of what to say and how to say it. - Edward R. Murrow (1964)

  6. #6
    Community Member Wanesa's Avatar
    Join Date
    Sep 2012
    Location
    Prague
    Posts
    527

    Default

    Quote Originally Posted by sebastianosmith View Post
    That's pretty much all SSD.
    /agree

    Quote Originally Posted by sebastianosmith View Post
    I don't know what Turbine is doing when game assets are being loaded, but is has got to use the most random read pattern ever devised. They really need to address the file structures or loading scheme to minimize disk thrash. It gets worse with every update.
    IMHO, the file "'client_gamelogic.dat" is like internal filesystem. During each update, you can see a lot of small files created in the "forward"(?) folder named "client_gamelogic.NNN.dat" and one file "client_gamelogic.new.dat". During final phase, these files are glued up to a new client_gamelogic.dat file. It seems that files are randomly placed there. Turbine should consider some ordering of these pieces of the data ... to keep the order in which pieces are accessed during load.

    Quote Originally Posted by sebastianosmith View Post
    A complete uninstall/reinstall cycle (to internally defrag the "unfraggable" data files) no longer has a significant impact on load times. There are simply too many resources to load all at once.
    Because the official downloadable version is very old and it always updates self complete.

    Quote Originally Posted by sebastianosmith View Post
    And many thanks to Wanesa ...
    YW!

    Thelanis: Shewind the Airbender (Sorc20/Epic5 -> Bard20/Epic8 -> Rog20/Epic8/Epic2 -> Harper_FvS20/Epic4 -> Art19), Azaxe (Rog18/Wiz2 -> Sorc20/Epic6)

  7. #7
    Community Member vampiregoat69's Avatar
    Join Date
    Sep 2009
    Location
    Sneaking up behind you
    Posts
    241

    Default

    I have 4 gigs of memory and installed into the directory it says but gives me a memory allocation error every time.

  8. #8
    Hero AZgreentea's Avatar
    Join Date
    Dec 2009
    Location
    Lost in the Swamp
    Posts
    3,638

    Default

    Has anyone noticed if Hybrid drives give improvement to DDO, or is SSD pretty much the way to go with the constantly dropping prices? Granted, I can get a bigger drive if I go Hybrid vs SSD.
    The problem is never how to get new, innovative thoughts into your mind, but how to get old ones out. Every mind is a building filled with archaic furniture. Clean out a corner of your mind and creativity will instantly fill it.
    Dee Hock

  9. #9
    Community Member Wanesa's Avatar
    Join Date
    Sep 2012
    Location
    Prague
    Posts
    527

    Default

    Quote Originally Posted by vampiregoat69 View Post
    I have 4 gigs of memory and installed into the directory it says but gives me a memory allocation error every time.
    EDIT: Should be fixed now! Try to download again from the same location.

    -----------------------------
    Thank you for bug-report.
    This issue can happen if your desktop contains many "hook" tools (plugins, fun apps, optimizers, etc. mostly when you have a lot of icons in the systray) that install system wide ddls and fragment memory a lot. I can promise improvement of this phase soon.
    Last edited by Wanesa; 07-19-2013 at 10:40 AM.

    Thelanis: Shewind the Airbender (Sorc20/Epic5 -> Bard20/Epic8 -> Rog20/Epic8/Epic2 -> Harper_FvS20/Epic4 -> Art19), Azaxe (Rog18/Wiz2 -> Sorc20/Epic6)

  10. #10
    Scholar Of Adventure & Hero Missing_Minds's Avatar
    Join Date
    Jun 2006
    Location
    work....
    Posts
    30,169

    Default

    Quote Originally Posted by Wanesa View Post
    IMHO, the file "'client_gamelogic.dat" is like internal filesystem. During each update, you can see a lot of small files created in the "forward"(?) folder named "client_gamelogic.NNN.dat" and one file "client_gamelogic.new.dat". During final phase, these files are glued up to a new client_gamelogic.dat file. It seems that files are randomly placed there. Turbine should consider some ordering of these pieces of the data ... to keep the order in which pieces are accessed during load.
    One large file vs millions of smaller ones. It really won't make a whole lot of hard drive access/thrashing difference. At least I think it won't. Order won't help a whole lot as each person wears different skins, and not everyone loads in at the same places.

    The biggest difference between the two is how much space on your drive is used. The smaller ones will take up more just on the issue of block usage.

    Something I want to do at some point is to just delete one of the data files and try to repair. I would think it should download the whole file which should in turn also defrag it internally.

  11. #11
    Community Member Wanesa's Avatar
    Join Date
    Sep 2012
    Location
    Prague
    Posts
    527

    Default

    Quote Originally Posted by Missing_Minds View Post
    One large file vs millions of smaller ones. It really won't make a whole lot of hard drive access/thrashing difference. At least I think it won't.
    There isn't problem with the size. There is problem with ordering.

    Quote Originally Posted by Missing_Minds View Post
    Order won't help a whole lot as each person wears different skins, and not everyone loads in at the same places.
    client_gamelogics.dat takes longest to load. And that file is preloaded by DDOPreloader. It does contain neither graphics nor geometry nor music nor textures. It contains ... game logics. For example values for all characters, races, spells, crafting recipes (see how long takes load Cannith Crafting Hall, during this only client_gamelogic.dat is accessed). I am pretty sure, that these informations are loaded almost everytime in the same order but their location in this file doesn't respect the order of loading

    I guess, that gamelogics.dat also contains all icons.

    Thelanis: Shewind the Airbender (Sorc20/Epic5 -> Bard20/Epic8 -> Rog20/Epic8/Epic2 -> Harper_FvS20/Epic4 -> Art19), Azaxe (Rog18/Wiz2 -> Sorc20/Epic6)

  12. #12
    Scholar Of Adventure & Hero Missing_Minds's Avatar
    Join Date
    Jun 2006
    Location
    work....
    Posts
    30,169

    Default

    Quote Originally Posted by Wanesa View Post
    There isn't problem with the size. There is problem with ordering.
    You say ordering. Define ordering in this context, we may be thinking different "order" here.

    You seemed to imply the order in which the "foo" is placed in the file would help.

    I am saying it would not. The only thing that would make it worse for access time is if the data is internally fragmented. That is something players say is happening but Turbine says is no longer an issue.
    The only way I can see "order" of the file coming into play is with mechanical hard drives due to the "random" place the data is stored on the drive. This is something out of Turbine's hands.
    Last edited by Missing_Minds; 07-19-2013 at 11:20 AM.

  13. #13
    Community Member
    Join Date
    Oct 2009
    Location
    3rd star from the left, and on til morning...
    Posts
    2,629

    Default

    I've been using a version of the preloader for a few weeks and I swear by it.

    My 4GB WinXP system loads DDO from character selection to gameplay MUCH faster and never a disconnect on first startup.

    Without it, since a release or two ago, every first connection takes forever resulting in a disconnect.

    I love it, I thank you for it, and I use it.

  14. #14
    Community Member Wanesa's Avatar
    Join Date
    Sep 2012
    Location
    Prague
    Posts
    527

    Default

    Quote Originally Posted by Missing_Minds View Post
    You say ordering. Define ordering in this context, we may be thinking different "order" here.

    You seemed to imply the order in which the "foo" is placed in the file would help.

    I am saying it would not. The only thing that would make it worse for access time is if the data is internally fragmented. That is something players say is happening but Turbine says is no longer an issue.
    The only way I can see "order" of the file coming into play is with mechanical hard drives due to the "random" place the data is stored on the drive. This is something out of Turbine's hands.

    Let the loading script of the game says
    1. Load A
    2. Load B
    3. Load C
    4. Load D
    etc

    And the gamelogic looks this (just example)
    A is located at offset 120.237.320 , size 100 bytes
    B is located at offset 65.230, size 2048
    C is located at offset 237.987.800, size 232 bytes
    D is located at offset 78.320.050, size 786 bytes
    etc-


    When the game starts loading such a file, your HDD will go insane soon or later.

    Lets start some math: Assume, that file has about 270MB. Smallest piece of file is 4KB (size of memory page). Yoú have 65917 pages for this file. If you start to read that file from random offsets, with caching you will access each page in aprx 10ms (access time of your HDD). Result is about 10 minutes to read whole file. That is the theory. In reality, situation is much better, about 4-5 minutes.

    "but Turbine says is no longer an issue"

    ppl can say anything. True is, that preloader works and it works because Turbine isn't telling the truth
    Last edited by Wanesa; 07-19-2013 at 12:22 PM.

    Thelanis: Shewind the Airbender (Sorc20/Epic5 -> Bard20/Epic8 -> Rog20/Epic8/Epic2 -> Harper_FvS20/Epic4 -> Art19), Azaxe (Rog18/Wiz2 -> Sorc20/Epic6)

  15. #15
    Community Member
    Join Date
    Jan 2012
    Posts
    4,643

    Default

    I see. It's like vacuuming. It is much faster to vacuum up a single long line of dust than it is to vacuum up the same amount of dust that is scattered about in several different piles.
    http://myaccount.turbine.com

    Je ne suis pas
    DDO Alpha Tester

  16. #16
    Founder
    Join Date
    Feb 2006
    Posts
    4,595

    Default

    Quote Originally Posted by Missing_Minds View Post
    You say ordering. Define ordering in this context, we may be thinking different "order" here.

    You seemed to imply the order in which the "foo" is placed in the file would help.

    I am saying it would not. The only thing that would make it worse for access time is if the data is internally fragmented. That is something players say is happening but Turbine says is no longer an issue.
    The only way I can see "order" of the file coming into play is with mechanical hard drives due to the "random" place the data is stored on the drive. This is something out of Turbine's hands.
    I am not a computer expert by any definition. However I do understand logic. I'm just curious: if the preloader works (makes login time shorter) what do you think its doing if it isn't doing what Wanesa says it does?

  17. #17
    Community Member ycheese123's Avatar
    Join Date
    May 2010
    Location
    Pittsburgh, PA
    Posts
    314

    Thumbs up

    Quote Originally Posted by Wanesa View Post
    After hours of the investigation, me and my friend have developed a small tool that can speed up this loading phase and definitively solve problem with red icon in the first enter to the game. DDOPreloader. This program reads all DDO's necessary files into Windows cache which is apparently much faster than any classical HDD. DDOClient can benefit from the windows cache and starts faster.
    I've been using the previous version, it's worked 100% of the time and still does. Is this new one better in some way? And thank you so much by the way!
    Orien
    | Klorox | Suppressor | Vellien | Knost | & more |
    ~ Youtube ~ DDO Adventures ~

  18. #18
    Scholar Of Adventure & Hero Missing_Minds's Avatar
    Join Date
    Jun 2006
    Location
    work....
    Posts
    30,169

    Default

    Quote Originally Posted by HAL View Post
    I am not a computer expert by any definition. However I do understand logic. I'm just curious: if the preloader works (makes login time shorter) what do you think its doing if it isn't doing what Wanesa says it does?
    In no fashion at all am I saying what he has provided does not help people get into the game faster. In fact, first time running the game after a reboot it deff should.

    What I do agree with is internal fragmentation of the .dat files is an issue.

    What I am disagreeing (on the worth it effort side) is for a reordering of the individual files in the .dat files. The gains will be too minimal vs. the effort spent.

    Lets just use the below as a representation of files, where upon that line is one .dat file.
    AAAAAAAAAAAABBBBBBBBBBBBBBCCCCCCCCCCCDDEEEEEEEEEWW WWPPPPPPPPPPPP

    What Wansa is saying is that if we reorder the internal files, it would be better because of how much data is held per block of memory. (I don't know if the below is utterly accurate because I'm not defining a how large a block is. But you get the idea. Also the memory block size can be different depending on systems and size of harddrives. It isn't universal which throws another "ugh" into a definitive optimized order.)

    AAAAAAAAAAAABBBBBBBBBBBBBBCCCCCCCCCCCPPPPPPPPPPPPE EEEEEEEEWWWWDD

    Given how computers are today, there really isn't much of any gain for that. And certainly not worth the effort of hiring a contractor to spend time on. (I'll hit another point on that in a moment.)


    Now what we do agree with is internal file fragmentation is an issue. You can see what I mean by the below.
    AAAAAAAABBAAAABBBBBBBBBCCCCCPPPPPCCCCCCDDEEEBBBEEE EEEWWWWPPPPPPP

    Now there is a difference between file ordering and internal file defragmentation even though the latter will inevitably change order as well (typically.) And that is with the first, you want a definitive order AND no defragmentation to optimize speed of access. This would have to be analyzed EACH time a patch/update was released. (Now you see why it isn't worth the cost.) Where as internal file defragmentation you just want the internal files contiguous without worry of order.

    TLDR?
    I just disagree with Wansa that specializing the order of the data held within the .dat files is worth the effort for the minimal access speed gained.
    Last edited by Missing_Minds; 07-19-2013 at 02:46 PM.

  19. #19
    Scholar Of Adventure & Hero Missing_Minds's Avatar
    Join Date
    Jun 2006
    Location
    work....
    Posts
    30,169

    Default

    Quote Originally Posted by Tscheuss View Post
    I see. It's like vacuuming. It is much faster to vacuum up a single long line of dust than it is to vacuum up the same amount of dust that is scattered about in several different piles.
    Yes, but two things.
    1. The dust is colored. You can not mix colors.
    2. The vacuum works in pulses it isn't continuous.

  20. #20
    Founder
    Join Date
    Feb 2006
    Posts
    4,595

    Default

    Quote Originally Posted by Missing_Minds View Post
    TLDR?
    I just disagree with Wansa that specializing the order of the data held within the .dat files is worth the effort for the minimal access speed gained.
    Ok, now I see what you're disagreeing about. Thanks for the explanation.

    How do other MMOs manage to keep their loading times short? I've never had a problem loading an MMO (in recent history) like I have with DDO.

    And frankly I didn't even have a problem loading DDO until MotU. Although I know other people did so that: DDO doesn't play well with some computers; and now DDO either doesn't play well with even more computers (including mine) or there is an additional loading issue separate from the first.

Page 1 of 17 1234511 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

This form's session has expired. You need to reload the page.

Reload