[Genma Saotome]

I've expressed reservations about this... perhaps I'm not grasping the benefits, so whenever possible examples would be helpful.

Why the insistence on the unix forward slash?

[YoRyan]

 conductorchris, on 22 August 2020 - 05:57 PM, said:

Ryan, I am very pleased to see your work and clear thinking. I think this is very important. I appreciate your knowledge and experience that you bring to your suggestions.
Christopher

Thanks; you're too kind. :D

 Genma Saotome, on 22 August 2020 - 08:45 PM, said:

I've expressed reservations about this... perhaps I'm not grasping the benefits, so whenever possible examples would be helpful.

Certainly.


Scenario A: A pair of native OR content packages in ZIP format.

=YoRyan_SuperlinerInteriors.zip=
    ORTS/
        YoRyan_SuperlinerInteriors/
            Shapes/
                supercoach.s
                superdiner.s
                superlounge.s
            Textures/
                supercoach.ace
                superdiner.ace
                superlounge.ace

=YoRyan_DerailFlag.zip=
    ORTS/
        YoRyan_DerailFlag/
            Shapes/
                derailflag.s
            Textures/
                derailflag.ace

The simulator would combine both of them into a single virtual filesystem.

ORTS/
    YoRyan_SuperlinerInteriors/
        Shapes/
            supercoach.s
            superdiner.s
            superlounge.s
        Textures/
            supercoach.ace
            superdiner.ace
            superlounge.ace
    YoRyan_DerailFlag/
        Shapes/
            derailflag.s
        Textures/
            derailflag.ace

Scenario B: The monkey patch. You want to customize my Superliner coach interior texture, so you add a new package that overrides my texture. (Put aside how a "proper" reskin with a separate wagon specification would be defined, as that is outside of the scope of this proposal.)

=YoRyan_SuperlinerInteriors.zip=
    ORTS/
        YoRyan_SuperlinerInteriors/
            Shapes/
                supercoach.s
                superdiner.s
                superlounge.s
            Textures/
                supercoach.ace
                superdiner.ace
                superlounge.ace

=You_SupercoachReplacementSeats=
    ORTS/
        YoRyan_SuperlinerInteriors/
            Textures/
                supercoach.ace

With the following load order (this would be a user preference),

• YoRyan_SuperlinerInteriors.zip
  
• You_SupercoachReplacementSeats.zip

the result is that your texture is loaded in place of my original.

ORTS/
    YoRyan_SuperlinerInteriors/
        Shapes/
            supercoach.s
            superdiner.s
            superlounge.s
        Textures/
            supercoach.ace    *from You_SupercoachReplacementSeats.zip
            superdiner.ace
            superlounge.ace

Scenario C: Now, imagine you have your MSTS content organized as packages instead of uncompressed files in a single directory. You can have a base package for the Kuju stock stuff, an XTracks package, packages for activities and timetables, etc., and all shareable by multiple installation profiles to boot. Upon loading, Open Rails would combine all of them into a single, virtual installation directory. Very elegant, if I do say so myself.

=MicrosoftTrainSimulator1.2.zip=
    MSTS/
        GLOBAL/
            ...
        SOUND/
            ...
        ROUTES/
            ...
        TRAINS/
            ...

=YoRyan_MSTSTimetables.zip=
    MSTS/
        ROUTES/
            USA1/
                ACTIVITIES/
                    YoRyan_USA1_Amtrak.timetable-or
                PATHS/
                    ...
            USA2/
                ACTIVITIES/
                    YoRyan_USA2_EmpireBuilder.timetable-or
                PATHS/
                    ...
        TRAINS/
            CONSISTS/
                ...

=XTracks_Nov2019.zip=
    MSTS/
        GLOBAL/
            SHAPES/
                ...

=Surfliner2.zip=
    MSTS/
        ROUTES/
            Surfliner2/
                ...

Packages further down in the load order would also be able to override previously loaded files, as in scenario B.

 Genma Saotome, on 22 August 2020 - 08:45 PM, said:

Why the insistence on the unix forward slash?

Two reasons. First, it further differentiates an Open Rails virtual path from a real Windows file path. Second, backslash is the escape character in JSON strings, so our file formats would look rather silly with every backslash separator doubled up ("\\ORTS\\SomePackage\\Shapes\\SomeShape.s").

[gpz]

Ryan, having such a zip package system would be really welcome. It is great that you brought up this discussion, I haven't even read about it in the past at all!

However I would not be eager at all adding a C library like PhysicsFS as a reference for handling such a basic and essential task as reading from files. I would not even like to use a native 3rd-party C# library for this task either, given that the standard C# classes are more than capable of handling this. (See e.g. the ZipArchive class.) I would rather like to see a custom FileManager class with a GetFile() function, that could easily handle this stuff.

[Genma Saotome]

Ryan, thank you for the examples, very much appreciated!

I see now I did not previously understand the "mount" aspect of the proposal. Your example made that clear and I can see now why there would be a certain appeal to those end users who, for the most part, just download or do minimal subsitutions. Speaking as a route developer it appears to add no useful features and add an extra layer of work. EVERY file I work with is subject to replacement or editing on any day. My larger \route directory has 87,000 files. I expect fewer than 200 were downloaded from public sources. 2300 custom shapes, 2400 custom textures. There is no way I'd package 2400 individual shape packages. Packaging things as whole directories (e.g., \myshapes) would be simple enough but provide nothing whatsoever for the end user that isn't present today.

Am I still missing the point? Perhaps. So what have I still got wrong about this?

[YoRyan]

 gpz, on 23 August 2020 - 01:03 AM, said:

However I would not be eager at all adding a C library like PhysicsFS as a reference for handling such a basic and essential task as reading from files. I would not even like to use a native 3rd-party C# library for this task either, given that the standard C# classes are more than capable of handling this. (See e.g. the ZipArchive class.) I would rather like to see a custom FileManager class with a GetFile() function, that could easily handle this stuff.

Well, there is a native C# wrapper available. If it's performance you're concerned about, a slight reduction is inevitable, yes, given that we are loading from ZIP and other kinds of archives.

Ultimately, the use of this library is a call for the management team - but given our extremely stretched resources, I would strongly advocate for a battle-tested solution, used by many other video game engines, that supports a vast array of archive formats (not just ZIP) rather than trying to homebrew something ourselves that might well turn out to perform worse. If only for my sake. :)

 Genma Saotome, on 23 August 2020 - 07:05 AM, said:

Speaking as a route developer it appears to add no useful features and add an extra layer of work. EVERY file I work with is subject to replacement or editing on any day. My larger \route directory has 87,000 files. I expect fewer than 200 were downloaded from public sources. 2300 custom shapes, 2400 custom textures. There is no way I'd package 2400 individual shape packages. Packaging things as whole directories (e.g., \myshapes) would be simple enough but provide nothing whatsoever for the end user that isn't present today.

Am I still missing the point? Perhaps. So what have I still got wrong about this?

You're not required to migrate your content to the ORTS namespace (a completely different, yet-to-be-determined structure), nor to store it in a ZIP file - these are optional steps. All of your legacy content would remain available under the "MSTS/" namespace and behave as before. "Packages" can also be plain directories instead of ZIP files, making editing easy.

This is useful for 1) players who want to organize their MSTS directories, and 2) content creators who want to release their work. You know that feeling when you install an activity from trainsim.com, and now your path and consist folders are littered with files of unknown origin? This is a way to track that information and make clean uninstalling a breeze.

[gpz]

Dave, the average end user's content folder is hopelessly smashed up, files from all different places are all around in rotes, trains, etc. folders, mixed all together. For them it is an unmaintainable mess, they are able to only save the install packages to somewhere, and reinstall the whole stuff ftom time to time.

And here comes this genuine feature in subject: just point OR to the folder of the downloaded zip-s, and provided these zip-s are properly arranged, everything just starts magically work. No mess, no thinking of where-this-wrong-file-came-from, what version of xtracks I have, what files a concurrent package has overwritten, etc. Everything is just in order, in its correct place, back-searchable.

[YoRyan]

Well, there are a couple of complications to what Peter added. First, I feel like the user needs control of the package load order and which packages are included in a given installation profile, so rather than auto-loading all from a specific directory, I am envisioning a page in the Settings screen where one can add and reorder content packages. Second, for content that uses the MSTS/ namespace, there would still be no protection against the shortcomings of Kuju's file structure. You could still be dealing with an incompatible XTracks version, activities that override each other's paths, consists that override each other, etc. It's just that with the content separated into packages, it would be easier to track down and debug these kinds of issues.

[gpz]

 YoRyan, on 23 August 2020 - 08:02 AM, said:

Well, there is a native C# wrapper available. If it's performance you're concerned about, a slight reduction is inevitable, yes, given that we are loading from ZIP and other kinds of archives.

Ultimately, the use of this library is a call for the management team - but given our extremely stretched resources, I would strongly advocate for a battle-tested solution, used by many other video game engines, that supports a vast array of archive formats (not just ZIP) rather than trying to homebrew something ourselves that might well turn out to perform worse. If only for my sake. :)

Most of those other archive formats are just proprietary garbage, OR cannot benefit from them. Not just the performance is my concern. (Although it is also important.) Come on, is it really a good software design, to replace the standard MS-maintained, well defined, managed StreamReader, File.Open, ZipArchive class, etc. with an unmanaged C library, where 1-2 developers are fighting with the maintenance? For such a not too complex task of maintaining a dictionary of files, assigned to their containing prioritized package name? To me it is a big no-no. You may call for the management team, in this case I would especially interested in James's opinion about this. :)

[YoRyan]

I take your point about the risks of calling a C library. And a global mutex? Ouch!

As for pure C# solutions - SharpFileSystem looks pretty promising (and even uses my proposed path structure), but it lacks ZIP support, and it's been unmaintained for some years now. So rolling our own might well be the way to go after all. Note that in addition to ZIP support, we would also need to support raw directories, and 7ZIP support would also be nice to have.

[Eldorado.Railroad]

 gpz, on 23 August 2020 - 09:25 AM, said:

Most of those other archive formats are just proprietary garbage, OR cannot benefit from them.

Absolutely!. Case in point is the obfuscation in "Railworks" "serz" garbage and recently in Auran Trainz ".trainzmesh". Keep it simple please. The average user will want to edit those .zip files for content, if not personal fixes. In my experience multi-threaded 7zip nets very little in compression performance (overall size of 7zip file) with heavy use of all cores of the CPU.

Steve