[James Ross]

smmille1, on 30 December 2011 - 04:19 PM, said:

1. Why do the zip files contain a top level directory (and why don't we then care about the zip names)? Assuming we're allowing folders or zip files, wouldn't it make more sense to have the zip name AS the package folder name?

As I explained previously in this thread, the zip filename is excluded because websites and browsers are bad at preserving the filename reliably. An alternative, which I may have also mentioned previously, is to have some metadata in the zip (i.e. a specific file in the root) that provides the package name instead of the directory (which would also work for directory-packages and let them be freely renamed). I'm not too fussed how we get the package name but I'm pretty sure the zip filename is out.

smmille1, on 30 December 2011 - 04:19 PM, said:

2. If we're referring to files outside of the current package, why the ".."? If there aren't going to be separately aliased content folders and we're not actually referring to filesystem locations, then I think you can make the assumption of if it starts with a slash or backslash (there's a mini-discussion in itself), then it starts with a package, and if not, then it's local to the current package.

It makes no difference, if we implement the virtual file structure properly, but people are familiar with the ..\ notation so that's what I've used in my examples.

smmille1, on 30 December 2011 - 04:19 PM, said:

3. What is the exact benefit to segregating by file type and then inferring the type folder beyond saving a couple of bytes in the referencing string? I think you've said that you can have subfolders inside the type folders (textures, shapes, etc.), but it would seem to me that if I create a package with several components, I might rather organize things by relationship, especially if I have several different groups of things (buildings, trees, signs, etc.) in the package.

Saving those bytes also reduces the chance of errors, like typos, and misunderstandings about what it references. The intention with the strict 1st level within a package is to make the contents of packages more discoverable/explorable by both people and programs, such that neither I nor OR needs to, for example, search the entire zip file for wherever you've hidden the activity files. I agree that the groups of shape (.s), shape data (.sd) and textures (.ace) for particular objects might seem like grouping, but you can share the textures between shapes, so I think pretty much everything should be grouped by type ultimately. There could be exceptions, such as .s and .sd files both being in "Shape", and the special consideration for routes I've mentioned before.

[longiron]

wacampbell, on 30 December 2011 - 02:55 PM, said:

I certainly agree that what MSTS is using, and what James proposes are actually RDBMS. It makes me chuckle when I hear people say 'lets use a rdbms'.

Wayne,
When I say use a RDBMS, I'm talking about a true database. Tables, Primary and secondary keys, rows, columns, blob data and outside references. No more individual files for train data. It works well for eng, wag files as they share common parameters and data structures, and may work for routes. It makes it extremely easy to use common data sets as many items can reference the same table/columns/rows. It does demand a tool set to manage data and some kind of tool to package data for sharing.

[cjakeman]

I've had some new commercial routes for Christmas :-) Some very impressive scenery from Making Tracks (UK) Great Eastern and Greater Eastern and also the local preserved Nene Valley Railway.

I had to re-install MSTS (for other reasons) and discovered that these routes will only play nicely together if they are installed in a certain sequence. (OR pointed the finger at TSection.dat) Maybe there are even combinations of routes which can never co-exist.

Perhaps this is a well-known problem (and I apologise if I'm going over old ground) but will these proposals for Overall File System and Data Structure avoid this particular issue?

[longiron]

You've discovered the dreaded Tsection.dat issue. Since trackwork in MSTS is like model RR sectional track, all the track pieces must be defined for the game. Different routes were built using different Tsection.dat. In general, later versions of Tsection should work with older routes. But you must be careful during installation that it doesn't get overwritten with an earlier version.

OR proposal is to use procedural track. Essentially a profile extruded along the railway route. So the MSTS type of issues should not be a problem.

[Genma Saotome]

James Ross, on 30 December 2011 - 03:55 PM, said:

Example Files
\Some File I Downloaded From The Internet.zip

\3DTS Cajon

\Shape\Bungalow.s

\My Package.zip

\James's Package

\Shape\House.s
\World\-123456-123456.w

For "\My Package.zip\James's Package\World\-123456-123456.w" to reference "House.s" in its own package it would use FileName("House.s"). To reference "Bungalow.s" in "Some File I Downloaded From The Internet.zip" it would use FileName("..\3DTS Cajon\Bungalow.s"). Because OR knows it's looking for a shape, it will be using the shape virtual file structure, starting in the "Shape" directory for "James's Package". Up one level from this is the virtual directory listing all packages with "Shape" directories, so "..\3DTS Cajon" takes it to the "Shape" directory in the package called "3DTS Cajon".

Kickin life into an old thread... I do understand the above example. Questions:

• How does a route designer find all of the packages he has made use of (for the purpose of gathering them all up to make a distribution package of his own)?
  
• How does he gather those packages together into his own about-to-be-distributed package (call it "Dave's Project")?
  
• After installing "..\3DTS Cajon\Bungalow.s" as part of "Dave's Project", Wayne does "Wayne's Project" and makes use of "Dave's Project\..\3DTS Cajon\Bungalow.s" and wants to package that up to give to Tom, where upon Tom does the same for his project and distributes to Dick, who does the same and gives it to Harry, who does the same and gives it to Chris Van Wagoner. He finds "Harry's Project\Dicks Project\Tom's Project\Waynes Project\Dave's Project\..\3DTS Cajon\Bungalow.s" How is what Chris receives better than the RSC example he cited above?

[James Ross]

Genma Saotome, on 04 November 2012 - 12:01 PM, said:

• How does a route designer find all of the packages he has made use of (for the purpose of gathering them all up to make a distribution package of his own)?

This would be provided by tooling; either we require packages to have metadata listing other packages they've used (which would be updated by RE and related tools when editing) or we have a standalone function that scans a package and finds all external references. This would be a useful function anyway, to find external references that point to things which don't exist, too.

Genma Saotome, on 04 November 2012 - 12:01 PM, said:

• How does he gather those packages together into his own about-to-be-distributed package (call it "Dave's Project")?

I'd personally prefer people simply define their dependencies and avoid the duplication that occurs when you redistribute others' content (and you'd obviously need permission to do so when you do), but this could be provided as a tool function or done manually (copy each package directory in to the "Dave's Project" zip file).

Genma Saotome, on 04 November 2012 - 12:01 PM, said:

• After installing "..\3DTS Cajon\Bungalow.s" as part of "Dave's Project", Wayne does "Wayne's Project" and makes use of "Dave's Project\..\3DTS Cajon\Bungalow.s" and wants to package that up to give to Tom, where upon Tom does the same for his project and distributes to Dick, who does the same and gives it to Harry, who does the same and gives it to Chris Van Wagoner. He finds "Harry's Project\Dicks Project\Tom's Project\Waynes Project\Dave's Project\..\3DTS Cajon\Bungalow.s" How is what Chris receives better than the RSC example he cited above?

I'm not sure where you've ended up with that path from, as including another package with yours will not add any extra levels of referencing (or paths in the file system), but I would expect this situation to result in the following file structure:

• Downloaded file 1234.zip
  
  • 3DTS Cajon
    
    • Shapes\Bungalow.s
  • Dave's Project
    
    • Files Dave added
  • Wayne's Project
    
    • Files Wayne added
  • Tom's Project
    
    • Files Tom added
  • Dick's Project
    
    • Files Dick added
  • Harry's Project
    
    • Files Harry added

Anything outside of the 3DTS Cajon package would still use the reference "..\3DTS Cajon\Bungalow.s".

[Genma Saotome]

Bump.

[YoRyan]

I've been thinking a lot about this topic during my work on the new consist format, and I'm now at a point where I'm not comfortable proceeding further with that work without first defining a new OR content directory structure. This is because, unlike what we did with the timetable format, I'd like to avoid shoehorning the new consist files into Kuju's directory structure.

So, 9 years later, let's open the floor.

I'd like to adopt James Ross' proposal, which you can read here, to use a virtual filesystem composed of multiple cascading content directories and archives. It's how modding is done for other video games and there is a popular off-the-shelf library, called PhysicsFS, that would ease the implementation of such a system. However, I would also like to propose some additional changes.

• James drew a distinction between the archive directory structure, with the package name as the top-level directory and the types of files as subdirectories, and the directory structure as it would be populated by OR, with the different file types as the top-level directories and the packages as subdirectories (so reversed). This would add technical complexity, as well as introduce a layer of indirection that might catch content creators off-guard, so I suggest sticking with the packages as the top-level directories.
  
• It should be possible to for OR content to reference MSTS content and vice-versa, because for the foreseeable future, we will all have to deal with MSTS content. The way this should work is that we would have top-level "ORTS" and "MSTS" namespaces in our virtual filesystem that both kinds of file formats could utilize.
  
  • Think "/ORTS/SomePackage/Activites/SomeActivity.activity-or" and "/MSTS/ROUTES/SomeRoute/ACTIVITIES/SomeActivity.act".
    
  • When you select an MSTS directory for your installation profile, you would be "mounting" that directory as "/MSTS" in the virtual filesystem.
    
  • This would allow players to take advantage of the benefits of a modding-friendly virtual filesystem while still using their legacy MSTS content, by installing (or creating) zip packages that place files into the "/MSTS" namespace. For example, I could create a package that contains my XTracks assets, and then share that single copy among all of my MSTS installation profiles without having to resort to hardlinks or softlinks.
• Speaking of, let's clarify how paths would work, because there should be a well-defined system (call it an "OR path") for files that take advantage of the new virtual filesystem. For this new system, I suggest deprecating relative paths altogether, as well as adopting the "/" character as the path separator. All paths would be absolute, insofar as absolute within the virtual filesystem ("/ORTS/SomePackage/Shapes/bungalow.s"), as opposed to on-disk ("C:\Users\Ryan\Open Rails\SomePackage.zip\SomePackage\Shapes\bungalow.s").
  
  • The use of the ".." token would be prohibited - except for legacy MSTS files still using old-style paths, so long as they don't attempt to "break out" of the "/MSTS" namespace by chaining multiple ".."'s.
    
  • An OR path would be distinguished from an old-style path by its leading "/" character.
• For OR assets that are inextricably linked with other assets (paths and activities vis-a-vis routes), the link should be explicitly made through a JSON attribute with a file path ("'Route': '/ORTS/SomePackage/Routes/SomeRoute.route-or'") rather than a particular location in the filesystem (the "ROUTES\SomeRoute\ACTIVITIES" folder as defined by Kuju).

Overall, the virtual filesystem concept is intuitive to understand and makes modding easy, because you can "patch" files by injecting your own version into an archive with a higher load priority. I would not seriously consider any approaches that involve RDBMS software.

I am prepared to suspend my new consists project and work on the virtual filesystem first (hopefully, with some collaborators), but it will have to wait until after the Monogame merge, as this work would be just as invasive - all of the file I/O calls would have to be swapped for PhysicsFS calls, the launcher would have to be reworked, the command-line interface would have to be re-architected, etc. All of this would entail a massive amount of effort, but IMO it would be worth it to make the content-installing experience much more pleasant for players in the short-term, and to allow Open Rails to expand its horizons in the long-term. It would also open the door to a package management system, a centralized content repository, and multiplayer content sharing if we so choose to go down those roads.

[cjakeman]

YoRyan, on 22 August 2020 - 12:21 PM, said:

I am prepared to suspend my new consists project and work on the virtual filesystem first (hopefully, with some collaborators), but it will have to wait until after the Monogame merge, as this work would be just as invasive.

I see the GitHub page for James work on the Monogame merge now shows up to 34/37 items completed, which is very pleasing.

[conductorchris]

Ryan, I am very pleased to see your work and clear thinking. I think this is very important. I appreciate your knowledge and experience that you bring to your suggestions.
Christopher

[Genma Saotome]

I've expressed reservations about this... perhaps I'm not grasping the benefits, so whenever possible examples would be helpful.

Why the insistence on the unix forward slash?

[YoRyan]

conductorchris, on 22 August 2020 - 05:57 PM, said:

Ryan, I am very pleased to see your work and clear thinking. I think this is very important. I appreciate your knowledge and experience that you bring to your suggestions.
Christopher

Thanks; you're too kind. :D

Genma Saotome, on 22 August 2020 - 08:45 PM, said:

I've expressed reservations about this... perhaps I'm not grasping the benefits, so whenever possible examples would be helpful.

Certainly.


Scenario A: A pair of native OR content packages in ZIP format.

=YoRyan_SuperlinerInteriors.zip=
    ORTS/
        YoRyan_SuperlinerInteriors/
            Shapes/
                supercoach.s
                superdiner.s
                superlounge.s
            Textures/
                supercoach.ace
                superdiner.ace
                superlounge.ace

=YoRyan_DerailFlag.zip=
    ORTS/
        YoRyan_DerailFlag/
            Shapes/
                derailflag.s
            Textures/
                derailflag.ace

The simulator would combine both of them into a single virtual filesystem.

ORTS/
    YoRyan_SuperlinerInteriors/
        Shapes/
            supercoach.s
            superdiner.s
            superlounge.s
        Textures/
            supercoach.ace
            superdiner.ace
            superlounge.ace
    YoRyan_DerailFlag/
        Shapes/
            derailflag.s
        Textures/
            derailflag.ace

Scenario B: The monkey patch. You want to customize my Superliner coach interior texture, so you add a new package that overrides my texture. (Put aside how a "proper" reskin with a separate wagon specification would be defined, as that is outside of the scope of this proposal.)

=YoRyan_SuperlinerInteriors.zip=
    ORTS/
        YoRyan_SuperlinerInteriors/
            Shapes/
                supercoach.s
                superdiner.s
                superlounge.s
            Textures/
                supercoach.ace
                superdiner.ace
                superlounge.ace

=You_SupercoachReplacementSeats=
    ORTS/
        YoRyan_SuperlinerInteriors/
            Textures/
                supercoach.ace

With the following load order (this would be a user preference),

• YoRyan_SuperlinerInteriors.zip
  
• You_SupercoachReplacementSeats.zip

the result is that your texture is loaded in place of my original.

ORTS/
    YoRyan_SuperlinerInteriors/
        Shapes/
            supercoach.s
            superdiner.s
            superlounge.s
        Textures/
            supercoach.ace    *from You_SupercoachReplacementSeats.zip
            superdiner.ace
            superlounge.ace

Scenario C: Now, imagine you have your MSTS content organized as packages instead of uncompressed files in a single directory. You can have a base package for the Kuju stock stuff, an XTracks package, packages for activities and timetables, etc., and all shareable by multiple installation profiles to boot. Upon loading, Open Rails would combine all of them into a single, virtual installation directory. Very elegant, if I do say so myself.

=MicrosoftTrainSimulator1.2.zip=
    MSTS/
        GLOBAL/
            ...
        SOUND/
            ...
        ROUTES/
            ...
        TRAINS/
            ...

=YoRyan_MSTSTimetables.zip=
    MSTS/
        ROUTES/
            USA1/
                ACTIVITIES/
                    YoRyan_USA1_Amtrak.timetable-or
                PATHS/
                    ...
            USA2/
                ACTIVITIES/
                    YoRyan_USA2_EmpireBuilder.timetable-or
                PATHS/
                    ...
        TRAINS/
            CONSISTS/
                ...

=XTracks_Nov2019.zip=
    MSTS/
        GLOBAL/
            SHAPES/
                ...

=Surfliner2.zip=
    MSTS/
        ROUTES/
            Surfliner2/
                ...

Packages further down in the load order would also be able to override previously loaded files, as in scenario B.

Genma Saotome, on 22 August 2020 - 08:45 PM, said:

Why the insistence on the unix forward slash?

Two reasons. First, it further differentiates an Open Rails virtual path from a real Windows file path. Second, backslash is the escape character in JSON strings, so our file formats would look rather silly with every backslash separator doubled up ("\\ORTS\\SomePackage\\Shapes\\SomeShape.s").

[gpz]

Ryan, having such a zip package system would be really welcome. It is great that you brought up this discussion, I haven't even read about it in the past at all!

However I would not be eager at all adding a C library like PhysicsFS as a reference for handling such a basic and essential task as reading from files. I would not even like to use a native 3rd-party C# library for this task either, given that the standard C# classes are more than capable of handling this. (See e.g. the ZipArchive class.) I would rather like to see a custom FileManager class with a GetFile() function, that could easily handle this stuff.

[Genma Saotome]

Ryan, thank you for the examples, very much appreciated!

I see now I did not previously understand the "mount" aspect of the proposal. Your example made that clear and I can see now why there would be a certain appeal to those end users who, for the most part, just download or do minimal subsitutions. Speaking as a route developer it appears to add no useful features and add an extra layer of work. EVERY file I work with is subject to replacement or editing on any day. My larger \route directory has 87,000 files. I expect fewer than 200 were downloaded from public sources. 2300 custom shapes, 2400 custom textures. There is no way I'd package 2400 individual shape packages. Packaging things as whole directories (e.g., \myshapes) would be simple enough but provide nothing whatsoever for the end user that isn't present today.

Am I still missing the point? Perhaps. So what have I still got wrong about this?

[YoRyan]

gpz, on 23 August 2020 - 01:03 AM, said:

However I would not be eager at all adding a C library like PhysicsFS as a reference for handling such a basic and essential task as reading from files. I would not even like to use a native 3rd-party C# library for this task either, given that the standard C# classes are more than capable of handling this. (See e.g. the ZipArchive class.) I would rather like to see a custom FileManager class with a GetFile() function, that could easily handle this stuff.

Well, there is a native C# wrapper available. If it's performance you're concerned about, a slight reduction is inevitable, yes, given that we are loading from ZIP and other kinds of archives.

Ultimately, the use of this library is a call for the management team - but given our extremely stretched resources, I would strongly advocate for a battle-tested solution, used by many other video game engines, that supports a vast array of archive formats (not just ZIP) rather than trying to homebrew something ourselves that might well turn out to perform worse. If only for my sake. :)

Genma Saotome, on 23 August 2020 - 07:05 AM, said:

Speaking as a route developer it appears to add no useful features and add an extra layer of work. EVERY file I work with is subject to replacement or editing on any day. My larger \route directory has 87,000 files. I expect fewer than 200 were downloaded from public sources. 2300 custom shapes, 2400 custom textures. There is no way I'd package 2400 individual shape packages. Packaging things as whole directories (e.g., \myshapes) would be simple enough but provide nothing whatsoever for the end user that isn't present today.

Am I still missing the point? Perhaps. So what have I still got wrong about this?

You're not required to migrate your content to the ORTS namespace (a completely different, yet-to-be-determined structure), nor to store it in a ZIP file - these are optional steps. All of your legacy content would remain available under the "MSTS/" namespace and behave as before. "Packages" can also be plain directories instead of ZIP files, making editing easy.

This is useful for 1) players who want to organize their MSTS directories, and 2) content creators who want to release their work. You know that feeling when you install an activity from trainsim.com, and now your path and consist folders are littered with files of unknown origin? This is a way to track that information and make clean uninstalling a breeze.