[James Ross]

 smmille1, on 30 December 2011 - 04:19 PM, said:

1. Why do the zip files contain a top level directory (and why don't we then care about the zip names)? Assuming we're allowing folders or zip files, wouldn't it make more sense to have the zip name AS the package folder name?

As I explained previously in this thread, the zip filename is excluded because websites and browsers are bad at preserving the filename reliably. An alternative, which I may have also mentioned previously, is to have some metadata in the zip (i.e. a specific file in the root) that provides the package name instead of the directory (which would also work for directory-packages and let them be freely renamed). I'm not too fussed how we get the package name but I'm pretty sure the zip filename is out.

 smmille1, on 30 December 2011 - 04:19 PM, said:

2. If we're referring to files outside of the current package, why the ".."? If there aren't going to be separately aliased content folders and we're not actually referring to filesystem locations, then I think you can make the assumption of if it starts with a slash or backslash (there's a mini-discussion in itself), then it starts with a package, and if not, then it's local to the current package.

It makes no difference, if we implement the virtual file structure properly, but people are familiar with the ..\ notation so that's what I've used in my examples.

 smmille1, on 30 December 2011 - 04:19 PM, said:

3. What is the exact benefit to segregating by file type and then inferring the type folder beyond saving a couple of bytes in the referencing string? I think you've said that you can have subfolders inside the type folders (textures, shapes, etc.), but it would seem to me that if I create a package with several components, I might rather organize things by relationship, especially if I have several different groups of things (buildings, trees, signs, etc.) in the package.

Saving those bytes also reduces the chance of errors, like typos, and misunderstandings about what it references. The intention with the strict 1st level within a package is to make the contents of packages more discoverable/explorable by both people and programs, such that neither I nor OR needs to, for example, search the entire zip file for wherever you've hidden the activity files. I agree that the groups of shape (.s), shape data (.sd) and textures (.ace) for particular objects might seem like grouping, but you can share the textures between shapes, so I think pretty much everything should be grouped by type ultimately. There could be exceptions, such as .s and .sd files both being in "Shape", and the special consideration for routes I've mentioned before.

[longiron]

 wacampbell, on 30 December 2011 - 02:55 PM, said:

I certainly agree that what MSTS is using, and what James proposes are actually RDBMS. It makes me chuckle when I hear people say 'lets use a rdbms'.

Wayne,
When I say use a RDBMS, I'm talking about a true database. Tables, Primary and secondary keys, rows, columns, blob data and outside references. No more individual files for train data. It works well for eng, wag files as they share common parameters and data structures, and may work for routes. It makes it extremely easy to use common data sets as many items can reference the same table/columns/rows. It does demand a tool set to manage data and some kind of tool to package data for sharing.

[cjakeman]

I've had some new commercial routes for Christmas :-) Some very impressive scenery from Making Tracks (UK) Great Eastern and Greater Eastern and also the local preserved Nene Valley Railway.

I had to re-install MSTS (for other reasons) and discovered that these routes will only play nicely together if they are installed in a certain sequence. (OR pointed the finger at TSection.dat) Maybe there are even combinations of routes which can never co-exist.

Perhaps this is a well-known problem (and I apologise if I'm going over old ground) but will these proposals for Overall File System and Data Structure avoid this particular issue?

[longiron]

You've discovered the dreaded Tsection.dat issue. Since trackwork in MSTS is like model RR sectional track, all the track pieces must be defined for the game. Different routes were built using different Tsection.dat. In general, later versions of Tsection should work with older routes. But you must be careful during installation that it doesn't get overwritten with an earlier version.

OR proposal is to use procedural track. Essentially a profile extruded along the railway route. So the MSTS type of issues should not be a problem.

[Genma Saotome]

 James Ross, on 30 December 2011 - 03:55 PM, said:

Example Files
\Some File I Downloaded From The Internet.zip

\3DTS Cajon

\Shape\Bungalow.s

\My Package.zip

\James's Package

\Shape\House.s
\World\-123456-123456.w

For "\My Package.zip\James's Package\World\-123456-123456.w" to reference "House.s" in its own package it would use FileName("House.s"). To reference "Bungalow.s" in "Some File I Downloaded From The Internet.zip" it would use FileName("..\3DTS Cajon\Bungalow.s"). Because OR knows it's looking for a shape, it will be using the shape virtual file structure, starting in the "Shape" directory for "James's Package". Up one level from this is the virtual directory listing all packages with "Shape" directories, so "..\3DTS Cajon" takes it to the "Shape" directory in the package called "3DTS Cajon".

Kickin life into an old thread... I do understand the above example. Questions:

• How does a route designer find all of the packages he has made use of (for the purpose of gathering them all up to make a distribution package of his own)?
  
• How does he gather those packages together into his own about-to-be-distributed package (call it "Dave's Project")?
  
• After installing "..\3DTS Cajon\Bungalow.s" as part of "Dave's Project", Wayne does "Wayne's Project" and makes use of "Dave's Project\..\3DTS Cajon\Bungalow.s" and wants to package that up to give to Tom, where upon Tom does the same for his project and distributes to Dick, who does the same and gives it to Harry, who does the same and gives it to Chris Van Wagoner. He finds "Harry's Project\Dicks Project\Tom's Project\Waynes Project\Dave's Project\..\3DTS Cajon\Bungalow.s" How is what Chris receives better than the RSC example he cited above?

[James Ross]

 Genma Saotome, on 04 November 2012 - 12:01 PM, said:

• How does a route designer find all of the packages he has made use of (for the purpose of gathering them all up to make a distribution package of his own)?

This would be provided by tooling; either we require packages to have metadata listing other packages they've used (which would be updated by RE and related tools when editing) or we have a standalone function that scans a package and finds all external references. This would be a useful function anyway, to find external references that point to things which don't exist, too.

 Genma Saotome, on 04 November 2012 - 12:01 PM, said:

• How does he gather those packages together into his own about-to-be-distributed package (call it "Dave's Project")?

I'd personally prefer people simply define their dependencies and avoid the duplication that occurs when you redistribute others' content (and you'd obviously need permission to do so when you do), but this could be provided as a tool function or done manually (copy each package directory in to the "Dave's Project" zip file).

 Genma Saotome, on 04 November 2012 - 12:01 PM, said:

• After installing "..\3DTS Cajon\Bungalow.s" as part of "Dave's Project", Wayne does "Wayne's Project" and makes use of "Dave's Project\..\3DTS Cajon\Bungalow.s" and wants to package that up to give to Tom, where upon Tom does the same for his project and distributes to Dick, who does the same and gives it to Harry, who does the same and gives it to Chris Van Wagoner. He finds "Harry's Project\Dicks Project\Tom's Project\Waynes Project\Dave's Project\..\3DTS Cajon\Bungalow.s" How is what Chris receives better than the RSC example he cited above?

I'm not sure where you've ended up with that path from, as including another package with yours will not add any extra levels of referencing (or paths in the file system), but I would expect this situation to result in the following file structure:

• Downloaded file 1234.zip
  
  • 3DTS Cajon
    
    • Shapes\Bungalow.s
  • Dave's Project
    
    • Files Dave added
  • Wayne's Project
    
    • Files Wayne added
  • Tom's Project
    
    • Files Tom added
  • Dick's Project
    
    • Files Dick added
  • Harry's Project
    
    • Files Harry added

Anything outside of the 3DTS Cajon package would still use the reference "..\3DTS Cajon\Bungalow.s".

[Genma Saotome]

Bump.

[YoRyan]

I've been thinking a lot about this topic during my work on the new consist format, and I'm now at a point where I'm not comfortable proceeding further with that work without first defining a new OR content directory structure. This is because, unlike what we did with the timetable format, I'd like to avoid shoehorning the new consist files into Kuju's directory structure.

So, 9 years later, let's open the floor.

I'd like to adopt James Ross' proposal, which you can read here, to use a virtual filesystem composed of multiple cascading content directories and archives. It's how modding is done for other video games and there is a popular off-the-shelf library, called PhysicsFS, that would ease the implementation of such a system. However, I would also like to propose some additional changes.

• James drew a distinction between the archive directory structure, with the package name as the top-level directory and the types of files as subdirectories, and the directory structure as it would be populated by OR, with the different file types as the top-level directories and the packages as subdirectories (so reversed). This would add technical complexity, as well as introduce a layer of indirection that might catch content creators off-guard, so I suggest sticking with the packages as the top-level directories.
  
• It should be possible to for OR content to reference MSTS content and vice-versa, because for the foreseeable future, we will all have to deal with MSTS content. The way this should work is that we would have top-level "ORTS" and "MSTS" namespaces in our virtual filesystem that both kinds of file formats could utilize.
  
  • Think "/ORTS/SomePackage/Activites/SomeActivity.activity-or" and "/MSTS/ROUTES/SomeRoute/ACTIVITIES/SomeActivity.act".
    
  • When you select an MSTS directory for your installation profile, you would be "mounting" that directory as "/MSTS" in the virtual filesystem.
    
  • This would allow players to take advantage of the benefits of a modding-friendly virtual filesystem while still using their legacy MSTS content, by installing (or creating) zip packages that place files into the "/MSTS" namespace. For example, I could create a package that contains my XTracks assets, and then share that single copy among all of my MSTS installation profiles without having to resort to hardlinks or softlinks.
• Speaking of, let's clarify how paths would work, because there should be a well-defined system (call it an "OR path") for files that take advantage of the new virtual filesystem. For this new system, I suggest deprecating relative paths altogether, as well as adopting the "/" character as the path separator. All paths would be absolute, insofar as absolute within the virtual filesystem ("/ORTS/SomePackage/Shapes/bungalow.s"), as opposed to on-disk ("C:\Users\Ryan\Open Rails\SomePackage.zip\SomePackage\Shapes\bungalow.s").
  
  • The use of the ".." token would be prohibited - except for legacy MSTS files still using old-style paths, so long as they don't attempt to "break out" of the "/MSTS" namespace by chaining multiple ".."'s.
    
  • An OR path would be distinguished from an old-style path by its leading "/" character.
• For OR assets that are inextricably linked with other assets (paths and activities vis-a-vis routes), the link should be explicitly made through a JSON attribute with a file path ("'Route': '/ORTS/SomePackage/Routes/SomeRoute.route-or'") rather than a particular location in the filesystem (the "ROUTES\SomeRoute\ACTIVITIES" folder as defined by Kuju).

Overall, the virtual filesystem concept is intuitive to understand and makes modding easy, because you can "patch" files by injecting your own version into an archive with a higher load priority. I would not seriously consider any approaches that involve RDBMS software.

I am prepared to suspend my new consists project and work on the virtual filesystem first (hopefully, with some collaborators), but it will have to wait until after the Monogame merge, as this work would be just as invasive - all of the file I/O calls would have to be swapped for PhysicsFS calls, the launcher would have to be reworked, the command-line interface would have to be re-architected, etc. All of this would entail a massive amount of effort, but IMO it would be worth it to make the content-installing experience much more pleasant for players in the short-term, and to allow Open Rails to expand its horizons in the long-term. It would also open the door to a package management system, a centralized content repository, and multiplayer content sharing if we so choose to go down those roads.

[cjakeman]

 YoRyan, on 22 August 2020 - 12:21 PM, said:

I am prepared to suspend my new consists project and work on the virtual filesystem first (hopefully, with some collaborators), but it will have to wait until after the Monogame merge, as this work would be just as invasive.

I see the GitHub page for James work on the Monogame merge now shows up to 34/37 items completed, which is very pleasing.

[conductorchris]

Ryan, I am very pleased to see your work and clear thinking. I think this is very important. I appreciate your knowledge and experience that you bring to your suggestions.
Christopher