[Genma Saotome]

 gpz, on 23 August 2020 - 08:05 AM, said:

Dave, the average end user's content folder is hopelessly smashed up, files from all different places are all around in rotes, trains, etc. folders, mixed all together. For them it is an unmaintainable mess, they are able to only save the install packages to somewhere, and reinstall the whole stuff ftom time to time.

And here comes this genuine feature in subject: just point OR to the folder of the downloaded zip-s, and provided these zip-s are properly arranged, everything just starts magically work. No mess, no thinking of where-this-wrong-file-came-from, what version of xtracks I have, what files a concurrent package has overwritten, etc. Everything is just in order, in its correct place, back-searchable.

Yes, I know. Been there myself. Anything i download now goes into \To Install where it gets worked over before it goes into any actionable OR content area. I just about re-write every .wag file I see. .engs take longer so there is a to-do list for them. So I do understand this matter is a complete mess for just about everyone.

It could help route builders who download loads of content for inclusion in their work. My observations, above, are meant to show that any route builder who has already "blown up" the downloaded zip and/or is creating original content cannot be expected to do much more than simply zip the entire \route folder, perhaps taking hte time to do \shapes and \textures as individual zips. This will do nothing for end users. It won't be helping reuse and it won't be helping mods. Assuming there will always be a flow of new content replacing old stuf, then years down the road a route builder will be able to distribute the archives he has obtained. But the flow of new content is only a trickle now compared to years past and if libraries like Trainsim.com are any indication it will be largely limited to rolling stock and Indian content.

My point here is this would be best served as a proposal if it is pitched as a limited improvement. It's not going to be a panacea and it's not too likely to be retro-fitted into the great mass of existing content. Which begs the question of are all interesting ideas going to pay off? Nobody has a crystal ball so no sure answer will be found w/o the passage of time. Given those doubts, might it simply add to the mess? I dunno. It could really help w/ rolling stock. beyond that I have my doubts.


p.s. one more thought... doesn't the use of zip files add an impediment to direct editing? You have to pull files out to edit them and then put them back. Might the idea work as some sort of command file placed in an ordinary directory? All content in sort of a library of content directories and all references to that in some sort of file directory scheme where command files point to he library folders? A bit bulkier yes, but disk is cheap.

[YoRyan]

We don't have a mechanism to handle dependencies, it's true; it's up to users to install prerequisites first, or for route builders to include them in their own packages. Unfortunately there's not really a good solution except to design a package management system with a centralized repository of content, which has been brought up before in this thread, and I don't think we are prepared to do.

It's also true that existing content will not benefit as much from the virtual filesystem - by definition, of course, since we're maintaining backwards compatibility with MSTS. But I disagree with your contention that it will not do anything at all. First off, as an (admittedly very biased) end user, it would allow me to reorganize all of my MSTS content in ZIP files - starting with the 1GB of Kuju data that currently has to be duplicated across all of my installation profiles - and likewise for any new content I download. New route? Create a new installation profile, add the MSTS ZIP, add the XTracks ZIP, add the NewRoads ZIP, and then add the downloaded route folder. Easy, saves disk space, and is neat and tidy. Second, content creators would be able to provide a vastly improved install experience for end users. Instead of going through the error-prone unzip/copy song and dance, you would download the package, add it to your load order, and start playing.

By the way, packages needn't be contained in ZIP's. We will also support plain directories to make editing easy.

[Genma Saotome]

At a larger level of granularity the proposal doesn't appear to do anything that cannot already be done by symbolic links. I already use symbolic links for \global,\sound, and all duplicate instances of \routes. Down at the level of individual folders, such as inside of \trains, the proposal should be quite helpful simply because there are so many folders that get duplicated the use of symbolic links because potentially problematic. That's also where most mods will occur so it is a plus for that as well. I suspect it will also be a plus for the things inside of the activities folders as it will help in backdating one instance of a route while leaving another instance modern. In this regard it should also be useful for backdating one instance of \consists.

So I'll repeat myself here... it's no panacea for all of the messes but it could be quite good for specific messes. IMO "Marketing" the proposal as a useful tool for those specific situations will gain it more support -- and use -- than trying to sell it for all situations, something that implies everybody has to do it everywhere in order to keep things working. Just a suggestion with good intentions.

[YoRyan]

But very few people use symbolic links because of the difficulty and administrative headache involved. (I myself use jdupes to identify and hardlink duplicate files.) The virtual filesystem is intended to be much easier to use.

It's also not so much about "preventing duplication" and "backdating" so much as it is about "composing game data from multiple possible sources," is how I'd pitch it. The target audience is content creators packaging new stuff. It's also completely optional, so if you decide to stick with the old-fashioned way of unzipping stuff in Explorer, you can still do that - I'm neither expecting nor demanding 100% adoption following the next stable release.

Finally, note that any future OR formats will use the ORTS/ namespace, which will be structured completely differently, and will avoid many of the pitfalls of the Kuju directory structure.

[James Ross]

 YoRyan, on 22 August 2020 - 12:21 PM, said:

• James drew a distinction between the archive directory structure, with the package name as the top-level directory and the types of files as subdirectories, and the directory structure as it would be populated by OR, with the different file types as the top-level directories and the packages as subdirectories (so reversed). This would add technical complexity, as well as introduce a layer of indirection that might catch content creators off-guard, so I suggest sticking with the packages as the top-level directories.

I think there was a benefit to this, as you could anchor paths to the appropriate content type - e.g. a train file referencing a wagon would only be able to reference files in the approved location for wagon files (while still allowing content creators to add subdirectories).

I'm going to have a think about this as I am concerned this part of the proposal might be just a little bit too free-form, but the rest sounds good!

 YoRyan, on 22 August 2020 - 12:21 PM, said:

• It should be possible to for OR content to reference MSTS content and vice-versa, because for the foreseeable future, we will all have to deal with MSTS content. The way this should work is that we would have top-level "ORTS" and "MSTS" namespaces in our virtual filesystem that both kinds of file formats could utilize.

Absolutely yes for cross-references, in both directions.

I would lean towards "OR" instead of "ORTS" but I'm not strongly opinionated about that - we have a long history of using OR and ORTS interchangeably, even though they might have meant different things at some point.

 YoRyan, on 22 August 2020 - 12:21 PM, said:

• This would allow players to take advantage of the benefits of a modding-friendly virtual filesystem while still using their legacy MSTS content, by installing (or creating) zip packages that place files into the "/MSTS" namespace. For example, I could create a package that contains my XTracks assets, and then share that single copy among all of my MSTS installation profiles without having to resort to hardlinks or softlinks.

We will have to think about how we want the packages to interact; I know you've continued to talk about installation profiles, but I'd like to see that feature be less visible (certainly not forcing the user to pick one before they can pick anything else).

 YoRyan, on 22 August 2020 - 12:21 PM, said:

• Speaking of, let's clarify how paths would work, because there should be a well-defined system (call it an "OR path") for files that take advantage of the new virtual filesystem. For this new system, I suggest deprecating relative paths altogether, as well as adopting the "/" character as the path separator. All paths would be absolute, insofar as absolute within the virtual filesystem ("/ORTS/SomePackage/Shapes/bungalow.s"), as opposed to on-disk ("C:\Users\Ryan\Open Rails\SomePackage.zip\SomePackage\Shapes\bungalow.s").
  
  • The use of the ".." token would be prohibited - except for legacy MSTS files still using old-style paths, so long as they don't attempt to "break out" of the "/MSTS" namespace by chaining multiple ".."'s.
    
  • An OR path would be distinguished from an old-style path by its leading "/" character.

:thumbup3:

 YoRyan, on 22 August 2020 - 12:21 PM, said:

• For OR assets that are inextricably linked with other assets (paths and activities vis-a-vis routes), the link should be explicitly made through a JSON attribute with a file path ("'Route': '/ORTS/SomePackage/Routes/SomeRoute.route-or'") rather than a particular location in the filesystem (the "ROUTES\SomeRoute\ACTIVITIES" folder as defined by Kuju).

This is where I am very slightly concerned, that you could put that route-or file in any directory inside the package. Maybe that's fine, and obviously certain file types will need to be in specific places for the game to auto-find them (like routes), but that's not true of everything.

 YoRyan, on 22 August 2020 - 12:21 PM, said:

I am prepared to suspend my new consists project and work on the virtual filesystem first (hopefully, with some collaborators), but it will have to wait until after the Monogame merge, as this work would be just as invasive - all of the file I/O calls would have to be swapped for PhysicsFS calls, the launcher would have to be reworked, the command-line interface would have to be re-architected, etc. All of this would entail a massive amount of effort, but IMO it would be worth it to make the content-installing experience much more pleasant for players in the short-term, and to allow Open Rails to expand its horizons in the long-term. It would also open the door to a package management system, a centralized content repository, and multiplayer content sharing if we so choose to go down those roads.

Absolutely want this work, after MonoGame as you say. I've had the multi-player content sharing idea rattling around for a few years since I built the Content Manager contributed tool - it would be super cool to do, but that's a way off yet. :)

 gpz, on 23 August 2020 - 01:03 AM, said:

However I would not be eager at all adding a C library like PhysicsFS as a reference for handling such a basic and essential task as reading from files. I would not even like to use a native 3rd-party C# library for this task either, given that the standard C# classes are more than capable of handling this. (See e.g. the ZipArchive class.) I would rather like to see a custom FileManager class with a GetFile() function, that could easily handle this stuff.

I would be okay with a 3rd party open-source library written entirely in .NET, but I am quite wary of adding a dependency on native code, no matter how high quality/battle-tested it is. It provides a sticking point for both 64bit support and cross-platform support, two things I would like us to start addressing (or at least not making any worse!).

 YoRyan, on 23 August 2020 - 08:02 AM, said:

Ultimately, the use of this library is a call for the management team - but given our extremely stretched resources, I would strongly advocate for a battle-tested solution, used by many other video game engines, that supports a vast array of archive formats (not just ZIP) rather than trying to homebrew something ourselves that might well turn out to perform worse. If only for my sake. :)

I would actually rather we didn't support a "vast array of archive formats" and only supported ZIP, because that makes everyone's life easier in the long-term. Yes, maybe the files won't be as small, but everyone will be able to open them up and poke around inside without any 3rd party software.

 gpz, on 23 August 2020 - 09:25 AM, said:

Most of those other archive formats are just proprietary garbage, OR cannot benefit from them. Not just the performance is my concern. (Although it is also important.) Come on, is it really a good software design, to replace the standard MS-maintained, well defined, managed StreamReader, File.Open, ZipArchive class, etc. with an unmanaged C library, where 1-2 developers are fighting with the maintenance? For such a not too complex task of maintaining a dictionary of files, assigned to their containing prioritized package name? To me it is a big no-no. You may call for the management team, in this case I would especially interested in James's opinion about this. :)

I'm definitely with Peter here; the built-in .NET APIs should provide enough standardised, performant, and reliable basis on which to implement this.

[YoRyan]

 James Ross, on 26 August 2020 - 09:26 AM, said:

I think there was a benefit to this, as you could anchor paths to the appropriate content type - e.g. a train file referencing a wagon would only be able to reference files in the approved location for wagon files (while still allowing content creators to add subdirectories).

...

This is where I am very slightly concerned, that you could put that route-or file in any directory inside the package. Maybe that's fine, and obviously certain file types will need to be in specific places for the game to auto-find them (like routes), but that's not true of everything.

If I'm honest, part of the reason I leaned toward flexibility was to make the PhysicsFs integration simpler; as that is no longer a factor, I'd be fine with revisiting this. We'd have to decide how to handle paths, though, and how to make clear to content creators which "form" of abbreviated VFS path applies for which fields.

 James Ross, on 26 August 2020 - 09:26 AM, said:

I would lean towards "OR" instead of "ORTS" but I'm not strongly opinionated about that - we have a long history of using OR and ORTS interchangeably, even though they might have meant different things at some point.

Fine by me. Might as well save the 2-8 bytes. :)

 James Ross, on 26 August 2020 - 09:26 AM, said:

We will have to think about how we want the packages to interact; I know you've continued to talk about installation profiles, but I'd like to see that feature be less visible (certainly not forcing the user to pick one before they can pick anything else).

Alas, I suspect we are stuck with that feature given the limitations of the Kuju file structure. One example off of the top of my head is UK XTracks versus standard XTracks vis-a-vis the GLOBAL\ folder.

So going forward, one strategy would be to retain profiles for the /MSTS/ namespace, but use a single common /OR/ namespace. Another would be to de-emphasize the profiles by listing all routes from all profiles in the same menu, and likewise for the consists, etc. We could even do both...

 James Ross, on 26 August 2020 - 09:26 AM, said:

I'm definitely with Peter here; the built-in .NET APIs should provide enough standardised, performant, and reliable basis on which to implement this.

Fair enough; I'll stop being lazy and start planning that code. ;)

[Genma Saotome]

Ryan, would you please do me the favor of making an example of how an update to a route would occur? Typically it would contain files for \textures, \shapes, \world, the root of the route, and perhaps tiles. Use as many or as few of those as you want. FWIW the Monon rte did 16 such versioning releases.

What I'm curious about is this: with the above example what's the difference between sending out only net changes vs. whole directories.

Thanks in advance.

[YoRyan]

Well, to start, I'm going to assume the continued use of the MSTS directory structure, as we still have yet to define the OR one, whatever that would look like.

As a content creator releasing your first version, you would release a package that includes the entire route and all of its files. When you release an update, you would have two options: either issue a package that completely replaces the original, or issue a much smaller net-change package that depends on the original and must come after it in the load order. In the first scenario, the update package would contain new copies of all of the route's files. In the second scenario, it would include only those files that have changed between versions.

So it's not substantially different compared to the way we do things today, except that OR would handle the unzip/copy/paste steps automatically, and users would now have the opportunity to cleanly uninstall content.

[RR1]

 YoRyan, on 27 August 2020 - 10:27 PM, said:

Well, to start, I'm going to assume the continued use of the MSTS directory structure, as we still have yet to define the OR one, whatever that would look like.

As a content creator releasing your first version, you would release a package that includes the entire route and all of its files. When you release an update, you would have two options: either issue a package that completely replaces the original, or issue a much smaller net-change package that depends on the original and must come after it in the load order. In the first scenario, the update package would contain new copies of all of the route's files. In the second scenario, it would include only those files that have changed between versions.

So it's not substantially different compared to the way we do things today, except that OR would handle the unzip/copy/paste steps automatically, and users would now have the opportunity to cleanly uninstall content.

So, assuming the Monon example, are you saying that in the second case OR would load the original route then load each of the 15 updates until it reached the 16th? Also, at the 16th update, could not the entire route now be 3, 4, 5 times or (even way) more bigger than a single version? What am I not understanding here? Thanks.

[gpz]

By reading the directories of the packages first, it is straightforward to select the newest files only to load.