[Genma Saotome]

conductorchris, on 12 May 2021 - 10:26 AM, said:

Dave, I think it is essentially something you've been asking for: it would allow you to (for example) organize your roster with multiple trainset folders.
Christopher

Yes, I think it could be rather useful.

The part I do not understand is how and where the path statements are recorded. I saw something before that said full path, no relative references and that's fine by me. I'd just like to know where those path statements are going to be. I'm assuming in many cases is everywhere a filename is written in any current or equivalent-to MSTS file. The part I'm most curious about is what happens when the software itself "assumes" the file it wants is where KUJU designated it belongs. For instance, KUJU "assumes" the textures for a locomotive are in the same folder as the .s file. That does make sense and there is, at least initially, no reason to think that would ever change. OTOH KUJU "assumed" it would find all the textures for static objects in \Textures. But what if they are someplace else? The path to wherever the are is easy to write... but where does that path statement reside so it gets conveyed to the program?

[James Ross]

YoRyan, on 26 August 2020 - 11:34 PM, said:

So going forward, one strategy would be to retain profiles for the /MSTS/ namespace, but use a single common /OR/ namespace. Another would be to de-emphasize the profiles by listing all routes from all profiles in the same menu, and likewise for the consists, etc. We could even do both...

The latter is what I was going for - the profiles can still exist, but you wouldn't need to navigate them every time. If you can add multiple MSTS folders to a profile, you might be able to get away with only one profile for MSTS content too, although often not.

Genma Saotome, on 12 May 2021 - 10:49 AM, said:

The part I do not understand is how and where the path statements are recorded. I saw something before that said full path, no relative references and that's fine by me. I'd just like to know where those path statements are going to be. I'm assuming in many cases is everywhere a filename is written in any current or equivalent-to MSTS file. The part I'm most curious about is what happens when the software itself "assumes" the file it wants is where KUJU designated it belongs. For instance, KUJU "assumes" the textures for a locomotive are in the same folder as the .s file. That does make sense and there is, at least initially, no reason to think that would ever change. OTOH KUJU "assumed" it would find all the textures for static objects in \Textures. But what if they are someplace else? The path to wherever the are is easy to write... but where does that path statement reside so it gets conveyed to the program?

For MSTS-organised content, the files would be subject to the same restrictions on location within the virtual file system. So if the locomotive shape is in virtual folder "/MSTS/TRAINS/TRAINSET/KIHA140", the textures would also have to be in virtual folder "/MSTS/TRAINS/TRAINSET/KIHA140". The change is that those files would not need to necessarily be in the same physical folder, as long as the packages and virtual paths lined up as described.

For OR-organised content, the current plan is for everything to be referenced by a complete virtual path. That means nothing will be assumed to be in any particular folder and all paths would be absolute, which is fine because it is all virtual.

I do not believe we have currently planned how the program will find "origin" files in OR-organised content yet, e.g. routes (equivalent to the .trk files), consists, timetables, activities, etc.

[Genma Saotome]

James Ross, on 19 May 2021 - 11:35 AM, said:

For MSTS-organised content, the files would be subject to the same restrictions on location within the virtual file system. So if the locomotive shape is in virtual folder "/MSTS/TRAINS/TRAINSET/KIHA140", the textures would also have to be in virtual folder "/MSTS/TRAINS/TRAINSET/KIHA140". The change is that those files would not need to necessarily be in the same physical folder, as long as the packages and virtual paths lined up as described.

In the case of textures would there be a mechanism present that allows the target texture to have a physical name different than what is recorded in the .s file? The utility here is for reskins. For example locomotive model SP1234.s uses SP1234.ace. That shape might be perfectly suitable skinned for other railroads and anyone doing that would probably not be inclined to use the same ace file name -- the letters SP do mean something significant. A different texture for the locomotive might be NKP4567 and so it is reasonable that the reskinner would want to use that name for his reskin file. Somewhere there needs to be something simple that tells the software SP1234.ace has been replaced by NKP5678.ace

James Ross, on 19 May 2021 - 11:35 AM, said:

For OR-organised content, the current plan is for everything to be referenced by a complete virtual path. That means nothing will be assumed to be in any particular folder and all paths would be absolute, which is fine because it is all virtual.

Will you consider use of a token for a complete path? I mean something like
$Path "Dales Textures" = {full physical path} and then where ever that full path is needed WITHIN that archive one would type
"Dales Textures\acleantrack1.ace". It's a lot more terse than typing "E:\Bin\Open Rails - US West 1966-1980\Routes\Siskiyou\Textures\acleantrack1.ace", especially if that path is needed many dozens of times.

[YoRyan]

Genma Saotome, on 19 May 2021 - 09:16 PM, said:

In the case of textures would there be a mechanism present that allows the target texture to have a physical name different than what is recorded in the .s file?

That could be a useful feature, but it's a discussion for another day. It's unrelated to the virtual filesystem.

Genma Saotome, on 19 May 2021 - 09:16 PM, said:

Will you consider use of a token for a complete path?

Full physical paths (with the drive letter in front) constitute a security hole and Open Rails should never accept them. If they work right now, it is a bug.

James was talking about full virtual paths.

As for some sort of abbreviation token, I dunno, maybe? It's impossible to say without having actually defined any new OR formats. If we ever actually have this discussion, it's going to be in conjunction with the introduction of new tooling, so you won't be writing these files by hand anyway.

[James Ross]

Genma Saotome, on 19 May 2021 - 09:16 PM, said:

In the case of textures would there be a mechanism present that allows the target texture to have a physical name different than what is recorded in the .s file?

Such a system could be supported with or without the virtual file system and as such is not part of the current discussion.

We can potentially come back to this when we start thinking about locomotive/wagon data files and model formats, since there might be a much nicer way of building it into the OR content formats.

Genma Saotome, on 19 May 2021 - 09:16 PM, said:

Will you consider use of a token for a complete path? I mean something like
$Path "Dales Textures" = {full physical path} and then where ever that full path is needed WITHIN that archive one would type
"Dales Textures\acleantrack1.ace". It's a lot more terse than typing "E:\Bin\Open Rails - US West 1966-1980\Routes\Siskiyou\Textures\acleantrack1.ace", especially if that path is needed many dozens of times.

You cannot have an absolute physical path inside content when using the virtual file system. The only paths that will work are:

• Absolute virtual paths
  
• Relative virtual paths but probably only in MSTS content

The most likely absolute virtual path to that texture is "/MSTS/Routes/Siskiyou/Textures/acleantrack1.ace". It does not matter where you physically have the files, and you will never use those physical paths anywhere other than in setting up Open Rails profiles.

If you want to use alternative textures on the same route, such as for different periods in the railway's operations, you would take advantage of the virtual file system's layering:

Profile "US West 1966-1980" might have two folders, in this order:

• "E:\Bin\Open Rails - original" (which contains the original "Routes\Siskiyou")
  
• "E:\Bin\Open Rails - US West 1966-1980" (which contains files like "Routes\Siskiyou\Textures\acleantrack1.ace" to override the originals)

The virtual file system is very similar to using relative paths everywhere, except that you can combine and override files without editing the originals.

[Genma Saotome]

James Ross, on 20 May 2021 - 08:34 AM, said:

Such a system could be supported with or without the virtual file system and as such is not part of the current discussion.

We can potentially come back to this when we start thinking about locomotive/wagon data files and model formats, since there might be a much nicer way of building it into the OR content formats.

Thanks.

The reason behind the request is that the file name is inside the .s file as one of the data recorded by the 3dmodeling software. It doesn't appear anywhere else and therein is the problem when it comes to reskins. The reskinner either uses the original file name for his completely new art even tho it may not be relevant at all to the new art... or he makes a copy of the .s file, decompresses it, and edits the file name he finds inside to match his new art file. I routinely do the later, but that's me. My point is the problem could be solved by any sort of replacement statement made somewhere editable. I personally favor using the .sd file that as it seems the .sd file is where KUJU put game data that isn't recorded by the modeling software. Your plans for using paths perked my recalling the topic in that perhaps the answer can be found there as well.

James Ross, on 20 May 2021 - 08:34 AM, said:

Profile "US West 1966-1980" might have two folders, in this order:

"E:\Bin\Open Rails - original" (which contains the original "Routes\Siskiyou")
"E:\Bin\Open Rails - US West 1966-1980" (which contains files like "Routes\Siskiyou\Textures\acleantrack1.ace" to override the originals)


The virtual file system is very similar to using relative paths everywhere, except that you can combine and override files without editing the originals.

I do understand your plans for the virtual layering. What I am ignorant of is this: where within an archive does the text you wrote above belong?

[YoRyan]

Profiles are managed by users themselves. The archives will be quite simple—their only function will be to contain the files themselves, plus possibly some basic metadata like an author list and some documentation.

Users will add archives and directories to their load orders using physical paths. But, to reiterate, physical paths in content files will be (and should already be) illegal.

[gpz]

I re-read some comments of this super long thread started 10 years ago, and got into a mood of making a sprint with the development of the virtual file system. (It took a week or so.) It can't be so complicated, I thought, and indeed, its complexity is just a fraction of the Monogame migration's. So here it is, interested developers please see the pull request on GitHub. I made the PR in a hope that Ryan had not done much coding work on this feature yet. At least this is what I concluded from this discussion thread.

Comments I also documented in the code for not losing it:

The virtual file system can be initialized from a config file with lines e.g. C:\dir\archive.zip\subdir\ /MSTS/mountpoint/, or from a directory. Found subsequent archives are auto-mounted to their respective locations. The configuration file consits of lines of format:

Drive:\SystemPath\ /MountPoint/
SystemPath
- may use "/" or "\", but MountPoint may use "/" only as a directory separator.
- with spaces in it must be quoted, e.g. "C:\My Routes\USA85\" /MSTS/ROUTES/USA85/
- referring to a directory must end with "/" or "\" to assure that only _below_ dirs and files will be visible _below_ the mount point.
- referring to an archive must not end with "/" or "\", e.g. C:\TEMP\MSTS1.2.zip /MSTS/
- may refer to an archive subdirectory, e.g.: C:\routes.zip\USA3\ /MSTS/ROUTES/USA3/
- may _not_ refer to a single non-archive file, neither a non-archive file within an archive.
- is case-insensitive.
- nested archives are unsupported. (archive inside another archive)
MountPoint
- must end with "/" to avoid confusion and to assure that mounting will be done _below_ the given path.
- may not contain spaces, may not be quoted.
- is case-insensitive, will be converted to all-uppercase internally.

Since I am not a software architect, and I wrote this whole thing from the top of my head, a deep review would be very welcome. If James or Chris is still around here, it would be really helpful from them to take a look onto the code. The code is based on SharpCompress to extract the archives. I haven't made changes to contrib software in the OpenRails repository, but as long as someone uses a simple directory as the VFS "configuration profile", everything should work as before, in this case the change is transparent. The user interface in the Menu Options surely needs some refinement, I just made a quick change on it without much design. One can select a file as well on the Content frame now, and there is an Experimental setting to possible log all VFS file accesses. I left the OpenRails code where files get written intact, although this VFS also has writing abilities by my intention, to system directories only, not to archives. (The code is there theoretically, but I have not tested that extensively.)

[cjakeman]

gpz, on 04 December 2021 - 06:54 AM, said:

If James or Chris is still around here, it would be really helpful from them to take a look onto the code.

Yes, indeed.

Many thanks for moving this forward.

[Genma Saotome]

Good to see you here again Peter.

[gpz]

Two basic file operations are missing from this PR yet: DELETE and LINK inside the VFS.

Let's take an example of the CoolPackage_1.0.zip having the following entries:

/ROUTES/USA85/TEXTURES/test.ace
/ROUTES/USA85/TEXTURES/reallycool.ace

It comes to the package creator's mind, that he made a mistake, because the test.ace causes problems there for some reason, and he also needs the "boring.ace" from the "boring" route, but with name "notsoboring.ace", and he doesn't want to redistribute that file, only link it from the original place.

A possible solution is that the VFS would allow two special entry types in an archive, like to following:
CoolPackage_1.1.zip would contain the following entries:

/ROUTES/USA85/TEXTURES/test.ace.or-delete
/ROUTES/USA85/TEXTURES/notsoboring.ace.or-link

with the latter one containing the following text:

/MSTS/ROUTES/BORING/BORING.ACE

Then if both packages were mounted to the /MSTS/ mountpoint, the latter package could make the necessary changes:

"C:\downloadedpackages\BoringRoute.zip\Train Simulator\" /MSTS/
C:\downloadedpackages\CoolPackage_1.0.zip /MSTS/
C:\downloadedpackages\CoolPackage_1.1.zip /MSTS/

But the format of these special files needs to be decided by the OR community, (If such features are necessary at all, of course.) I haven't implemented any of these "proposed" features yet in my PR.

[Genma Saotome]

A couple of real world practices:

What happens if a route name has been edited to something else?
What happens when you decide the boring route really is boring and so you delete it to save disk space?
What happens when you install something in a check-it-out-first area before moving it to where it will be used?
What does one do with multiple occurrences of the same route? This could be \development, \testing, and \release or it could be \1930-52, \1953-65, and \1966-99?

I expect many of the answers will be procedural but to whatever degree that's not all cases will be an issue.

[gpz]

Genma Saotome, on 05 December 2021 - 12:59 PM, said:

What happens if a route name has been edited to something else?

A player user doesn't edit route names (I assume you are mentioning directory names, as that is the only thing matters), but if the downloaded newer package has the route directory renamed, they can still mount the route to its original directory name by modifying the mount configuration file, without altering the downloaded archive.

Genma Saotome, on 05 December 2021 - 12:59 PM, said:

What happens when you decide the boring route really is boring and so you delete it to save disk space?

Obviously, it is the scope of your backup software. :-) But answering your real question: A good Package Manager Software's role is to reliably handle dependecies. However the VFS is not a package manager, it is just providing the background for developing one.

Genma Saotome, on 05 December 2021 - 12:59 PM, said:

What happens when you install something in a check-it-out-first area before moving it to where it will be used?

Just add a line to your VFS mounting configuration file, like

C:\check-it-out-first\just-downloaded.zip /MSTS/

and try it out.

Genma Saotome, on 05 December 2021 - 12:59 PM, said:

What does one do with multiple occurrences of the same route? This could be \development, \testing, and \release or it could be \1930-52, \1953-65, and \1966-99?

You may have multiple mounting configuration files as "profiles", with each one referencing a different directory as your route base dir, while having the rest of the references the same. Only alter the following line in the 3 configuration files while leaving everything else intact:

C:\MSTS-Route-Development\MyRoute\1930-52\ /MSTS/ROUTES/MYROUTE/

[Genma Saotome]

Thank you for the answers. This is going to be interesting.

[cjakeman]

gpz, on 04 December 2021 - 06:54 AM, said:

Since I am not a software architect, and I wrote this whole thing from the top of my head, a deep review would be very welcome.

Anyone who can write this whole thing from the top of his head is very talented. (76 files changed because it's pretty fundamental.)

I took your code diff today for a look-see. Here's what I did, in case anyone else wants to do the same:




I spotted the new button "File". (The old Change ... is now labelled Dir, presumably to save space).

I created a configuration file for this File button to read. My file was at "D:\OR\VFS\MSTS\VFS.ini" but I think the path and filename could be anything at all.


I added 3 statements to the VFS.ini file which mounted Zip archives to the virtual filesystem:


(I've added comments on the right as I don't expect comments directly in the file are supported yet.)

I used Peter Newell's CTN Test Route as an example because it's small.I split it into 3 Zip archives just to see if that would work - and it did, perfectly.

It was very nice to be using Zip archives instead of uncompressed folders. So much simpler (and smaller).

Note that the names of the Zip archives must match the folder they contain. Not sure if this is a permanent restriction.

After this set up, I have just scratched the surface, so will be exploring the functionality (and the code) much further. But I hope I've "broken the ice" and encouraged a few other people to explore the possibilities.


P.S. Peter's work is not yet included in the Unstable Version of Open Rails, so if anyone wants to know how to get that code running on their PC, please send me a PR.