[Genma Saotome]

Ryan, it might help everyone if you explain what you mean by virtual data directories that only exist in memory (people do tend to to think in terms of some sort of organizing principle on their disks).

AFAIK there are three approaches to consider when storing all of these files:

• Exactly as KUJU defines things -- everything in one place but problems with both duplicate names and too many files in any non-route branch.
  
• Minroutes -- many installations, but problems with shape, texture, and rolling stock duplication.
  
• Master library -- everything in one place but not using KUJU's tree, with problems "expressing" these master files into those places where the game software looks for files.

For myself, I use #3 where I can (but I rather doubt more than handful of other people do the same) in combination with #2. If I understand where you guys are trying to go it seems to look more like #3 than anything else I can figure out except that unlike a library w/ books sorted onto shelves it sounds more like a storage locker filled w/ unmarked boxes containing who knows what.

[YoRyan]

All of the above, and none of the above. Using the VFS is like Ctrl-C'ing, Ctrl-V'ing the contents of five different directories into a single MSTS directory, except the game will do this for you. Does that make things more clear?

(This should arguably be its own topic, because the virtual filesystem has little or no relation to the eventual Open Rails directory structure. That is a separate thing.)

[Amtrak115]

Ryan,

I was going to jump in again....but I hadn't read the earlier posts....I'll back out now..or until I have another question....

Barry

[Csantucci]

It's the moment also for me to try to understand better.
What about mini-routes installations? How can a trainset be made common to more MSTS-like installations (e.g. base installation, and a mini-route installation)?
Are there actual content structures that will be no more permitted with this new arrangement?

[YoRyan]

By sharing the same VFS source among multiple installation profiles, you can avoid duplicating stuff that is common to multiple installations.

Source: XTracks.zip
  /Global

Source: DekosoftTrains
  /Trains

Source: Surfliner2
  /Routes/Surfliner2

Source: NECv4
  /Routes/USA1A

Profile: Surfliner
  Load order: [XTracks.zip, Surfliner2, DekosoftTrains]

Profile: Northeast Corridor
  Load order: [XTracks.zip, USA1A, DekosoftTrains]

If you do nothing, all of your existing installation profiles will continue to work as normal. So, no, the VFS will not constrain any existing configurations.

[R H Steele]

So a VFS is not like junction or symbolic links? In that OR is providing the VFS through code changes...and the user does not have to create links? Pardon my confusionhttp://www.elvastower.com/forums/public/style_emoticons/default/ko2.gif

[YoRyan]

That's one way to think of it, I guess: OR will build its own data "directory" full of "symbolic links" pointing to the physical files.

The simplest explanation I can muster is still "the VFS combines multiple data directories into one." It is a standard feature in just about every video game released post-MSTS, including all of Dovetail Games' products (Train Simulator, Train Sim World).

Further reading:
https://trac.wildfir...ual_File_System
https://www.randygau...stems-in-games/

[Genma Saotome]

YoRyan, on 10 May 2021 - 02:52 PM, said:

That's one way to think of it, I guess: OR will build its own data "directory" full of "symbolic links" pointing to the physical files.
https://www.randygau...stems-in-games/

Virtual symbolic links, right?

As a statement of objective, it is, IMO, an excellent idea that will solve many problems. I do fear the emphasis on zip files will be a problem on account of lower visibility of what they contain and I suspect the lack of a single, standard directory structure will flummox many people. Bet's lets see how it turns out.

[pschlik]

I think it can be very easy for experienced users to underestimate just how unusual and complicated an exposed file structure like MSTS's GLOBAL, ROUTES, TRAINSET is in 2021. A lot of newbies will come from other games and be completely confused by the added complexity of needing to navigate folders to copy and paste files. The typical user in the present day could not care less about what is in a mod, they expect to be able to paste the file into the folder the readme says, then to be done. Needing to copy shapes, sounds, cabviews, and whatnot is an awful user experience that will be very nice to leave behind. Maybe I can stop having inexperienced players pestering me with advice on how to install something...when it turns out they simply installed it to the TRAINSET folder when it belonged in the TRAINS folder.

[Genma Saotome]

pschlik, on 10 May 2021 - 07:56 PM, said:

I think it can be very easy for experienced users to underestimate just how unusual and complicated an exposed file structure like MSTS's GLOBAL, ROUTES, TRAINSET is in 2021. A lot of newbies will come from other games and be completely confused by the added complexity of needing to navigate folders to copy and paste files. The typical user in the present day could not care less about what is in a mod, they expect to be able to paste the file into the folder the readme says, then to be done. Needing to copy shapes, sounds, cabviews, and whatnot is an awful user experience that will be very nice to leave behind. Maybe I can stop having inexperienced players pestering me with advice on how to install something...when it turns out they simply installed it to the TRAINSET folder when it belonged in the TRAINS folder.

I rather doubt what you hope for will occur with the proposal we're discussing. AFAIK (Ryan, please correct me if I get this wrong) it goes like this: Inside every package will either be every single file necessary or a list of firm paths to those files not included. If the package is a route, and it includes everything, it will include, say all Scalerail shapes, every piece of rolling stock, etc. etc, WITHOUT REGARD TO whether you already have those things and have tweeked values to suit your tastes. That keeps it simple but at a price of considerable duplication w/o regard to what you want.

The alternative is the package has complete paths recorded so instead of including all Scalerail shapes it points to where the person who distributed the package thinks the ScaleRail package exists. If it is there, it works. If it is not there but instead is located somewhere else you either have to change the string for that path or move a copy of the Scalerail package to whatever disk location the newly installed stuff expects to find it.

The solution to the first problem is to require all packages to be dumped into one directory. Good luck finding anything later on. There is no easy solution to the second problem (unless you are patient and don't mind editing all sorts of file paths inside an archive).

In both cases there are circumstances where things can go wrong for your end user.

The reality is we operate within an extremely complex environment. This is not like a commercial game where you install it once and you've got it all. I make content. My \users\dave\MSTS directory is where I keep my SOURCE files. Right now there are 4,742 folders in that tree, holding 128,456 files, spanning 389gb. Backups are stored elsewhere, so are mini-routes. The smaller of the two routes I am building has 8,808 files (that's just what is in the \Routes\Goose Island tree. The other one is larger; it has 89,900 files. Thousands of files are common to both projects. Naturally I don't expect anyone else who is looking over my shoulder to know what's there no matter how carefully I try to explain it because it is very complex -- I do know what is there, and where it is. Now think about dumping all those folders into individual archives and dumping all the archives into one folder. Now try and find something.

In an ideal world any recipient of either of these two routes would not ne4ed to know anything at all about these files, in large part because I already have done that for them. But as soon as you add rolling stock that ideal becomes almost impossible to retain and I would argue that rolling stock probably represents 80-90% of the difficulty in getting a new user going.

This proposal, AFAIK, should help with this, but again, there will be issues, as I noted above. Will it be better? Beats me. It'll solve some major issues for me so long as I can pry things out of those installation packages. But I don't expect everyone do try that. We'll just have to see how it works out.

[YoRyan]

Genma Saotome, on 10 May 2021 - 10:18 PM, said:

AFAIK (Ryan, please correct me if I get this wrong) it goes like this: Inside every package will either be every single file necessary or a list of firm paths to those files not included. If the package is a route, and it includes everything, it will include, say all Scalerail shapes, every piece of rolling stock, etc. etc, WITHOUT REGARD TO whether you already have those things and have tweeked values to suit your tastes. That keeps it simple but at a price of considerable duplication w/o regard to what you want.

...

The solution to the first problem is to require all packages to be dumped into one directory. Good luck finding anything later on.

Again, I have to emphasize—I suppose you could have gotten the wrong impression from the GitHub document—that the virtual filesystem will make no changes to the existing Kuju-derived file formats. The path references inside of them will continue to work as before. (Well, using sheer numbers of ".." tokens to break out of the MSTS directory and using absolute paths will be prohibited, but such malice should never have been permitted in the first place.) Notice the "/OR" namespace at the beginning of all of those file paths in the document? That is for a namespace that does not yet exist, because we have no new editors or file formats.

You still seem to be having trouble understanding the virtual filesystem, because the point of it is that you don't need to "dump everything into one directory"—you can sort your content into multiple physical directories, grouped by however as you see logical, and the game will unite them into a single one.

I think you should go back and read the articles I linked. They do a far better job explaining the concept than I ever could in a 10-minute forum post. Or, spend a few minutes poking around the asset pack files included with Dovetail Games's Train Simulator, a game that also uses a virtual filesystem.

If you're still skeptical, you're absolutely free to continue using your personal setup—Excel spreadsheets and RDBMS databases and all. But speaking for myself, as someone who is also intimately familiar with the MSTS directory structure, I am very interested in having a virtual filesystem like this. It will help me organize the sheer mess that the MSTS folder can become, save disk space by sharing common assets among multiple mini-routes, and reduce my loading times by packing things into archives. And, as pschlik pointed out, it will also help new users who lack the background knowledge that we have. (We still care about new users, right?)

[Genma Saotome]

YoRyan, on 11 May 2021 - 11:26 AM, said:

You still seem to be having trouble understanding the virtual filesystem, because the point of it is that you don't need to "dump everything into one directory"—you can sort your content into multiple physical directories, grouped by however as you see logical, and the game will unite them into a single one.

Hmmm... I thought I described that as "The Alternative"... the use of path statements to point to wherever the files have been placed. I was asking for that in .con files many years ago. I like that as I'm already working with a congruent solution (using symbolic links instead of paths). I think it will solve lots of problems in managing rolling stock and certain widely used assets (e.g. Scalerail). The only drawback I can guess at is dealing with any difference between one's own way of organizing things and whatever is in the mind of the person who put together the package. When the two are identical, great, it all works. When they are not it will require a fair bit of editing to maintain one's own method or you just give y up and use the other guys (and by guys I mean every other person who ever puts together a package).


p.s. when I find time I'm going to stick a full path into some rows of a .con file, just to see if it works.

[YoRyan]

No, it's not the same thing. A virtual filesystem alone does not impose any constraints on "path statements." See the Wildfire Games article.

I'll add that in addition to accepting packages distributed by other people, you will also have the ability to create your own. This is really no different than rearranging content within your MSTS folder; the VFS just gives you an additional axis with which to organize things, plus potentially faster loading times.

[Amtrak115]

Ryan,

I'm going to assume (bad precedence on my part) that you are working on the design of the VFS, I'm also going to assume you have defined somewhere in your head the structure within the VFS that OR is going to use. Can I also assume that eventually, (please not now) you will document that structure somewhere such that developers of editors will know what it is??

Barry

[conductorchris]

I think this sounds great.
Virtual symbolic links as part of open rails are a lot better than those links as part of windows because they have trouble copying and pasting, for example in backups.
It does indeed solve a lot of problems.
Dave, I think it is essentially something you've been asking for: it would allow you to (for example) organize your roster with multiple trainset folders.
Christopher