Elvas Tower: Overall File System and Data Structure - Elvas Tower

Jump to content

Posting Rules

All new threads will be started by members of the Open Rails team, Staff, and/or Admins. Existing threads started in other forums may get moved here when it makes sense to do so.

Once a thread is started any member may post replies to it.
  • 17 Pages +
  • « First
  • 11
  • 12
  • 13
  • 14
  • 15
  • Last »
  • You cannot start a new topic
  • You cannot reply to this topic

Overall File System and Data Structure Rate Topic: ***** 1 Votes

#121 User is offline   Genma Saotome 

  • Owner Emeritus and Admin
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 15,307
  • Joined: 11-January 04
  • Gender:Male
  • Location:United States
  • Simulator:Open Rails
  • Country:

Posted 12 May 2021 - 10:49 AM

 conductorchris, on 12 May 2021 - 10:26 AM, said:

Dave, I think it is essentially something you've been asking for: it would allow you to (for example) organize your roster with multiple trainset folders.
Christopher


Yes, I think it could be rather useful.

The part I do not understand is how and where the path statements are recorded. I saw something before that said full path, no relative references and that's fine by me. I'd just like to know where those path statements are going to be. I'm assuming in many cases is everywhere a filename is written in any current or equivalent-to MSTS file. The part I'm most curious about is what happens when the software itself "assumes" the file it wants is where KUJU designated it belongs. For instance, KUJU "assumes" the textures for a locomotive are in the same folder as the .s file. That does make sense and there is, at least initially, no reason to think that would ever change. OTOH KUJU "assumed" it would find all the textures for static objects in \Textures. But what if they are someplace else? The path to wherever the are is easy to write... but where does that path statement reside so it gets conveyed to the program?

#122 User is offline   James Ross 

  • Open Rails Developer
  • Group: Status: Elite Member
  • Posts: 5,488
  • Joined: 30-June 10
  • Gender:Not Telling
  • Simulator:Open Rails
  • Country:

Posted 19 May 2021 - 11:35 AM

 YoRyan, on 26 August 2020 - 11:34 PM, said:

So going forward, one strategy would be to retain profiles for the /MSTS/ namespace, but use a single common /OR/ namespace. Another would be to de-emphasize the profiles by listing all routes from all profiles in the same menu, and likewise for the consists, etc. We could even do both...

The latter is what I was going for - the profiles can still exist, but you wouldn't need to navigate them every time. If you can add multiple MSTS folders to a profile, you might be able to get away with only one profile for MSTS content too, although often not.

 Genma Saotome, on 12 May 2021 - 10:49 AM, said:

The part I do not understand is how and where the path statements are recorded. I saw something before that said full path, no relative references and that's fine by me. I'd just like to know where those path statements are going to be. I'm assuming in many cases is everywhere a filename is written in any current or equivalent-to MSTS file. The part I'm most curious about is what happens when the software itself "assumes" the file it wants is where KUJU designated it belongs. For instance, KUJU "assumes" the textures for a locomotive are in the same folder as the .s file. That does make sense and there is, at least initially, no reason to think that would ever change. OTOH KUJU "assumed" it would find all the textures for static objects in \Textures. But what if they are someplace else? The path to wherever the are is easy to write... but where does that path statement reside so it gets conveyed to the program?

For MSTS-organised content, the files would be subject to the same restrictions on location within the virtual file system. So if the locomotive shape is in virtual folder "/MSTS/TRAINS/TRAINSET/KIHA140", the textures would also have to be in virtual folder "/MSTS/TRAINS/TRAINSET/KIHA140". The change is that those files would not need to necessarily be in the same physical folder, as long as the packages and virtual paths lined up as described.

For OR-organised content, the current plan is for everything to be referenced by a complete virtual path. That means nothing will be assumed to be in any particular folder and all paths would be absolute, which is fine because it is all virtual.

I do not believe we have currently planned how the program will find "origin" files in OR-organised content yet, e.g. routes (equivalent to the .trk files), consists, timetables, activities, etc.

#123 User is offline   Genma Saotome 

  • Owner Emeritus and Admin
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 15,307
  • Joined: 11-January 04
  • Gender:Male
  • Location:United States
  • Simulator:Open Rails
  • Country:

Posted 19 May 2021 - 09:16 PM

 James Ross, on 19 May 2021 - 11:35 AM, said:

For MSTS-organised content, the files would be subject to the same restrictions on location within the virtual file system. So if the locomotive shape is in virtual folder "/MSTS/TRAINS/TRAINSET/KIHA140", the textures would also have to be in virtual folder "/MSTS/TRAINS/TRAINSET/KIHA140". The change is that those files would not need to necessarily be in the same physical folder, as long as the packages and virtual paths lined up as described.


In the case of textures would there be a mechanism present that allows the target texture to have a physical name different than what is recorded in the .s file? The utility here is for reskins. For example locomotive model SP1234.s uses SP1234.ace. That shape might be perfectly suitable skinned for other railroads and anyone doing that would probably not be inclined to use the same ace file name -- the letters SP do mean something significant. A different texture for the locomotive might be NKP4567 and so it is reasonable that the reskinner would want to use that name for his reskin file. Somewhere there needs to be something simple that tells the software SP1234.ace has been replaced by NKP5678.ace



 James Ross, on 19 May 2021 - 11:35 AM, said:

For OR-organised content, the current plan is for everything to be referenced by a complete virtual path. That means nothing will be assumed to be in any particular folder and all paths would be absolute, which is fine because it is all virtual.

Will you consider use of a token for a complete path? I mean something like
$Path "Dales Textures" = {full physical path} and then where ever that full path is needed WITHIN that archive one would type
"Dales Textures\acleantrack1.ace". It's a lot more terse than typing "E:\Bin\Open Rails - US West 1966-1980\Routes\Siskiyou\Textures\acleantrack1.ace", especially if that path is needed many dozens of times.

#124 User is offline   YoRyan 

  • Conductor
  • Group: Status: Active Member
  • Posts: 391
  • Joined: 19-February 20
  • Gender:Male
  • Location:California, United States
  • Simulator:Open Rails/unstable
  • Country:

Posted 19 May 2021 - 10:17 PM

 Genma Saotome, on 19 May 2021 - 09:16 PM, said:

In the case of textures would there be a mechanism present that allows the target texture to have a physical name different than what is recorded in the .s file?

That could be a useful feature, but it's a discussion for another day. It's unrelated to the virtual filesystem.

 Genma Saotome, on 19 May 2021 - 09:16 PM, said:

Will you consider use of a token for a complete path?

Full physical paths (with the drive letter in front) constitute a security hole and Open Rails should never accept them. If they work right now, it is a bug.

James was talking about full virtual paths.

As for some sort of abbreviation token, I dunno, maybe? It's impossible to say without having actually defined any new OR formats. If we ever actually have this discussion, it's going to be in conjunction with the introduction of new tooling, so you won't be writing these files by hand anyway.

#125 User is offline   James Ross 

  • Open Rails Developer
  • Group: Status: Elite Member
  • Posts: 5,488
  • Joined: 30-June 10
  • Gender:Not Telling
  • Simulator:Open Rails
  • Country:

Posted 20 May 2021 - 08:34 AM

 Genma Saotome, on 19 May 2021 - 09:16 PM, said:

In the case of textures would there be a mechanism present that allows the target texture to have a physical name different than what is recorded in the .s file?

Such a system could be supported with or without the virtual file system and as such is not part of the current discussion.

We can potentially come back to this when we start thinking about locomotive/wagon data files and model formats, since there might be a much nicer way of building it into the OR content formats.


 Genma Saotome, on 19 May 2021 - 09:16 PM, said:

Will you consider use of a token for a complete path? I mean something like
$Path "Dales Textures" = {full physical path} and then where ever that full path is needed WITHIN that archive one would type
"Dales Textures\acleantrack1.ace". It's a lot more terse than typing "E:\Bin\Open Rails - US West 1966-1980\Routes\Siskiyou\Textures\acleantrack1.ace", especially if that path is needed many dozens of times.

You cannot have an absolute physical path inside content when using the virtual file system. The only paths that will work are:

  • Absolute virtual paths
  • Relative virtual paths but probably only in MSTS content

The most likely absolute virtual path to that texture is "/MSTS/Routes/Siskiyou/Textures/acleantrack1.ace". It does not matter where you physically have the files, and you will never use those physical paths anywhere other than in setting up Open Rails profiles.

If you want to use alternative textures on the same route, such as for different periods in the railway's operations, you would take advantage of the virtual file system's layering:

Profile "US West 1966-1980" might have two folders, in this order:

  • "E:\Bin\Open Rails - original" (which contains the original "Routes\Siskiyou")
  • "E:\Bin\Open Rails - US West 1966-1980" (which contains files like "Routes\Siskiyou\Textures\acleantrack1.ace" to override the originals)

The virtual file system is very similar to using relative paths everywhere, except that you can combine and override files without editing the originals.

#126 User is offline   Genma Saotome 

  • Owner Emeritus and Admin
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 15,307
  • Joined: 11-January 04
  • Gender:Male
  • Location:United States
  • Simulator:Open Rails
  • Country:

Posted 20 May 2021 - 11:11 AM

 James Ross, on 20 May 2021 - 08:34 AM, said:

Such a system could be supported with or without the virtual file system and as such is not part of the current discussion.

We can potentially come back to this when we start thinking about locomotive/wagon data files and model formats, since there might be a much nicer way of building it into the OR content formats.


Thanks.

The reason behind the request is that the file name is inside the .s file as one of the data recorded by the 3dmodeling software. It doesn't appear anywhere else and therein is the problem when it comes to reskins. The reskinner either uses the original file name for his completely new art even tho it may not be relevant at all to the new art... or he makes a copy of the .s file, decompresses it, and edits the file name he finds inside to match his new art file. I routinely do the later, but that's me. My point is the problem could be solved by any sort of replacement statement made somewhere editable. I personally favor using the .sd file that as it seems the .sd file is where KUJU put game data that isn't recorded by the modeling software. Your plans for using paths perked my recalling the topic in that perhaps the answer can be found there as well.



 James Ross, on 20 May 2021 - 08:34 AM, said:

Profile "US West 1966-1980" might have two folders, in this order:

"E:\Bin\Open Rails - original" (which contains the original "Routes\Siskiyou")
"E:\Bin\Open Rails - US West 1966-1980" (which contains files like "Routes\Siskiyou\Textures\acleantrack1.ace" to override the originals)


The virtual file system is very similar to using relative paths everywhere, except that you can combine and override files without editing the originals.


I do understand your plans for the virtual layering. What I am ignorant of is this: where within an archive does the text you wrote above belong?

#127 User is offline   YoRyan 

  • Conductor
  • Group: Status: Active Member
  • Posts: 391
  • Joined: 19-February 20
  • Gender:Male
  • Location:California, United States
  • Simulator:Open Rails/unstable
  • Country:

Posted 20 May 2021 - 11:51 AM

Profiles are managed by users themselves. The archives will be quite simple—their only function will be to contain the files themselves, plus possibly some basic metadata like an author list and some documentation.

Users will add archives and directories to their load orders using physical paths. But, to reiterate, physical paths in content files will be (and should already be) illegal.

#128 User is offline   gpz 

  • Superintendant
  • Group: Status: Elite Member
  • Posts: 1,771
  • Joined: 27-October 12
  • Gender:Male
  • Location:Budapest
  • Simulator:OpenRails
  • Country:

Posted 04 December 2021 - 06:54 AM

I re-read some comments of this super long thread started 10 years ago, and got into a mood of making a sprint with the development of the virtual file system. (It took a week or so.) It can't be so complicated, I thought, and indeed, its complexity is just a fraction of the Monogame migration's. So here it is, interested developers please see the pull request on GitHub. I made the PR in a hope that Ryan had not done much coding work on this feature yet. At least this is what I concluded from this discussion thread.

Comments I also documented in the code for not losing it:

The virtual file system can be initialized from a config file with lines e.g. C:\dir\archive.zip\subdir\ /MSTS/mountpoint/, or from a directory. Found subsequent archives are auto-mounted to their respective locations. The configuration file consits of lines of format:

Drive:\SystemPath\ /MountPoint/
SystemPath
- may use "/" or "\", but MountPoint may use "/" only as a directory separator.
- with spaces in it must be quoted, e.g. "C:\My Routes\USA85\" /MSTS/ROUTES/USA85/
- referring to a directory must end with "/" or "\" to assure that only _below_ dirs and files will be visible _below_ the mount point.
- referring to an archive must not end with "/" or "\", e.g. C:\TEMP\MSTS1.2.zip /MSTS/
- may refer to an archive subdirectory, e.g.: C:\routes.zip\USA3\ /MSTS/ROUTES/USA3/
- may _not_ refer to a single non-archive file, neither a non-archive file within an archive.
- is case-insensitive.
- nested archives are unsupported. (archive inside another archive)
MountPoint
- must end with "/" to avoid confusion and to assure that mounting will be done _below_ the given path.
- may not contain spaces, may not be quoted.
- is case-insensitive, will be converted to all-uppercase internally.

Since I am not a software architect, and I wrote this whole thing from the top of my head, a deep review would be very welcome. If James or Chris is still around here, it would be really helpful from them to take a look onto the code. The code is based on SharpCompress to extract the archives. I haven't made changes to contrib software in the OpenRails repository, but as long as someone uses a simple directory as the VFS "configuration profile", everything should work as before, in this case the change is transparent. The user interface in the Menu Options surely needs some refinement, I just made a quick change on it without much design. One can select a file as well on the Content frame now, and there is an Experimental setting to possible log all VFS file accesses. I left the OpenRails code where files get written intact, although this VFS also has writing abilities by my intention, to system directories only, not to archives. (The code is there theoretically, but I have not tested that extensively.)

#129 User is offline   cjakeman 

  • Vice President
  • PipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 2,857
  • Joined: 03-May 11
  • Gender:Male
  • Location:Peterborough, UK
  • Simulator:Open Rails
  • Country:

Posted 04 December 2021 - 10:45 AM

 gpz, on 04 December 2021 - 06:54 AM, said:

If James or Chris is still around here, it would be really helpful from them to take a look onto the code.

Yes, indeed.

Many thanks for moving this forward.

#130 User is offline   Genma Saotome 

  • Owner Emeritus and Admin
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 15,307
  • Joined: 11-January 04
  • Gender:Male
  • Location:United States
  • Simulator:Open Rails
  • Country:

Posted 04 December 2021 - 11:50 AM

Good to see you here again Peter.

  • 17 Pages +
  • « First
  • 11
  • 12
  • 13
  • 14
  • 15
  • Last »
  • You cannot start a new topic
  • You cannot reply to this topic

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users