Elvas Tower: Overall File System and Data Structure - Elvas Tower

Jump to content

Posting Rules

All new threads will be started by members of the Open Rails team, Staff, and/or Admins. Existing threads started in other forums may get moved here when it makes sense to do so.

Once a thread is started any member may post replies to it.
  • 17 Pages +
  • 1
  • 2
  • 3
  • 4
  • Last »
  • You cannot start a new topic
  • You cannot reply to this topic

Overall File System and Data Structure Rate Topic: ***** 1 Votes

#11 User is offline   wacampbell 

  • Member since Nov. 2003
  • Group: Fan: Traction Nuts
  • Posts: 2,342
  • Joined: 22-November 03
  • Gender:Male
  • Location:British Columbia, Canada
  • Country:

Posted 09 December 2011 - 06:36 PM

View PostGenma Saotome, on 09 December 2011 - 05:47 PM, said:

Anyway, one thing I want to speak to is the unfortunate need to occasionally rely on the old ms-dos ..\..\ scheme, something I always thought was highly prone to both error and confusion.


This was a clever work around developed by the MSTS community so that they could share sounds , cabs and the like among different folders. Its seems there should be a better way.

As Rob already mentioned, there is the fundamental issue. If we use a scheme of shared folders for common itesm, we get issues with missing dependancies and finding all the related file components. If everything is included in the loco package, then its easy to install with no dependancies. But there's a waste of disk storage space. There is also potential performance issues although some clever programming could detect the duplicates and deal properly to prevent wasted resources.

As for the RW directory structure, it is never meant to be worked on directly. Users are meant to add rolling stock and other assets using the menu tools they provide. They are similarly removed via menu. Routes are installed, copied, deleted the same way - all from menus. Some people do indeed go in and hack at the files directly, but there is really no need for most users. But your point is well taken keep it simple and clear at the directory level.

#12 User is offline   smmille1 

  • Open Rails Developer
  • Group: Private - Open Rails Developer
  • Posts: 82
  • Joined: 22-November 10
  • Gender:Male
  • Location:Monee, IL
  • Country:

Posted 09 December 2011 - 07:19 PM

There are so many aspects to this, I can't even think of where to begin.

I don't think we want to impose a system where the user doesn't have full control over their local content. Of course, this means that they also have the ability to royally screw things up.

I don't think UIDs are going to do us any benefits unless we plan to do some sort of centralized content management system - and I don't see that happening. Without the central control, there really isn't anything to enforce it, so why bother trying? By and large, I think most members of the train sim community want to be respectful (although it's certainly not always the case). So in terms of per-user organization, I think we need to perhaps adopt a convention, but not enforce anything.

As much as I like databases, i don't a think full-blown RDBMS belongs on a user machine for just one program. So we're left with files. We should probably avoid having to index every file to look for keys that might be referenced from inside other files since this leads to scalability issues, so that leaves us with linking the files directly.

In terms of file references, I think we need to keep it as flexible as possible. I propose that file links should all support absolute, relative, and aliased (both named and a default) paths. I don't think absolute paths (i.e. 'C:\My Folder\Something\thefile.abc') will be used except maybe in testing, but I don't see why we should specifically exclude them. Relative paths ('textures\mytexture.dds' or '..\..\common\tree.cfg') will be useful for child content, with the parent directory pathing probably having some occasional valid reason for use. Aliased paths will be a big benefit in flexibility of configuration. As I brought up here, we could use the tilde (~) as a special character to denote an alias. ~\ would be the default (whatever we decide that should be) and then ~[name]\ would resolve to whatever the [name] path was set to. I don't like the concept of asking for a file, and then it looks in one place and then another if it doesn't find it in the first place, and I don't think it's necessary if we aren't planning on having a massive pile of stuff in a global folder.

This would allow people to organize their content however they see fit, but still allow for commonality between users using aliasing. We could offer a set of out-of-the box aliases pointing to folders intended for certain types of content (track, cabviews, sounds, etc.) with the hopes that people will latch on to that. It would be possible to create a sort of package installer that could copy files into the various aliased locations to aid in setup.

For rolling stock, we could have a configurable lists of locations using paths as designated above that will be included when putting together consists. This would probably work for shapes too during route construction (although I haven't looked at the next gen route editor doc to see what direction that's going).

Regarding keeping related content together versus spreading it around, so long as people are going to latch on to other peoples content, I would advocate splitting it up by type rather than by project. What I've proposed above won't preclude either method, but I think we should be encouraging reuse, and having to reference a cabview off in some folder that was originally specific to one locomotive is not the cleanest solution. As long as the content creators play nice and pick unique references, then you could have your BZZ folder underneath several locations that you could park your content in depending upon type.

#13 User is offline   James Ross 

  • Open Rails Developer
  • Group: Status: Elite Member
  • Posts: 5,488
  • Joined: 30-June 10
  • Gender:Not Telling
  • Simulator:Open Rails
  • Country:

Posted 10 December 2011 - 02:53 AM

Some of my ideas have been mentioned previously, but let's include them here...

Firstly, the game would have one or more designated "content directories". These would be where content is installed, either by the game or the user, and could include a directory in the program area (for stuff we include) and a directory for the user to install things.

Content within a content directory would be allowed in two formats:
  • A directory.
  • A zip file.


The zip file would contain a directory structure identical to the directory-style of content and would be treated exactly as if it had been unzipped into the content directory. The purpose of this would be to allow simple development (directory-style) and simple distribution (zip-style).

The content of a package directory would be a pre-defined list of directories (only those used need to be included), each containing a single file for each item of that type. For things that are genuinely related, maybe one extra level of directory could be required; we could also keep something similar to MSTS regarding texture selection with weather/season. In general, though, everything inside the pre-defined directories is under the content author's control - they can create sub-directories to group things if they want.

  • All references to other files would know what type of file is expected and would allow only relative paths.
  • If the source and reference file type are from the same pre-defined directory, the base of the relative path would be the source file's path, e.g. a track file referencing a track database of "foo.tdb" would use "foo.tdb" from its own directory.
  • If the source and reference file type are from different pre-defined directories, the base of the relative path would be the pre-defined directory of the appropriate file type within the package, e.g. a shape referencing a texture of "foo.ace" would use "\Textures\foo.ace" from the package it is within.
  • References would be allowed to go up a single directory beyond the pre-defined directory and back down in to a different package, e.g. "..\package_name\foo.ace" would use "\Textures\foo.ace" from package "package_name". The package name would be the directory name (the directory inside the zip for zip-style packages), but we could add a better uniqueness to this (e.g. by having packages have UUIDs and using that) if needed.


Actual files

\UserContentPackage1
\Shapes
\Whatever.s
\SoundGroups
\Whatever.sms
\Sounds
\Whatever.wav
\Textures
\Whatever.ace
\Snow
\Whatever.ace
\TrainCars
\Whatever.wag
\Trains
\Whatever.con
\Download_1234_UCP2.zip
\UserContentPackage2
\Routes
\Whatever
\Whatever.trk
\Whatever.rdb
\Whatever.rit
\Shapes
\Whatever.s
\SoundGroups
\Whatever.sms
\Sounds
\Whatever.wav
\Textures
\Whatever.ace
\Snow
\Whatever.ace


Virtual files

Routes
\UserContentPackage2
\Whatever
\Whatever.trk
\Whatever.rdb
\Whatever.rit
Shapes
\UserContentPackage1
\Whatever.s
\UserContentPackage2
\Whatever.s
Sound groups
\UserContentPackage1
\Whatever.sms
\UserContentPackage2
\Whatever.sms
Sounds
\UserContentPackage1
\Whatever.wav
\UserContentPackage2
\Whatever.wav
Textures
\UserContentPackage1
\Whatever.ace
\Snow
\Whatever.ace
\UserContentPackage2
\Whatever.ace
\Snow
\Whatever.ace
Trains
\UserContentPackage1
\Whatever.con
Train cars
\UserContentPackage1
\Whatever.wag


#14 User is offline   wacampbell 

  • Member since Nov. 2003
  • Group: Fan: Traction Nuts
  • Posts: 2,342
  • Joined: 22-November 03
  • Gender:Male
  • Location:British Columbia, Canada
  • Country:

Posted 10 December 2011 - 06:10 AM

I appreciate the comments. And thanks for referencing previous discussions along on the topic.

So far I am hearing
- simplify deployment and removal of addons - make it drag and drop to install
- ensure we can support sharing and reuse of ... everything...
- ensure we can manage huge collections of addons ( PS Chris please elaborate - what are the problems now wrt file structure)
- make it easy for content creators to 'code' these links into their files

James said:

The zip file would contain a directory structure identical to the directory-style of content and would be treated exactly as if it had been unzipped into the content directory. The purpose of this would be to allow simple development (directory-style) and simple distribution (zip-style).


I definitely like the sound of this. Saves disk space .. simpler deployment .. and probably faster to parse ( on magnetic hard disks at least ).

James, a few questions to help me understand your concept a little more:

1. What problems of the MSTS file structure does your concept aim to solve?
2. What do you mean by the 'Virtual files' section on your diagram?
3. Could you provide an example of what the directories would look like inside the zip file for someone distributing a new boxcar, for example.

#15 User is offline   James Ross 

  • Open Rails Developer
  • Group: Status: Elite Member
  • Posts: 5,488
  • Joined: 30-June 10
  • Gender:Not Telling
  • Simulator:Open Rails
  • Country:

Posted 10 December 2011 - 07:04 AM

View Postwacampbell, on 10 December 2011 - 06:10 AM, said:

1. What problems of the MSTS file structure does your concept aim to solve?


Mostly it's about self-containment; whatever I want to distribute, it can be done in a single directory or zip that can be installed with a single copy/save in the right directory - no file overwriting, no file copying scripts (because you can reference other packages) - and removed with a single delete. Each package could contain as much or as little as it wanted - from an entire route with scenery, rolling stock and activities, right down to a package containing only vegetation textures for others to use. Everything would be in this structure, with no special global locations or such like - though you could include a package called "global" with default stuff people can use.

View Postwacampbell, on 10 December 2011 - 06:10 AM, said:

2. What do you mean by the 'Virtual files' section on your diagram?


This was an attempt to visualise how the relative file references work; the point being that the package names make up one directory level, but the pre-defined directories don't (as they're implied by the type of reference). E.g. "\UserContentPackage1\Shapes\Whatever.s" could reference another package's shape with "..\UserContentPackage2\Whatever.s" instead of the more verbose "..\..\UserContentPackage2\Shapes\Whatever.s".

View Postwacampbell, on 10 December 2011 - 06:10 AM, said:

3. Could you provide an example of what the directories would look like inside the zip file for someone distributing a new boxcar, for example.


\My_Boxcar.zip
\James_Ross_Boxcar_1
\Shapes
\boxcar.s
\SoundGroups
\boxcar.sms
\Sounds
\boxcar1.wav
\boxcar2.wav
\boxcar3.wav
\Textures
\boxcaralpha.ace
\boxcarcolor.ace
\boxcarmisc.ace
\TrainCars
\boxcar.wag


If I wanted to make a consist in another package use this boxcar, it would reference is as "..\James_Ross_Boxcar_1\boxcar.wag". All the references within would just be a filename - no path components would be needed, as they're all at the top of their respective pre-defined directories. They don't have to be, though; if I put boxcar1.wav, etc., in to "\My_Boxcar.zip\James_Ross_Boxcar_1\Sounds\Boxcar" then boxcar.sms would use "Boxcar\boxcar1.wav", etc..

I'm not a content creator, so I might've forgotten some part you need here - but the idea should be clear.

#16 User is offline   wacampbell 

  • Member since Nov. 2003
  • Group: Fan: Traction Nuts
  • Posts: 2,342
  • Joined: 22-November 03
  • Gender:Male
  • Location:British Columbia, Canada
  • Country:

Posted 10 December 2011 - 07:55 AM

My first thoughts are to support as many content directories as a user wants. People will squawk if we limit them to one. However to keep it simple, the OpenRails code would always treat them as if they were all one directory. That way linked references would never have to specify a content directory ... only the package name.

It would require package names to be unique across the community, but I think that is solvable in a number of ways. ( lets not get into solutions ideas in this thread )

I think you have created a convert ... good discussion.

#17 User is offline   wacampbell 

  • Member since Nov. 2003
  • Group: Fan: Traction Nuts
  • Posts: 2,342
  • Joined: 22-November 03
  • Gender:Male
  • Location:British Columbia, Canada
  • Country:

Posted 10 December 2011 - 08:11 AM

Another clarification about James' proposal ...

Do I get that the name of the zip file doesn't matter at all. ie the package will be identified by the package folder name inside the zip? I like this because the zip file name is often set by the web site where things are stored. This way it doesn't matter what the zip file is called. In fact a user could rename it if there are naming conflicts. The only thing that would be important is that it be unique in the content directory where its stored.

#18 User is offline   James Ross 

  • Open Rails Developer
  • Group: Status: Elite Member
  • Posts: 5,488
  • Joined: 30-June 10
  • Gender:Not Telling
  • Simulator:Open Rails
  • Country:

Posted 10 December 2011 - 08:59 AM

View Postwacampbell, on 10 December 2011 - 07:55 AM, said:

My first thoughts are to support as many content directories as a user wants. People will squawk if we limit them to one. However to keep it simple, the OpenRails code would always treat them as if they were all one directory. That way linked references would never have to specify a content directory ... only the package name.


Yeah, the idea was that we have multiple "content directories" and OR merges its view across all of them in a defined way. Everything would just see a single list of packages.

View Postwacampbell, on 10 December 2011 - 08:11 AM, said:

Do I get that the name of the zip file doesn't matter at all. ie the package will be identified by the package folder name inside the zip? I like this because the zip file name is often set by the web site where things are stored. This way it doesn't matter what the zip file is called. In fact a user could rename it if there are naming conflicts. The only thing that would be important is that it be unique in the content directory where its stored.


You are correct; websites controlling the filename was one of the reasons I went for using the directory name inside the zip only.

It occurs to me that a zip could contain multiple packages, but I'm not sure how useful that would be. (You should never need to have more than one package for a collecdtion of things.)

#19 User is offline   Genma Saotome 

  • Owner Emeritus and Admin
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 15,308
  • Joined: 11-January 04
  • Gender:Male
  • Location:United States
  • Simulator:Open Rails
  • Country:

Posted 10 December 2011 - 10:05 AM

Interesting.

One concern brings a question: For reuse you create a relative reference to another "package", the reference is complete text rather that half dots and slashes (good)... now repeat that many hundreds of times and then decide that package 143 is actually pretty bad now that you seen Joe's replacement. How do you remove (or over-write) package 143, in ignorance of how it is referenced elsewhere? What it looks like is Package 143 is there "for the duration".

It's a shame really that there isn't the sort of referential integrity at the file level that there is inside of a dbms... I keep wondering if we should give going down that path that some serious consideration.

Anyway, I know disk is cheap but time isn't... trying to sort & find a directory with tens of thousands of things in it sometimes a PITA and full backups take time too. So the idea of once it's installed it's there forever strikes me as a bit problematic.

#20 User is offline   wacampbell 

  • Member since Nov. 2003
  • Group: Fan: Traction Nuts
  • Posts: 2,342
  • Joined: 22-November 03
  • Gender:Male
  • Location:British Columbia, Canada
  • Country:

Posted 10 December 2011 - 10:14 AM

View PostGenma Saotome, on 10 December 2011 - 10:05 AM, said:

and then decide that package 143 is actually pretty bad now that you seen Joe's replacement. How do you remove (or over-write) package 143, in ignorance of how it is referenced elsewhere?

I can see this might lead to another level of indirection or macro substitution or something like that. Really more an issue for how things are encoded in the file - not so much an issue for the directory structure itself. Even so, I am not sure I see the occasion where you could substitute one package for another. Would someone else's package have all the same files in it as the one you originally referenced?

Quote

So the idea of once it's installed it's there forever strikes me as a bit problematic.

Could you elaborate. Why do you think the proposed file and directory structure leads to this problem?

  • 17 Pages +
  • 1
  • 2
  • 3
  • 4
  • Last »
  • You cannot start a new topic
  • You cannot reply to this topic

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users