[wacampbell]

My first thoughts are to support as many content directories as a user wants. People will squawk if we limit them to one. However to keep it simple, the OpenRails code would always treat them as if they were all one directory. That way linked references would never have to specify a content directory ... only the package name.

It would require package names to be unique across the community, but I think that is solvable in a number of ways. ( lets not get into solutions ideas in this thread )

I think you have created a convert ... good discussion.

[wacampbell]

Another clarification about James' proposal ...

Do I get that the name of the zip file doesn't matter at all. ie the package will be identified by the package folder name inside the zip? I like this because the zip file name is often set by the web site where things are stored. This way it doesn't matter what the zip file is called. In fact a user could rename it if there are naming conflicts. The only thing that would be important is that it be unique in the content directory where its stored.

[James Ross]

wacampbell, on 10 December 2011 - 07:55 AM, said:

My first thoughts are to support as many content directories as a user wants. People will squawk if we limit them to one. However to keep it simple, the OpenRails code would always treat them as if they were all one directory. That way linked references would never have to specify a content directory ... only the package name.

Yeah, the idea was that we have multiple "content directories" and OR merges its view across all of them in a defined way. Everything would just see a single list of packages.

wacampbell, on 10 December 2011 - 08:11 AM, said:

Do I get that the name of the zip file doesn't matter at all. ie the package will be identified by the package folder name inside the zip? I like this because the zip file name is often set by the web site where things are stored. This way it doesn't matter what the zip file is called. In fact a user could rename it if there are naming conflicts. The only thing that would be important is that it be unique in the content directory where its stored.

You are correct; websites controlling the filename was one of the reasons I went for using the directory name inside the zip only.

It occurs to me that a zip could contain multiple packages, but I'm not sure how useful that would be. (You should never need to have more than one package for a collecdtion of things.)

[Genma Saotome]

Interesting.

One concern brings a question: For reuse you create a relative reference to another "package", the reference is complete text rather that half dots and slashes (good)... now repeat that many hundreds of times and then decide that package 143 is actually pretty bad now that you seen Joe's replacement. How do you remove (or over-write) package 143, in ignorance of how it is referenced elsewhere? What it looks like is Package 143 is there "for the duration".

It's a shame really that there isn't the sort of referential integrity at the file level that there is inside of a dbms... I keep wondering if we should give going down that path that some serious consideration.

Anyway, I know disk is cheap but time isn't... trying to sort & find a directory with tens of thousands of things in it sometimes a PITA and full backups take time too. So the idea of once it's installed it's there forever strikes me as a bit problematic.

[wacampbell]

Genma Saotome, on 10 December 2011 - 10:05 AM, said:

and then decide that package 143 is actually pretty bad now that you seen Joe's replacement. How do you remove (or over-write) package 143, in ignorance of how it is referenced elsewhere?

I can see this might lead to another level of indirection or macro substitution or something like that. Really more an issue for how things are encoded in the file - not so much an issue for the directory structure itself. Even so, I am not sure I see the occasion where you could substitute one package for another. Would someone else's package have all the same files in it as the one you originally referenced?

Quote

So the idea of once it's installed it's there forever strikes me as a bit problematic.

Could you elaborate. Why do you think the proposed file and directory structure leads to this problem?

[James Ross]

Genma Saotome, on 10 December 2011 - 10:05 AM, said:

Anyway, I know disk is cheap but time isn't... trying to sort & find a directory with tens of thousands of things in it sometimes a PITA and full backups take time too. So the idea of once it's installed it's there forever strikes me as a bit problematic.

The proposed structure actually makes it simpler to remove things, as they're contained with single objects (directory or zip); there's also nothing stopping someone providing a tool to report which packages reference which other packages. We could include such a tool with OR, for example. (I've actually thought about a metadata file in each package could list which others it uses, but it's not critical to this design.)

[Genma Saotome]

Sorry I wasn't clear... the proposal addresses installation and ordinary use. When you begin to add re-use the reference is only known on the "requesting" side of the relationship... the providing side knows nothing of how many times or where it is being made use of. Absent other tools that will revel such information, the average end user will will be better off leaving all installed packages installed forever for fear of breaking something if it is removed. MSTS does the same and I find that problematic How many references are there out there to files from the Flying Scotsman or ... whats-it-called now... the ESE sound file (some early payware stuff)?

I've never run the Flying Scotsman... never intend on running the Flying Scotsman, have no idea how many different places its cab view might still be referenced, and have the unhappy experience of learning that all sorts of things break when I deleted its folder.

I'm not seeing anything in the proposal that solves that problem. Something as simple as \content\reused\"many packages" and \content\standalone\"many packages" might help.
===================

I'm also a bit concerned about the top level folder name used on content. I rename (and prune) a lot of the stuff I get because I find it vastly easier to use a directory named \SP #5637 GP-9 than I would if it were the original \Bob's GP-9 Collection.zip; Will the proposal allow me to rename and organize sets of files as I like w/o all sorts of stuff breaking? I'm concerned that it won't... not unless we also implement the feature that Scott and I have mentioned (tilde and %) to create an editable shortcut that works over a wide spread of files. Add that feature and then you enable files to hold logical references while in a control file you map them over to physical names and locations.

[James Ross]

Genma Saotome, on 10 December 2011 - 11:07 AM, said:

Sorry I wasn't clear... the proposal addresses installation and ordinary use. When you begin to add re-use the reference is only known on the "requesting" side of the relationship... the providing side knows nothing of how many times or where it is being made use of. Absent other tools that will revel such information, the average end user will will be better off leaving all installed packages installed forever for fear of breaking something if it is removed.

This problem is only solvable with tools, in the general case; no amount of arranging files will solve it IMHO. What do you do when you want to prune your "reused" folder, for example? I think we're better off defining the linking cleanly and neatly and then providing tooling based upon that.

Genma Saotome, on 10 December 2011 - 11:07 AM, said:

I'm also a bit concerned about the top level folder name used on content. I rename (and prune) a lot of the stuff I get because I find it vastly easier to use a directory named \SP #5637 GP-9 than I would if it were the original \Bob's GP-9 Collection.zip; Will the proposal allow me to rename and organize sets of files as I like w/o all sorts of stuff breaking? I'm concerned that it won't... not unless we also implement the feature that Scott and I have mentioned (tilde and %) to create an editable shortcut that works over a wide spread of files. Add that feature and then you enable files to hold logical references while in a control file you map them over to physical names and locations.

Zip files can be renamed freely; the top-level directory (in a content directory or a zip) cannot. Honestly, I'm not sure we should be catering to this need, but we can do it with only a small addition: a simple file in any content directory, containing package names and locations (directory or inside a zip file).

[smmille1]

Have you given any thought to the technical aspects of having an entire route get loaded starting from a zip file? It could be too big to load into RAM. Use the %temp% folder?

[James Ross]

smmille1, on 10 December 2011 - 11:56 AM, said:

Have you given any thought to the technical aspects of having an entire route get loaded starting from a zip file? It could be too big to load into RAM. Use the %temp% folder?

Yes, I have. You don't need the whole zip file in RAM to read individual files, or the file list, from it, and you don't even need to write the files to disk as an intermediate step for loading. Obviously some experimentation will be needed here, but the "worst" case is that we cache an expanded copy in a private folder.

[wacampbell]

smmille1, on 10 December 2011 - 11:56 AM, said:

Have you given any thought to the technical aspects of having an entire route get loaded starting from a zip file? It could be too big to load into RAM. Use the %temp% folder?

Certainly something that would have to be investigated. The world files and terrain files are loaded dynamically as the camera moves along the route. Ideally we will find that zip reading libraries can quickly reposition themselves in the zip file stream to any file needed.

Definitely worth investigating.

[Genma Saotome]

There has been something rolling around in my mind that says "there is a problem with this idea" and I think I've pulled it out of the goo to look at and it's this: When a revision / patch occurs how do you find the thing to patch? You cannot assume what's in hand to install even came from the same person who created the thing to be revised and there's no assurance the name of top level container has any similarity to the other. I know that's not different than what MSTS does but at least with MSTS there is an extra layer of structure there whether it's routes, cars, sounds, or textures. OTOH if we set this up so everything is content (period) with author provided names for the next layer, I'm afraid the task of locating the old to over-write it with the new might be more difficult that what Chris showed us in the Railworks method.

Second, am I correct in understanding that the proposal allows for any sort of directory structure to fall under the top level directory of content? Everyone can do something different as it suits them? If so, that's flexible... perhaps waaaay too flexible for Joe Average to wrap his mind around on managing tens of thousand of files.

OTOH, if my understanding is incorrect, that there is an expected structure that exists under the top level directory... what is it?

=====================

All things considered I can't but think the reason this issue is hard is because there is a need to see the relationships -- which are Joe's files? Where did I get this zip? Let me see my installed F-units, where is that cool sound file? and so on. The stuff that sometimes, in more simple situations, file names, extensions, and the directory tree can adequately solve. But perhaps this software environment is, in fact, too complex and is in need of an app that provides varied views of the installed data and software that manages the relationships... a dbms database. Put that between the user and the files, between the OR software and the files, and suddenly what the file extensions are, what the file names are, and the directory tree and its names all fade into almost complete insignificance. Problem solved.

[wacampbell]

Genma Saotome, on 11 December 2011 - 08:36 PM, said:

I know that's not different than what MSTS does but at least with MSTS there is an extra layer of structure there whether it's routes, cars, sounds, or textures.

I don't think James' proposal is any worse in this regard, and perhaps better. We allow a user to set up as many content directories as they like. Consider I might have the following content directories:

1920 Rolling Stock
Traction Loco's
German Locos
1920 Scenery
Modern Scenery
Vegetation
European Routes
NA Routes

It should be pretty easy to find Tims PE1606 should he decide to issue an update.

Quote

... perhaps waaaay too flexible for Joe Average to wrap his mind around on managing tens of thousand of files.

I don't see how it could be 'too flexible' Some people are good at organizing their addons. Some are not, but this gives them a pretty flexible tool.

Quote

OTOH, if my understanding is incorrect, that there is an expected structure that exists under the top level directory... what is it?

I am not sure what James' envisioned, but I wouldn't see that we have a top level content directory at all. The user adds his individual content directories to OpenRails through a configuration process. He can store the content directories wherever he wants even on different hard disks if needed. And as James' suggests, the OpenRails provided content would live in a content directory in the program folder.

Quote

... a dbms database. Put that between the user and the files, between the OR software and the files

I definitely see the appeal of this. a DBMS is a collection of data on the disk controlled by a program that indexes it and allows users to query it in flexible ways. But then isn't the proposed method also a DBMS where the records are represented by files and folders. All thats missing is the 'program' to index them and allow users to query it in flexible ways. ... and that could be added.

[James Ross]

Genma Saotome, on 11 December 2011 - 08:36 PM, said:

There has been something rolling around in my mind that says "there is a problem with this idea" and I think I've pulled it out of the goo to look at and it's this: When a revision / patch occurs how do you find the thing to patch?

Finding the thing to patch is easy. The readme with the patch tells you what it's called and you open that thing in the content directory you've dumped everything in. (Or you can add an extra level of organisation by using multiple content directories as Wayne mentioned.) Actually patching things is a more interesting subject than finding them... (well, for zip files anyway; directory patching is identical to MSTS).

Genma Saotome, on 11 December 2011 - 08:36 PM, said:

Second, am I correct in understanding that the proposal allows for any sort of directory structure to fall under the top level directory of content? Everyone can do something different as it suits them? If so, that's flexible... perhaps waaaay too flexible for Joe Average to wrap his mind around on managing tens of thousand of files.

OTOH, if my understanding is incorrect, that there is an expected structure that exists under the top level directory... what is it?

It's not clear to me what directory you are refering to by "top level" here. Let's see if I can name everything:

\OpenRails\Content This is a content directory. There can be any number of these and OR will merge its view as if only one existed.

\UserContentPackage1 This is a package directory. Its name is the name of the package.

\Shapes This is a pre-defined directory. All the directories at this level are fixed in name. Extra directories will be ignored and you will not be able to access them from other files.
\Routes This is another pre-defined directory.

\Route1 This is a required sub-directory. Certain pre-defined directories will need required sub-directories because each "item" they contain is made up of multiple files - in this case, a route will probably need multiple files to define it (the tiles and textures and so on will not be in here though).

Within each pre-defined directory, or required sub-directory when appropriate, the author can do pretty much anything they like with files and folders. So they could have all the shape files in "\UserContentPackage1\Shapes" or they could put them in sub-directories. If they put them in sub-directories, they'll need to use the sub-directory names when referencing the shapes from elsewhere.

The intention is that it is similar to MSTS, with pre-defined directory names and basic structure, but improved so that e.g. shapes are not "part" of the route and specifically allowing flexibility in organisation within the pre-defined directories.

[wacampbell]

So with respect to James' proposal, what are everyone's thoughts on how MSTS addons would be integrated. I see a couple of possibilities:

1. We have an import process where they are brought in, validated and converted to the OR file structure

2. We support drag and drop of MSTS items into our content directory for rolling stock folders and route folders. We might call these package folders 'Legacy Packages'. OR detects that its a MSTS legacy 'package' folder and modifies its behavior as needed.