[James Ross]

 Genma Saotome, on 10 December 2011 - 10:05 AM, said:

Anyway, I know disk is cheap but time isn't... trying to sort & find a directory with tens of thousands of things in it sometimes a PITA and full backups take time too. So the idea of once it's installed it's there forever strikes me as a bit problematic.

The proposed structure actually makes it simpler to remove things, as they're contained with single objects (directory or zip); there's also nothing stopping someone providing a tool to report which packages reference which other packages. We could include such a tool with OR, for example. (I've actually thought about a metadata file in each package could list which others it uses, but it's not critical to this design.)

[Genma Saotome]

Sorry I wasn't clear... the proposal addresses installation and ordinary use. When you begin to add re-use the reference is only known on the "requesting" side of the relationship... the providing side knows nothing of how many times or where it is being made use of. Absent other tools that will revel such information, the average end user will will be better off leaving all installed packages installed forever for fear of breaking something if it is removed. MSTS does the same and I find that problematic How many references are there out there to files from the Flying Scotsman or ... whats-it-called now... the ESE sound file (some early payware stuff)?

I've never run the Flying Scotsman... never intend on running the Flying Scotsman, have no idea how many different places its cab view might still be referenced, and have the unhappy experience of learning that all sorts of things break when I deleted its folder.

I'm not seeing anything in the proposal that solves that problem. Something as simple as \content\reused\"many packages" and \content\standalone\"many packages" might help.
===================

I'm also a bit concerned about the top level folder name used on content. I rename (and prune) a lot of the stuff I get because I find it vastly easier to use a directory named \SP #5637 GP-9 than I would if it were the original \Bob's GP-9 Collection.zip; Will the proposal allow me to rename and organize sets of files as I like w/o all sorts of stuff breaking? I'm concerned that it won't... not unless we also implement the feature that Scott and I have mentioned (tilde and %) to create an editable shortcut that works over a wide spread of files. Add that feature and then you enable files to hold logical references while in a control file you map them over to physical names and locations.

[James Ross]

 Genma Saotome, on 10 December 2011 - 11:07 AM, said:

Sorry I wasn't clear... the proposal addresses installation and ordinary use. When you begin to add re-use the reference is only known on the "requesting" side of the relationship... the providing side knows nothing of how many times or where it is being made use of. Absent other tools that will revel such information, the average end user will will be better off leaving all installed packages installed forever for fear of breaking something if it is removed.

This problem is only solvable with tools, in the general case; no amount of arranging files will solve it IMHO. What do you do when you want to prune your "reused" folder, for example? I think we're better off defining the linking cleanly and neatly and then providing tooling based upon that.

 Genma Saotome, on 10 December 2011 - 11:07 AM, said:

I'm also a bit concerned about the top level folder name used on content. I rename (and prune) a lot of the stuff I get because I find it vastly easier to use a directory named \SP #5637 GP-9 than I would if it were the original \Bob's GP-9 Collection.zip; Will the proposal allow me to rename and organize sets of files as I like w/o all sorts of stuff breaking? I'm concerned that it won't... not unless we also implement the feature that Scott and I have mentioned (tilde and %) to create an editable shortcut that works over a wide spread of files. Add that feature and then you enable files to hold logical references while in a control file you map them over to physical names and locations.

Zip files can be renamed freely; the top-level directory (in a content directory or a zip) cannot. Honestly, I'm not sure we should be catering to this need, but we can do it with only a small addition: a simple file in any content directory, containing package names and locations (directory or inside a zip file).

[smmille1]

Have you given any thought to the technical aspects of having an entire route get loaded starting from a zip file? It could be too big to load into RAM. Use the %temp% folder?

[James Ross]

 smmille1, on 10 December 2011 - 11:56 AM, said:

Have you given any thought to the technical aspects of having an entire route get loaded starting from a zip file? It could be too big to load into RAM. Use the %temp% folder?

Yes, I have. You don't need the whole zip file in RAM to read individual files, or the file list, from it, and you don't even need to write the files to disk as an intermediate step for loading. Obviously some experimentation will be needed here, but the "worst" case is that we cache an expanded copy in a private folder.

[wacampbell]

 smmille1, on 10 December 2011 - 11:56 AM, said:

Have you given any thought to the technical aspects of having an entire route get loaded starting from a zip file? It could be too big to load into RAM. Use the %temp% folder?

Certainly something that would have to be investigated. The world files and terrain files are loaded dynamically as the camera moves along the route. Ideally we will find that zip reading libraries can quickly reposition themselves in the zip file stream to any file needed.

Definitely worth investigating.

[Genma Saotome]

There has been something rolling around in my mind that says "there is a problem with this idea" and I think I've pulled it out of the goo to look at and it's this: When a revision / patch occurs how do you find the thing to patch? You cannot assume what's in hand to install even came from the same person who created the thing to be revised and there's no assurance the name of top level container has any similarity to the other. I know that's not different than what MSTS does but at least with MSTS there is an extra layer of structure there whether it's routes, cars, sounds, or textures. OTOH if we set this up so everything is content (period) with author provided names for the next layer, I'm afraid the task of locating the old to over-write it with the new might be more difficult that what Chris showed us in the Railworks method.

Second, am I correct in understanding that the proposal allows for any sort of directory structure to fall under the top level directory of content? Everyone can do something different as it suits them? If so, that's flexible... perhaps waaaay too flexible for Joe Average to wrap his mind around on managing tens of thousand of files.

OTOH, if my understanding is incorrect, that there is an expected structure that exists under the top level directory... what is it?

=====================

All things considered I can't but think the reason this issue is hard is because there is a need to see the relationships -- which are Joe's files? Where did I get this zip? Let me see my installed F-units, where is that cool sound file? and so on. The stuff that sometimes, in more simple situations, file names, extensions, and the directory tree can adequately solve. But perhaps this software environment is, in fact, too complex and is in need of an app that provides varied views of the installed data and software that manages the relationships... a dbms database. Put that between the user and the files, between the OR software and the files, and suddenly what the file extensions are, what the file names are, and the directory tree and its names all fade into almost complete insignificance. Problem solved.

[wacampbell]

 Genma Saotome, on 11 December 2011 - 08:36 PM, said:

I know that's not different than what MSTS does but at least with MSTS there is an extra layer of structure there whether it's routes, cars, sounds, or textures.

I don't think James' proposal is any worse in this regard, and perhaps better. We allow a user to set up as many content directories as they like. Consider I might have the following content directories:

1920 Rolling Stock
Traction Loco's
German Locos
1920 Scenery
Modern Scenery
Vegetation
European Routes
NA Routes

It should be pretty easy to find Tims PE1606 should he decide to issue an update.

Quote

... perhaps waaaay too flexible for Joe Average to wrap his mind around on managing tens of thousand of files.

I don't see how it could be 'too flexible' Some people are good at organizing their addons. Some are not, but this gives them a pretty flexible tool.

Quote

OTOH, if my understanding is incorrect, that there is an expected structure that exists under the top level directory... what is it?

I am not sure what James' envisioned, but I wouldn't see that we have a top level content directory at all. The user adds his individual content directories to OpenRails through a configuration process. He can store the content directories wherever he wants even on different hard disks if needed. And as James' suggests, the OpenRails provided content would live in a content directory in the program folder.

Quote

... a dbms database. Put that between the user and the files, between the OR software and the files

I definitely see the appeal of this. a DBMS is a collection of data on the disk controlled by a program that indexes it and allows users to query it in flexible ways. But then isn't the proposed method also a DBMS where the records are represented by files and folders. All thats missing is the 'program' to index them and allow users to query it in flexible ways. ... and that could be added.

[James Ross]

 Genma Saotome, on 11 December 2011 - 08:36 PM, said:

There has been something rolling around in my mind that says "there is a problem with this idea" and I think I've pulled it out of the goo to look at and it's this: When a revision / patch occurs how do you find the thing to patch?

Finding the thing to patch is easy. The readme with the patch tells you what it's called and you open that thing in the content directory you've dumped everything in. (Or you can add an extra level of organisation by using multiple content directories as Wayne mentioned.) Actually patching things is a more interesting subject than finding them... (well, for zip files anyway; directory patching is identical to MSTS).

 Genma Saotome, on 11 December 2011 - 08:36 PM, said:

Second, am I correct in understanding that the proposal allows for any sort of directory structure to fall under the top level directory of content? Everyone can do something different as it suits them? If so, that's flexible... perhaps waaaay too flexible for Joe Average to wrap his mind around on managing tens of thousand of files.

OTOH, if my understanding is incorrect, that there is an expected structure that exists under the top level directory... what is it?

It's not clear to me what directory you are refering to by "top level" here. Let's see if I can name everything:

\OpenRails\Content This is a content directory. There can be any number of these and OR will merge its view as if only one existed.

\UserContentPackage1 This is a package directory. Its name is the name of the package.

\Shapes This is a pre-defined directory. All the directories at this level are fixed in name. Extra directories will be ignored and you will not be able to access them from other files.
\Routes This is another pre-defined directory.

\Route1 This is a required sub-directory. Certain pre-defined directories will need required sub-directories because each "item" they contain is made up of multiple files - in this case, a route will probably need multiple files to define it (the tiles and textures and so on will not be in here though).

Within each pre-defined directory, or required sub-directory when appropriate, the author can do pretty much anything they like with files and folders. So they could have all the shape files in "\UserContentPackage1\Shapes" or they could put them in sub-directories. If they put them in sub-directories, they'll need to use the sub-directory names when referencing the shapes from elsewhere.

The intention is that it is similar to MSTS, with pre-defined directory names and basic structure, but improved so that e.g. shapes are not "part" of the route and specifically allowing flexibility in organisation within the pre-defined directories.

[wacampbell]

So with respect to James' proposal, what are everyone's thoughts on how MSTS addons would be integrated. I see a couple of possibilities:

1. We have an import process where they are brought in, validated and converted to the OR file structure

2. We support drag and drop of MSTS items into our content directory for rolling stock folders and route folders. We might call these package folders 'Legacy Packages'. OR detects that its a MSTS legacy 'package' folder and modifies its behavior as needed.