[Genma Saotome]

May I add that one thing I learned from study of what MSTS does is that there are registry & system calls every time the value of a path changes. I've since sorted my world files so anything that is in the \global\shapes directory is read first and when that's done there is one switch over to \routename\shapes and all the remaining files for the tile are read from there. The de-fragger I have can operate on directories too so I also defrag the same directories one at a time. I've seen no evidence that sorting by filename adds value.

I'm assuming this applies to OR as well.

[Genma Saotome]

One of the other threads got me thinking so I went and searched on *.wav in the Train Simulator directory tree... 0.67gb in > 5000 files. Now I don't have that many routes and I think by most considerations a rather small \trainset tree and so for me these numbers are a surprise.

I havn't bothered to determine exactly how many dups are present... just eyeballin the list I see a lot of two's and three's of stuff, but there are also 9 of these and 13 of those in there too.

I mention this WRT James's proposal for wholly self-contained packages of content. It seems to me that the sound files in the MSTS \trainset directory tree would serve as a very good indicator of the kind of duplication that would exist were we to adopt that proposal as-is -- a lot of unnecessary duplication. The same duplication would occur with cars or locomotives distributed as part of a new activity.

In thinking on the issue, it strikes me that James's ideas are good for distribution purposes as you get everything you need, probably easier for the programmer, and not that good for the purpose of reuse, not good for the task of locating and replacing files with something better.

What we have here is a long list of content -- mesh, texture, and sound files that can be used in many places -- a simple one to many relationship. The dilemma to solve is how to ensure only one instance of content and having done so to put it into master list that is known "globally" to all other content. In business systems the term I've used to describe that task is merge/purge: a comparison of candidates to existing master data that takes in new content as new stuff and on matches performs a reconciliation between master data and the in-coming data (the merge part) so it can get rid of the duplicate candidate (the purge part).

Were we to use something like that, the objective would be to put commonly used content would be located in "global" directories and unique content left with the downloaded stuff. A sophisticated process might also differentiate global freeware from vender-specific "global" content. Prior to the merge/purge step distribution package content would be as James has suggested -- you get it all, altho we might also see proactive content creators might pre-segregate files into candidates for "global" consideration and everything else. As part of the merge / purge process our code would have to replace the original references to candidate file with the correct values of the master data. It might sound knarly but it might not be much more than doing a string search on all installed files and looping thru the list changing the string.

For OR the reconciliation step might be based on certain file attributes (e.g., name, size, date, etc.), values of tags, and so on. To be on the safe side, it might also ask for the end-users agreement to accept each recommended merge/purge. It might also be wise to limit what types of files that go thru this process -- sound, mesh, and texture files come to mind as good choices for master data but not consist, activity, or world files. I've no opinion on whether this step occurs on install... or as a result of optimization for Open Rails.

The end result would be some directories holding single instances of "global" master content that were referenced many times elsewhere.

[James Ross]

 Genma Saotome, on 27 December 2011 - 06:09 PM, said:

I mention this WRT James's proposal for wholly self-contained packages of content. It seems to me that the sound files in the MSTS \trainset directory tree would serve as a very good indicator of the kind of duplication that would exist were we to adopt that proposal as-is -- a lot of unnecessary duplication. The same duplication would occur with cars or locomotives distributed as part of a new activity.

In thinking on the issue, it strikes me that James's ideas are good for distribution purposes as you get everything you need, probably easier for the programmer, and not that good for the purpose of reuse, not good for the task of locating and replacing files with something better.

Were we to use something like that, the objective would be to put commonly used content would be located in "global" directories and unique content left with the downloaded stuff.

The entire point of the organised packages is that commonly used content would be available in a package of its own, which others would then reference, to avoid duplication. A bit like how things use XTracks or NewRoads currently, but actually designed for that.

[longiron]

I think these proposals are excellent. On the duplication issue, I suggest we have to be careful. There are many wave, ace, s files with the same file name, but are completely different content. The community has already moved in the "shared" direction with common.snd and common.cab folders instead of individual sets within each train folder.

Whatever the proposal we adopt, I do not want to get into the content management business - i.e. writing a program to manage the users files. I suggest this is better left to the community to do.

[wacampbell]

Jame's proposal does give us the flexibility to set up shared content.

There may still be cases where a builder improperly distributes his work with certain duplicate files contained in his package instead of referencing the proper external shared pakage. If the distributor were worried that the end user might not have the required external package, he could, with permission, include the necessary parts of that external package in his distribution as a separate package. In other words his distribution contains his package, and the necessary parts of the common package. We'll have to set up some recommended guidelines for builders to use. And it would be easy for others in the community to make some tools to allow removal of duplicates and substitutions.

[Genma Saotome]

 James Ross, on 28 December 2011 - 02:10 AM, said:

The entire point of the organised packages is that commonly used content would be available in a package of its own, which others would then reference, to avoid duplication. A bit like how things use XTracks or NewRoads currently, but actually designed for that.

I have to say it was not obvious to me that the "entire point" of the proposal was to manage global files... it appeared that the "entire point" was solving the problem of a downloader getting everything he needed in just one download. That said. I agree the proposal could be used as you describe... but let's look at some reality here: Xtracks is one package only because Osakra built it and was good enough to invest his time and energy to integrate contributions from others. At any time over the last 10 or so years someone could have done exactly the same thing with sound file... but nobody did. Ditto for commonly used textures... but nobody did. So while the proposal can accommodate a distribution of commonly used files... what basis is there for believing the user community will do anything to take advantage of it?

The polar opposite approach is perhaps exampled by Trainz where AFAIK everything must be registered with the vendor. A handy way to get a unique identifier but it requires a long term commitment and IMO I do not think the OR team wants (or should do) that.

What I was trying to outline is an idea that I see as a middle road solution: let people distribute complete packages... and then given them some tools they could optionally use to identify and reconcile duplication. I don't see that there should be any requirement to use such tools but to whatever degree they do get used would certainly get the ball moving towards building a master repository of global files.
=====================
Taking one step back... what is your proposal for a directory structure? The post you made earlier is about content distrubtion and tehre is an implied directory structure there... is that complete? What is is meant by the second half, labeled virtual? What directories are global and where are they? In what way is this different from Kuju's directory design?

[Genma Saotome]

 longiron, on 28 December 2011 - 05:46 AM, said:

I think these proposals are excellent. On the duplication issue, I suggest we have to be careful. There are many wave, ace, s files with the same file name, but are completely different content. The community has already moved in the "shared" direction with common.snd and common.cab folders instead of individual sets within each train folder.

Whatever the proposal we adopt, I do not want to get into the content management business - i.e. writing a program to manage the users files. I suggest this is better left to the community to do.

IMO a very large percentage of identically named but digitally different files are caused by hardcoding the physical file name of each texture into the mesh file. Were OR to follow a standard of the mesh containing the logical name and then asking content developers to equate that logical name to a physical file name within a describing file (.sd or .md or .whatever) then it would be easy to give new art new names. For examaple:

Mesh File
-- Texture01.ace
-- Texture02.ace
-- Texture03.ace

Original Mesh Description File
-- Texture01.ace = Texture01.ace
-- Texture02.ace = Texture02.ace
-- Texture03.ace = Texture03.ace

New Mesh Description #21 File
-- Texture01.ace = MyTextureA.ace
-- Texture02.ace = MyTextureB.ace
-- Texture03.ace = Texture03.ace

New Mesh Description #22 File
-- Texture01.ace = MyTextureD.ace
-- Texture02.ace = MyTextureB.ace
-- Texture03.ace = MyTextureC.ace

New Mesh Description #23 File
-- Texture01.ace = MyTextureD.ace
-- Texture02.ace = MyTextureF.ace
-- Texture03.ace = MyTextureC.ace
etc.

[James Ross]

 Genma Saotome, on 28 December 2011 - 11:02 AM, said:

Taking one step back... what is your proposal for a directory structure? The post you made earlier is about content distrubtion and tehre is an implied directory structure there... is that complete? What is is meant by the second half, labeled virtual? What directories are global and where are they? In what way is this different from Kuju's directory design?

Go re-read my posts in this thread, please. I've explained all of this already, but here are the bits I can be bothered typing out again...

The structure is not 100% complete because it's a principal more than an explicit design in stone. The idea is that each "type of thing" is a seperate directory inside each package (pre-defined directories). So you don't have cab view textures in the trains directory, you have them in the textures directory, but you can organise within each of these pre-defined directories as you like. As it was intended for OR's new content, the exact selection of "types" isn't known yet either...

If you still don't know what the virtual section was (having explained it before), I suggest just ignoring it. I'm not rehashing the explanation yet again for you.

There are no global directories, that's the point. Everything is a package and every package is equal. We could ship packages with OR but wouldn't have to.

[wacampbell]

 Genma Saotome, on 28 December 2011 - 11:20 AM, said:

IMO a very large percentage of identically named but digitally different files are caused by hardcoding the physical file name of each texture into the mesh file. Were OR to follow a standard of the mesh containing the logical name and then asking content developers to equate that logical name to a physical file name within a describing file (.sd or .md or .whatever) then it would be easy to give new art new names. For examaple:

I do agree that we need some sort of texture substitution mechanism. Encoding it into an SD file seems logical. It doesn't seem very elegant from a programming point of view, but I can see it would be incredibly useful for the end user.

[Genma Saotome]

 wacampbell, on 28 December 2011 - 04:43 PM, said:

I do agree that we need some sort of texture substitution mechanism. Encoding it into an SD file seems logical. It doesn't seem very elegant from a programming point of view, but I can see it would be incredibly useful for the end user.

Actually if you go back far enough in time (pre widespread use of Unix) it was a standard programming technique to deal w/ the same problem: a compiler requirement on Programmer A to write a file name in the source code and a requirement for Programmer B to maintain the working environment w/o access to that source code. Person B did have access & authority over the script that invoked the program and the substitution occurred there as needed.