[Genma Saotome]

James Ross, on 28 December 2011 - 02:35 PM, said:

... but here are the bits I can be bothered typing out again...

Alrighty then.

For the record, I couldn't care less if this or that proposal makes things easier to code. All I care about is whether any idea is going to be better or worse for the end users and / or content providers as they're the ones who will have to live with it for n years to come. I can see this proposal probably does address the issue of downloaded packages having a chance of being complete. Whether it would be used in that manner is plausible but not assured. The likelihood of duplication appears to me to be greater than what Kuju's design created. The proposal tells me nothing about anything else other than it is simply different than the Kuju design.

As for "Virtual Files" the string appears nowhere else in these forums.

If someone besides James wants to explain anything about this... here, via PM, via e-mail, feel free.

[longiron]

I guess I must go back to the beginning, to make heads of this discussion. The original assumption were that the current MSTS storage plan has some deficiencies as detailed below.

- difficult to manage at scale, lots of time & effort
- must use other's conventions (folder name, file name, etc) to share stuff (Trainset/Activities/Paths)
- lots of repetitious stuff (wav, ace, etc) that needs to be managed
- clean up (delete) is impossible unless you understand the dependencies

Based on everyone's comments, we seem to have discussed two approaches to solving these problems. One, build a tool that abstracts all the complexities, allows for easy re-use of content, and makes clean up simple within a traditional file/folder structure. Or move everything into a RDBMS, which by its nature addresses many of the issues. Is that really the basic choice before us?

[wacampbell]

longiron, on 28 December 2011 - 05:53 PM, said:

Or move everything into a RDBMS, which by its nature addresses many of the issues. Is that really the basic choice before us?

I'd like to hear more from the proponents of the RDBMS solution. How would addons be distributed and merged into the RDBMS? Can these RDBMS's handle binary data such as mesh files and image files?

[Genma Saotome]

longiron, on 28 December 2011 - 05:53 PM, said:

Based on everyone's comments, we seem to have discussed two approaches to solving these problems. One, build a tool that abstracts all the complexities, allows for easy re-use of content, and makes clean up simple within a traditional file/folder structure. Or move everything into a RDBMS, which by its nature addresses many of the issues. Is that really the basic choice before us?

There are several potential solutions Chris, not just two.

We could always just simply duplicate the Kuju design. People are familiar with it and the code already handles it. Whether it would be wise to do that or not is another discussion.

There is also one or more ways to make revisions to the Kuju design to address some of it's problems. That could produce several reasonable proposals varying in degree of change and/or what changes. Merits... another discussion.

James has put forward... something.

I have expressed the opinion that many of the problems are of a nature similar to what DBMS are designed to solve... but I've not firmed up the idea into a proposal.

Given that, it seems to me that the best description of the present state is we're brainstorming ideas. Nothing wrong with that at all. But to move forward it might be best to set aside potential solutions for a bit and instead revisit the list of what problems of the Kuju design we want to avoid repeating and what new capabilities we think are worthy enough to introduce. Whatever comes up in either are likely to be germaine to the design.

Right now I've got diner to deal with and I want to go see if I can get it down w/o throwing a plate across the room.

[smmille1]

I've seen a fair number of local (as in deployed on the client machine) database implementations that I was less than satisfied with. That being said, I figure there have been some that I didn't "see", which is what I would have as one of my main goals of a local installation of a RDBMS. I don't like cases that require a separate install (SQL Server CE, Oracle Lite, MSDE, etc.) because at least in some cases, they're sitting there eating CPU cycles even when the relevant program isn't running. And even if they aren't, it's still a piece of the puzzle that's not fully under your control.

The only thing I've seen by searching so far that I haven't seen in action (that I know of) and could be promising is SQLite. It doesn't require a separate install and appears to run in-process. It looks like the capacity has a hard limit at 2TB, but that could be worked around by storing the large binary content (textures, meshes) outside of the database itself that we'd have to manage separately. With the presumption that the user won't place this content, but rather will feed it to the DB through some sort of interface, I see pros and cons to keeping the files themselves outside. Biggest pro is that if we keep the data in some sort of directory structure, we could hand the path for a given file back to users so that they could update a file in place if they choose - good for content developers who are rapidly tweaking a model or texture. Biggest con is that we've obviously introduced a way for the users to wantonly mess things up by move/rename/delete actions. But that's still an issue if we go with a pure filesystem-based organization too.

[captain_bazza]

Quote

Given that, it seems to me that the best description of the present state is we're brainstorming ideas. Nothing wrong with that at all. But to move forward it might be best to set aside potential solutions for a bit and instead revisit the list of what problems of the Kuju design we want to avoid repeating and what new capabilities we think are worthy enough to introduce. Whatever comes up in either are likely to be germaine to the design.

As a content producer, I concur with Dave's comment.

Cheers Bazza

[Genma Saotome]

WRT relational databases (rdbms)... to begin with, I regard rdbms as just another set of functionality for which certain toolsets have packaged up an instance for easy use by programmers. By saying I think many of our problems could be dealt with a dbms I'm not saying which toolset to use -- or even to use any packaged toolset. One could implement the valued features of a rdbms entirely in "home grown" code. So it's the use of the functionality in designing a solution I'm interested in discussing not the toolset itself.

As an example of a "home grown" solution, there are numerous instances in our environment: one mesh gets placed many times in a route. The master data (the .s and .sd files) are defined once and many references to them are made in world files; each reference has a system generated unique identifier (e.g., UiD(), some data specific to that instance (e.g., location), and a foreign key (e.g., filename) that points back to the (one) copy of master data. THAT's a relational link done w/o a packaged toolset.

Rather than holding all of that inside a rdbms packaged toolset, Kuju implemented the relational design in text and directory structure. The text defaults to an implied path thru the well defined directory tree to the master data (n.b., we know from examples that a route designer can explicitly write in a different path if he wants to). Essentially the value of the path, implied or spelt out in text, is equivalent to the hashing that a packaged rdbms uses when it does a join of tables.

Kuju made a design decision that this relational path was assumed. They could have decided to use a textual entry of more / most of the path, something like "\Train Simulator"... etc, etc. and had they done so it would have been fairly easy for any person to change all occurrences of "\Train Simulator\routes\Whitefish9\shapes\PonderosaPine.s" to "\Train Simulator\global\shapes\PonderosaPine.s" without fear of error. It would be verbose... but transparently clear to all... and perhaps most important, obvious to even the dullest end user.

Kuju could have used a packaged rdbms toolset to do the same... one table called WorldPlacement, another called Shape. The WorldPlacement table would have a foreign key aiming back to the unique identifier on a record of Shape. Whether the Shape table subsequently had a key into some other table that held binary data, or itself had a field for binary data, or had a string containing a path and file name that located the binary data... would be just another design choice, probably made by the technical team based on performance considerations.

My point is Kuju used relational techniques... we're using relational techniques by following their design... and the discussion in this thread is, IMO, essentially one of what is the design for the path between an instance and the master definition? Is the value of the path implied by a well defined directory tree or Is it defined by whatever construct is contained within a zip file? Is it fixed for all time (in practical terms, not just by what's in the code) or can it be changed by an end user w/o fear of corrupting the works? Is it going to be transparently clear to all so it's trivially easy to work with -- and if not, will the OR team provide the tools to manage the hidden complexity so it can be trivial for end users to work with?

WRT James's proposal: I don't know what is required by that design as far as managing these relational paths... do I change a world file value of "FileName ( House.s)" to "..\\..\\someplace else\Downloads\From 3dtrains\Cajon.zip\routes\Cajon\Shapes\Bungalow.s" and give my newly edited world file to someone else who, in turn, need to replace all instances of "..\\..\\someplace else\Downloads\From 3dtrains\Cajon.zip\routes\Cajon\Shapes\Bungalow.s" with a string that is correct on their PC? I know I could be nice and just make a copy of Bungalow.s and put it my route... but what if I'm lazy and I don't bother? Or I do bother... and now the average end user has 14 copies of bungalow.s?

All of which is to say my primary objection to the proposal is in the absence of understand how to manage the relational paths in a clean, transparently simply way.

FWIW, IMO the use of MS-Dos relative paths doesn't qualify as a clean, transparently simply solution. And I have my doubts that an ordinary text editor is sufficiently powerful tool to properly manage the complexity. Last, it isn't obvious the need for the identification and use of master data is provided for... except by virtue of hoping a few sharp minds figure out the opportunity and spend the time to implement something.

[wacampbell]

Genma Saotome, on 30 December 2011 - 12:21 PM, said:

WRT James's proposal: I don't know what is required by that design as far as managing these relational paths... do I change a world file value of "FileName ( House.s)" to "..\\..\\someplace else\Downloads\From 3dtrains\Cajon.zip\routes\Cajon\Shapes\Bungalow.s" and give my newly edited world file to someone else who, in turn, need to replace all instances of "..\\..\\someplace else\Downloads\From 3dtrains\Cajon.zip\routes\Cajon\Shapes\Bungalow.s" with a string that is correct on their PC? I know I could be nice and just make a copy of Bungalow.s and put it my route... but what if I'm lazy and I don't bother? Or I do bother... and now the average end user has 14 copies of bungalow.s?

I certainly agree that what MSTS is using, and what James proposes are actually RDBMS. It makes me chuckle when I hear people say 'lets use a rdbms'. But I fear that electronic communication is failing us. When I read your concerns, they seem to be perfectly resolved by James' proposal, so it must be that I just don't understand your worry. For discussin here lets say we retain a text based world file. To reference Bungalow.s, it will contain an entry like shape( 3DTrainsCajon, Bungalow.s ). There is no path there. It tells OR to find the Bungalow.s in the 3DTrainsCajon package. That 3DTrainsCajon package could be located anywhere on the users harddisk. The 3DTrainsCajon package is a folder in any of the registered content directories. We haven't decided whether that folder will be identified by metadata, or if it simply will have the name 3DTrainsCajon, but the end user has incredible flexibility on where it is located.

[James Ross]

Genma Saotome, on 30 December 2011 - 12:21 PM, said:

WRT James's proposal: I don't know what is required by that design as far as managing these relational paths... do I change a world file value of "FileName ( House.s)" to "..\\..\\someplace else\Downloads\From 3dtrains\Cajon.zip\routes\Cajon\Shapes\Bungalow.s" and give my newly edited world file to someone else who, in turn, need to replace all instances of "..\\..\\someplace else\Downloads\From 3dtrains\Cajon.zip\routes\Cajon\Shapes\Bungalow.s" with a string that is correct on their PC? I know I could be nice and just make a copy of Bungalow.s and put it my route... but what if I'm lazy and I don't bother? Or I do bother... and now the average end user has 14 copies of bungalow.s?

The paths are relative to the virtual file structure, which exists seperately for each "type" (shape, texture, etc.) and is independant of the location of the packages (OR will scan a list of content directories and pretend they're all together using the virtual structure). The virtual file paths start with a package name and then can be any number of directories down (including 0) and a file - depending on how the author of that package layed out that type of file. Paths are resolved relative to their own package.

Example Files
\Some File I Downloaded From The Internet.zip

\3DTS Cajon

\Shape\Bungalow.s

\My Package.zip

\James's Package

\Shape\House.s
\World\-123456-123456.w

For "\My Package.zip\James's Package\World\-123456-123456.w" to reference "House.s" in its own package it would use FileName("House.s"). To reference "Bungalow.s" in "Some File I Downloaded From The Internet.zip" it would use FileName("..\3DTS Cajon\Bungalow.s"). Because OR knows it's looking for a shape, it will be using the shape virtual file structure, starting in the "Shape" directory for "James's Package". Up one level from this is the virtual directory listing all packages with "Shape" directories, so "..\3DTS Cajon" takes it to the "Shape" directory in the package called "3DTS Cajon".

[smmille1]

If we're going into semantics, it's the MS part of RDBMS that the current and proposed solutions are lacking - keeping piles of files is inherently unmanaged. You can try to approximate an RDBMS on top of those files, but it won't be a true RDBMS since you can't guarantee any sort of integrity.

---

And to continue contributing to the discussion, here are my current questions regarding James' proposal:

1. Why do the zip files contain a top level directory (and why don't we then care about the zip names)? Assuming we're allowing folders or zip files, wouldn't it make more sense to have the zip name AS the package folder name?

So you get
...Content\MyPackage.zip containing folder\file
or
...Content\MyPackage\folder\file

and both are resolved the same way logically with MyPackage as the package name.

2. If we're referring to files outside of the current package, why the ".."? If there aren't going to be separately aliased content folders and we're not actually referring to filesystem locations, then I think you can make the assumption of if it starts with a slash or backslash (there's a mini-discussion in itself), then it starts with a package, and if not, then it's local to the current package.

3. What is the exact benefit to segregating by file type and then inferring the type folder beyond saving a couple of bytes in the referencing string? I think you've said that you can have subfolders inside the type folders (textures, shapes, etc.), but it would seem to me that if I create a package with several components, I might rather organize things by relationship, especially if I have several different groups of things (buildings, trees, signs, etc.) in the package.

[James Ross]

smmille1, on 30 December 2011 - 04:19 PM, said:

1. Why do the zip files contain a top level directory (and why don't we then care about the zip names)? Assuming we're allowing folders or zip files, wouldn't it make more sense to have the zip name AS the package folder name?

As I explained previously in this thread, the zip filename is excluded because websites and browsers are bad at preserving the filename reliably. An alternative, which I may have also mentioned previously, is to have some metadata in the zip (i.e. a specific file in the root) that provides the package name instead of the directory (which would also work for directory-packages and let them be freely renamed). I'm not too fussed how we get the package name but I'm pretty sure the zip filename is out.

smmille1, on 30 December 2011 - 04:19 PM, said:

2. If we're referring to files outside of the current package, why the ".."? If there aren't going to be separately aliased content folders and we're not actually referring to filesystem locations, then I think you can make the assumption of if it starts with a slash or backslash (there's a mini-discussion in itself), then it starts with a package, and if not, then it's local to the current package.

It makes no difference, if we implement the virtual file structure properly, but people are familiar with the ..\ notation so that's what I've used in my examples.

smmille1, on 30 December 2011 - 04:19 PM, said:

3. What is the exact benefit to segregating by file type and then inferring the type folder beyond saving a couple of bytes in the referencing string? I think you've said that you can have subfolders inside the type folders (textures, shapes, etc.), but it would seem to me that if I create a package with several components, I might rather organize things by relationship, especially if I have several different groups of things (buildings, trees, signs, etc.) in the package.

Saving those bytes also reduces the chance of errors, like typos, and misunderstandings about what it references. The intention with the strict 1st level within a package is to make the contents of packages more discoverable/explorable by both people and programs, such that neither I nor OR needs to, for example, search the entire zip file for wherever you've hidden the activity files. I agree that the groups of shape (.s), shape data (.sd) and textures (.ace) for particular objects might seem like grouping, but you can share the textures between shapes, so I think pretty much everything should be grouped by type ultimately. There could be exceptions, such as .s and .sd files both being in "Shape", and the special consideration for routes I've mentioned before.

[longiron]

wacampbell, on 30 December 2011 - 02:55 PM, said:

I certainly agree that what MSTS is using, and what James proposes are actually RDBMS. It makes me chuckle when I hear people say 'lets use a rdbms'.

Wayne,
When I say use a RDBMS, I'm talking about a true database. Tables, Primary and secondary keys, rows, columns, blob data and outside references. No more individual files for train data. It works well for eng, wag files as they share common parameters and data structures, and may work for routes. It makes it extremely easy to use common data sets as many items can reference the same table/columns/rows. It does demand a tool set to manage data and some kind of tool to package data for sharing.

[cjakeman]

I've had some new commercial routes for Christmas :-) Some very impressive scenery from Making Tracks (UK) Great Eastern and Greater Eastern and also the local preserved Nene Valley Railway.

I had to re-install MSTS (for other reasons) and discovered that these routes will only play nicely together if they are installed in a certain sequence. (OR pointed the finger at TSection.dat) Maybe there are even combinations of routes which can never co-exist.

Perhaps this is a well-known problem (and I apologise if I'm going over old ground) but will these proposals for Overall File System and Data Structure avoid this particular issue?

[longiron]

You've discovered the dreaded Tsection.dat issue. Since trackwork in MSTS is like model RR sectional track, all the track pieces must be defined for the game. Different routes were built using different Tsection.dat. In general, later versions of Tsection should work with older routes. But you must be careful during installation that it doesn't get overwritten with an earlier version.

OR proposal is to use procedural track. Essentially a profile extruded along the railway route. So the MSTS type of issues should not be a problem.

[Genma Saotome]

James Ross, on 30 December 2011 - 03:55 PM, said:

Example Files
\Some File I Downloaded From The Internet.zip

\3DTS Cajon

\Shape\Bungalow.s

\My Package.zip

\James's Package

\Shape\House.s
\World\-123456-123456.w

For "\My Package.zip\James's Package\World\-123456-123456.w" to reference "House.s" in its own package it would use FileName("House.s"). To reference "Bungalow.s" in "Some File I Downloaded From The Internet.zip" it would use FileName("..\3DTS Cajon\Bungalow.s"). Because OR knows it's looking for a shape, it will be using the shape virtual file structure, starting in the "Shape" directory for "James's Package". Up one level from this is the virtual directory listing all packages with "Shape" directories, so "..\3DTS Cajon" takes it to the "Shape" directory in the package called "3DTS Cajon".

Kickin life into an old thread... I do understand the above example. Questions:

• How does a route designer find all of the packages he has made use of (for the purpose of gathering them all up to make a distribution package of his own)?
  
• How does he gather those packages together into his own about-to-be-distributed package (call it "Dave's Project")?
  
• After installing "..\3DTS Cajon\Bungalow.s" as part of "Dave's Project", Wayne does "Wayne's Project" and makes use of "Dave's Project\..\3DTS Cajon\Bungalow.s" and wants to package that up to give to Tom, where upon Tom does the same for his project and distributes to Dick, who does the same and gives it to Harry, who does the same and gives it to Chris Van Wagoner. He finds "Harry's Project\Dicks Project\Tom's Project\Waynes Project\Dave's Project\..\3DTS Cajon\Bungalow.s" How is what Chris receives better than the RSC example he cited above?