[Genma Saotome]

 James Ross, on 28 December 2011 - 02:35 PM, said:

... but here are the bits I can be bothered typing out again...

Alrighty then.

For the record, I couldn't care less if this or that proposal makes things easier to code. All I care about is whether any idea is going to be better or worse for the end users and / or content providers as they're the ones who will have to live with it for n years to come. I can see this proposal probably does address the issue of downloaded packages having a chance of being complete. Whether it would be used in that manner is plausible but not assured. The likelihood of duplication appears to me to be greater than what Kuju's design created. The proposal tells me nothing about anything else other than it is simply different than the Kuju design.

As for "Virtual Files" the string appears nowhere else in these forums.

If someone besides James wants to explain anything about this... here, via PM, via e-mail, feel free.

[longiron]

I guess I must go back to the beginning, to make heads of this discussion. The original assumption were that the current MSTS storage plan has some deficiencies as detailed below.

- difficult to manage at scale, lots of time & effort
- must use other's conventions (folder name, file name, etc) to share stuff (Trainset/Activities/Paths)
- lots of repetitious stuff (wav, ace, etc) that needs to be managed
- clean up (delete) is impossible unless you understand the dependencies

Based on everyone's comments, we seem to have discussed two approaches to solving these problems. One, build a tool that abstracts all the complexities, allows for easy re-use of content, and makes clean up simple within a traditional file/folder structure. Or move everything into a RDBMS, which by its nature addresses many of the issues. Is that really the basic choice before us?

[wacampbell]

 longiron, on 28 December 2011 - 05:53 PM, said:

Or move everything into a RDBMS, which by its nature addresses many of the issues. Is that really the basic choice before us?

I'd like to hear more from the proponents of the RDBMS solution. How would addons be distributed and merged into the RDBMS? Can these RDBMS's handle binary data such as mesh files and image files?

[Genma Saotome]

 longiron, on 28 December 2011 - 05:53 PM, said:

Based on everyone's comments, we seem to have discussed two approaches to solving these problems. One, build a tool that abstracts all the complexities, allows for easy re-use of content, and makes clean up simple within a traditional file/folder structure. Or move everything into a RDBMS, which by its nature addresses many of the issues. Is that really the basic choice before us?

There are several potential solutions Chris, not just two.

We could always just simply duplicate the Kuju design. People are familiar with it and the code already handles it. Whether it would be wise to do that or not is another discussion.

There is also one or more ways to make revisions to the Kuju design to address some of it's problems. That could produce several reasonable proposals varying in degree of change and/or what changes. Merits... another discussion.

James has put forward... something.

I have expressed the opinion that many of the problems are of a nature similar to what DBMS are designed to solve... but I've not firmed up the idea into a proposal.

Given that, it seems to me that the best description of the present state is we're brainstorming ideas. Nothing wrong with that at all. But to move forward it might be best to set aside potential solutions for a bit and instead revisit the list of what problems of the Kuju design we want to avoid repeating and what new capabilities we think are worthy enough to introduce. Whatever comes up in either are likely to be germaine to the design.

Right now I've got diner to deal with and I want to go see if I can get it down w/o throwing a plate across the room.

[smmille1]

I've seen a fair number of local (as in deployed on the client machine) database implementations that I was less than satisfied with. That being said, I figure there have been some that I didn't "see", which is what I would have as one of my main goals of a local installation of a RDBMS. I don't like cases that require a separate install (SQL Server CE, Oracle Lite, MSDE, etc.) because at least in some cases, they're sitting there eating CPU cycles even when the relevant program isn't running. And even if they aren't, it's still a piece of the puzzle that's not fully under your control.

The only thing I've seen by searching so far that I haven't seen in action (that I know of) and could be promising is SQLite. It doesn't require a separate install and appears to run in-process. It looks like the capacity has a hard limit at 2TB, but that could be worked around by storing the large binary content (textures, meshes) outside of the database itself that we'd have to manage separately. With the presumption that the user won't place this content, but rather will feed it to the DB through some sort of interface, I see pros and cons to keeping the files themselves outside. Biggest pro is that if we keep the data in some sort of directory structure, we could hand the path for a given file back to users so that they could update a file in place if they choose - good for content developers who are rapidly tweaking a model or texture. Biggest con is that we've obviously introduced a way for the users to wantonly mess things up by move/rename/delete actions. But that's still an issue if we go with a pure filesystem-based organization too.

[captain_bazza]

Quote

Given that, it seems to me that the best description of the present state is we're brainstorming ideas. Nothing wrong with that at all. But to move forward it might be best to set aside potential solutions for a bit and instead revisit the list of what problems of the Kuju design we want to avoid repeating and what new capabilities we think are worthy enough to introduce. Whatever comes up in either are likely to be germaine to the design.

As a content producer, I concur with Dave's comment.

Cheers Bazza

[Genma Saotome]

WRT relational databases (rdbms)... to begin with, I regard rdbms as just another set of functionality for which certain toolsets have packaged up an instance for easy use by programmers. By saying I think many of our problems could be dealt with a dbms I'm not saying which toolset to use -- or even to use any packaged toolset. One could implement the valued features of a rdbms entirely in "home grown" code. So it's the use of the functionality in designing a solution I'm interested in discussing not the toolset itself.

As an example of a "home grown" solution, there are numerous instances in our environment: one mesh gets placed many times in a route. The master data (the .s and .sd files) are defined once and many references to them are made in world files; each reference has a system generated unique identifier (e.g., UiD(), some data specific to that instance (e.g., location), and a foreign key (e.g., filename) that points back to the (one) copy of master data. THAT's a relational link done w/o a packaged toolset.

Rather than holding all of that inside a rdbms packaged toolset, Kuju implemented the relational design in text and directory structure. The text defaults to an implied path thru the well defined directory tree to the master data (n.b., we know from examples that a route designer can explicitly write in a different path if he wants to). Essentially the value of the path, implied or spelt out in text, is equivalent to the hashing that a packaged rdbms uses when it does a join of tables.

Kuju made a design decision that this relational path was assumed. They could have decided to use a textual entry of more / most of the path, something like "\Train Simulator"... etc, etc. and had they done so it would have been fairly easy for any person to change all occurrences of "\Train Simulator\routes\Whitefish9\shapes\PonderosaPine.s" to "\Train Simulator\global\shapes\PonderosaPine.s" without fear of error. It would be verbose... but transparently clear to all... and perhaps most important, obvious to even the dullest end user.

Kuju could have used a packaged rdbms toolset to do the same... one table called WorldPlacement, another called Shape. The WorldPlacement table would have a foreign key aiming back to the unique identifier on a record of Shape. Whether the Shape table subsequently had a key into some other table that held binary data, or itself had a field for binary data, or had a string containing a path and file name that located the binary data... would be just another design choice, probably made by the technical team based on performance considerations.

My point is Kuju used relational techniques... we're using relational techniques by following their design... and the discussion in this thread is, IMO, essentially one of what is the design for the path between an instance and the master definition? Is the value of the path implied by a well defined directory tree or Is it defined by whatever construct is contained within a zip file? Is it fixed for all time (in practical terms, not just by what's in the code) or can it be changed by an end user w/o fear of corrupting the works? Is it going to be transparently clear to all so it's trivially easy to work with -- and if not, will the OR team provide the tools to manage the hidden complexity so it can be trivial for end users to work with?

WRT James's proposal: I don't know what is required by that design as far as managing these relational paths... do I change a world file value of "FileName ( House.s)" to "..\\..\\someplace else\Downloads\From 3dtrains\Cajon.zip\routes\Cajon\Shapes\Bungalow.s" and give my newly edited world file to someone else who, in turn, need to replace all instances of "..\\..\\someplace else\Downloads\From 3dtrains\Cajon.zip\routes\Cajon\Shapes\Bungalow.s" with a string that is correct on their PC? I know I could be nice and just make a copy of Bungalow.s and put it my route... but what if I'm lazy and I don't bother? Or I do bother... and now the average end user has 14 copies of bungalow.s?

All of which is to say my primary objection to the proposal is in the absence of understand how to manage the relational paths in a clean, transparently simply way.

FWIW, IMO the use of MS-Dos relative paths doesn't qualify as a clean, transparently simply solution. And I have my doubts that an ordinary text editor is sufficiently powerful tool to properly manage the complexity. Last, it isn't obvious the need for the identification and use of master data is provided for... except by virtue of hoping a few sharp minds figure out the opportunity and spend the time to implement something.

[wacampbell]

 Genma Saotome, on 30 December 2011 - 12:21 PM, said:

WRT James's proposal: I don't know what is required by that design as far as managing these relational paths... do I change a world file value of "FileName ( House.s)" to "..\\..\\someplace else\Downloads\From 3dtrains\Cajon.zip\routes\Cajon\Shapes\Bungalow.s" and give my newly edited world file to someone else who, in turn, need to replace all instances of "..\\..\\someplace else\Downloads\From 3dtrains\Cajon.zip\routes\Cajon\Shapes\Bungalow.s" with a string that is correct on their PC? I know I could be nice and just make a copy of Bungalow.s and put it my route... but what if I'm lazy and I don't bother? Or I do bother... and now the average end user has 14 copies of bungalow.s?

I certainly agree that what MSTS is using, and what James proposes are actually RDBMS. It makes me chuckle when I hear people say 'lets use a rdbms'. But I fear that electronic communication is failing us. When I read your concerns, they seem to be perfectly resolved by James' proposal, so it must be that I just don't understand your worry. For discussin here lets say we retain a text based world file. To reference Bungalow.s, it will contain an entry like shape( 3DTrainsCajon, Bungalow.s ). There is no path there. It tells OR to find the Bungalow.s in the 3DTrainsCajon package. That 3DTrainsCajon package could be located anywhere on the users harddisk. The 3DTrainsCajon package is a folder in any of the registered content directories. We haven't decided whether that folder will be identified by metadata, or if it simply will have the name 3DTrainsCajon, but the end user has incredible flexibility on where it is located.

[James Ross]

 Genma Saotome, on 30 December 2011 - 12:21 PM, said:

WRT James's proposal: I don't know what is required by that design as far as managing these relational paths... do I change a world file value of "FileName ( House.s)" to "..\\..\\someplace else\Downloads\From 3dtrains\Cajon.zip\routes\Cajon\Shapes\Bungalow.s" and give my newly edited world file to someone else who, in turn, need to replace all instances of "..\\..\\someplace else\Downloads\From 3dtrains\Cajon.zip\routes\Cajon\Shapes\Bungalow.s" with a string that is correct on their PC? I know I could be nice and just make a copy of Bungalow.s and put it my route... but what if I'm lazy and I don't bother? Or I do bother... and now the average end user has 14 copies of bungalow.s?

The paths are relative to the virtual file structure, which exists seperately for each "type" (shape, texture, etc.) and is independant of the location of the packages (OR will scan a list of content directories and pretend they're all together using the virtual structure). The virtual file paths start with a package name and then can be any number of directories down (including 0) and a file - depending on how the author of that package layed out that type of file. Paths are resolved relative to their own package.

Example Files
\Some File I Downloaded From The Internet.zip

\3DTS Cajon

\Shape\Bungalow.s

\My Package.zip

\James's Package

\Shape\House.s
\World\-123456-123456.w

For "\My Package.zip\James's Package\World\-123456-123456.w" to reference "House.s" in its own package it would use FileName("House.s"). To reference "Bungalow.s" in "Some File I Downloaded From The Internet.zip" it would use FileName("..\3DTS Cajon\Bungalow.s"). Because OR knows it's looking for a shape, it will be using the shape virtual file structure, starting in the "Shape" directory for "James's Package". Up one level from this is the virtual directory listing all packages with "Shape" directories, so "..\3DTS Cajon" takes it to the "Shape" directory in the package called "3DTS Cajon".

[smmille1]

If we're going into semantics, it's the MS part of RDBMS that the current and proposed solutions are lacking - keeping piles of files is inherently unmanaged. You can try to approximate an RDBMS on top of those files, but it won't be a true RDBMS since you can't guarantee any sort of integrity.

---

And to continue contributing to the discussion, here are my current questions regarding James' proposal:

1. Why do the zip files contain a top level directory (and why don't we then care about the zip names)? Assuming we're allowing folders or zip files, wouldn't it make more sense to have the zip name AS the package folder name?

So you get
...Content\MyPackage.zip containing folder\file
or
...Content\MyPackage\folder\file

and both are resolved the same way logically with MyPackage as the package name.

2. If we're referring to files outside of the current package, why the ".."? If there aren't going to be separately aliased content folders and we're not actually referring to filesystem locations, then I think you can make the assumption of if it starts with a slash or backslash (there's a mini-discussion in itself), then it starts with a package, and if not, then it's local to the current package.

3. What is the exact benefit to segregating by file type and then inferring the type folder beyond saving a couple of bytes in the referencing string? I think you've said that you can have subfolders inside the type folders (textures, shapes, etc.), but it would seem to me that if I create a package with several components, I might rather organize things by relationship, especially if I have several different groups of things (buildings, trees, signs, etc.) in the package.