[James Ross]

Quote

Open Rails consist editor for new file formats (JSON)

• Load/save OR consists
  
• Import MSTS consists
  
• Select from MSTS rolling stock

Blueprint: https://blueprints.l.../consist-editor
Roadmap: https://trello.com/c...-consist-editor

I have not yet written a full specification for this, but wanted to kick-off the discussion anyway to see what people thought early, as this would be our first official Open Rails editor.

The goal here is to have a simple editor, which can be improved upon later, that provides the very basic function of creating a train consist from an available collection of locomotives and wagons. This would not be a complex UI, but would involve a minimum of the following specific features:

• Design and implementation of an OR-specific consist format - which can at minimum specify an ordered list of locomotives and wagons, and flip individual cars.
  
• Conversion of MSTS consist files into OR consist files (not with 100% fidelity).
  
• Identification/reference system for linking to MSTS rolling stock from OR file formats.

I would also like to explore the ideas in the "Content Manager" contributed project, where there is an abstraction layer between the core game's "load car X from consist Y"-style function and the physical files on disk. I believe this abstraction would be useful for a number of reasons:

• Each downloaded content pack can be kept separate (rather than all installed into a single mini-route, for example) while still being able to easily have one pack reference another.
  
• Allowing the mixture of OR and MSTS formats more freely, as cross-references can be resolved in ways other than simply looking at an exact file path.
  
• Multiplayer content sharing, wherein players could receive another player's content over the network seamlessly.

If we do include the ideas from the "Content Manager", this would also be the ideal time to implement a new directory structure that is less about a single game's install and more about the content, e.g. moving all shapes, textures, etc., outside of routes into a top-level folder, and such like. This would make the cross-referencing clearer and more obvious: each content pack simply has a list of shapes, textures, locomotives, etc., that it contains.

Questions: Do the above points seem appropriate to start work on now? Is there anything obvious missing above that we should consider before embarking on this?

[cr-stagg]

Concerning your proposal

Quote

this would also be the ideal time to implement a new directory structure that is less about a single game's install and more about the content, e.g. moving all shapes, textures, etc., outside of routes into a top-level folder

How would you know that Station.s from Route A is exactly the same as Station.s from Route B? You could end up with a Route in California that has a Station with a signboard that says New York instead of San Diego. I think it would be better to keep Route Shapes within the Route folder structure.

[Genma Saotome]

James Ross, on 02 April 2017 - 01:22 PM, said:

If we do include the ideas from the "Content Manager", this would also be the ideal time to implement a new directory structure that is less about a single game's install and more about the content, e.g. moving all shapes, textures, etc., outside of routes into a top-level folder, and such like. This would make the cross-referencing clearer and more obvious: each content pack simply has a list of shapes, textures, locomotives, etc., that it contains.

It's well understood that how KUJU chose to handle multiple uses of a single file (or folder) across multiple route directories has it's problems. An abstraction layer should go a long way to addressing those issues.

The concern I have is WRT abandoning the "traditional" directory structure and (eventually) moving everything (presumably into into zip files) under a single content directory. Speaking as a content developer there are lots and lots of situations where you want multiple directories containing the same list of files where the content of the like-named files are not the same. For example, one directory holding the what was released last month (now unchanging) and another that remains the development directory tree. For the most part, the same files using the same names but where the first directory had Rev 6 bits the second has Rev 7 -- and depending on the task you need to look today at Rev 6 and tomorrow at Rev 7.

To bring the example into clear focus, today we could be taking about a .con, .act., .srv, or .trf file(s). Names are all the same, lines are all different. Tomorrow we could be talking about Rev 7 of an .ace file, a world file, and 30 brand new .s and .sd files.

My point here is to show that a content developer works in a different environment than both a passive end user and a content re-user much in the sme way a software developer's environment and needs are quite different from his end users.

Am I correct in guessing your answer will be two different packages (zip files), one for Rev 6 and the other for rev 7?

[Genma Saotome]

Another example of the complexity here for your consideration:

Route A is installed in 2 different directories. What makes them different is the era which is shown by which rolling stock is displayed -- steam era vs. diesel era.

One solution is simply to change the rows of the .con file to select the correct locomotives and rolling stock. Because loose consists are not managed as consists (IMO a problem) this means the .act file needs to be edited as well. The .com and .act file names remain the same while having different rows because they are in two different directory trees.

An alternative to the above would be to change the file names of the service and traffic files so they refer to brand new consist files, and changing the rows of the .act file to reflect that. This too works.

I prefer the first method but the second works equally well making the choice pretty subjective.

In the proposed future, how would this situation be addressed?




postscript: Upon further reflection it occurs to me to suggest the root issue is this: Global files work perfectly well until you have a situation where things no longer should be global but instead handled on a case by case basis. Then global stuff gets in the way.

How does the proposal address this fact?

[James Ross]

cr-stagg, on 02 April 2017 - 01:58 PM, said:

How would you know that Station.s from Route A is exactly the same as Station.s from Route B? You could end up with a Route in California that has a Station with a signboard that says New York instead of San Diego. I think it would be better to keep Route Shapes within the Route folder structure.

There are two approaches I can see to this:

• Content created in this new arrangement will have more distinctive names, perhaps including the author's initials, world region, or such.
  
• Content will be referenced via a pair of identifiers: package ID + file name.

(As mentioned at the end, also) For name-only referencing, I would still want to look for content locally in the same package first.

Genma Saotome, on 02 April 2017 - 04:06 PM, said:

My point here is to show that a content developer works in a different environment than both a passive end user and a content re-user much in the sme way a software developer's environment and needs are quite different from his end users.

Am I correct in guessing your answer will be two different packages (zip files), one for Rev 6 and the other for rev 7?

Yes, but perhaps not quite in the way you were expecting.

Firstly, let's set aside any zip files - they would be a convenience for distribution of finished content only. Development can and will work with files directly in the file system.

As for revisions and alternate versions, there is a potentially interesting way of doing this with the 2nd option above (i.e. using IDs + names to reference content): you can have packages that override other packages. In other words, you can have 7 different packages with the same ID but different version numbers (or something else to identify which overrides which) but the game will search through them from newest to oldest when locating content (we can do this more efficiently than this sounds) so you only have new/updated content in each package. The burden of creating a new "revision" of the package could be handled as a single button in the tools, as could created a "merged" package for release.

That's one idea, anyway.

I mean, at the basic level we could use it the same as mini-routes work now: the current package is the default search location anyway (even when using ID + name we can have no ID imply the current package) so you can just create a new copy of the package and the only thing that might need any tweaking is either how we cope with duplicates, or you'd change the ID of the new copy (but only need to do it in one place).

Genma Saotome, on 02 April 2017 - 04:47 PM, said:

Route A is installed in 2 different directories. What makes them different is the era which is shown by which rolling stock is displayed -- steam era vs. diesel era.

One solution is simply to change the rows of the .con file to select the correct locomotives and rolling stock. Because loose consists are not managed as consists (IMO a problem) this means the .act file needs to be edited as well. The .com and .act file names remain the same while having different rows because they are in two different directory trees.

An alternative to the above would be to change the file names of the service and traffic files so they refer to brand new consist files, and changing the rows of the .act file to reflect that. This too works.

I prefer the first method but the second works equally well making the choice pretty subjective.

In the proposed future, how would this situation be addressed?

One of the things you could do out-of-the-box (as it were) would be to use any of your rolling stock on any of your routes. That would at least let you use steam vs diesel as your choice for exploring.

For the activities (and timetables) side of things, you would probably have to have two copies of whichever files reference the rollingstock, each referencing their respective version, but any shared files, including the route itself, would likely still exist only once.

Genma Saotome, on 02 April 2017 - 04:47 PM, said:

postscript: Upon further reflection it occurs to me to suggest the root issue is this: Global files work perfectly well until you have a situation where things no longer should be global but instead handled on a case by case basis. Then global stuff gets in the way.

How does the proposal address this fact?

This depends on which route we take with the content references at the top of this post.

If we use just names, the idea would be to look locally before looking globally, and when looking globally always look in a defined order. This would mean that if a file in a content pack references a particular shape (for example), and that shape exists in that same content pack, it would always use that one. And if it doesn't, you'd be given some level of control over how it found it.

If we use pack IDs + names, I think this is pretty much a non-issue as you will always be referring to a very specific item across the whole content ecosystem. You might still be able to override one pack with another, but it would still be very deterministic.

[thegrindre]

There is one thing I'd be interested in and that would be a small graphic profile from the shape file for display.
I'm not so fond of Convoi's method (Takes up too much room on the hard drive) but at least something to look at, anyway.

http://www.elvastower.com/forums/public/style_emoticons/default/oldstry.gif

[Genma Saotome]

James, I remain skeptical but at this point in time I think it best to reserve further comment until such time there is more available to comment on.

May I suggest to you that a change of this magnitude had better be obviously way better than how things are done today because the real risk here is it won't be, that all you wind up with is replacing a bunch on known problems for an equal number of unknown problems and in doing so you've wasted not only your own time and skill but that of everyone else as well. So take your time and work the problem really well with as much feedback as you can get.



For the record, I'm reasonably happy with a combination of symbolic links and ordinary copy/paste to create a number of associative directories between two reference directory trees -- one of routes and the other of rolling stock. This lets me logically present any route several times, each in a directory tree specific to an era, at no cost in disk space. I do the same with other big "global" directories, like \global\shapes and \sound as well as "largish" folders inside of \trainset. The Beyond Compare software, which I highly recommend, does an excellent job of keeping the copy/paste stuff in sync. The only disappointment I have in this setup is with regard to the activity files that have loose consists, something that could quickly be solved by allowing a .con file to replace each set of rows of individual car.

The value in this solution is it did not require the OR team to invest their scarce -- and therefore valuable -- time and skill into building something I could do on my own. Please take this as food for thought.

[Genma Saotome]

On second thought, there is something I want to comment on and that is the idea of appending a personal identifier on some files to help distinguish them from otherwise identically named files from somebody else. You do touch on a real problem there, one not easily solved. Initials, or initials plus rev id (or date) is worthwhile looking into -- I suggested the same idea for Camera Sets. The loophole that spoils the idea is this: how do you prevent somebody from changing something inside the package and then forwarding it on to someone else without changing the package id?

[Mike B]

Way back in ancient days, library sites for FS4 (like the FS section of Compuserve) had voluntary (but generally used) file naming conventions for scenery, especially. Those were in 8.3 filename days, of course, so things had to be a tad terse. IIRC it commonly went along the lines of VORSCNii where VOR was the central VOR, airport ID, or the like. For very large-area package, it might be something more descriptive, like NORCAL. ii was the creator's initials. Actually worked out quite well, and most people followed the script. So yes, it can be done.

As for the loophole - you can't fix that unless you have a way to guarantee that files are automatically versioned when changed, and checkouts are logged. IOW some kind of Subversion or Github setup. Probably way too much trouble. However, having a Library where the "master" version is always available to compare with what you got from $someplace_else is a Good Thing. I suppose you could post hashes for the library files so people could check whether they got the Real Thing. EDIT: sorry; thought you were talking about something else.

[James Ross]

thegrindre, on 03 April 2017 - 12:41 PM, said:

There is one thing I'd be interested in and that would be a small graphic profile from the shape file for display.
I'm not so fond of Convoi's method (Takes up too much room on the hard drive) but at least something to look at, anyway.

I'd start with a text-only version for simplicity first, but that is probably the first thing we'd add after it was up and running. :)

Genma Saotome, on 03 April 2017 - 03:48 PM, said:

James, I remain skeptical but at this point in time I think it best to reserve further comment until such time there is more available to comment on.

Absolutely; we can start off with the new consist file format being allowed/used inside the MSTS file structure, without any content manager, so that we can focus on one thing at a time.

Genma Saotome, on 03 April 2017 - 03:48 PM, said:

May I suggest to you that a change of this magnitude had better be obviously way better than how things are done today because the real risk here is it won't be, that all you wind up with is replacing a bunch on known problems for an equal number of unknown problems and in doing so you've wasted not only your own time and skill but that of everyone else as well. So take your time and work the problem really well with as much feedback as you can get.

The more we discuss it, the more I feel we should separate it from the consist editor's initial version - while we can think about better arrangements for files, referencing, etc., there's actually no need to tie it together. We can work on the JSON format for consists in MSTS directories, and subsequently add the new structure (assuming it meets said criteria of being way better).

This is by no means exhaustive, but some things we might be able to do if a new directory structure/package format was created:

• Share all content items - including textures and shapes - between all places (e.g. avoiding the need for "installme.bat")
  
• Install/uninstall content packs without overwriting / deleting other pack's files
  
• Automatically identify dependencies on other required content packs (maybe even with web page links)

Genma Saotome, on 03 April 2017 - 03:48 PM, said:

For the record, I'm reasonably happy with a combination of symbolic links and ordinary copy/paste to create a number of associative directories between two reference directory trees -- one of routes and the other of rolling stock. This lets me logically present any route several times, each in a directory tree specific to an era, at no cost in disk space. I do the same with other big "global" directories, like \global\shapes and \sound as well as "largish" folders inside of \trainset. The Beyond Compare software, which I highly recommend, does an excellent job of keeping the copy/paste stuff in sync. The only disappointment I have in this setup is with regard to the activity files that have loose consists, something that could quickly be solved by allowing a .con file to replace each set of rows of individual car.

These types of arrangement are a motivating factor for me; whilst obviously you and I can do them, and software like Beyond Compare makes it much easier than it otherwise would be, if we can bring some of the same ability to all users in a more "native" way, I'd be happy. This could involve us producing a tool that manages the symbolic links for you with a train simulator-focused UI on top, although I'd rather not as I take your point below on scares resources. But for new content, the less we need to do this kind of "patching" of files to get where we want, the better IMHO.

Genma Saotome, on 03 April 2017 - 03:48 PM, said:

The value in this solution is it did not require the OR team to invest their scarce -- and therefore valuable -- time and skill into building something I could do on my own. Please take this as food for thought.

Perhaps you could pen a section for the manual on how you set this up?

Genma Saotome, on 03 April 2017 - 03:57 PM, said:

On second thought, there is something I want to comment on and that is the idea of appending a personal identifier on some files to help distinguish them from otherwise identically named files from somebody else. You do touch on a real problem there, one not easily solved. Initials, or initials plus rev id (or date) is worthwhile looking into -- I suggested the same idea for Camera Sets. The loophole that spoils the idea is this: how do you prevent somebody from changing something inside the package and then forwarding it on to someone else without changing the package id?

Although the JavaScript package ecosystem is not a pretty place for many reasons, their use of a "package.json" to identify the name, version, description, authors, and some other key metadata for each package is a pretty nice idea. It is something I would like to do with the content manager packages, but how much information to include is still very much unknown.

On the point of package IDs specifically, there's a number of mitigations that can be employed:

• Package hosting sites put in a small amount of effort to prevent duplicate package IDs on their site
  
• Create a meta-package site that doesn't host, but does track metadata - which the hosting sites can check against before allowing uploads
  
• Open Rails itself ignores and/or shows an error on any duplicate IDs in the installed packages
  
• Distributing packages in zip files (and having them run from those zip files) may make it clearer that modifications aren't the norm (i.e. modifying content from other people requires an extra step or two, but doesn't prevent it)

But at the end of the day, it's really a social problem; technical solutions get help spot mistakes, but it would be hard to stop deliberate actions.

[Genma Saotome]

>Perhaps you could pen a section for the manual on how you set this up?

This thread describes what I was doing at the time. I expect today it is slightly different but not significantly so.

[Goku]

thegrindre, on 03 April 2017 - 12:41 PM, said:

There is one thing I'd be interested in and that would be a small graphic profile from the shape file for display.
I'm not so fond of Convoi's method (Takes up too much room on the hard drive) but at least something to look at, anyway.

Today's computers are so fast, that I think it is better to always render shapes in realtime, like TSRE Consist Editor do, instead of saving shape images on the disk.

[thegrindre]

Sure, doesn't matter to me. Convoi uses bitmat images and that takes up hard drive space so another way is fine IMO.

http://www.elvastower.com/forums/public/style_emoticons/default/oldstry.gif

[Eldorado.Railroad]

Trying to be upbeat about this is going to be hard, but I will try!

There have been others who have created consist editors, and sadly either they are payware with a terms of service that I do not want, or the creators have added to the list of abandonware without source code.

At least OpenRails is supposed to outlive us all. The code is still there to compile, if the tools to compile them are still around.

Some ideas have been centered around databases of sorts, but the creation of those databases and how they "look" has always been a bit of a conundrum. We are going to have to settle on some kind of key, yes that kind a key, the database kind, maybe even multiple keys. When your collection of wagons and engines goes well past 1000 finding things is a problem. There are numerous .eng and .wag files out there from various sources, payware or otherwise, that all have "their own ways" of doing things. A user ends up having to edit every .eng/.wag file to conform to some kind of "standard" so that you can find things. This is a huge PITA.

The consist editor, MUST accommodate long names, explicit ones. The consist editor MUST allow for various font sizes too, one size cannot fit all. I speak from experience here. The level of frustration and compromise on having to re-edit .eng/.wag every time you realize that you cannot use more than X characters is a major PITA.

In a very short list and not conclusive, you will have to deal with dead engines, loaded and unloaded wagons, etc. All of these need to be clear to the user, somehow. I agree with Goku, a selected piece of rolling stock should be immediately (well within reason) visible. As a sidebar, for the longest time, I have wished for OpenRails to have its own shapeviewer, that understands, non square textures, DDS textures, etc. Other things like random consist creation are also very handy to have.

The proof will be in the pudding, I hope I can signal the shortcomings in the beta of this idea, once it is released.

Thank you for undertaking this challenge, along with a good activity editor, it is crucial.

[Genma Saotome]

Many interesting points made in the previous post....

Eldorado.Railroad, on 09 April 2017 - 08:15 AM, said:

Some ideas have been centered around databases of sorts, but the creation of those databases and how they "look" has always been a bit of a conundrum. We are going to have to settle on some kind of key, yes that kind a key, the database kind, maybe even multiple keys. When your collection of wagons and engines goes well past 1000 finding things is a problem. There are numerous .eng and .wag files out there from various sources, payware or otherwise, that all have "their own ways" of doing things. A user ends up having to edit every .eng/.wag file to conform to some kind of "standard" so that you can find things. This is a huge PITA.

Years ago I argued on behalf of using a dbms to solve all of the unmanaged relationships between track and intereactives but it was far too early in the project for anyone to care about. Perhaps it'll return as a point of serious conversation.

The big problem here is not database design or the applications used to put in or take out information, it's what happens when the end user does something (or his PC does something) and the integrity of either the database or of rows in the database is broken. I don't know of any good way to help him out of the jam he will be in.

But for those willing to take the risk -- probably a rather small risk -- using a database as a repository from which either flat files are produced or is accessed by the OR code, a whole lot of issues would be successfully addressed, from proper referential integrity for interactives to 3rd normal form for creator-attributed versions and their revisions of content like .engs and .wags.

Eldorado.Railroad, on 09 April 2017 - 08:15 AM, said:

The consist editor, MUST accommodate long names, explicit ones. The consist editor MUST allow for various font sizes too, one size cannot fit all. I speak from experience here. The level of frustration and compromise on having to re-edit .eng/.wag every time you realize that you cannot use more than X characters is a major PITA.

Which argues for display windows that can be expanded or shrunk as needed.

The fundamental issue here is absent complete management within a custom application, one that presents you with all the metadata you need (thereby allowing the file name to be both hidden from view and probably just a serial number), we're going to put a lot of that metadata into the filenames: Road initials, unit number, type of unit, overall appearance of unit, optional but important features (like a white extra flag), and date range within which the model, in this appearance, was used. Or as they say, a real mouthful: "WP F7A #913 Silver small letters extra 1950-54"

Eldorado.Railroad, on 09 April 2017 - 08:15 AM, said:

In a very short list and not conclusive, you will have to deal with dead engines, loaded and unloaded wagons, etc. All of these need to be clear to the user, somehow.

IMO there is a need for some conceptual re-thinking: Take the mesh file and call it a class from which a variety of subtyped objects can spring, each of which get defined as individual game objects ahnd you solve lots of problems. For example, take a mesh for a particular U.S. diesel locomotive: A EMD SD-40. One file. Now add a subtyped variant: the skins it will display (yes, I know those are defined in the mesh but here I'm speaking of what the OR software will actually display. One mesh, many sets of skins, varying by railroad, railroad paint scheme, unit number, etc. etc. Maybe hundreds of variations. Yet there are a whole lot of data parameters specific to the mesh rather than any one variant, in fact, the majority of data parameters are probably true for the mesh. One mesh, one set of data parameters, many variants each with their own list of skins. What the game needs is a copy of all of that for each displayed loc0omotive. What the user should have on his PC is one mesh file, one set of data parameters (there may be multiple files here because a lot of the data is actually common to a lot of different locomotives -- think couplers), and a big bunch of tiny files listing the skins... and maybe a lot of metadata too. Reskinners then distribute only the last kind of file. End users would no longer have to put all of those reskinner files in separate directories -- they could have a single folder labeled UP "SD-40" and have 100 reskinning file sin there instead of having 100 folders labeled UP SD-40 6001, UP SD-40 6002....

IOW .eng and .wag files as we know them would be obsolete and the number of folders in \trainsets would be reduced considerably. The original mesh file and all of those data parameters could be read whether the game object was going to be a UP skinned SD-40 or a Santa Fe SD-40, or any other railroads SD-40. Who knows, maybe they get stored in a directory called Mesh Files and the convention is to mark everything their with the creators name -- "Barry_Munro_s_SD-40". If we're lucky, that'll be a folder name and it will contain an assembly file to points to as many individual mesh files as are needed to make up a whole locomotive. Meaning physical variants are dealt with here. What this really doess is it make the physical nature of the model wholly independent of its skinning and assigns most of the data parameters to the physical model. One model folder set off tot he side, many reskinning files placed into \trainsets, each of which is so tiny who cares about the cost of doing a copy/paste everywhere!

Eldorado.Railroad, on 09 April 2017 - 08:15 AM, said:

As a sidebar, for the longest time, I have wished for OpenRails to have its own shapeviewer, that understands, non square textures, DDS textures, etc. Other things like random consist creation are also very handy to have.

IMO Shapeviewer needed to be replaced at least 8-10 years ago. The graphics are bad, the displayed area far too small, and having one, permanent point for rotation is no good at all for anything larger than a small locomotive or car.

The only really insightful thing Shapeviewer does in the hierarchy window -- it shows the draw counts.