Elvas Tower: OR consist editor - Elvas Tower

Jump to content

Posting Rules

All new threads will be started by members of the Open Rails team, Staff, and/or Admins. Existing threads started in other forums may get moved here when it makes sense to do so.

Once a thread is started any member may post replies to it.
  • 5 Pages +
  • 1
  • 2
  • 3
  • Last »
  • You cannot start a new topic
  • You cannot reply to this topic

OR consist editor Rate Topic: -----

#1 User is offline   James Ross 

  • Open Rails Developer
  • Group: Status: Elite Member
  • Posts: 5,488
  • Joined: 30-June 10
  • Gender:Not Telling
  • Simulator:Open Rails
  • Country:

Posted 02 April 2017 - 01:22 PM

Quote

Open Rails consist editor for new file formats (JSON)

  • Load/save OR consists
  • Import MSTS consists
  • Select from MSTS rolling stock

Blueprint: https://blueprints.l.../consist-editor
Roadmap: https://trello.com/c...-consist-editor

I have not yet written a full specification for this, but wanted to kick-off the discussion anyway to see what people thought early, as this would be our first official Open Rails editor.

The goal here is to have a simple editor, which can be improved upon later, that provides the very basic function of creating a train consist from an available collection of locomotives and wagons. This would not be a complex UI, but would involve a minimum of the following specific features:

  • Design and implementation of an OR-specific consist format - which can at minimum specify an ordered list of locomotives and wagons, and flip individual cars.
  • Conversion of MSTS consist files into OR consist files (not with 100% fidelity).
  • Identification/reference system for linking to MSTS rolling stock from OR file formats.

I would also like to explore the ideas in the "Content Manager" contributed project, where there is an abstraction layer between the core game's "load car X from consist Y"-style function and the physical files on disk. I believe this abstraction would be useful for a number of reasons:

  • Each downloaded content pack can be kept separate (rather than all installed into a single mini-route, for example) while still being able to easily have one pack reference another.
  • Allowing the mixture of OR and MSTS formats more freely, as cross-references can be resolved in ways other than simply looking at an exact file path.
  • Multiplayer content sharing, wherein players could receive another player's content over the network seamlessly.

If we do include the ideas from the "Content Manager", this would also be the ideal time to implement a new directory structure that is less about a single game's install and more about the content, e.g. moving all shapes, textures, etc., outside of routes into a top-level folder, and such like. This would make the cross-referencing clearer and more obvious: each content pack simply has a list of shapes, textures, locomotives, etc., that it contains.

Questions: Do the above points seem appropriate to start work on now? Is there anything obvious missing above that we should consider before embarking on this?

#2 User is offline   cr-stagg 

  • Foreman Of Engines
  • Group: Status: Contributing Member
  • Posts: 909
  • Joined: 16-May 05
  • Gender:Male
  • Simulator:OR
  • Country:

Posted 02 April 2017 - 01:58 PM

Concerning your proposal

Quote

this would also be the ideal time to implement a new directory structure that is less about a single game's install and more about the content, e.g. moving all shapes, textures, etc., outside of routes into a top-level folder


How would you know that Station.s from Route A is exactly the same as Station.s from Route B? You could end up with a Route in California that has a Station with a signboard that says New York instead of San Diego. I think it would be better to keep Route Shapes within the Route folder structure.

#3 User is online   Genma Saotome 

  • Owner Emeritus and Admin
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 15,307
  • Joined: 11-January 04
  • Gender:Male
  • Location:United States
  • Simulator:Open Rails
  • Country:

Posted 02 April 2017 - 04:06 PM

View PostJames Ross, on 02 April 2017 - 01:22 PM, said:


If we do include the ideas from the "Content Manager", this would also be the ideal time to implement a new directory structure that is less about a single game's install and more about the content, e.g. moving all shapes, textures, etc., outside of routes into a top-level folder, and such like. This would make the cross-referencing clearer and more obvious: each content pack simply has a list of shapes, textures, locomotives, etc., that it contains.



It's well understood that how KUJU chose to handle multiple uses of a single file (or folder) across multiple route directories has it's problems. An abstraction layer should go a long way to addressing those issues.

The concern I have is WRT abandoning the "traditional" directory structure and (eventually) moving everything (presumably into into zip files) under a single content directory. Speaking as a content developer there are lots and lots of situations where you want multiple directories containing the same list of files where the content of the like-named files are not the same. For example, one directory holding the what was released last month (now unchanging) and another that remains the development directory tree. For the most part, the same files using the same names but where the first directory had Rev 6 bits the second has Rev 7 -- and depending on the task you need to look today at Rev 6 and tomorrow at Rev 7.

To bring the example into clear focus, today we could be taking about a .con, .act., .srv, or .trf file(s). Names are all the same, lines are all different. Tomorrow we could be talking about Rev 7 of an .ace file, a world file, and 30 brand new .s and .sd files.

My point here is to show that a content developer works in a different environment than both a passive end user and a content re-user much in the sme way a software developer's environment and needs are quite different from his end users.

Am I correct in guessing your answer will be two different packages (zip files), one for Rev 6 and the other for rev 7?

#4 User is online   Genma Saotome 

  • Owner Emeritus and Admin
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 15,307
  • Joined: 11-January 04
  • Gender:Male
  • Location:United States
  • Simulator:Open Rails
  • Country:

Posted 02 April 2017 - 04:47 PM

Another example of the complexity here for your consideration:

Route A is installed in 2 different directories. What makes them different is the era which is shown by which rolling stock is displayed -- steam era vs. diesel era.

One solution is simply to change the rows of the .con file to select the correct locomotives and rolling stock. Because loose consists are not managed as consists (IMO a problem) this means the .act file needs to be edited as well. The .com and .act file names remain the same while having different rows because they are in two different directory trees.

An alternative to the above would be to change the file names of the service and traffic files so they refer to brand new consist files, and changing the rows of the .act file to reflect that. This too works.

I prefer the first method but the second works equally well making the choice pretty subjective.

In the proposed future, how would this situation be addressed?




postscript: Upon further reflection it occurs to me to suggest the root issue is this: Global files work perfectly well until you have a situation where things no longer should be global but instead handled on a case by case basis. Then global stuff gets in the way.

How does the proposal address this fact?

#5 User is offline   James Ross 

  • Open Rails Developer
  • Group: Status: Elite Member
  • Posts: 5,488
  • Joined: 30-June 10
  • Gender:Not Telling
  • Simulator:Open Rails
  • Country:

Posted 03 April 2017 - 11:54 AM

View Postcr-stagg, on 02 April 2017 - 01:58 PM, said:

How would you know that Station.s from Route A is exactly the same as Station.s from Route B? You could end up with a Route in California that has a Station with a signboard that says New York instead of San Diego. I think it would be better to keep Route Shapes within the Route folder structure.

There are two approaches I can see to this:

  • Content created in this new arrangement will have more distinctive names, perhaps including the author's initials, world region, or such.
  • Content will be referenced via a pair of identifiers: package ID + file name.

(As mentioned at the end, also) For name-only referencing, I would still want to look for content locally in the same package first.

View PostGenma Saotome, on 02 April 2017 - 04:06 PM, said:

My point here is to show that a content developer works in a different environment than both a passive end user and a content re-user much in the sme way a software developer's environment and needs are quite different from his end users.

Am I correct in guessing your answer will be two different packages (zip files), one for Rev 6 and the other for rev 7?

Yes, but perhaps not quite in the way you were expecting.

Firstly, let's set aside any zip files - they would be a convenience for distribution of finished content only. Development can and will work with files directly in the file system.

As for revisions and alternate versions, there is a potentially interesting way of doing this with the 2nd option above (i.e. using IDs + names to reference content): you can have packages that override other packages. In other words, you can have 7 different packages with the same ID but different version numbers (or something else to identify which overrides which) but the game will search through them from newest to oldest when locating content (we can do this more efficiently than this sounds) so you only have new/updated content in each package. The burden of creating a new "revision" of the package could be handled as a single button in the tools, as could created a "merged" package for release.

That's one idea, anyway.

I mean, at the basic level we could use it the same as mini-routes work now: the current package is the default search location anyway (even when using ID + name we can have no ID imply the current package) so you can just create a new copy of the package and the only thing that might need any tweaking is either how we cope with duplicates, or you'd change the ID of the new copy (but only need to do it in one place).

View PostGenma Saotome, on 02 April 2017 - 04:47 PM, said:

Route A is installed in 2 different directories. What makes them different is the era which is shown by which rolling stock is displayed -- steam era vs. diesel era.

One solution is simply to change the rows of the .con file to select the correct locomotives and rolling stock. Because loose consists are not managed as consists (IMO a problem) this means the .act file needs to be edited as well. The .com and .act file names remain the same while having different rows because they are in two different directory trees.

An alternative to the above would be to change the file names of the service and traffic files so they refer to brand new consist files, and changing the rows of the .act file to reflect that. This too works.

I prefer the first method but the second works equally well making the choice pretty subjective.

In the proposed future, how would this situation be addressed?

One of the things you could do out-of-the-box (as it were) would be to use any of your rolling stock on any of your routes. That would at least let you use steam vs diesel as your choice for exploring.

For the activities (and timetables) side of things, you would probably have to have two copies of whichever files reference the rollingstock, each referencing their respective version, but any shared files, including the route itself, would likely still exist only once.

View PostGenma Saotome, on 02 April 2017 - 04:47 PM, said:

postscript: Upon further reflection it occurs to me to suggest the root issue is this: Global files work perfectly well until you have a situation where things no longer should be global but instead handled on a case by case basis. Then global stuff gets in the way.

How does the proposal address this fact?

This depends on which route we take with the content references at the top of this post.

If we use just names, the idea would be to look locally before looking globally, and when looking globally always look in a defined order. This would mean that if a file in a content pack references a particular shape (for example), and that shape exists in that same content pack, it would always use that one. And if it doesn't, you'd be given some level of control over how it found it.

If we use pack IDs + names, I think this is pretty much a non-issue as you will always be referring to a very specific item across the whole content ecosystem. You might still be able to override one pack with another, but it would still be very deterministic.

#6 User is offline   thegrindre 

  • Member, Board of Directors
  • Group: Status: Elite Member
  • Posts: 8,349
  • Joined: 10-September 08
  • Gender:Male
  • Location:Now in central Arkansas
  • Simulator:MSTS & Trainz '04 & Open Rails
  • Country:

Posted 03 April 2017 - 12:41 PM

There is one thing I'd be interested in and that would be a small graphic profile from the shape file for display.
I'm not so fond of Convoi's method (Takes up too much room on the hard drive) but at least something to look at, anyway.

http://www.elvastower.com/forums/public/style_emoticons/default/oldstry.gif

#7 User is online   Genma Saotome 

  • Owner Emeritus and Admin
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 15,307
  • Joined: 11-January 04
  • Gender:Male
  • Location:United States
  • Simulator:Open Rails
  • Country:

Posted 03 April 2017 - 03:48 PM

James, I remain skeptical but at this point in time I think it best to reserve further comment until such time there is more available to comment on.

May I suggest to you that a change of this magnitude had better be obviously way better than how things are done today because the real risk here is it won't be, that all you wind up with is replacing a bunch on known problems for an equal number of unknown problems and in doing so you've wasted not only your own time and skill but that of everyone else as well. So take your time and work the problem really well with as much feedback as you can get.



For the record, I'm reasonably happy with a combination of symbolic links and ordinary copy/paste to create a number of associative directories between two reference directory trees -- one of routes and the other of rolling stock. This lets me logically present any route several times, each in a directory tree specific to an era, at no cost in disk space. I do the same with other big "global" directories, like \global\shapes and \sound as well as "largish" folders inside of \trainset. The Beyond Compare software, which I highly recommend, does an excellent job of keeping the copy/paste stuff in sync. The only disappointment I have in this setup is with regard to the activity files that have loose consists, something that could quickly be solved by allowing a .con file to replace each set of rows of individual car.

The value in this solution is it did not require the OR team to invest their scarce -- and therefore valuable -- time and skill into building something I could do on my own. Please take this as food for thought.

#8 User is online   Genma Saotome 

  • Owner Emeritus and Admin
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 15,307
  • Joined: 11-January 04
  • Gender:Male
  • Location:United States
  • Simulator:Open Rails
  • Country:

Posted 03 April 2017 - 03:57 PM

On second thought, there is something I want to comment on and that is the idea of appending a personal identifier on some files to help distinguish them from otherwise identically named files from somebody else. You do touch on a real problem there, one not easily solved. Initials, or initials plus rev id (or date) is worthwhile looking into -- I suggested the same idea for Camera Sets. The loophole that spoils the idea is this: how do you prevent somebody from changing something inside the package and then forwarding it on to someone else without changing the package id?

#9 User is offline   Mike B 

  • Superintendant
  • Group: Status: Elite Member
  • Posts: 1,085
  • Joined: 18-January 13
  • Gender:Not Telling
  • Location:Pacific Time
  • Simulator:Mostly ORTS these days
  • Country:

Posted 03 April 2017 - 07:55 PM

Way back in ancient days, library sites for FS4 (like the FS section of Compuserve) had voluntary (but generally used) file naming conventions for scenery, especially. Those were in 8.3 filename days, of course, so things had to be a tad terse. IIRC it commonly went along the lines of VORSCNii where VOR was the central VOR, airport ID, or the like. For very large-area package, it might be something more descriptive, like NORCAL. ii was the creator's initials. Actually worked out quite well, and most people followed the script. So yes, it can be done.

As for the loophole - you can't fix that unless you have a way to guarantee that files are automatically versioned when changed, and checkouts are logged. IOW some kind of Subversion or Github setup. Probably way too much trouble. However, having a Library where the "master" version is always available to compare with what you got from $someplace_else is a Good Thing. I suppose you could post hashes for the library files so people could check whether they got the Real Thing. EDIT: sorry; thought you were talking about something else.

#10 User is offline   James Ross 

  • Open Rails Developer
  • Group: Status: Elite Member
  • Posts: 5,488
  • Joined: 30-June 10
  • Gender:Not Telling
  • Simulator:Open Rails
  • Country:

Posted 04 April 2017 - 02:18 PM

View Postthegrindre, on 03 April 2017 - 12:41 PM, said:

There is one thing I'd be interested in and that would be a small graphic profile from the shape file for display.
I'm not so fond of Convoi's method (Takes up too much room on the hard drive) but at least something to look at, anyway.

I'd start with a text-only version for simplicity first, but that is probably the first thing we'd add after it was up and running. :)

View PostGenma Saotome, on 03 April 2017 - 03:48 PM, said:

James, I remain skeptical but at this point in time I think it best to reserve further comment until such time there is more available to comment on.

Absolutely; we can start off with the new consist file format being allowed/used inside the MSTS file structure, without any content manager, so that we can focus on one thing at a time.

View PostGenma Saotome, on 03 April 2017 - 03:48 PM, said:

May I suggest to you that a change of this magnitude had better be obviously way better than how things are done today because the real risk here is it won't be, that all you wind up with is replacing a bunch on known problems for an equal number of unknown problems and in doing so you've wasted not only your own time and skill but that of everyone else as well. So take your time and work the problem really well with as much feedback as you can get.

The more we discuss it, the more I feel we should separate it from the consist editor's initial version - while we can think about better arrangements for files, referencing, etc., there's actually no need to tie it together. We can work on the JSON format for consists in MSTS directories, and subsequently add the new structure (assuming it meets said criteria of being way better).

This is by no means exhaustive, but some things we might be able to do if a new directory structure/package format was created:

  • Share all content items - including textures and shapes - between all places (e.g. avoiding the need for "installme.bat")
  • Install/uninstall content packs without overwriting / deleting other pack's files
  • Automatically identify dependencies on other required content packs (maybe even with web page links)


View PostGenma Saotome, on 03 April 2017 - 03:48 PM, said:

For the record, I'm reasonably happy with a combination of symbolic links and ordinary copy/paste to create a number of associative directories between two reference directory trees -- one of routes and the other of rolling stock. This lets me logically present any route several times, each in a directory tree specific to an era, at no cost in disk space. I do the same with other big "global" directories, like \global\shapes and \sound as well as "largish" folders inside of \trainset. The Beyond Compare software, which I highly recommend, does an excellent job of keeping the copy/paste stuff in sync. The only disappointment I have in this setup is with regard to the activity files that have loose consists, something that could quickly be solved by allowing a .con file to replace each set of rows of individual car.

These types of arrangement are a motivating factor for me; whilst obviously you and I can do them, and software like Beyond Compare makes it much easier than it otherwise would be, if we can bring some of the same ability to all users in a more "native" way, I'd be happy. This could involve us producing a tool that manages the symbolic links for you with a train simulator-focused UI on top, although I'd rather not as I take your point below on scares resources. But for new content, the less we need to do this kind of "patching" of files to get where we want, the better IMHO.

View PostGenma Saotome, on 03 April 2017 - 03:48 PM, said:

The value in this solution is it did not require the OR team to invest their scarce -- and therefore valuable -- time and skill into building something I could do on my own. Please take this as food for thought.

Perhaps you could pen a section for the manual on how you set this up?

View PostGenma Saotome, on 03 April 2017 - 03:57 PM, said:

On second thought, there is something I want to comment on and that is the idea of appending a personal identifier on some files to help distinguish them from otherwise identically named files from somebody else. You do touch on a real problem there, one not easily solved. Initials, or initials plus rev id (or date) is worthwhile looking into -- I suggested the same idea for Camera Sets. The loophole that spoils the idea is this: how do you prevent somebody from changing something inside the package and then forwarding it on to someone else without changing the package id?

Although the JavaScript package ecosystem is not a pretty place for many reasons, their use of a "package.json" to identify the name, version, description, authors, and some other key metadata for each package is a pretty nice idea. It is something I would like to do with the content manager packages, but how much information to include is still very much unknown.

On the point of package IDs specifically, there's a number of mitigations that can be employed:

  • Package hosting sites put in a small amount of effort to prevent duplicate package IDs on their site
  • Create a meta-package site that doesn't host, but does track metadata - which the hosting sites can check against before allowing uploads
  • Open Rails itself ignores and/or shows an error on any duplicate IDs in the installed packages
  • Distributing packages in zip files (and having them run from those zip files) may make it clearer that modifications aren't the norm (i.e. modifying content from other people requires an extra step or two, but doesn't prevent it)

But at the end of the day, it's really a social problem; technical solutions get help spot mistakes, but it would be hard to stop deliberate actions.

  • 5 Pages +
  • 1
  • 2
  • 3
  • Last »
  • You cannot start a new topic
  • You cannot reply to this topic

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users