[YoRyan]

And C# for the script files themselves... okay, that's being pedantic. :)

[YoRyan]

For the record, I intend to follow OR policy, which means using JSON for new formats going forward. I didn't decide that, but I've already explained why I think it's a good policy. Should a performance problem arise, I will attempt to optimize the parser, and should that still be insufficient, I will investigate off-the-shelf binary formats.

[wacampbell]

YoRyan, on 25 April 2021 - 12:32 PM, said:

And C# for the script files themselves... okay, that's being pedantic. :)

Yah, I had really hoped c# would become our universal language for signal scripts, engine scripts and all. These would all be first class components using the full power of .net for performance. But alas too much resistance.

[YoRyan]

C# is a heavy language that basically requires you to work in Visual Studio. Maybe a more scripting-oriented language like Lua would have been a friendlier choice for content creators? That's also what they use for Train Simulator/RailWorks. (But it's still not what MSTS used, and therefore it would have still been difficult to convince people to switch.)

[wacampbell]

YoRyan, on 25 April 2021 - 12:56 PM, said:

C# is a heavy language that basically requires you to work in Visual Studio.

Not really. You can edit a .cs in a text editor ( as you would Lua ) and Open Rails would open , jit compile and execute. All this comes free with .net. An end user would see little difference between scripting in c# vs msts scripts other than brackets and semi’s in different places.

[James Ross]

jonas, on 15 April 2021 - 12:52 PM, said:

• It may be that there are good reasons why the * .trk file in the OpenRails folder needs a blank line at the beginning and begins with "include (" ../___. Trk ")".
• It may be that there are good reasons why there can be another Openrails folder in the MSTS world folder, which contains additional world files with extensions, but which do not need a blank line at the beginning and also not with "include ("..\thisAndThat.w"). Instead, this world extension file begins just like any ordinary world file from MSTS.
• It may be that there are Open Rails policies that provide for MSTS extensions in Openrails folders and OR own files are to be saved in the root folder of the route.

All SIMIS files start with a header; for binary/compressed files, this clearly identifies how to load the file and is used by Open Rails for shapes, terrain, and world files only. For all other SIMIS files, Open Rails only supports the Unicode text format (STF), where the header is one line of text, which is ignored by OR. This is the "blank line" in these files.

The rule for the "OpenRails" subfolder inside MSTS content is that Open Rails loads "OpenRails\foo.dat" instead of "foo.dat", if it exists, but there is no difference in content format or features supported. The arrangement is provided so that content can be MSTS compatible while also taking advantage of Open Rails features in a way that is not otherwise compatible with MSTS, by having two different files for one thing.

The exception is world files, which cannot use "include" because they are not STF files (they can be, but not always), so we do some extra work to merge the OpenRails and base versions as if you'd used "include".

jonas, on 15 April 2021 - 12:52 PM, said:

• It may be that there are good reasons why the carspawn.dat in the Openrails folder contains a list of CarSpawnItems, but is not called, for example, CarspawnList.dat, because its content differs fundamentally from the carspawn.dat from MSTS and can not just count as an extension. But on the other hand, if it were actually called CarspawnList.dat, it would finally be clear that it represents an new OR format and, according to OR policy, would have to be in the root folder of the route.

In hindsight, this was probably a mistake. It should have been either:

• Extension to the existing format (such that the same data could also be used in the base "carspawn.dat", if you didn't care about MSTS compatibility)
  
• A separate filename and/or extension (whose presence would override the base "carspawn.dat")

jonas, on 15 April 2021 - 12:52 PM, said:

• It may be that there are good reasons why the clocks.dat is in the Openrails folder, but the "animated.clocks-or" file generated from it is then in the root folder of the route.
• It may be that there are good reasons why the turntable.dat is seen differently than the clocks.dat, but I haven't hear any reason why so far (@Chris, you didn't answer me about this in this post :-)).

I personally see no reason for "clocks.dat" or "turntables.dat" to be in the OpenRails folder, but we did the same thing for timetable files (which are in "Activities\OpenRails").

One thing you absolutely have to bear in mind is that Open Rails has been around many years now; it has grown organically and has historically not had a particularly tight control over how new features were implemented, including where new files go. Some of them will be accidents, most of them will be what the author thought was best at the time, but few will have been placed according to a grand plan (because there really isn't one).

jonas, on 15 April 2021 - 12:52 PM, said:

• It may be that there are good reasons why a user of a route should be forced to switch to a Json file. But if it is obviously possible with an Open Rails converter to convert the clocks.dat at runtime into the Json file animated.clocks-or, then this can always "virtual" happen unnoticed by the user as long as a clocks.dat is already available. After all, we are assured that clocks.dat will always be readable in the future. I think politics is being made here to "educate" the user to use Json formated files, so to speak.

Csantucci, on 19 April 2021 - 11:58 PM, said:

IMHO keeping the clocks.dat file should have been the solution here.

We have not released any official "clocks.dat" support. Officially, we have a completely free choice of how to implement this brand new feature, for which JSON is preferred. However, we have a problem whereby there may exist third party content creators that have created unofficial files that we are being asked to support.

There are no good solutions here.

• We support JSON format only - any existing content must be updated
  
• We support STF format only - void our own policy
  
• We support both formats - extra maintenance burden forever more

I agree with Rob that this file is a great example for implementing in JSON (option 1), not least because we should gain more experience with it before embarking on larger works. The failure mode is basically non-existant, and the fix is very simple for the content creator.

wacampbell, on 18 April 2021 - 09:48 AM, said:

One thing of note were the speed tests that I ran. I converted my L&PS .t files to .json and modified openrails to read them. Sadly reading the .json files was ten times slower than the msts format. I think there are some fast JSON readers, ones that don't use code reflection, but some effort would have to be spent to choose one that's fast.

It is obviously an unfair comparison to use JSON with reflection vs STF with custom tokenisation parsing.

My motivation for JSON is entirely the standardised nature of it - everyone knows exactly how it works, so we don't have to spend years bug fixing the parsing and 3rd parties can much more easily get involved by using their standard library.

We still don't know how to parse STF properly. Only last July were we trying to fix bugs in StfReader. It might look like a nice simple format, but without a specification we have spent years guessing how to parse it correctly and we're still not 100% there.

I did a lot of my own reverse engineering and tool-writing before joining Open Rails and I am certain we still have STF parsing bugs.

So while JSON is a new syntax that people in this community may end up learning, you can still happily edit with Notepad if you so wish:

roeter, on 20 April 2021 - 01:54 PM, said:

If files are readable (i.e. not binary), there will always be users who will edit the file directly, whether there is an editor or not.
Users are now editing .con files, .w files and even the .tdb if necessary, even though these are (or can) be created through editors.
There are always two groups of people - those with no or limited knowledge of the data structure who therefor are happy to use the editor. And those who have that knowledge and just find editors cumbersome or too restricted, especially when things go wrong. That's how it always has been, that's how it always will be.

wacampbell, on 25 April 2021 - 10:13 AM, said:

This comes down to end user convenience vs programmer convenience. I don't think there are any content developers or end users who prefer JSON. They would prefer to have only one familiar format used cohesively across the project.

I guess that depends on whether you believe in the vision of the future where we have editors for the new Open Rails file structure.

In that future, almost all files will be JSON (exceptions may be models and textures, which will be BSON or another standardised binary format) and users will have GUI editors that allow them to almost never see the contents of those files. And when they do need to go into the files, it's a well-defined format that will have support from any popular text editor (potentially including JSON Schema) making any manual editing easier and less error prone.

I do not care about whether I can serialise/deserialise JSON using reflection or any other "developer magic"; I care about the file format being properly and completely defined, which JSON is but STF is not.

wacampbell, on 25 April 2021 - 10:13 AM, said:

Not to mention the technical limitations of JSON such as:
- more verbose
- more picky about syntax
- can't handle line spanning strings ( such as route descriptions etc )
- lack of a fast loading binary version.

The only one of these that might matter to us is the last one, which, if we need it, binary JSON formats (BSON) already exists and are standardised.

[jonas]

Thank you James for this really comprehensive and detailed post!
I feel like I'm somewhat up to date now and I see even better what the current problems and solutions are with the different formats. For me it was a good school on this topic, I hope for others as well and not just a repetition of what was already standard knowledge for many.

And thanks also to all others who have written here on the subject so far! I'm looking forward to more posts here in case there come more.

Best regards
Jonas

[YoRyan]

James Ross, on 26 April 2021 - 12:31 PM, said:

My motivation for JSON is entirely the standardised nature of it - everyone knows exactly how it works, so we don't have to spend years bug fixing the parsing and 3rd parties can much more easily get involved by using their standard library.

We still don't know how to parse STF properly. Only last July were we trying to fix bugs in StfReader. It might look like a nice simple format, but without a specification we have spent years guessing how to parse it correctly and we're still not 100% there.

I did a lot of my own reverse engineering and tool-writing before joining Open Rails and I am certain we still have STF parsing bugs.

...

I do not care about whether I can serialise/deserialise JSON using reflection or any other "developer magic"; I care about the file format being properly and completely defined, which JSON is but STF is not.

I'd just like to emphasize that this is the single biggest strike against expanding the use of STF. Drops microphone.

[Genma Saotome]

James Ross, on 26 April 2021 - 12:31 PM, said:

We still don't know how to parse STF properly. Only last July were we trying to fix bugs in StfReader. It might look like a nice simple format, but without a specification we have spent years guessing how to parse it correctly and we're still not 100% there.

I did a lot of my own reverse engineering and tool-writing before joining Open Rails and I am certain we still have STF parsing bugs.

How much of that is actually the fault of sloppy work by KUJU that is kept around as-is simply to keep OR files working in train.exe?

Something to consider: Drop the need to run those files thru train.exe and the effort to fix the slop might be less than what will be required to fully implement .json

[cjakeman]

jonas, on 18 April 2021 - 07:03 AM, said:

Chris, I see that the whole thing is really not an easy story and I know the content of some of the links you mention here.

But we have now landed here, where two (sorry) extortionate dialogues pop up before the activity starts.

These dialogs have been the cause of some disquiet, popping up as they do on pressing "Start" if a user has a clocks.dat but not yet an animated.clocks-or

I have therefore removed them from the submission for the official Open Rails and conversion is no longer prompted for.

Users who want to convert from clocks.dat to animated.clocks-or so that their clocks will be animated in the official version are advised to use the Contrib.DataConverter.exe which is supplied with OpenRails.

For example:

Contrib.DataConverter.exe /input "C:\OR\MyRoute\ROUTES\OpenRails\clocks.dat" /output "C:\OR\MyRoute\ROUTES\animated.clocks-or"