[YoRyan]

And C# for the script files themselves... okay, that's being pedantic. :)

[YoRyan]

For the record, I intend to follow OR policy, which means using JSON for new formats going forward. I didn't decide that, but I've already explained why I think it's a good policy. Should a performance problem arise, I will attempt to optimize the parser, and should that still be insufficient, I will investigate off-the-shelf binary formats.

[wacampbell]

YoRyan, on 25 April 2021 - 12:32 PM, said:

And C# for the script files themselves... okay, that's being pedantic. :)

Yah, I had really hoped c# would become our universal language for signal scripts, engine scripts and all. These would all be first class components using the full power of .net for performance. But alas too much resistance.

[YoRyan]

C# is a heavy language that basically requires you to work in Visual Studio. Maybe a more scripting-oriented language like Lua would have been a friendlier choice for content creators? That's also what they use for Train Simulator/RailWorks. (But it's still not what MSTS used, and therefore it would have still been difficult to convince people to switch.)

[wacampbell]

YoRyan, on 25 April 2021 - 12:56 PM, said:

C# is a heavy language that basically requires you to work in Visual Studio.

Not really. You can edit a .cs in a text editor ( as you would Lua ) and Open Rails would open , jit compile and execute. All this comes free with .net. An end user would see little difference between scripting in c# vs msts scripts other than brackets and semi’s in different places.

[James Ross]

jonas, on 15 April 2021 - 12:52 PM, said:

• It may be that there are good reasons why the * .trk file in the OpenRails folder needs a blank line at the beginning and begins with "include (" ../___. Trk ")".
• It may be that there are good reasons why there can be another Openrails folder in the MSTS world folder, which contains additional world files with extensions, but which do not need a blank line at the beginning and also not with "include ("..\thisAndThat.w"). Instead, this world extension file begins just like any ordinary world file from MSTS.
• It may be that there are Open Rails policies that provide for MSTS extensions in Openrails folders and OR own files are to be saved in the root folder of the route.

All SIMIS files start with a header; for binary/compressed files, this clearly identifies how to load the file and is used by Open Rails for shapes, terrain, and world files only. For all other SIMIS files, Open Rails only supports the Unicode text format (STF), where the header is one line of text, which is ignored by OR. This is the "blank line" in these files.

The rule for the "OpenRails" subfolder inside MSTS content is that Open Rails loads "OpenRails\foo.dat" instead of "foo.dat", if it exists, but there is no difference in content format or features supported. The arrangement is provided so that content can be MSTS compatible while also taking advantage of Open Rails features in a way that is not otherwise compatible with MSTS, by having two different files for one thing.

The exception is world files, which cannot use "include" because they are not STF files (they can be, but not always), so we do some extra work to merge the OpenRails and base versions as if you'd used "include".

jonas, on 15 April 2021 - 12:52 PM, said:

• It may be that there are good reasons why the carspawn.dat in the Openrails folder contains a list of CarSpawnItems, but is not called, for example, CarspawnList.dat, because its content differs fundamentally from the carspawn.dat from MSTS and can not just count as an extension. But on the other hand, if it were actually called CarspawnList.dat, it would finally be clear that it represents an new OR format and, according to OR policy, would have to be in the root folder of the route.

In hindsight, this was probably a mistake. It should have been either:

• Extension to the existing format (such that the same data could also be used in the base "carspawn.dat", if you didn't care about MSTS compatibility)
  
• A separate filename and/or extension (whose presence would override the base "carspawn.dat")

jonas, on 15 April 2021 - 12:52 PM, said:

• It may be that there are good reasons why the clocks.dat is in the Openrails folder, but the "animated.clocks-or" file generated from it is then in the root folder of the route.
• It may be that there are good reasons why the turntable.dat is seen differently than the clocks.dat, but I haven't hear any reason why so far (@Chris, you didn't answer me about this in this post :-)).

I personally see no reason for "clocks.dat" or "turntables.dat" to be in the OpenRails folder, but we did the same thing for timetable files (which are in "Activities\OpenRails").

One thing you absolutely have to bear in mind is that Open Rails has been around many years now; it has grown organically and has historically not had a particularly tight control over how new features were implemented, including where new files go. Some of them will be accidents, most of them will be what the author thought was best at the time, but few will have been placed according to a grand plan (because there really isn't one).

jonas, on 15 April 2021 - 12:52 PM, said:

• It may be that there are good reasons why a user of a route should be forced to switch to a Json file. But if it is obviously possible with an Open Rails converter to convert the clocks.dat at runtime into the Json file animated.clocks-or, then this can always "virtual" happen unnoticed by the user as long as a clocks.dat is already available. After all, we are assured that clocks.dat will always be readable in the future. I think politics is being made here to "educate" the user to use Json formated files, so to speak.

Csantucci, on 19 April 2021 - 11:58 PM, said:

IMHO keeping the clocks.dat file should have been the solution here.

We have not released any official "clocks.dat" support. Officially, we have a completely free choice of how to implement this brand new feature, for which JSON is preferred. However, we have a problem whereby there may exist third party content creators that have created unofficial files that we are being asked to support.

There are no good solutions here.

• We support JSON format only - any existing content must be updated
  
• We support STF format only - void our own policy
  
• We support both formats - extra maintenance burden forever more

I agree with Rob that this file is a great example for implementing in JSON (option 1), not least because we should gain more experience with it before embarking on larger works. The failure mode is basically non-existant, and the fix is very simple for the content creator.

wacampbell, on 18 April 2021 - 09:48 AM, said:

One thing of note were the speed tests that I ran. I converted my L&PS .t files to .json and modified openrails to read them. Sadly reading the .json files was ten times slower than the msts format. I think there are some fast JSON readers, ones that don't use code reflection, but some effort would have to be spent to choose one that's fast.

It is obviously an unfair comparison to use JSON with reflection vs STF with custom tokenisation parsing.

My motivation for JSON is entirely the standardised nature of it - everyone knows exactly how it works, so we don't have to spend years bug fixing the parsing and 3rd parties can much more easily get involved by using their standard library.

We still don't know how to parse STF properly. Only last July were we trying to fix bugs in StfReader. It might look like a nice simple format, but without a specification we have spent years guessing how to parse it correctly and we're still not 100% there.

I did a lot of my own reverse engineering and tool-writing before joining Open Rails and I am certain we still have STF parsing bugs.

So while JSON is a new syntax that people in this community may end up learning, you can still happily edit with Notepad if you so wish:

roeter, on 20 April 2021 - 01:54 PM, said:

If files are readable (i.e. not binary), there will always be users who will edit the file directly, whether there is an editor or not.
Users are now editing .con files, .w files and even the .tdb if necessary, even though these are (or can) be created through editors.
There are always two groups of people - those with no or limited knowledge of the data structure who therefor are happy to use the editor. And those who have that knowledge and just find editors cumbersome or too restricted, especially when things go wrong. That's how it always has been, that's how it always will be.

wacampbell, on 25 April 2021 - 10:13 AM, said:

This comes down to end user convenience vs programmer convenience. I don't think there are any content developers or end users who prefer JSON. They would prefer to have only one familiar format used cohesively across the project.

I guess that depends on whether you believe in the vision of the future where we have editors for the new Open Rails file structure.

In that future, almost all files will be JSON (exceptions may be models and textures, which will be BSON or another standardised binary format) and users will have GUI editors that allow them to almost never see the contents of those files. And when they do need to go into the files, it's a well-defined format that will have support from any popular text editor (potentially including JSON Schema) making any manual editing easier and less error prone.

I do not care about whether I can serialise/deserialise JSON using reflection or any other "developer magic"; I care about the file format being properly and completely defined, which JSON is but STF is not.

wacampbell, on 25 April 2021 - 10:13 AM, said:

Not to mention the technical limitations of JSON such as:
- more verbose
- more picky about syntax
- can't handle line spanning strings ( such as route descriptions etc )
- lack of a fast loading binary version.

The only one of these that might matter to us is the last one, which, if we need it, binary JSON formats (BSON) already exists and are standardised.

[jonas]

Thank you James for this really comprehensive and detailed post!
I feel like I'm somewhat up to date now and I see even better what the current problems and solutions are with the different formats. For me it was a good school on this topic, I hope for others as well and not just a repetition of what was already standard knowledge for many.

And thanks also to all others who have written here on the subject so far! I'm looking forward to more posts here in case there come more.

Best regards
Jonas

[YoRyan]

James Ross, on 26 April 2021 - 12:31 PM, said:

My motivation for JSON is entirely the standardised nature of it - everyone knows exactly how it works, so we don't have to spend years bug fixing the parsing and 3rd parties can much more easily get involved by using their standard library.

We still don't know how to parse STF properly. Only last July were we trying to fix bugs in StfReader. It might look like a nice simple format, but without a specification we have spent years guessing how to parse it correctly and we're still not 100% there.

I did a lot of my own reverse engineering and tool-writing before joining Open Rails and I am certain we still have STF parsing bugs.

...

I do not care about whether I can serialise/deserialise JSON using reflection or any other "developer magic"; I care about the file format being properly and completely defined, which JSON is but STF is not.

I'd just like to emphasize that this is the single biggest strike against expanding the use of STF. Drops microphone.

[Genma Saotome]

James Ross, on 26 April 2021 - 12:31 PM, said:

We still don't know how to parse STF properly. Only last July were we trying to fix bugs in StfReader. It might look like a nice simple format, but without a specification we have spent years guessing how to parse it correctly and we're still not 100% there.

I did a lot of my own reverse engineering and tool-writing before joining Open Rails and I am certain we still have STF parsing bugs.

How much of that is actually the fault of sloppy work by KUJU that is kept around as-is simply to keep OR files working in train.exe?

Something to consider: Drop the need to run those files thru train.exe and the effort to fix the slop might be less than what will be required to fully implement .json

[cjakeman]

jonas, on 18 April 2021 - 07:03 AM, said:

Chris, I see that the whole thing is really not an easy story and I know the content of some of the links you mention here.

But we have now landed here, where two (sorry) extortionate dialogues pop up before the activity starts.

These dialogs have been the cause of some disquiet, popping up as they do on pressing "Start" if a user has a clocks.dat but not yet an animated.clocks-or

I have therefore removed them from the submission for the official Open Rails and conversion is no longer prompted for.

Users who want to convert from clocks.dat to animated.clocks-or so that their clocks will be animated in the official version are advised to use the Contrib.DataConverter.exe which is supplied with OpenRails.

For example:

Contrib.DataConverter.exe /input "C:\OR\MyRoute\ROUTES\OpenRails\clocks.dat" /output "C:\OR\MyRoute\ROUTES\animated.clocks-or" 

[James Ross]

Genma Saotome, on 26 April 2021 - 08:17 PM, said:

How much of that is actually the fault of sloppy work by KUJU that is kept around as-is simply to keep OR files working in train.exe?

Something to consider: Drop the need to run those files thru train.exe and the effort to fix the slop might be less than what will be required to fully implement .json

There are certainly some uncommon elements to the STF format, such as the variety of ways of ignoring sections/commenting ("Skip", "Info", "Comment" blocks and any block name starting with "_" are skipped, and "//" is an inline comment which counts parentheses to work correctly).

However, the problem with creating a better version of STF just for Open Rails is that we still have to read all the existing content created for MSTS, so we'd still have two formats to read - like we do with JSON and STF - but this time we'd have to manually implement both of them. Implementing a parser is a lot of hard work, and you couldn't use the same parser for both without a toggle for which features to use - and a single parser with a toggle is still two parsers when it comes to correctness, testing, and bug fixes (this is the same reason each new game option is resisted).

This is why JSON is a better choice - we are offloading all the hard work of designing, specifying, and implementing the format (syntax, tokeniser/serialiser/deserialiser, etc.) and only working on the bits specific to Open Rails. It will save us a huge amount of time in the long run.

cjakeman, on 27 April 2021 - 05:35 AM, said:

These dialogs have been the cause of some disquiet, popping up as they do on pressing "Start" if a user has a clocks.dat but not yet an animated.clocks-or

I have therefore removed them from the submission for the official Open Rails and conversion is no longer prompted for.

Thanks.

[Goku]

Quote

There are certainly some uncommon elements to the STF format, such as the variety of ways of ignoring sections/commenting

I don't see a problem with new files in STF format. Just avoid using these uncommon elements. Use only the simple elements.
I think that files extending route/tsection STF files should be also made using the same format.

For files unrelated to existing STF files, JSON seems to be good.

[Genma Saotome]

James Ross, on 27 April 2021 - 10:41 AM, said:

However, the problem with creating a better version of STF just for Open Rails is that we still have to read all the existing content created for MSTS, so we'd still have two formats to read - like we do with JSON and STF - but this time we'd have to manually implement both of them. Implementing a parser is a lot of hard work, and you couldn't use the same parser for both without a toggle for which features to use - and a single parser with a toggle is still two parsers when it comes to correctness, testing, and bug fixes (this is the same reason each new game option is resisted).
Thanks.

Ok. Let's take the example of Position() in world files. If I understand the problem described in this thread, Position() is a "problem" because it is not in this format:
Position (

X(nnn.nnn)
Y(nnn.nnn)
Z(nnn.nnn)

)

I know of many similar data. My question: If you need to convert the current string for Position() for .json... you'd have to do almost the exact same thing to fix that problem in .stf. Is that correct? Assuming for discussion that it is, then what real advantage do all the braces, commas, and brackets bring to end users in .json -- or you -- other than cluttering things up (and apparently slowing down the loader)?

Other questions: Am I correct in guessing that most of the parsing problems that concern you are found in files in the route's home folder? Might some of those benefit from a complete reworking rather than a format change?

FWIW, other than the route's own .ref file I rather doubt there are more than a handful of people worldwide who would have an opinion about the format used for files in the home directory. That strikes me as far less controversial than anything else that comes to mind... and what also comes to mins is >80% of the parsing problems can be dealt with in <20% of extant files.

[YoRyan]

Uhm... no. Frankly, the more you interrogate James over the format question, the less sense you start to make.

The "problem" has nothing to do with what the data looks like in your text editor. The problem is that there's no documentation out there telling you exactly what to do to parse an STF file, so we have to make our best guess. And sometimes, we get it wrong, or the answer itself is ambiguous. Go ahead, start in Kuju's SD40-2 on Marias Pass and watch as Open Rails fails to load the locomotive shape. In this sense, JSON is already "fixed," because it is a meticulously documented format with built-in support in all programming languages. So rather than compound the problem, it's much easier to simply avoid using the bad, ill-defined format going forward.

JSON performed slower in performance in Wayne's test because he used an exceptionally slow method that sandbagged the JSON loader, not because the extra characters in JSON make the format inherently "slower" to read. That just isn't how parsers work.

[Genma Saotome]

YoRyan, on 27 April 2021 - 01:46 PM, said:

The problem is that there's no documentation out there telling you exactly what to do to parse an STF file, so we have to make our best guess.

To which I'll ask again: What files are most problematic? Is the problem KUJU slop or people hand editing files? These are basic quality control questions meant to identify the more consequential problems to address.

Something else... MANY years ago I had a conversation w/ James about his experience contributing to another volunteer software project (I think it might have been Firefox). The gist of it was his inclination was to remove as many things as possible from direct editing by end users because they'll just screw it up and for my part of the conversation was pretty much if you lock everything down we'll be left with flat cardboard for paint by numbers instead of a canvas, palate, paint, and brushes. While I can appreciate James' concern over what messing around does to the programmers I need to note that controlling everything is deleterious to the other half of the population.

The creativity we see in routes and models is, IMO, largely because people found ways to work around the constraints KUJU gave us, many of which were stupid (can't run electric locomotives with any other kind of power in the same route), a product of an incomplete product, or in the case of proper referential integrity in flat files, rather hard to do.

My suspicions are the files that are most problematic may be the least tweeked by users; If that's the case, go for it. OTOH locking up files that are most often fiddled with will be very problematic.