Route extending Open Rails files
#71
Posted 25 April 2021 - 12:32 PM
#72
Posted 25 April 2021 - 12:37 PM
#73
Posted 25 April 2021 - 12:51 PM
YoRyan, on 25 April 2021 - 12:32 PM, said:
Yah, I had really hoped c# would become our universal language for signal scripts, engine scripts and all. These would all be first class components using the full power of .net for performance. But alas too much resistance.
#74
Posted 25 April 2021 - 12:56 PM
#75
Posted 25 April 2021 - 01:03 PM
YoRyan, on 25 April 2021 - 12:56 PM, said:
Not really. You can edit a .cs in a text editor ( as you would Lua ) and Open Rails would open , jit compile and execute. All this comes free with .net. An end user would see little difference between scripting in c# vs msts scripts other than brackets and semi’s in different places.
#76
Posted 26 April 2021 - 12:31 PM
jonas, on 15 April 2021 - 12:52 PM, said:
• It may be that there are good reasons why there can be another Openrails folder in the MSTS world folder, which contains additional world files with extensions, but which do not need a blank line at the beginning and also not with "include ("..\thisAndThat.w"). Instead, this world extension file begins just like any ordinary world file from MSTS.
• It may be that there are Open Rails policies that provide for MSTS extensions in Openrails folders and OR own files are to be saved in the root folder of the route.
All SIMIS files start with a header; for binary/compressed files, this clearly identifies how to load the file and is used by Open Rails for shapes, terrain, and world files only. For all other SIMIS files, Open Rails only supports the Unicode text format (STF), where the header is one line of text, which is ignored by OR. This is the "blank line" in these files.
The rule for the "OpenRails" subfolder inside MSTS content is that Open Rails loads "OpenRails\foo.dat" instead of "foo.dat", if it exists, but there is no difference in content format or features supported. The arrangement is provided so that content can be MSTS compatible while also taking advantage of Open Rails features in a way that is not otherwise compatible with MSTS, by having two different files for one thing.
The exception is world files, which cannot use "include" because they are not STF files (they can be, but not always), so we do some extra work to merge the OpenRails and base versions as if you'd used "include".
jonas, on 15 April 2021 - 12:52 PM, said:
In hindsight, this was probably a mistake. It should have been either:
- Extension to the existing format (such that the same data could also be used in the base "carspawn.dat", if you didn't care about MSTS compatibility)
- A separate filename and/or extension (whose presence would override the base "carspawn.dat")
jonas, on 15 April 2021 - 12:52 PM, said:
• It may be that there are good reasons why the turntable.dat is seen differently than the clocks.dat, but I haven't hear any reason why so far (@Chris, you didn't answer me about this in this post :-)).
I personally see no reason for "clocks.dat" or "turntables.dat" to be in the OpenRails folder, but we did the same thing for timetable files (which are in "Activities\OpenRails").
One thing you absolutely have to bear in mind is that Open Rails has been around many years now; it has grown organically and has historically not had a particularly tight control over how new features were implemented, including where new files go. Some of them will be accidents, most of them will be what the author thought was best at the time, but few will have been placed according to a grand plan (because there really isn't one).
jonas, on 15 April 2021 - 12:52 PM, said:
Csantucci, on 19 April 2021 - 11:58 PM, said:
We have not released any official "clocks.dat" support. Officially, we have a completely free choice of how to implement this brand new feature, for which JSON is preferred. However, we have a problem whereby there may exist third party content creators that have created unofficial files that we are being asked to support.
There are no good solutions here.
- We support JSON format only - any existing content must be updated
- We support STF format only - void our own policy
- We support both formats - extra maintenance burden forever more
I agree with Rob that this file is a great example for implementing in JSON (option 1), not least because we should gain more experience with it before embarking on larger works. The failure mode is basically non-existant, and the fix is very simple for the content creator.
wacampbell, on 18 April 2021 - 09:48 AM, said:
It is obviously an unfair comparison to use JSON with reflection vs STF with custom tokenisation parsing.
My motivation for JSON is entirely the standardised nature of it - everyone knows exactly how it works, so we don't have to spend years bug fixing the parsing and 3rd parties can much more easily get involved by using their standard library.
We still don't know how to parse STF properly. Only last July were we trying to fix bugs in StfReader. It might look like a nice simple format, but without a specification we have spent years guessing how to parse it correctly and we're still not 100% there.
I did a lot of my own reverse engineering and tool-writing before joining Open Rails and I am certain we still have STF parsing bugs.
So while JSON is a new syntax that people in this community may end up learning, you can still happily edit with Notepad if you so wish:
roeter, on 20 April 2021 - 01:54 PM, said:
Users are now editing .con files, .w files and even the .tdb if necessary, even though these are (or can) be created through editors.
There are always two groups of people - those with no or limited knowledge of the data structure who therefor are happy to use the editor. And those who have that knowledge and just find editors cumbersome or too restricted, especially when things go wrong. That's how it always has been, that's how it always will be.
wacampbell, on 25 April 2021 - 10:13 AM, said:
I guess that depends on whether you believe in the vision of the future where we have editors for the new Open Rails file structure.
In that future, almost all files will be JSON (exceptions may be models and textures, which will be BSON or another standardised binary format) and users will have GUI editors that allow them to almost never see the contents of those files. And when they do need to go into the files, it's a well-defined format that will have support from any popular text editor (potentially including JSON Schema) making any manual editing easier and less error prone.
I do not care about whether I can serialise/deserialise JSON using reflection or any other "developer magic"; I care about the file format being properly and completely defined, which JSON is but STF is not.
wacampbell, on 25 April 2021 - 10:13 AM, said:
- more verbose
- more picky about syntax
- can't handle line spanning strings ( such as route descriptions etc )
- lack of a fast loading binary version.
The only one of these that might matter to us is the last one, which, if we need it, binary JSON formats (BSON) already exists and are standardised.
#77
Posted 26 April 2021 - 02:22 PM
I feel like I'm somewhat up to date now and I see even better what the current problems and solutions are with the different formats. For me it was a good school on this topic, I hope for others as well and not just a repetition of what was already standard knowledge for many.
And thanks also to all others who have written here on the subject so far! I'm looking forward to more posts here in case there come more.
Best regards
Jonas
#78
Posted 26 April 2021 - 03:02 PM
James Ross, on 26 April 2021 - 12:31 PM, said:
We still don't know how to parse STF properly. Only last July were we trying to fix bugs in StfReader. It might look like a nice simple format, but without a specification we have spent years guessing how to parse it correctly and we're still not 100% there.
I did a lot of my own reverse engineering and tool-writing before joining Open Rails and I am certain we still have STF parsing bugs.
...
I do not care about whether I can serialise/deserialise JSON using reflection or any other "developer magic"; I care about the file format being properly and completely defined, which JSON is but STF is not.
I'd just like to emphasize that this is the single biggest strike against expanding the use of STF. Drops microphone.
#79
Posted 26 April 2021 - 08:17 PM
James Ross, on 26 April 2021 - 12:31 PM, said:
We still don't know how to parse STF properly. Only last July were we trying to fix bugs in StfReader. It might look like a nice simple format, but without a specification we have spent years guessing how to parse it correctly and we're still not 100% there.
I did a lot of my own reverse engineering and tool-writing before joining Open Rails and I am certain we still have STF parsing bugs.
How much of that is actually the fault of sloppy work by KUJU that is kept around as-is simply to keep OR files working in train.exe?
Something to consider: Drop the need to run those files thru train.exe and the effort to fix the slop might be less than what will be required to fully implement .json
#80
Posted 27 April 2021 - 05:35 AM
jonas, on 18 April 2021 - 07:03 AM, said:
But we have now landed here, where two (sorry) extortionate dialogues pop up before the activity starts.
These dialogs have been the cause of some disquiet, popping up as they do on pressing "Start" if a user has a clocks.dat but not yet an animated.clocks-or
I have therefore removed them from the submission for the official Open Rails and conversion is no longer prompted for.
Users who want to convert from clocks.dat to animated.clocks-or so that their clocks will be animated in the official version are advised to use the Contrib.DataConverter.exe which is supplied with OpenRails.
For example:
Contrib.DataConverter.exe /input "C:\OR\MyRoute\ROUTES\OpenRails\clocks.dat" /output "C:\OR\MyRoute\ROUTES\animated.clocks-or"