[jonas]

Thanks for replying, I was just curious if it is known how to fix the sd40-2 model.
I am trying to program an editor for the clocks files. I have to read the JSON animated.clocks-or and the STF clocks.dat. So I just realize what a considerable effort is required for this.

Best regards
Jonas

[YoRyan]

Well, there's loader code and a converter already in Open Rails, so there's no need to write your own readers. In addition, mainline OR does not need support for clocks.dat.

[jonas]

YoRyan, on 27 April 2021 - 06:38 PM, said:

Well, there's loader code and a converter already in Open Rails...

Thanks for the tip. I forgot to mention that I don't program the editor in C#. It will run (if I can get it to work at all :-) as a stand alone, independend from OR. It's more of a template how such an editor could look like...maybe just for me.

[YoRyan]

Do note that we only accept C# into our repository, so if you choose not to use it, you will not be able to contribute your editor to OR.

[jonas]

I am perfectly clear about this.

[Csantucci]

Ryan, I don't think that the SD40 issue demonstrates that STF is unreliable or badly defined. Assume you have a shape editor that generates loadable .json shape files. I could modify them, still maintaining the .json syntax, and get something that is unloadable. That wouldn't mean that Json is unreliable or badly defined.
By the way, do you mean SD40 or SD40-2? The latter loads here.

[Laci1959]

Why is manual repair sometimes necessary? A very typical example:

https://msts.steam4m...ls/gantlet.html
https://msts.steam4m..._dbl_slips.html

OR does not handle this to date. TSRE is already yes but not complete. This is when manual editing comes in.
There will always be something in the situation that the app creator does not think about.
Mindig kíváncsian figyelem az újdonságokat, mert ezek viszik előbbre az Open Railst.

[cjakeman]

Genma Saotome, on 27 April 2021 - 05:12 PM, said:

Consider, if the goal is to completely replace stf so that code can be retired, how long will that take? How many programmers? What will be done with people who don't convert or don't convert fully, as in some gets down this week, some next, etc. etc, and in three months his will all be in the new format?

. . .

I'm checking out.

Dave, I think the message has become muddled. We are not taking away the parsing of STF for existing types of file. JSON is only for new types of file.

Our policy is that "Open Rails will continue to maintain compatibility with content from previous official versions of Open Rails."

I will be publishing this and other policies on the Open Rails website shortly.

[Goku]

YoRyan, on 27 April 2021 - 06:13 PM, said:

There's a backslash at the end of one of the lines that fools OR's parser. Just one of many ambiguous corner cases in the STF syntax.

If you decide to use custom fast libs for JSON loading, then different libs can treat this kind of bugs differently too.

I don't think that pointing to bugs in some STF files is any reasonable argument against STF file format.

[YoRyan]

Csantucci, on 27 April 2021 - 11:10 PM, said:

Ryan, I don't think that the SD40 issue demonstrates that STF is unreliable or badly defined. Assume you have a shape editor that generates loadable .json shape files. I could modify them, still maintaining the .json syntax, and get something that is unloadable. That wouldn't mean that Json is unreliable or badly defined.
By the way, do you mean SD40 or SD40-2? The latter loads here.

The SD40-2. Here's the file that doesn't load; it's straight from the MSTS 1.2 update. I have just tested it on master and on NewYear and it doesn't work on either. It's possible you've already repaired your own copy.

 SD402.zip (12.14MB)
Number of downloads: 340

It's the syntax that's badly defined about STF, not the names of the properties. In the case of the SD40-2, there's a backslash at the end of one of the lines. What is that supposed to mean to the parser? Kuju apparently just ignores it, but Open Rails doesn't. Should we change the behavior to match MSTS's? If we do, does this break any existing OR files that assume our current behavior?

You don't have these kinds of problems with JSON. If your data doesn't load correctly in your web browser, then it's not valid JSON.

Goku, on 28 April 2021 - 08:52 AM, said:

If you decide to use custom fast libs for JSON loading, then different libs can treat this kind of bugs differently too.

Actually, they don't. There's one and only one way to parse JSON. This information does not exist for STF.

Goku, on 28 April 2021 - 08:52 AM, said:

I don't think that pointing to bugs in some STF files is any reasonable argument against STF file format.

But it's a file produced by Kuju themselves. It's not a bug - it was intentional. The bug is in Open Rails itself, a consequence of the extreme difficulty of writing a consistent STF parser.

[jonas]

Just for my understanding - If this is the correct json file:

[
  {
    "Name": "Foo.s",
    "Type": "tree"
  }
]

And this would be the same json file with a typo:

[
  {
    "Name": "Foo.s",
    "Type": "tree"\
  }
]

Would the json parsers in general be robust enough to read the latter file flawless?

[jonas]

Meanwhile I've test it with "animated.clocks-or" and get this box:


To be fair: At least an error box pops up in OR with a hint where the error comes from.
But no error box with the OR-STF-Parser when reading the SD40-2, but then the SD40-2 isn't visible ...and one does not know why.

[YoRyan]

As you can see by running your snippet through JSONLint, it's not considered valid JSON:

Error: Parse error on line 3:
....s",	"Type": "tree"\}]
----------------------^
Expecting 'EOF', '}', ':', ',', ']', got 'undefined'

The point is not so much that the parser is "robust" - it's that you can expect every JSON reader ever written to follow the exact same logic. If it doesn't pass this validator, it's not JSON. Period.

Is the SD40-2 shape valid STF? Kuju thinks so. MSTS thinks so. Open Rails does not. But Open Rails has become the de-facto STF reader.

[Goku]

YoRyan, on 28 April 2021 - 08:57 AM, said:

The SD40-2. Here's the file that doesn't load; it's straight from the MSTS 1.2 update. I have just tested it on master and on NewYear and it doesn't work on either. It's possible you've already repaired your own copy.

SD402.zip

It's the syntax that's badly defined about STF, not the names of the properties. In the case of the SD40-2, there's a backslash at the end of one of the lines. What is that supposed to mean to the parser? Kuju apparently just ignores it, but Open Rails doesn't. Should we change the behavior to match MSTS's? If we do, does this break any existing OR files that assume our current behavior?

But the case with this file may be just an user made bug.

YoRyan, on 28 April 2021 - 08:57 AM, said:

You don't have these kinds of problems with JSON. If your data doesn't load correctly in your web browser, then it's not valid JSON.

So, maybe the SD40-2 file is not a valid file and MSTS parser just ignores this bug?

YoRyan, on 28 April 2021 - 08:57 AM, said:

Actually, they don't. There's one and only one way to parse JSON. This information does not exist for STF.

No programmer is perfect and many of these fast libs don't have to follow the specs 100% correctly. Try opening advanced ODF file id different apps and the result may be completely different. It's common.

YoRyan, on 28 April 2021 - 08:57 AM, said:

But it's a file produced by Kuju themselves. It's not a bug - it was intentional. The bug is in Open Rails itself, a consequence of the extreme difficulty of writing a consistent STF parser.

How do you know that it was intentional? It doesn't mean anything.

I think that you are looking for problems where there are none. If you don't know what to do with "\" character in STF file when parsing it, just don't allow using it in OR files.

[Goku]

Look, TSRE parser doesn't know nothing about "\" character, but this file works fine here:

https://i.imgur.com/WcSYOhq.png

I'm just a noob programmer and TSRE doesn't care about looking for errors. In the time of MSTS development it was common, handling errors makes things slower.