[jonas]

Just for my understanding - If this is the correct json file:

[
  {
    "Name": "Foo.s",
    "Type": "tree"
  }
]

And this would be the same json file with a typo:

[
  {
    "Name": "Foo.s",
    "Type": "tree"\
  }
]

Would the json parsers in general be robust enough to read the latter file flawless?

[jonas]

Meanwhile I've test it with "animated.clocks-or" and get this box:


To be fair: At least an error box pops up in OR with a hint where the error comes from.
But no error box with the OR-STF-Parser when reading the SD40-2, but then the SD40-2 isn't visible ...and one does not know why.

[YoRyan]

As you can see by running your snippet through JSONLint, it's not considered valid JSON:

Error: Parse error on line 3:
....s",	"Type": "tree"\}]
----------------------^
Expecting 'EOF', '}', ':', ',', ']', got 'undefined'

The point is not so much that the parser is "robust" - it's that you can expect every JSON reader ever written to follow the exact same logic. If it doesn't pass this validator, it's not JSON. Period.

Is the SD40-2 shape valid STF? Kuju thinks so. MSTS thinks so. Open Rails does not. But Open Rails has become the de-facto STF reader.

[Goku]

YoRyan, on 28 April 2021 - 08:57 AM, said:

The SD40-2. Here's the file that doesn't load; it's straight from the MSTS 1.2 update. I have just tested it on master and on NewYear and it doesn't work on either. It's possible you've already repaired your own copy.

SD402.zip

It's the syntax that's badly defined about STF, not the names of the properties. In the case of the SD40-2, there's a backslash at the end of one of the lines. What is that supposed to mean to the parser? Kuju apparently just ignores it, but Open Rails doesn't. Should we change the behavior to match MSTS's? If we do, does this break any existing OR files that assume our current behavior?

But the case with this file may be just an user made bug.

YoRyan, on 28 April 2021 - 08:57 AM, said:

You don't have these kinds of problems with JSON. If your data doesn't load correctly in your web browser, then it's not valid JSON.

So, maybe the SD40-2 file is not a valid file and MSTS parser just ignores this bug?

YoRyan, on 28 April 2021 - 08:57 AM, said:

Actually, they don't. There's one and only one way to parse JSON. This information does not exist for STF.

No programmer is perfect and many of these fast libs don't have to follow the specs 100% correctly. Try opening advanced ODF file id different apps and the result may be completely different. It's common.

YoRyan, on 28 April 2021 - 08:57 AM, said:

But it's a file produced by Kuju themselves. It's not a bug - it was intentional. The bug is in Open Rails itself, a consequence of the extreme difficulty of writing a consistent STF parser.

How do you know that it was intentional? It doesn't mean anything.

I think that you are looking for problems where there are none. If you don't know what to do with "\" character in STF file when parsing it, just don't allow using it in OR files.

[Goku]

Look, TSRE parser doesn't know nothing about "\" character, but this file works fine here:

https://i.imgur.com/WcSYOhq.png

I'm just a noob programmer and TSRE doesn't care about looking for errors. In the time of MSTS development it was common, handling errors makes things slower.

[YoRyan]

Goku, on 28 April 2021 - 12:01 PM, said:

No programmer is perfect and many of these fast libs don't have to follow the specs 100% correctly. Try opening advanced ODF file id different apps and the result may be completely different. It's common.

The fact remains that there are JSON and ODF specs and there is no such spec for STF.

---

Let's flip the script: Is there a reason why this community is coming out of the woodwork to defend STF? Is there an argument being made here to change OR's new formats policy? Because if not, all of this talk really amounts to nothing more than a waste of everyone's time.

[jonas]

YoRyan, on 28 April 2021 - 11:56 AM, said:

As you can see by running your snippet through JSONLint, it's not considered valid JSON: ...The point is not so much that the parser is "robust" - it's that you can expect every JSON reader ever written to follow the exact same logic. If it doesn't pass this validator, it's not JSON. ...

Yes, I see. Can we (OR) then make the external json files pass a "JSONLint test" so to speak first before reading them in to prevent the error box above?
Would certainly take too much loading time while the game is running with the many shapes and world files, but maybe with the smaller files that only read once, like the animated.clocks-or similar, that the end user "likes" to manipulate?

[YoRyan]

That error box provides the exact same information that a JSON linter would. How do you think OR should behave differently?

[jonas]

YoRyan, on 28 April 2021 - 12:13 PM, said:

That error box provides the exact same information that a JSON linter would. How do you think OR should behave differently?

Oh, ok, I probably have a misconception of the hierarchical flows in OR code.
My way of thinking was assuming to avoid the parsing causing the error box and to test beforehand if it is correct json. Now I understand that the error box itself already comes from the parser. Is that correct?

[YoRyan]

That is correct, yes.

[jonas]

Goku, on 28 April 2021 - 12:01 PM, said:

... How do you know that it was intentional? ...

I'm also pretty sure it was simply a typo that the backslash "ended up" in SD40-2.s, wasn't it? Maybe someone even wanted to open a comment line...don't know.
Otherwise, what intention could be the reason for this...just curious?

[jonas]

YoRyan, on 28 April 2021 - 12:22 PM, said:

That is correct, yes.

Is there a way or do you see one, Ryan, how to prevent such error boxes that cause OR to crash due to "bad" json (i.e. no json :-), despite the "bad" json?
In other words, is the json parser error message catchable OR-internally and could better lead to skipping of the "bad" json files instead of a complete OR crash?

[YoRyan]

Yes, the structure you're looking for is called a try-catch block. You can handle exceptions instead of allowing them to crash the simulator. Usually this means logging the error and skipping the object you were trying to load.

[conductorchris]

Ryan, you asked why the community was defending STF? I think it is because there is something STF does for the community that people value, which is the ease of digging into the file and modifying it. JSON looks complicated and difficult, especially compared to what we are used to. I value this and would not relish making .wag or .eng files with a text file in JSON.

I think you should go build the editor and then the conversation will be about the editor and everything cool and new it can do. That's where I think the conversation should be. The file format will fall into place.

Christopher

[YoRyan]

Well, policy is policy. If it happens to make anyone uncomfortable, and they would like to see it changed, then I would suggest they formulate an alternative, explicate the benefits and tradeoffs (taking into special consideration James' desire for a formal specification), and present as succinct a case as possible to the management team. Holy wars are not going to do anything for anyone.

To be clear, OR policy is: JSON or BSON shall be the preferred file format for new OR data files not derived from those of MSTS.

Sound advice, Chris. I'm bowing out.