[James Ross]

 Genma Saotome, on 26 April 2021 - 08:17 PM, said:

How much of that is actually the fault of sloppy work by KUJU that is kept around as-is simply to keep OR files working in train.exe?

Something to consider: Drop the need to run those files thru train.exe and the effort to fix the slop might be less than what will be required to fully implement .json

There are certainly some uncommon elements to the STF format, such as the variety of ways of ignoring sections/commenting ("Skip", "Info", "Comment" blocks and any block name starting with "_" are skipped, and "//" is an inline comment which counts parentheses to work correctly).

However, the problem with creating a better version of STF just for Open Rails is that we still have to read all the existing content created for MSTS, so we'd still have two formats to read - like we do with JSON and STF - but this time we'd have to manually implement both of them. Implementing a parser is a lot of hard work, and you couldn't use the same parser for both without a toggle for which features to use - and a single parser with a toggle is still two parsers when it comes to correctness, testing, and bug fixes (this is the same reason each new game option is resisted).

This is why JSON is a better choice - we are offloading all the hard work of designing, specifying, and implementing the format (syntax, tokeniser/serialiser/deserialiser, etc.) and only working on the bits specific to Open Rails. It will save us a huge amount of time in the long run.

 cjakeman, on 27 April 2021 - 05:35 AM, said:

These dialogs have been the cause of some disquiet, popping up as they do on pressing "Start" if a user has a clocks.dat but not yet an animated.clocks-or

I have therefore removed them from the submission for the official Open Rails and conversion is no longer prompted for.

Thanks.

[Goku]

Quote

There are certainly some uncommon elements to the STF format, such as the variety of ways of ignoring sections/commenting

I don't see a problem with new files in STF format. Just avoid using these uncommon elements. Use only the simple elements.
I think that files extending route/tsection STF files should be also made using the same format.

For files unrelated to existing STF files, JSON seems to be good.

[Genma Saotome]

 James Ross, on 27 April 2021 - 10:41 AM, said:

However, the problem with creating a better version of STF just for Open Rails is that we still have to read all the existing content created for MSTS, so we'd still have two formats to read - like we do with JSON and STF - but this time we'd have to manually implement both of them. Implementing a parser is a lot of hard work, and you couldn't use the same parser for both without a toggle for which features to use - and a single parser with a toggle is still two parsers when it comes to correctness, testing, and bug fixes (this is the same reason each new game option is resisted).
Thanks.

Ok. Let's take the example of Position() in world files. If I understand the problem described in this thread, Position() is a "problem" because it is not in this format:
Position (

X(nnn.nnn)
Y(nnn.nnn)
Z(nnn.nnn)

)

I know of many similar data. My question: If you need to convert the current string for Position() for .json... you'd have to do almost the exact same thing to fix that problem in .stf. Is that correct? Assuming for discussion that it is, then what real advantage do all the braces, commas, and brackets bring to end users in .json -- or you -- other than cluttering things up (and apparently slowing down the loader)?

Other questions: Am I correct in guessing that most of the parsing problems that concern you are found in files in the route's home folder? Might some of those benefit from a complete reworking rather than a format change?

FWIW, other than the route's own .ref file I rather doubt there are more than a handful of people worldwide who would have an opinion about the format used for files in the home directory. That strikes me as far less controversial than anything else that comes to mind... and what also comes to mins is >80% of the parsing problems can be dealt with in <20% of extant files.

[YoRyan]

Uhm... no. Frankly, the more you interrogate James over the format question, the less sense you start to make.

The "problem" has nothing to do with what the data looks like in your text editor. The problem is that there's no documentation out there telling you exactly what to do to parse an STF file, so we have to make our best guess. And sometimes, we get it wrong, or the answer itself is ambiguous. Go ahead, start in Kuju's SD40-2 on Marias Pass and watch as Open Rails fails to load the locomotive shape. In this sense, JSON is already "fixed," because it is a meticulously documented format with built-in support in all programming languages. So rather than compound the problem, it's much easier to simply avoid using the bad, ill-defined format going forward.

JSON performed slower in performance in Wayne's test because he used an exceptionally slow method that sandbagged the JSON loader, not because the extra characters in JSON make the format inherently "slower" to read. That just isn't how parsers work.

[Genma Saotome]

 YoRyan, on 27 April 2021 - 01:46 PM, said:

The problem is that there's no documentation out there telling you exactly what to do to parse an STF file, so we have to make our best guess.

To which I'll ask again: What files are most problematic? Is the problem KUJU slop or people hand editing files? These are basic quality control questions meant to identify the more consequential problems to address.

Something else... MANY years ago I had a conversation w/ James about his experience contributing to another volunteer software project (I think it might have been Firefox). The gist of it was his inclination was to remove as many things as possible from direct editing by end users because they'll just screw it up and for my part of the conversation was pretty much if you lock everything down we'll be left with flat cardboard for paint by numbers instead of a canvas, palate, paint, and brushes. While I can appreciate James' concern over what messing around does to the programmers I need to note that controlling everything is deleterious to the other half of the population.

The creativity we see in routes and models is, IMO, largely because people found ways to work around the constraints KUJU gave us, many of which were stupid (can't run electric locomotives with any other kind of power in the same route), a product of an incomplete product, or in the case of proper referential integrity in flat files, rather hard to do.

My suspicions are the files that are most problematic may be the least tweeked by users; If that's the case, go for it. OTOH locking up files that are most often fiddled with will be very problematic.

[YoRyan]

 Genma Saotome, on 27 April 2021 - 03:53 PM, said:

To which I'll ask again: What files are most problematic? Is the problem KUJU slop or people hand editing files? These are basic quality control questions meant to identify the more consequential problems to address.

It's... both, I guess? The problematic SD40-2 shape is an official Kuju file, while hand edited files are also a major liability because it's so easy to mess up the syntax. Indeed, any change to the STF parser is bound to break something out there and generate complaints.

So, for future formats, don't use STF. Minimize the problem as much as you can.

I'm sorry, I just don't understand how these questions are at all relevant. What are you proposing, a sort of "STF version 2.0" that corrects for the flaws in Kuju's STF? That looks like the original thing, so you can pretend it's 2002 again and you're modding MSTS, but has train.exe-breaking upgrades for OR? So we can have two STF parsers to test and maintain?

[Genma Saotome]

I have no reservations about changing what is in use now, especially is something specifically is a real nightmare. I am skeptical about the feasibility of that. Consider, if the goal is to completely replace stf so that code can be retired, how long will that take? How many programmers? What will be done with people who don't convert or don't convert fully, as in some gets down this week, some next, etc. etc, and in three months his will all be in the new format?

To some degree Ryan it sounds like this: We're gonna stick our fingers into your data data mes mess around with it over a 18 month period and while it won't actually ever get finished, nor will it do anything for you, we'll all be happier for trying cuz it's a PITA now and we hate it.

I understand the objective and why it has appeal. As I consider the possibilities for implementation I cringe. It leaves me worried. I dunno, maybe I should never have said anything, kept my nose stuck in a 3d modeling project.

I'm checking out.

[YoRyan]

 Genma Saotome, on 27 April 2021 - 05:12 PM, said:

Consider, if the goal is to completely replace stf so that code can be retired, how long will that take?

That's not what OR's policy says to do, so you needn't worry.

[jonas]

 YoRyan, on 27 April 2021 - 01:46 PM, said:

...Go ahead, start in Kuju's SD40-2 on Marias Pass and watch as Open Rails fails to load the locomotive shape. ...

Is it already known, why one can't see the SD40-2 in OR?

[YoRyan]

There's a backslash at the end of one of the lines that fools OR's parser. Just one of many ambiguous corner cases in the STF syntax.

Our current parser represents the second attempt at writing an STF loader, by the way, so it's possible it worked in version 1, but no longer works with the contemporary code.