[Goku]

STF files are superior to JSON because they can be binary. No one uses text files to store huge amounts of data for simulation/games etc., because it's so slow to process and takes a lot of space.

[wacampbell]

 YoRyan, on 20 April 2021 - 09:59 PM, said:

A 10x difference sounds very dramatic and I think it's impossible to draw any conclusions without knowing the circumstances of this test. If anything, I would expect the commercial JSON loaders to blow our do-it-yourself STF parsers out of the water; fast JSON loading is a well-studied problem. That is, assuming identical data for both formats. At the moment, we do not actually have JSON counterparts to MSTS world files, so no plausible comparison is possible.

This comes down to end user convenience vs programmer convenience. I don't think there are any content developers or end users who prefer JSON. They would prefer to have only one familiar format used cohesively across the project. Not to mention the technical limitations of JSON such as:
- more verbose
- more picky about syntax
- can't handle line spanning strings ( such as route descriptions etc )
- lack of a fast loading binary version.

Most of the programmer support for json derives from convenience. Its built into .net with automatic serialization. The problem is this automatic serialization is very slow. It uses code reflection at run time to look at the source and unravel what classes and types need to be loaded. Code reflection is slow. But as you say, there are a lot of fast JSON loading methods and techniques. But using these methods always involves writing a custom serializer for each of your classes. And doing so negates any of the programmer convenience advantages claimed over STF. I realize programmer convenience is important in an understaffed project, but end users and content developers could pay a high price for this convenience.

Wayne

PS my original 2012 JSON speed tests were discussed here.
Source code for test is in the dowload library here

PS Edit

Quote

we do not actually have JSON counterparts to MSTS world files

For my speed comparisons I used actual MSTS data from my L&PS route. I chose .t terrain files, created a JSON format for them, converted all that data to JSON, and measured its effect on route loading time. So it was as real a test case as I could create.

[YoRyan]

Personally even if I was not a programmer, I would much prefer JSON. Its syntax is well-defined and predictable, unlike that of STF. (There are still Kuju-produced STF files that will not load properly in Open Rails, like the SD40-2 shape.) Virtually every advanced text editor has support for JSON syntax highlighting and schema validation. And, here's the big one - you'll actually encounter JSON outside of the MSTS community, so there's very high value in learning it.

But, don't mind me. It's obvious that I'm in the extreme minority here.

[Csantucci]

 Goku, on 25 April 2021 - 08:22 AM, said:

STF files are superior to JSON because they can be binary. No one uses text files to store huge amounts of data for simulation/games etc., because it's so slow to process and takes a lot of space.

I think that this is a very important point.

[YoRyan]

I'll grant that sometimes there's so much data that performance has to be a key concern, as with terrain and world files, for example. But then you have existing technologies like Protobuf that are already available, easy to use, extensively tested, and have very strong support. And remember, binary formats are not modifiable by end users, so all arguments about "familiarity" and "prettier syntax" go out the window. There's simply no reason to build new STF binary formats.

[wacampbell]

Perhaps a compromise proposal...

If the file is used for input by content developers or end users, then use the STF format. That way there is consistency and familiarity from their point of view.

But files that are primarily internal can be programmer's choice of format. Examples might be eg terrain files, future shape files etc not normally edited by the end user. In choosing these formats programmers might consider loading speed along with programmer convenience.


Wayne

[YoRyan]

If the goal is consistency, then we've already got a problem on our hands, because at the moment we have:

• STF for content inherited from MSTS
  
• STF for paths produced by the Track Viewer
  
• CSV for timetable files
  
• (JSON became OR policy beyond this point?)
  
• JSON for timetable weather files
  
• STF for animated clocks (in NewYear-MG)
  
• JSON for animated clocks (in mainline OR, with a converter from STF)

[wacampbell]

 YoRyan, on 25 April 2021 - 12:08 PM, said:

If the goal is consistency, then we've already got a problem on our hands, because at the moment we have:

• STF for content inherited from MSTS
  
• STF for paths produced by the Track Viewer
  
• CSV for timetable files
  
• (JSON became OR policy beyond this point?)
  
• JSON for timetable weather files
  
• STF for animated clocks (in NewYear-MG)
  
• JSON for animated clocks (in mainline OR, with a converter from STF)

And more.
XML for dynamic track cross section ( I think )

... hence my concerns about consistency

[Csantucci]

.ini for script configuration files...

[jonas]

 wacampbell, on 25 April 2021 - 11:53 AM, said:

Perhaps a compromise proposal...

If the file is used for input by content developers or end users, then use the STF format. That way there is consistency and familiarity from their point of view.

But files that are primarily internal can be programmer's choice of format. Examples might be eg terrain files, future shape files etc not normally edited by the end user. In choosing these formats programmers might consider loading speed along with programmer convenience. ...

This is what I meant by the categories of "one-way" and "dual-way" files. Maybe I should have better called them
"user->machine" and "machine<->machine" files.
But in any case, for the OR-parsers of possible future STF "user->machine" files, the manpower of developers would be needed again and again - as YoRyan already objected here.
It really seems like a very double-edged problem to me, unfortunately.