[Genma Saotome]

 James Ross, on 29 April 2021 - 12:20 PM, said:

The intention has always been to build enough into the editors that people generally don't need to touch the files by hand. Of course, sometimes they will still need to, which is why I want the new format to be text.

Thank you.

[eric from trainsim]

I don't have much of a dog in the hunt, but I'd also push for any new format to be editor neutral, meaning stored in either ASCII or Unicode and able to be updated by any tool desired.

[conductorchris]

I can testify that unicode files have caused all sorts of problems for users over the years when they slipped into ascii format somehow and then stopped working.
This isn't hard for us old hands, but it is a big trip-wire for newbies.
Christopher

[Goku]

UTF-16 encoding used by MSTS is the best option if you want to support many languages. ASCII can't support that and it's user fault that they use obsolete MS Notepad to edit these files.
UTF-16 is also easier to support in software than UTF-8 although in modern programming libs it doesn't matter.

ASCII is only good for configuration files etc.

[darwins]

Quote

it's user fault that they use obsolete MS Notepad

http://www.elvastower.com/forums/public/style_emoticons/default/aggressive.gif
Apart from any suitable high level editor it might be worth recommending appropriate text editors for us ordinary folk to use with OR. I currently use ConTEXT for most things - but I am not sure it always puts stuff into Unicode when it should.

[James Ross]

 Goku, on 30 April 2021 - 07:03 AM, said:

UTF-16 encoding used by MSTS is the best option if you want to support many languages. ASCII can't support that and it's user fault that they use obsolete MS Notepad to edit these files.
UTF-16 is also easier to support in software than UTF-8 although in modern programming libs it doesn't matter.

ASCII is only good for configuration files etc.

Although we haven't talked about it for the new formats (AFAIK), I expect us to use UTF-8 for everything. The .NET built-in JSON libraries have support for reading/writing the UTF-8 bytes directly, skipping any internal UTF-16 representation of the text.

 darwins, on 30 April 2021 - 07:59 AM, said:

http://www.elvastower.com/forums/public/style_emoticons/default/aggressive.gif
Apart from any suitable high level editor it might be worth recommending appropriate text editors for us ordinary folk to use with OR. I currently use ConTEXT for most things - but I am not sure it always puts stuff into Unicode when it should.

If you're using Windows 10 1903 (19H1) or later, the built-in Notepad is enough to check what encoding your files are in. It has supported UTF-16 LE (what MSTS uses) since forever, but now displays the encoding in the bottom-right corner:

https://james-ross.co.uk/temp/Screenshot%202021-05-01%20135012.png

[markus_GE]

Jumping onto this discussion late as well, but having taken the time to read through all of it, I think the discussion currently comes down to two points:

• OTOH, we do not want to appall long-time users who have become accustomed to STF over many years and fear any new format will require them to learn everything all over again, from basic syntax rules to the overall concept of those new files. One might add to this that we seem to already e doing a reasonably good job of reading most STF files.
• OTOH, reading most STF files reasonably is far from being perfect. And that's because STF is a PITA to parse consistently because there is no good documentation on how to do it (or none at all?). That also means, it will be very hard to define new files catering to new program features based on STF - two problems that could be avoided by using JSON / BSON.

IMHO, those two main points seem to not be related too closely when one takes into account the proposal of only using JSON / BSON for files newly introduced to ORTS.

And this is exactly why I would like to put a STRONG EMPHASIS on what James has said above:

 James Ross, on 29 April 2021 - 12:20 PM, said:

For files in the new Open Rails directory structure, I strongly believe STF would be a bad decision. For new files in the MSTS directory structure, I feel less strongly but would still prefer using a new format so that:

• We get exposure to it (both community and code) before the most important usage starts (the new Open Rails directory structure)
• We can consistently say "new files are in the new format" (without the confusion between MSTS and OR directory structures)

Thus, everybody will get some hands-on time with JSON on new files everybody would have to learn about anyway, while anything else can stay the same for as long as it takes to really take the plunge and break away from only using legacy MSTS content. That will give all of us some time to re-evaluate JSON and its practicability.

Personally, I think this could be a good compromise for the time being.

Cheers, Markus

[jonInMaine]

One thing I don't see talked about here is that there are really 2 levels to be considered when representing data.

The base level is how the various elements are represented. For example XML the data language I am most familiar with, uses hierarchies of tags and attributes e.g.
<name>
<first>John</first>
<last>Smith</last>
</name>

JSON does something similar but with a more compact representation.
At this level you are just defining how elements of the language are represented but nothing about the content itself.

The next level is the data itself, the hierarchy, what is required or optional, any ordering required. For some languages e.g. XML you can define a schema which can be used to validate a file and determine if it has the required data specified, etc. I don't know if JSON has something like this.

To me it seems the problem with STF is that it doesn't seem to have a well defined base level of how elements are defined. You have a "magic" first line then various elements defined as a name then some data in parens which may be nested. The format of the data with the parens varies all over the place. I don't think you could ever generate anything like a schema equivalent for it.

But we all know the shortcomings of STF. I think what I am saying is it is not enough just to say "we are going to use JSON". You also need to define how the data is going to be represented in the JSON elements and how one can verify that a given file is valid i.e. has all the required elements specified and in the correct format.

Jon

[James Ross]

 jonInMaine, on 03 May 2021 - 04:17 AM, said:

The next level is the data itself, the hierarchy, what is required or optional, any ordering required. For some languages e.g. XML you can define a schema which can be used to validate a file and determine if it has the required data specified, etc. I don't know if JSON has something like this.

JSON Schema is a thing which exists and has built-in support in some text editors.

 jonInMaine, on 03 May 2021 - 04:17 AM, said:

But we all know the shortcomings of STF. I think what I am saying is it is not enough just to say "we are going to use JSON". You also need to define how the data is going to be represented in the JSON elements and how one can verify that a given file is valid i.e. has all the required elements specified and in the correct format.

The structure of the data is dependent on what the file is representing; we continue to discussion this for the first format, consists.

[Laci1959]

 jonas, on 17 April 2021 - 09:47 PM, said:

Of course, from Kuju's point of view, the really more complicated files, such as a local * .tdb, should never be fiddled with with Notepad.exe. Most user know that this is the death of any route of course.

Hello

You can fidget, and unfortunately sometimes you have to mess with MSTS v. to fix a bug in TSRE track editors. Of course, this requires expertise and practice.
Oh and a dense backup.
Anyway, it was from my heart.
I can handle a smooth MSTS format smoothly, I have been working on signaling scripts for years, including the possibilities offered by Open Rails. It was true that there were good descriptions from which we could learn everything.
I didn't find anything about JSON. I tried to translate mo_fog_mi_rain_ev_clear.dat to JSON in a self-taught way, but I got stuck.

Sincerely, Laci 1959