[jonas]

jonas, on 19 April 2021 - 03:34 PM, said:

... I think the argument of the amount of code for reading in is very smart, YoRyan, but it seems a bit 80s style to me when it comes to RAM limitations :-). ...

An addendum: I think that, among other things, this line in my penultimate post was a cause for anger. I apologize for that. I meant that rather jokingly and was not aware of the serious background. It wasn't on my mind that, among other things, rare manpower could be meant here and only memory limits came in to my mind as an idea - RAM memory of course - not my brain memory. Thats another question :-)

[YoRyan]

Jonas,

I should be the one apologizing, actually; I'm sorry. I didn't mean to come across as trying to shut you down or anything.

The context you may be missing (I'm not sure how much forum archaeology you've performed since you've come on board) is that we've discussed new file formats several times before, most recently a new consist format proposed by myself last year. And, to be blunt, I found it difficult to discuss such topics with non-programmers. Most people - it's only natural of them - think in terms of features, windows, buttons, and other tangible things they can interact with on their screens, while file formats are kind of abstract and easy to misunderstand. So while I was discussing a new way of storing sequences of rail vehicles, I was inundated with requests for new game operating modes and support for articulated engines - things that were tangentially related, but, I felt, were not properly within the purview of a replacement consist format.

Since then, I've felt that the way forward is to design the file formats around the eventual editor, rather than the other way around. So I'm happy to discuss file formats with you, but with the small caveat that we should not "miss the forest for the trees," as we say in US English. Let's put the user first. :)

(By the way, let me state that I vehemently agree with you that RunActivity.exe should never modify game content.)

[Csantucci]

IMHO keeping the clocks.dat file should have been the solution here. Another decision has been taken for the official OR.
It is my intention that ORNYMG will behave as follows:
- keep accepting clocks.dat and interpreting it natively, without conversion, to keep compatibility with existing content without user intervention and to avoid runtime or pre-runtime changes to content files;
- accept also the new JSON-format, because a goal of ORNYMG is that contents running on official OR should run also on ORNYMG;
- in case both files are present, consider only the file with the JSON format, again to keep compatibility with the official OR releases, and generate a warning message in the logfile stating that clocks.dat has been skipped due to the presence of the JSON file.

[xavivilla]

I will give my humble opinion.

As an end user, I don't care what format the extended files are. As an end user, what I'm interested in is starting an activity on a route, checking that I have all the elements involved in it, and executing it. At most, being able to create a paths, a consists and traverse a part of the path in explorer mode. But for all this, it never interacted with the .pat or .con files, I don't need to see them in a text editor, nor know what their structure is.

Things change when you are a content creator. So, if you are interested in the structure of the folders and the files within them. And if you have to know how these .dat files are written (in all their varieties). .ref, road and track databases, and other files related to routes, vehicles or activities. And here I see two options for the Open Rails team, either the structure of the new .json files is fully documented or specific editors are provided for each type of file in which the content creators enter the values ​​and these editors generate the files .json. Although the ideal would be both, good documentation and editors.

Regarding the controversy of the clocks.dat (STF) or animated.clocks-or (JSON) file, my opinion is that the route creator, at the time of publishing the route, should do the conversion and thus the end user would have nothing to do.

[Laci1959]

In Hungary, only one track, the "Alföld 7.3", is more than 2,000 kilometers today. They are trying to follow the renovation that is taking place in reality. This means 3-4 platform hours per station. So far we’ve been thinking about working, but now I’m very insecure. It may not be because we do not know the JSON language. There is no programmer among us

[cjakeman]

xavivilla, on 20 April 2021 - 01:41 AM, said:

Regarding the controversy of the clocks.dat (STF) or animated.clocks-or (JSON) file, my opinion is that the route creator, at the time of publishing the route, should do the conversion and thus the end user would have nothing to do.

Thanks, Rui. That is the intention of my proposal.

[YoRyan]

But, Chris, we have a dialog box that displays when you launch Open Rails and have a clocks.dat file in the current route. That's placing the burden upon the player, not the creator.

[jonas]

Hi,

I like Carlos suggestion above.

As far as the JSON format is concerned, it seems that I stung too naively into a wasp's nest, although I tried to read and overlook the relevant threads of the past. So I'm sorry again to everyone for the dust I raised when fundamentally question the JSON format.

My now modest opinion is that I would like to ask again to reflect on my categories of one-way and dual-way external files that I have brought into play. clock.dat and animated.clocks-or are one-way files in this sense. They pass on data from the user to the application, not the other way around.
It looks different if there will be an editor for the clocks. The whole question of the file format will fade away automatically. If an appropriate editor can be used, nobody will try to make a manual change (e.g. with Notepad.exe or similar) - neither in the clocks.dat nor in the animated.clocks-or.
In the case of dual-way external files, which are written by the machine and read by the machine, only the developers should decide which file format should be used - of course!
In any case, the aim is for all external files to become dual-way files in the future if I see this right.

But until the appropriate editors exist, the user is expected to write the content of clocks or turntables into the files by hand. And at least as long as we already have STF readers for such files, I would want to accommodate the user with the old STF format.

@Chris, I don't think we would violate the current Open Rails Policy in this way. At least as long as the "ExtClocksFile.cs" is still "on board". Is it still in use?

@YoRyan, my last thought here doesn't solve the manpower problem in general, I can see that. When I think about it, I realize that instead of coding the "ExtClocksFile.cs" (an adaptation of the "ExtCarSpawnerFiles.cs") ), I should rather have dealt with coding an editor for the clocks.dat / animated.clocks-or. But to be honest, that far exceeds my skills in C# :-). Here we have the lack of manpower - again, you are right about this.

Greetings
Jonas

[YoRyan]

I don't quite understand our collective obsession with STF, because it's just not a very good format for writing files by hand, either. Data types are ambiguous (is that a string or a number?), and there's no schema system to give you fancy autocompletion assistance, unlike with JSON. What about timetable files? They're not STF. Microsoft Train Simulator was a half-finished product rushed to market, not the pinnacle of software engineering.

I don't feel too strongly about this issue, so I'm not going to block the pull request over the presence of a clocks.dat STF reader, but I'm just not seeing the extenuating circumstance to keep it around, considering that in all likelihood only one clock file currently exists in the world.

Chris intends to make automatic converters from .eng, .wag, and so forth to JSON, the "dual-way" concept that you've discussed. Personally, I'm not convinced of the need for them at all. I'd rather have editors instead.

[Laci1959]

YoRyan, on 20 April 2021 - 07:50 AM, said:

But, Chris, we have a dialog box that displays when you launch Open Rails and have a clocks.dat file in the current route. That's placing the burden upon the player, not the creator.

Even if you no longer have a clocks.dat file because the track creator deleted it after the conversion? Even before release?

[roeter]

jonas, on 20 April 2021 - 10:32 AM, said:

It looks different if there will be an editor for the clocks. The whole question of the file format will fade away automatically. If an appropriate editor can be used, nobody will try to make a manual change (e.g. with Notepad.exe or similar) - neither in the clocks.dat nor in the animated.clocks-or.

Wrong. If files are readable (i.e. not binary), there will always be users who will edit the file directly, whether there is an editor or not.
Users are now editing .con files, .w files and even the .tdb if necessary, even though these are (or can) be created through editors.
There are always two groups of people - those with no or limited knowledge of the data structure who therefor are happy to use the editor. And those who have that knowledge and just find editors cumbersome or too restricted, especially when things go wrong. That's how it always has been, that's how it always will be.

Quote

In any case, the aim is for all external files to become dual-way files in the future if I see this right.

Well, even if there is such an aim (which I doubt), given the sparce development capacity it's just never going to happen. There will never be editors for all types of data files presently used in OR, and as long as new data is added without its own editor (like, in fact, turntables and clock) that problem is only growing.

Quote

But until the appropriate editors exist, the user is expected to write the content of clocks or turntables into the files by hand. And at least as long as we already have STF readers for such files, I would want to accommodate the user with the old STF format.

With that, you have just landed JSON as file format for OR onto the dustheap. Definitely. Forever.
For the next person to come along with new data can, of course, use this same argument. And the next. And the next. And ..... (ad infinitum).

In fact, the clock data is an ideal candidate for conversion to JSON, for the following reasons :

• It is a relatively small and simple file.
  
• It has not yet been spread in any large numbers.
  
• It is not aimed at actual users but at route builders, which is only a small subset of the total number of users.
  
• There is already a conversion available so it will take not much work to introduce it as JSON.

The decision to set JSON as standard for new data has been made some time ago. Of course there is no 100% consensus among all users or even developers for that decision, which such a varied group there never can be. But the decision has been made and ignoring it is not going to help to improve the overall OR data structure.

You started this thread by pointing out the hotch potch of data used in OR. There is no magic wand to change that overnight. Improving the data consistency can only be done in small steps. The first of those steps was the decision to set JSON as standard. The second step was the introduction of the first JSON data - the wheather files. You can make that very important third step. By insisting on your arguments to stick to STF you provide loads of ammunition to other developers to keep on using their own preferenced data formats, thus not only maintaining that hotch potch but even making it worse.
Or, you can accept the conversion to JSON and with that take away that ammunition. For then we have two files in JSON, so when other new data is introduced, we can say : "look, all new data has now been introduced as JSON so please follow suit."
So, you are actually holding the key for improving the OR data structure. The choice is yours.

Regards,
Rob Roeterdink

[jonas]

YoRyan, on 20 April 2021 - 11:29 AM, said:

I don't quite understand our collective obsession with STF, because it's just not a very good format for writing files by hand, either.

As far as I'm concerned, my obsession is only about the time span until there is an editor for the clocks. Thats all.

The users are used to use STF for 20 years and in the case of the animated clocks they (and not a route builder or the developer or OR as a mashine) have to write the clocks.dat. Because there is no editor for it yet.

However, this can also be generalized until there are editors for, for example: the carspawn.dat, the gantry.dat, speedpost.dat, ssource.dat, turntables.dat etc. and, above all, files like the *.ref.

I don't know what it's like elsewhere, but in Germany there is a vital scene within the remaining hard core of the MSTS/OR Simmer, which deals with the global abundance of shapes, routes and other virtual material. At least that's my impression. In the downloads packages there are a large number of introductions what you have to change or set where in which file (STF) so that you can get one or the other download to run. Downloads such as can be obtained from www.trainsim.com you know.

As I said, STF only as long as there is no editor for the respective file. As soon as there is an editor for a file type, the issue about the formats disappears on its own. Nobody of the users will then be interessted what format is used in a file.

YoRyan, on 20 April 2021 - 11:29 AM, said:

...I'd rather have editors instead.

Me too.

[YoRyan]

jonas, on 20 April 2021 - 02:32 PM, said:

The users are used to use STF for 20 years and in the case of the animated clocks they (and not a route builder or the developer or OR as a mashine) have to write the clocks.dat.

But users don't have to write their own clocks.dat files; only route builders do. And at the moment, you are the only route builder taking advantage of this feature. As Rob wrote, this is why you are in an unusually powerful position as the developer of this new clocks file.

Look, your argument boils down to "MSTS used STF files, and we have 20 years of practice using STF, so we need to use keep using STF until we can muster the resources to build editors for every single STF file." Wasn't it you who wrote

jonas, on 19 April 2021 - 06:57 PM, said:

Nobody, including myself, demands that such files you describe be written in STF now and in the future.

?

As a developer, I don't see why I should be obligated to copy whatever MSTS did. Indeed, when writing simulation software, a good rule of thumb is to do the exact opposite of whatever MSTS did. If we ever have our own world editor, who knows if carspawn.dat, gantry.dat, ssource.dat, and friends will even have 1:1 equivalents in Open Rails? We will almost certainly end up designing a radically different file structure that would be more powerful and easier to work with.

[jonas]

Hi,

I don’t know where to start. I give up on answering directly. In my posts I meanwhile repeat myself constantly.

I feel like in a religious war in which psalm-like slogans like "That's how it always has been, that's how it always will be." are called.
And the top maxim is:

"We, OR, must finally stay true to our self-imposed policy, new files may only be consistently allowed in JSON format!"

The terms of war rhetoric such as "loads of ammunition" fit in with this. As I said before, I seem to have stung a wasp's nest and this does not pleases me and wasn't my intension. And in this thread, to stay with the war picture, I'm not longer motivated to fight in this ditch. I wanted to understand, discuss, learn and bring up what I think was a small aspect, that of the one-way external files. Obviously, because of my ignorance, I went too far with exaggerated format suggestions.

This is, how I see it:
• I have already withdrawn the hybrid format suggestion here
• Personally, I have nothing against the JSON format as a dual-way file
• Developers should always use the JSON Format preferred by OR in dual-way files
• I hope OR will be more and more free of the one-way files with the help of editors in the future
• As far as I am concerned personal, there is no need to use the clocks.dat (STF)
• It is not about my route or how I can sell it better
• I am not trying to exploit my alleged position of power
• The OR code of the clocks can be changed until it is copyright unrecognizable
• I'm not not keen on be mentioned by name in any form
• of course, I do not think that the weaknesses of MSTS should be copied
• I hope OR remains compatible with MSTS
• I am not attacking the use of the Weather files in JSON format
and
• I think users will not like the JSON format in one-way files

The user I'm talking about is neither an only-activities-player nor a route builder-crack. My experience among the users of my route (the above 80 people) is that there is a wide range of users, including those who would manipulate the clocks.dat because they want on their favorite route (not mine) just have to try a different clock. It's just like with model railroaders, everyone tries to individualize their railway plate, no matter how much technical understanding they have. One repainted house shapes, the other modified vegetation shapes, the third tries to exchange only some signals in a place and a fourth has no idea about eng and wag files, but is an absolute specialist in changing the coupling distance in all of his eng and wag files exactly to his taste - and that by hand without inadvertently influencing the adjacent and not understanded values in the eng or wag!
And these users now have to make their changes in JSON formatted files for which there don't have editors. Editors that we have been talking about for years and that have always been a main argument for the introduction of JSON in OR.

However, these editors do not yet exist. In the war of religion they are always a paradise prospect only! This is argued constantly, so that even now (animated.clocks-or) the users are first forced to adopt a format which they can later forget again if there are editors available.

OR politics is made here, and as usual in wars, the victims are those who have no audible voice on the theater of war - the users described above. In some of the posts here it is fundamentally doubted whether such users even exist. But believe me, the user who is so one-dimensional that he only drives activities is rather the exception among the already smaller group of TS Simmers.

I think it is our chauvinism to assume that there is mostly only this "one-dimensional" user. The spectrum is broader and more colorful.

I look forward to the time of the editors - without any irony. Unfortunately, I can't give any additional help when coding editors. My skills are average in C# and foremost I have no good idea of the architecture of the hole OR code.

I learned continuously during this thread and believe that I have argued reasonably objectively in the posts here according to my each time current level of knowledge. I hope it has in some way contributed to the increase in knowledge of other readers here. If I insulted someone directly or indirectly, it was not my intention.
In any case, I am not offended, but tired of repeating the same thought in further posts here again and again.

Many Greetings
Jonas


One thing as an edit: I was never really asked about the question of whether clocks.dat (STF) or animated.clocks-or (JSON) when transferring to the official OR version, I was only informed! It was about an offer to the user to create the animated.clocks.dat in JSON, not about two dialog boxes when the activity starts.
My manpower for the ExtClocksFile.cs, which reads the clocks.dat (STF), played no role.
So I can't see any "unusually powerful position [of mine] as the developer of this new clocks file". Please take this into account.

[Genma Saotome]

YoRyan, on 20 April 2021 - 03:01 PM, said:

As a developer, I don't see why I should be obligated to copy whatever MSTS did. Indeed, when writing simulation software, a good rule of thumb is to do the exact opposite of whatever MSTS did. If we ever have our own world editor, who knows if carspawn.dat, gantry.dat, ssource.dat, and friends will even have 1:1 equivalents in Open Rails? We will almost certainly end up designing a radically different file structure that would be more powerful and easier to work with.

I have a long list of complaints about what KUJU did and didn't do w/ MSTS -- the stf format being one.

IMO .json looks worse tho but not bothering w/ one fact per line will help that problem a lot.

My major concern is (1) a comment from Wayne that in his testing .json was about 10X slower in getting thru the loader and (2) not forcing users to to their own conversions from stf. Let's (please) be sure there are no unforeseen consequences going forward w/ .json.

Q: Has anyone bothered to verify it's around 10x slower?
Q: Does it improve w/ compression?

I assume in many cases 10x slower is not going to have any negative effect but it prompts the question of what happens to t6he user experience when the loader thread hits 100%? 10km near view distance is quite different that 2km. World files w/ 32k lines is quite different from 2k lines. A .tdb w/ 42k lines?

Q: Any one know what happens to fps when the loader thread hits 100%?

I would hope not much since it's a different thread but best to know and not assume.


Q: Has anyone written a stf to .json conversion program?

It might be useful to pick a file type, write a converter, and let people test it.