Elvas Tower: Route extending Open Rails files - Elvas Tower

Jump to content

  • 13 Pages +
  • « First
  • 4
  • 5
  • 6
  • 7
  • 8
  • Last »
  • You cannot start a new topic
  • You cannot reply to this topic

Route extending Open Rails files Rate Topic: -----

#51 User is offline   roeter 

  • Vice President
  • Group: Status: Elite Member
  • Posts: 2,420
  • Joined: 25-October 11
  • Gender:Male
  • Country:

Posted 20 April 2021 - 01:54 PM

 jonas, on 20 April 2021 - 10:32 AM, said:

It looks different if there will be an editor for the clocks. The whole question of the file format will fade away automatically. If an appropriate editor can be used, nobody will try to make a manual change (e.g. with Notepad.exe or similar) - neither in the clocks.dat nor in the animated.clocks-or.

Wrong. If files are readable (i.e. not binary), there will always be users who will edit the file directly, whether there is an editor or not.
Users are now editing .con files, .w files and even the .tdb if necessary, even though these are (or can) be created through editors.
There are always two groups of people - those with no or limited knowledge of the data structure who therefor are happy to use the editor. And those who have that knowledge and just find editors cumbersome or too restricted, especially when things go wrong. That's how it always has been, that's how it always will be.

Quote

In any case, the aim is for all external files to become dual-way files in the future if I see this right.

Well, even if there is such an aim (which I doubt), given the sparce development capacity it's just never going to happen. There will never be editors for all types of data files presently used in OR, and as long as new data is added without its own editor (like, in fact, turntables and clock) that problem is only growing.

Quote

But until the appropriate editors exist, the user is expected to write the content of clocks or turntables into the files by hand. And at least as long as we already have STF readers for such files, I would want to accommodate the user with the old STF format.


With that, you have just landed JSON as file format for OR onto the dustheap. Definitely. Forever.
For the next person to come along with new data can, of course, use this same argument. And the next. And the next. And ..... (ad infinitum).

In fact, the clock data is an ideal candidate for conversion to JSON, for the following reasons :
  • It is a relatively small and simple file.
  • It has not yet been spread in any large numbers.
  • It is not aimed at actual users but at route builders, which is only a small subset of the total number of users.
  • There is already a conversion available so it will take not much work to introduce it as JSON.


The decision to set JSON as standard for new data has been made some time ago. Of course there is no 100% consensus among all users or even developers for that decision, which such a varied group there never can be. But the decision has been made and ignoring it is not going to help to improve the overall OR data structure.

You started this thread by pointing out the hotch potch of data used in OR. There is no magic wand to change that overnight. Improving the data consistency can only be done in small steps. The first of those steps was the decision to set JSON as standard. The second step was the introduction of the first JSON data - the wheather files. You can make that very important third step. By insisting on your arguments to stick to STF you provide loads of ammunition to other developers to keep on using their own preferenced data formats, thus not only maintaining that hotch potch but even making it worse.
Or, you can accept the conversion to JSON and with that take away that ammunition. For then we have two files in JSON, so when other new data is introduced, we can say : "look, all new data has now been introduced as JSON so please follow suit."
So, you are actually holding the key for improving the OR data structure. The choice is yours.

Regards,
Rob Roeterdink

#52 User is offline   jonas 

  • Engineer
  • Group: Status: Contributing Member
  • Posts: 548
  • Joined: 04-April 14
  • Gender:Male
  • Simulator:MSTS & OR
  • Country:

Posted 20 April 2021 - 02:32 PM

 YoRyan, on 20 April 2021 - 11:29 AM, said:

I don't quite understand our collective obsession with STF, because it's just not a very good format for writing files by hand, either.
As far as I'm concerned, my obsession is only about the time span until there is an editor for the clocks. Thats all.

The users are used to use STF for 20 years and in the case of the animated clocks they (and not a route builder or the developer or OR as a mashine) have to write the clocks.dat. Because there is no editor for it yet.

However, this can also be generalized until there are editors for, for example: the carspawn.dat, the gantry.dat, speedpost.dat, ssource.dat, turntables.dat etc. and, above all, files like the *.ref.

I don't know what it's like elsewhere, but in Germany there is a vital scene within the remaining hard core of the MSTS/OR Simmer, which deals with the global abundance of shapes, routes and other virtual material. At least that's my impression. In the downloads packages there are a large number of introductions what you have to change or set where in which file (STF) so that you can get one or the other download to run. Downloads such as can be obtained from www.trainsim.com you know.

As I said, STF only as long as there is no editor for the respective file. As soon as there is an editor for a file type, the issue about the formats disappears on its own. Nobody of the users will then be interessted what format is used in a file.

 YoRyan, on 20 April 2021 - 11:29 AM, said:

...I'd rather have editors instead.
Me too.

#53 User is offline   YoRyan 

  • Conductor
  • Group: Status: Active Member
  • Posts: 391
  • Joined: 19-February 20
  • Gender:Male
  • Location:California, United States
  • Simulator:Open Rails/unstable
  • Country:

Posted 20 April 2021 - 03:01 PM

 jonas, on 20 April 2021 - 02:32 PM, said:

The users are used to use STF for 20 years and in the case of the animated clocks they (and not a route builder or the developer or OR as a mashine) have to write the clocks.dat.

But users don't have to write their own clocks.dat files; only route builders do. And at the moment, you are the only route builder taking advantage of this feature. As Rob wrote, this is why you are in an unusually powerful position as the developer of this new clocks file.

Look, your argument boils down to "MSTS used STF files, and we have 20 years of practice using STF, so we need to use keep using STF until we can muster the resources to build editors for every single STF file." Wasn't it you who wrote

 jonas, on 19 April 2021 - 06:57 PM, said:

Nobody, including myself, demands that such files you describe be written in STF now and in the future.

?

As a developer, I don't see why I should be obligated to copy whatever MSTS did. Indeed, when writing simulation software, a good rule of thumb is to do the exact opposite of whatever MSTS did. If we ever have our own world editor, who knows if carspawn.dat, gantry.dat, ssource.dat, and friends will even have 1:1 equivalents in Open Rails? We will almost certainly end up designing a radically different file structure that would be more powerful and easier to work with.

#54 User is offline   jonas 

  • Engineer
  • Group: Status: Contributing Member
  • Posts: 548
  • Joined: 04-April 14
  • Gender:Male
  • Simulator:MSTS & OR
  • Country:

Posted 20 April 2021 - 06:11 PM

Hi,

I don’t know where to start. I give up on answering directly. In my posts I meanwhile repeat myself constantly.

I feel like in a religious war in which psalm-like slogans like "That's how it always has been, that's how it always will be." are called.
And the top maxim is:

"We, OR, must finally stay true to our self-imposed policy, new files may only be consistently allowed in JSON format!"

The terms of war rhetoric such as "loads of ammunition" fit in with this. As I said before, I seem to have stung a wasp's nest and this does not pleases me and wasn't my intension. And in this thread, to stay with the war picture, I'm not longer motivated to fight in this ditch. I wanted to understand, discuss, learn and bring up what I think was a small aspect, that of the one-way external files. Obviously, because of my ignorance, I went too far with exaggerated format suggestions.

This is, how I see it:
• I have already withdrawn the hybrid format suggestion here
• Personally, I have nothing against the JSON format as a dual-way file
• Developers should always use the JSON Format preferred by OR in dual-way files
• I hope OR will be more and more free of the one-way files with the help of editors in the future
• As far as I am concerned personal, there is no need to use the clocks.dat (STF)
• It is not about my route or how I can sell it better
• I am not trying to exploit my alleged position of power
• The OR code of the clocks can be changed until it is copyright unrecognizable
• I'm not not keen on be mentioned by name in any form
• of course, I do not think that the weaknesses of MSTS should be copied
• I hope OR remains compatible with MSTS
• I am not attacking the use of the Weather files in JSON format
and
• I think users will not like the JSON format in one-way files

The user I'm talking about is neither an only-activities-player nor a route builder-crack. My experience among the users of my route (the above 80 people) is that there is a wide range of users, including those who would manipulate the clocks.dat because they want on their favorite route (not mine) just have to try a different clock. It's just like with model railroaders, everyone tries to individualize their railway plate, no matter how much technical understanding they have. One repainted house shapes, the other modified vegetation shapes, the third tries to exchange only some signals in a place and a fourth has no idea about eng and wag files, but is an absolute specialist in changing the coupling distance in all of his eng and wag files exactly to his taste - and that by hand without inadvertently influencing the adjacent and not understanded values in the eng or wag!
And these users now have to make their changes in JSON formatted files for which there don't have editors. Editors that we have been talking about for years and that have always been a main argument for the introduction of JSON in OR.

However, these editors do not yet exist. In the war of religion they are always a paradise prospect only! This is argued constantly, so that even now (animated.clocks-or) the users are first forced to adopt a format which they can later forget again if there are editors available.

OR politics is made here, and as usual in wars, the victims are those who have no audible voice on the theater of war - the users described above. In some of the posts here it is fundamentally doubted whether such users even exist. But believe me, the user who is so one-dimensional that he only drives activities is rather the exception among the already smaller group of TS Simmers.

I think it is our chauvinism to assume that there is mostly only this "one-dimensional" user. The spectrum is broader and more colorful.

I look forward to the time of the editors - without any irony. Unfortunately, I can't give any additional help when coding editors. My skills are average in C# and foremost I have no good idea of the architecture of the hole OR code.

I learned continuously during this thread and believe that I have argued reasonably objectively in the posts here according to my each time current level of knowledge. I hope it has in some way contributed to the increase in knowledge of other readers here. If I insulted someone directly or indirectly, it was not my intention.
In any case, I am not offended, but tired of repeating the same thought in further posts here again and again.

Many Greetings
Jonas


One thing as an edit: I was never really asked about the question of whether clocks.dat (STF) or animated.clocks-or (JSON) when transferring to the official OR version, I was only informed! It was about an offer to the user to create the animated.clocks.dat in JSON, not about two dialog boxes when the activity starts.
My manpower for the ExtClocksFile.cs, which reads the clocks.dat (STF), played no role.
So I can't see any "unusually powerful position [of mine] as the developer of this new clocks file". Please take this into account.

#55 User is online   Genma Saotome 

  • Owner Emeritus and Admin
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 15,308
  • Joined: 11-January 04
  • Gender:Male
  • Location:United States
  • Simulator:Open Rails
  • Country:

Posted 20 April 2021 - 07:56 PM

 YoRyan, on 20 April 2021 - 03:01 PM, said:

As a developer, I don't see why I should be obligated to copy whatever MSTS did. Indeed, when writing simulation software, a good rule of thumb is to do the exact opposite of whatever MSTS did. If we ever have our own world editor, who knows if carspawn.dat, gantry.dat, ssource.dat, and friends will even have 1:1 equivalents in Open Rails? We will almost certainly end up designing a radically different file structure that would be more powerful and easier to work with.


I have a long list of complaints about what KUJU did and didn't do w/ MSTS -- the stf format being one.

IMO .json looks worse tho but not bothering w/ one fact per line will help that problem a lot.

My major concern is (1) a comment from Wayne that in his testing .json was about 10X slower in getting thru the loader and (2) not forcing users to to their own conversions from stf. Let's (please) be sure there are no unforeseen consequences going forward w/ .json.

Q: Has anyone bothered to verify it's around 10x slower?
Q: Does it improve w/ compression?

I assume in many cases 10x slower is not going to have any negative effect but it prompts the question of what happens to t6he user experience when the loader thread hits 100%? 10km near view distance is quite different that 2km. World files w/ 32k lines is quite different from 2k lines. A .tdb w/ 42k lines?

Q: Any one know what happens to fps when the loader thread hits 100%?

I would hope not much since it's a different thread but best to know and not assume.


Q: Has anyone written a stf to .json conversion program?

It might be useful to pick a file type, write a converter, and let people test it.

#56 User is offline   YoRyan 

  • Conductor
  • Group: Status: Active Member
  • Posts: 391
  • Joined: 19-February 20
  • Gender:Male
  • Location:California, United States
  • Simulator:Open Rails/unstable
  • Country:

Posted 20 April 2021 - 09:59 PM

 Genma Saotome, on 20 April 2021 - 07:56 PM, said:

Q: Has anyone bothered to verify it's around 10x slower?
Q: Does it improve w/ compression?

A 10x difference sounds very dramatic and I think it's impossible to draw any conclusions without knowing the circumstances of this test. If anything, I would expect the commercial JSON loaders to blow our do-it-yourself STF parsers out of the water; fast JSON loading is a well-studied problem. That is, assuming identical data for both formats. At the moment, we do not actually have JSON counterparts to MSTS world files, so no plausible comparison is possible.

#57 User is offline   Laci1959 

  • Foreman Of Engines
  • Group: Status: Contributing Member
  • Posts: 910
  • Joined: 01-March 15
  • Gender:Male
  • Simulator:Alföld
  • Country:

Posted 20 April 2021 - 10:40 PM

 roeter, on 20 April 2021 - 01:54 PM, said:

Wrong. If files are readable (i.e. not binary), there will always be users who will edit the file directly, whether there is an editor or not.
Users are now editing .con files, .w files and even the .tdb if necessary, even though these are (or can) be created through editors.
There are always two groups of people - those with no or limited knowledge of the data structure who therefor are happy to use the editor. And those who have that knowledge and just find editors cumbersome or too restricted, especially when things go wrong. That's how it always has been, that's how it always will be.

Regards,
Rob Roeterdink


Unfortunately, there are "bugs" that the editors of the editors didn't think of. These are very easy to fix with Notepad.
A concrete example.
The consist editors use the maximum speed of the locomotive when assembling the train. The scheduled speed of a freight train is 80km / h (due to the cars lined up in the train), but the consist editor sets this to 160km / h because that is the maximum speed of the locomotive. This is great for correcting in TimeTable mode, but not in Activity mode.
In this case, the incorrect value can be easily corrected quickly.
For simpler things, an editor would really be good, and then it doesn't matter the language. This would make the job easier for many track builders.

#58 User is offline   cjakeman 

  • Vice President
  • PipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 2,857
  • Joined: 03-May 11
  • Gender:Male
  • Location:Peterborough, UK
  • Simulator:Open Rails
  • Country:

Posted 21 April 2021 - 10:44 AM

 YoRyan, on 20 April 2021 - 07:50 AM, said:

But, Chris, we have a dialog box that displays when you launch Open Rails and have a clocks.dat file in the current route. That's placing the burden upon the player, not the creator.

That's only part of the picture.


The dialog box displays when you press "Start" and have a clocks.dat in the current route and you do not also have an animated.clocks-or file in the current route.
(It also displays if the clocks.dat is more recent than the animated.clocks-or.)

If someone distributes a route containing animated.clocks-or, then the users of that distribution will never see the dialog or need to. The aim has been to give the creator a convenient conversion and not to bother the player at all.

#59 User is offline   YoRyan 

  • Conductor
  • Group: Status: Active Member
  • Posts: 391
  • Joined: 19-February 20
  • Gender:Male
  • Location:California, United States
  • Simulator:Open Rails/unstable
  • Country:

Posted 21 April 2021 - 10:58 AM

Chris, here's the scenario I'm concerned about: NewYear is more popular than the mainline branch among savvy Open Rails users and content creators. Inevitably, someone is going to distribute route content with a clocks.dat file only, because it will work in NewYear, and as this thread has demonstrated, many people simply prefer writing STF files. Any end users of this route who are using mainline Open Rails will then encounter your prompt when they start their game. So, it's not going to affect just creators.

If NewYear retains support for clock.dat, I think that largely ties our hands and forces mainline to support it, too.

#60 User is offline   cjakeman 

  • Vice President
  • PipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 2,857
  • Joined: 03-May 11
  • Gender:Male
  • Location:Peterborough, UK
  • Simulator:Open Rails
  • Country:

Posted 21 April 2021 - 01:11 PM

 YoRyan, on 21 April 2021 - 10:58 AM, said:

Chris, here's the scenario I'm concerned about

Yes, my proposal is not ideal; it's a compromise and I wish I had responded to Jonas' plans in good time rather than after the code was in place.

Considering the small number and size of files involved, I think it's a reasonable compromise.

  • 13 Pages +
  • « First
  • 4
  • 5
  • 6
  • 7
  • 8
  • Last »
  • You cannot start a new topic
  • You cannot reply to this topic

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users