[Genma Saotome]

gpz, on 16 October 2014 - 12:36 PM, said:

I don't know if this conversion can be done on the fly. Previously the idea was to convert the routes prior of editing, then edit the new structure only.

Ok then, IF, by whatever means, all of the data in the KUJU files, as written by AE and RE, are processed by the new OR editors and written into the new OR file formats, then there is no "Big Fork".

That means everything relevant in KUJU's .act, .srv, .trf, .tdb, .rdb. .w, .t, .tit, .rit, .trk, *.dat, and *.haz files will be read by the new OR editors and their their content, edited or original, will be written to equivalent OR files. A lot of work but if that's the promise, good. :sweatingbullets:

[gpz]

Okay, but even if some files become editable, like e.g. the .w file in preliminar route editor, it is very early to speak about any Big Split, since in that case we still are talking about code that not even exists yet. So the current code will in neither case be split. And it is also ununderstandable to me why any split would be needed in the far future, since the program can easily distinguish between the route formats it is working on...

[Genma Saotome]

gpz, on 16 October 2014 - 01:02 PM, said:

And it is also ununderstandable to me why any split would be needed in the far future, since the program can easily distinguish between the route formats it is working on...

It's not really a technical issue but whether the development team is willing to take on the tasks necessary to deal with both old KUJU stuff and new OR equivalents. So long as the team remains willing, no fork. But should that change and there is a decision to ignore any hard-to-deal-with KUJU files (e.g., any tough ones like the .tdb) it would mean old routes get preserved in amber, just like the KUJU software... while new, OR specific routes, are alive by virtue of being fully editable by new, reliable code.

Am curious as to what James will say when he reads this thread tomorrow....

[Lindsayts]

Genma Saotome, on 16 October 2014 - 12:26 PM, said:

Ok... perhaps I've missed an option... are you saying you expect the new OR editors, one for Activities, one for the World, will be able to read KUJU files, optionally edit whatever is there, and write a fully functional corresponding file in whatever the new OR file format is? IOW these new editors will convert all existing data from the KUJU files that were created by AE and RE to whatever new files are native to the OR Editors?


If that is what you meant then, yeah, there is no "Big Fork" because all of the routes and their activities that people like are convertable to native OR files.

Is that really what you are saying?

Or currently reads most of the MSTS files very well, in fact better than MSTS ever did. "Native" OR files should only be used when it is clearly required for a new facility, an obvious example being more detailed terrain.
Changing file formats just to have new formats for OR use should ONLY be done if there is a clear need otherwise there is a great deal of work required for no real reason.

When I mentioned do not worry about backward compatibilty is, read and write the files as they were intended to be read, not aparently as MSTS does as make up for all sorts of little quirks in MSTS.

Lindsay

[Lindsayts]

Genma Saotome, on 16 October 2014 - 01:22 PM, said:

It's not really a technical issue but whether the development team is willing to take on the tasks necessary to deal with both old KUJU stuff and new OR equivalents. So long as the team remains willing, no fork. But should that change and there is a decision to ignore any hard-to-deal-with KUJU files (e.g., any tough ones like the .tdb) ...........

I cannot imagine any circumstances when this will occur. OR does a far better job of reading MSTS's files than MSTS ever did.

At track description file (Track data base file) will never be simple. One improvement would be the ability to be able to split it into smaller completely self contained sections. This would stop on error trashing the whole route.

Lindsay

[James Ross]

Genma Saotome, on 16 October 2014 - 01:22 PM, said:

Am curious as to what James will say when he reads this thread tomorrow....

So, my understanding of what the Open Rails project was planning to do is to:

• Be able to play MSTS file formats.
  
• Be able to play and edit OR file formats.
  
• Be able to convert some/most MSTS file formats to their OR equivalents (as a one-time conversion process).

This lack of editing of MSTS files is one of my motivations for thinking about and proposing various things regarding the new structures and possible file formats; if we're not going to edit MSTS files, we need OR files before OR editors.

It is clear here and now that people are actually thinking about editing MSTS files, or at least having editors that can load MSTS files. I think we need to be careful here. Doing an editor, for anything, is a big, complex task, and if you have to support not just loading and saving two different file formats, but exposing all the features and limitations of those formats, you're going to come unstuck IMHO.

What may work, however, is to start with loading just a small number of MSTS file formats and providing basic editing functions and the bare minimum of saving to MSTS formats, purely as a testbed. The purpose of this would be to get a handle on how we want editing to work, with the understanding that the final version will only load and save OR file formats. I believe it has been suggested before that simply being able to move/rotate scenery in world files would be a good choice here; I tend to agree, if we go this route. In that case, the editor would load and display terrain and scenery, allow moving and rotating scenery, and saving of just world files with the altered position/rotation. (Since I can see that being useful for MSTS content fixes too, we can keep the testbed version around after we've got the full OR version.)

Lindsayts, on 16 October 2014 - 01:34 PM, said:

Or currently reads most of the MSTS files very well, in fact better than MSTS ever did. "Native" OR files should only be used when it is clearly required for a new facility, an obvious example being more detailed terrain.
Changing file formats just to have new formats for OR use should ONLY be done if there is a clear need otherwise there is a great deal of work required for no real reason.

Amusingly, to me anyway, we don't actually need new file formats for higher resolution terrain (or smaller tiles), as they have all the information to support that as parameters in the files themselves IIRC.

I would disagree with the principal here though, since apart from audio and texture files, everything from MSTS that OR loads is a Simis (STF/SBR) file - while the structure looks quite well-defined (blocks with strings, numbers, units, etc.), parsing both versions (text and binary) is a bit of a pain. We still have a number of actual bugs parsing these formats. I would hate to have this format make its way (in either form) in to an OR structure and OR-specific content. I would consider that a failure on our part.

[James Ross]

Lindsayts, on 16 October 2014 - 12:37 PM, said:

One way to start this would be to have a section on say this forum where anyone could freely post an explanation of some kind on a definition in one of the formats. That way a repository of knowledge would gradually be built until a decent doc could easily be writen.

I would think the wiki is a better place for this, so that many people can contribute changes and we can track edits. Alternatively, we can write file format documentation in to the source code and/or Documentation folder (not sure the latter works well for editing given people keep using PDF though).

[Genma Saotome]

James Ross, on 16 October 2014 - 02:13 PM, said:

So, my understanding of what the Open Rails project was planning to do is to:

• Be able to play MSTS file formats.
  
• Be able to play and edit OR file formats.
  
• Be able to convert some/most MSTS file formats to their OR equivalents (as a one-time conversion process).

Highlights above are mine. What is quoted above was my understanding too, tho I was aware I might have misremembered. Anyway... the difference between play (as James says) MSTS files but play and edit OR files is what I thought would create "The Big Fork". Converting some/most MSTS files to their OR equivalent will minimize the issue but at this time we don't know which will be converted and which won't. If *.haz are not... probably no big deal. If .w are not, it's a real big deal.

Quote

This lack of editing of MSTS files is one of my motivations for thinking about and proposing various things regarding the new structures and possible file formats; if we're not going to edit MSTS files, we need OR files before OR editors.

Yes. And the original prinicple, perhaps now voided in some circumstances, was to not to mess with the contents of any MSTS file, at least not in any way that would cause a problem for that file if used in a MSTS environment. Considering what editors do that principle could force OR editors to write only to OR equivalent files. The alternative option is to toss the principle and don't worry if the files cause AE and RE to abort 'cuz hey, they're old and funky anyway. But, IMO, it has to be a team design decision rather than bumbling into something because one person felt like doing it.

Quote

It is clear here and now that people are actually thinking about editing MSTS files, or at least having editors that can load MSTS files. I think we need to be careful here. Doing an editor, for anything, is a big, complex task, and if you have to support not just loading and saving two different file formats, but exposing all the features and limitations of those formats, you're going to come unstuck IMHO.

What may work, however, is to start with loading just a small number of MSTS file formats and providing basic editing functions and the bare minimum of saving to MSTS formats, purely as a testbed. The purpose of this would be to get a handle on how we want editing to work, with the understanding that the final version will only load and save OR file formats. I believe it has been suggested before that simply being able to move/rotate scenery in world files would be a good choice here; I tend to agree, if we go this route. In that case, the editor would load and display terrain and scenery, allow moving and rotating scenery, and saving of just world files with the altered position/rotation. (Since I can see that being useful for MSTS content fixes too, we can keep the testbed version around after we've got the full OR version.)

See post #17 in this thread for one such suggestion. I agree fully with the caution James gives... dig too deep into doing a MSTS replacement RE and you open up a big can of worms. Does anybody want to take the time to train all those worms to behave properly? Much safer to keep it very limited... a testbed.

James has added another thought there -- "we can keep the testbed version around after we've got the full OR version". The implication is there is a fork... it might not be a big code fork but it could be very significant for routes because on one side are whatever MSTS files migrate over to being edited by OR programs after this fork and on the other is everything that remains MSTS ONLY... which, if we're talking about the consequences of new editors, might be all MSTS created routes... or, if there is good fortune looking down on us all, only those routes that, for whatever reason a few folks, working on their own, have chosen to continue to edit as pure MSTS routes, running in OR as is.

Quote

Amusingly, to me anyway, we don't actually need new file formats for higher resolution terrain (or smaller tiles), as they have all the information to support that as parameters in the files themselves IIRC.

I would disagree with the principal here though, since apart from audio and texture files, everything from MSTS that OR loads is a Simis (STF/SBR) file - while the structure looks quite well-defined (blocks with strings, numbers, units, etc.), parsing both versions (text and binary) is a bit of a pain. We still have a number of actual bugs parsing these formats. I would hate to have this format make its way (in either form) in to an OR structure and OR-specific content. I would consider that a failure on our part.

IIRC this topic has been kicked around many times... I believe the consensus of opinion at those times was let the programmers make technical decisions for technical reasons, allow users text editor access to the new formats when MSTS allowed it, even if the norm is for an OR program to edit them, and make a sincere but not herculean effort to migrate old data to the new files.

[Lindsayts]

James Ross, on 16 October 2014 - 02:13 PM, said:

So, my understanding of what the Open Rails project was planning to do is to:

• Be able to play MSTS file formats.
  
• Be able to play and edit OR file formats.
  
• Be able to convert some/most MSTS file formats to their OR equivalents (as a one-time conversion process).

This lack of editing of MSTS files is one of my motivations for thinking about and proposing various things regarding the new structures and possible file formats; if we're not going to edit MSTS files, we need OR files before OR editors.

It is clear here and now that people are actually thinking about editing MSTS files, or at least having editors that can load MSTS files. I think we need to be careful here. Doing an editor, for anything, is a big, complex task, and if you have to support not just loading and saving two different file formats, but exposing all the features and limitations of those formats, you're going to come unstuck IMHO.

Instead of new file formats what about any MSTS files the route editor uses is stored in a separate sub directory as is being done I believe already in OR. This would mean the files MSTS uses are not touched but the editor can still change a route. I understand the motavation to keep a route to be able to be run in MSTS but the more complex the path to the route editor is the less likley we are to see it and I do have the distinct impression quite a few OR people are afraid of the route editor.
This baffles me is there is clearly a good bit of programming talent working on OR.

This is a problem I believe the urgently needs to be solved, the better OR gets (and it is very good) the more glaring becomes the lack of any OR development tools. Note, MSTS is unsurported in the latter versions of windows, while it currently works there is no garuntee at all that this situation will continue on any future versions both full and bug fix versions. So to some extent OR is playing with fire.
Note: MSTS editor and tools runs quite OK on wine under Linux so this could always be a fall back position the problem here is a lot of OR people are hostile (to some degree) to Linux.

Lindsay

[gpz]

Genma Saotome, on 16 October 2014 - 01:22 PM, said:

It's not really a technical issue but whether the development team is willing to take on the tasks necessary to deal with both old KUJU stuff and new OR equivalents. So long as the team remains willing, no fork. But should that change and there is a decision to ignore any hard-to-deal-with KUJU files

There are two things got blended with the last quoted sentence, that must be thought over separately. KUJU files are hard to deal with only from the parsing point of view, which is a tiny bit in the whole code. But from the point of view of handling the resulting structure in the rest of the program, is not so at all. It fits perfectly, moreover this structure is the base of all further activities of the whole OR. The same resulting structure will be used for OR-specific file formats too, thus there is no point in splitting and abandoning it.

Thus dropping "backwards compatibility" wouldn't make any benefit, because from the whole program point of view these formats are not hard-to-deal-with at all. Only it must be kept in mind that parsing these files is not reliable enough, so extensive development in order to be able to write into them is pointless.