Elvas Tower: New Texture format and the memory problems - Elvas Tower

Jump to content

  • 9 Pages +
  • « First
  • 2
  • 3
  • 4
  • 5
  • 6
  • Last »
  • You cannot start a new topic
  • You cannot reply to this topic

New Texture format and the memory problems Rate Topic: -----

#31 User is online   James Ross 

  • Open Rails Developer
  • Group: Status: Elite Member
  • Posts: 5,492
  • Joined: 30-June 10
  • Gender:Not Telling
  • Simulator:Open Rails
  • Country:

Posted 05 February 2014 - 04:20 AM

View Postscottb613, on 05 February 2014 - 04:07 AM, said:

Just curious - what is DDS actually buying us here ? It's a more efficient so we should see better performance over all ? It's a better quality of compression graphic ? Is this leading into supporting bump maps ?


Generally speaking, at the moment it would only buy us a more standard format that is much easier to work with than ACE files. For example, what disc has been highlighting is that many ACE files currently use uncompressed RGBA formats instead of DXT3/5 because there's a distinct lack of tools that'll create DXT3/5 ACEs (they are possible, though). Tools to create DXT3/5 DDS files are much more readily available.

The benefits of DXT formats over RGBA is that, although they are lossy (so lose some sharp quality, similar to JPEG), they are supported in their compressed form on GPUs. Think about how much smaller a JPEG is than a BMP - it's like that, but in the graphics card's memory. That means you can fit a significant amount more in.

They may bring more features in the future, like bump maps, cube maps or other things. This hasn't been investigated thoroughly at this point, but DDS is regarded as one of the best, if not the best, format to use going forward.

#32 User is offline   disc 

  • Foreman Of Engines
  • Group: Private - Open Rails Developer
  • Posts: 818
  • Joined: 07-October 12
  • Gender:Male
  • Simulator:OpenRails
  • Country:

Posted 05 February 2014 - 07:16 AM

BTW the ACE's zlib compression wasn't a bad thing. Maybe that function could be moved on DDS too (under "zdds" extension?), just to save HDD space, and to lower the read count for the slow HDD. As DXT isn't an efficient, but fast compression, zlib compression can lower the file sizes significantly. Of course this need to be decompressed when loaded, but today's cpu-s can decompress zlib around 20 gbyte/sec speed in syntetic tests, so wouln't cause much performance loss.

#33 User is offline   scottb613 

  • Vice President
  • Group: Status: First Class
  • Posts: 2,973
  • Joined: 06-July 09
  • Gender:Male
  • Location:Downeast Maine (soon)
  • Simulator:ORTS
  • Country:

Posted 05 February 2014 - 07:30 AM

Sounds good - thanks for the info !!!
;)

Regards,
Scott

#34 User is offline   Genma Saotome 

  • Owner Emeritus and Admin
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 15,362
  • Joined: 11-January 04
  • Gender:Male
  • Location:United States
  • Simulator:Open Rails
  • Country:

Posted 05 February 2014 - 08:32 AM

View Postgpz, on 05 February 2014 - 03:33 AM, said:

Dave,

I'm just wondering why we should mess with these old .sd files at all. The modifications you propose mean a complete restructuring of the .eng format, which would mean we move away from it completely. Rather than trying to fit everything into the old format, a new tech data format specification could be made.


It is true a different file format could be created -- and in several instances should be created. In the meanwhile...

The only text file that is paired to every mesh file is the .sd; I'm of the opinion KUJU chose the letters .sd to signify shape description. They used it to designate texture replacement, calling it ESD_Alternative_Texture()... choosing to use a numeric code to designate a different path instead of a different filename. And so the principle of texture replacement is already in place. What should, IMO, be different is how that principle is implemented as an expansion of what KUJU has already done: Rather than use incomprehensible numeric codes to do more put in into ordinary structured language -- SIMIS of JSON... it doesn't matter to me (whichever is technically better to use) and so to example how that could open up new features I've written a few examples... how to replace one texture format with another... how to replace one LOD value with another. Stuff each end user can do on their own. If you can accept that as a broader principle of what KUJU gave us then it is just a small step further to add brand new features to a shape that were never coded into the mesh file: Time of day for the use of night textures for example.

So let's step back for a second.... There are a number of attributes of each mesh that are fixed by its creator but are often a problem for different end users. One is the texture, another is the file type of that texture. Add, bias, LOD distance too. There isn't any place for an end user to alter those outside of uncompressing the .s file and editing it which has certain risks attached to it. That need to change values becomes the question of where should such changes occur?

Now look at what KUJU saw as issues: how the mesh file dealt with seasonal textures, how the mesh file dealt with assembly of meshes, and how the mesh file did not deal with display loads on the PC. They chose the .sd file to deal with those because the mesh file could not. So why not use the .sd file to deal with those issues where the what's in the mesh file needs to be altered? Or features that could not be put into the mesh file?

Moving that new principle over to .wags and .engs reveals other possibilities... most useful of which is normalizing the data found in .wags. I don't know about your roster but most of my .wags are less than 100 lines line (no lights). I scroll down to around line 10, where IntertialTensor() is located and delete all of the remain lines. ALL of them. And then I plug in replacements: The same Coupler()data. The same Buffer() data. The same Brake() data. Friction & derailment data computed on total weight. New Name().

IOW 80-90% of what goes in is the same from one .wag to another. Have any idea of how many times ALL of that data got tweeked over the years as a better understanding of real world performance came to be expressed in MSTS terms? Loads of times. And EVERYTIME it did that work I resented the fact there were not a handful of standard reference files that contain that data in just one place: One coupler file to update, not hundreds.

Take one step forward: If there were those standard reference files... what's left of the .wag? Hardly anything. So little in fact what remains could have been put into the .sd of the car file. IOW it's quite plausible KUJU flattened the normalized data into a single file -- the .eng file because they had not anticipated the huge growth of content, the huge frequency of reskinning, or the huge amount of corrections to their original data that would occur. If they had known what would happen (and let's not be too harsh, crystal balls aren't too easy to find) they probably would have left the data normalized and there would not be a .wag file. Plausibly, no .eng either. But now there is a E_Type_Coupler.sd and a D_Type_Coupler.sd. Why not add .s file to go with them? Have to figure out how to do an assembly but that shouldn't be too hard to do. Right?

But that's not what we've got and so the question should be this: Is the KUJU design the go forward solution or are there good enough reasons to re-think it? For me, the answer is obvious: OF COURSE it should be re-thought because it opens up huge avenues of opportunity (equally obvious is that is should not released before 1.0).

Is the objective to do MSTS and let it go at that? Or finish MSTS compatibility and move on to Open Rails? If the later, then ignore all of my ideas (which AFAIK is the only result that has occurred). If the later... are there aspects that can be useful to do before 1.0? Yes: Texture replacement, LOD replacement, Time of day for using night textures... all of those can be implemented at any time -- and the ideas were proposed several years ago -- w/o interfering in any way w/ MSTS compatibility.

And so yes, the .sd files, an appendix to KUJU's work, are indeed important. The final OR version need not be called .sd but there should be one user editable "text-like" file that goes with each mesh file that allows people to enhance, alter, or turn off certain features encoded in the mesh. As for .wag and .eng files... if you look at how much of them is really in the mesh (e.g., wheel diameter), implied by the mesh (e.g., boiler capacity), or too application specific to put into the mesh (e.g., what to do with lights when a penalty occurs)... what are they but great big .sd files passing themselves off under another name?

#35 User is online   James Ross 

  • Open Rails Developer
  • Group: Status: Elite Member
  • Posts: 5,492
  • Joined: 30-June 10
  • Gender:Not Telling
  • Simulator:Open Rails
  • Country:

Posted 05 February 2014 - 09:07 AM

I feel this is getting a bit off-topic in this thread but I'll continue...

View PostGenma Saotome, on 05 February 2014 - 08:32 AM, said:

Is the objective to do MSTS and let it go at that? Or finish MSTS compatibility and move on to Open Rails? If the later, then ignore all of my ideas (which AFAIK is the only result that has occurred). If the later... are there aspects that can be useful to do before 1.0? Yes: Texture replacement, LOD replacement, Time of day for using night textures... all of those can be implemented at any time -- and the ideas were proposed several years ago -- w/o interfering in any way w/ MSTS compatibility.


All of your ideas on data and file structuring are (and will be) vital to Open Rails's own file formats. I am certain we'll have couplers in a separate file, for example. However, I am much more apprehensive about extensions to the MSTS file formats; even if they've been found to be 'safe' (in that MSTS doesn't choke) I would much prefer we put the effort into figuring out the new file formats than effectively retro-fitting features. There hasn't been much movement on this front recently, though, but perhaps we can kick-start that when 1.0 is released.

#36 User is offline   Genma Saotome 

  • Owner Emeritus and Admin
  • PipPipPipPipPipPipPipPipPipPipPipPipPip
  • Group: ET Admin
  • Posts: 15,362
  • Joined: 11-January 04
  • Gender:Male
  • Location:United States
  • Simulator:Open Rails
  • Country:

Posted 05 February 2014 - 10:52 AM

View PostJames Ross, on 05 February 2014 - 09:07 AM, said:

I feel this is getting a bit off-topic in this thread but I'll continue...



However, I am much more apprehensive about extensions to the MSTS file formats; even if they've been found to be 'safe' (in that MSTS doesn't choke) I would much prefer we put the effort into figuring out the new file formats than effectively retro-fitting features. There hasn't been much movement on this front recently, though, but perhaps we can kick-start that when 1.0 is released.


Concerns that certainly hasn't applied to the .eng file -- for content or timing.

Look, I am delighted the .dds format will finally be implemented. It was suggested oh, I dunno, maybe in the first 2 weeks of the project. How many years ago was that? It's a new feature, right? Well... to implement it one way or the other seems to be entirely dependent on a flip of a coin than any principle or design. I've had my say. I'll shut up now.

#37 User is offline   Csantucci 

  • Member, Board of Directors
  • Group: Status: Elite Member
  • Posts: 7,019
  • Joined: 31-December 11
  • Gender:Male
  • Country:

Posted 08 February 2014 - 05:36 AM

I would like to check OR behaviour in a full-dds texture route. So it would nice to have a fast to use .ace to .dds bulk conversion tool. Any suggestion? Do I have to pass to an intermediate format, e.g. .tga?

Second question (to Peter): is OR now able to load also terrtex files in .dds format?

#38 User is offline   Csantucci 

  • Member, Board of Directors
  • Group: Status: Elite Member
  • Posts: 7,019
  • Joined: 31-December 11
  • Gender:Male
  • Country:

Posted 08 February 2014 - 08:46 AM

I had a first try. To test with a route with few .aces I used the legacy Northeastern Corridor. First I bulk converted all daylight .ace files to .tga files with Route Riter. Then I downloaded the freeware Paint.net and a related bulk converter plugin and performed a bulk conversion from .tga to .dds format (that goes quite fast), and had a couple of very short runs, one loading the .dds textures and one loading the .ace textures.

Here some pictures:

.ace
Attached Image: Open Rails 2014-02-08 05-16-29.jpg

.dds
Attached Image: Open Rails 2014-02-08 05-11-12.jpg

.ace
Attached Image: Open Rails 2014-02-08 05-15-53.jpg

.dds
Attached Image: Open Rails 2014-02-08 05-11-32.jpg
Notice the display error in the trees in last image. I will check for this, maybe slightly different conversion parameters are needed.
The usage of memory was the same in the two cases.
Also differences in frame rate are unnoticeable. For the first image the .dds framerate is even lower, but the viewpoint is a bit different, and maybe this could be related to the transparency features selected for .dds conversion.
So all in all maybe I won't start a bulk conversion for existing routes (or maybe I will have a test with a more complex case) but... it works at the first trial, and this is remarkable, and of course it can surely be used for routes starting from scratch, thanks Peter !

#39 User is offline   gpz 

  • Superintendant
  • Group: Status: Elite Member
  • Posts: 1,772
  • Joined: 27-October 12
  • Gender:Male
  • Location:Budapest
  • Simulator:OpenRails
  • Country:

Posted 08 February 2014 - 08:50 AM

Every .ace file is now tried for a .dds version before actually loaded. If the texture contains sharp edge from a blended to opaque part (e.g. a window cutout), then convert it to DXT3, otherwise to DXT5. Thanks should go to Popescu Alexandru Cristian, he wrote the dds loader library for .net.

#40 User is offline   Csantucci 

  • Member, Board of Directors
  • Group: Status: Elite Member
  • Posts: 7,019
  • Joined: 31-December 11
  • Gender:Male
  • Country:

Posted 08 February 2014 - 09:09 AM

Thanks Peter, in the above test I converted everything to DX3. Selecting files to decide which ones to convert to DX3 and which ones to DX5 was too long. I removed all .ace files from the base Textures folder, to be sure that .dds files would be loaded :sign_thanks: .

  • 9 Pages +
  • « First
  • 2
  • 3
  • 4
  • 5
  • 6
  • Last »
  • You cannot start a new topic
  • You cannot reply to this topic

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users