[James Ross]

 ChrisD, on 09 July 2015 - 02:20 AM, said:

Error: System.Runtime.InteropServices.SEHException: External component has thrown an exception.
at new[](UInt32 )
at Microsoft.Xna.Framework.Graphics.VertexBuffer.CreateBuffer(UInt32 dwSize, UInt32 usage, _D3DPOOL pool)
at Microsoft.Xna.Framework.Graphics.VertexBuffer..ctor(GraphicsDevice graphicsDevice, Type vertexType, Int32 elementCount, BufferUsage usage)
at ORTS.Viewer3D.TerrainPrimitive.GetVertexBuffer(Single& averageElevation)
at ORTS.Viewer3D.TerrainPrimitive..ctor(Viewer viewer, TileManager tileManager, Tile tile, Int32 x, Int32 z)
at ORTS.Viewer3D.TerrainTile..ctor(Viewer viewer, TileManager tileManager, Tile tile)
at ORTS.Viewer3D.TerrainViewer.<>c__DisplayClass8.<Load>b__1(Tile tile)
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at System.Linq.Enumerable.<UnionIterator>d__81`1.MoveNext()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at ORTS.Viewer3D.TerrainViewer.Load()
at ORTS.Viewer3D.World.Load()
at ORTS.Viewer3D.Viewer.Load()
at ORTS.Processes.GameStateViewer3D.Load()
at ORTS.Processes.LoaderProcess.Load()
at ORTS.Processes.LoaderProcess.DoLoad()
at ORTS.Processes.LoaderProcess.LoaderThread()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

This is an external error, but it looks like it is the same kind of out-of-memory error but occurring in a component which cannot handle it as well as the texture loading. There's probably nothing we can do about this and it has the same cause System.OutOfMemoryException - namely that there's too much stuff loaded in to OR (virtual memory limits). I don't see any good reason why this would only happen with one version and not another.

[ChrisD]

 James Ross, on 09 July 2015 - 02:29 AM, said:

This is an external error, but it looks like it is the same kind of out-of-memory error but occurring in a component which cannot handle it as well as the texture loading. There's probably nothing we can do about this and it has the same cause System.OutOfMemoryException - namely that there's too much stuff loaded in to OR (virtual memory limits). I don't see any good reason why this would only happen with one version and not another.

Sorry, JR,

I do not understand.. :bigboss:

Max memory in use without crashing: 3,701,743,616 Bytes
Max managed Memory in use w/o crash: 417,098,184 Bytes

Memory in use when crashing: 3,560,153,088 Bytes, Managed memory 307,424,964 Bytes.

In short, X3181 seems to be able to utilize 135 MegaBytes more memory than X3191, and no less than 104 MegaBytes more Managed Memory wo. Crashing.

This sounds really strange to me...


BUT, You are the Boss. I will rest my case.


ChrisD

[James Ross]

 ChrisD, on 09 July 2015 - 09:59 AM, said:

In short, X3181 seems to be able to utilize 135 MegaBytes more memory than X3191, and no less than 104 MegaBytes more Managed Memory wo. Crashing.

This sounds really strange to me...

Don't worry, these things often don't make sense initially. You should not consider the numbers directly comparable in most cases because of a thing called memory fragmentation (the Wikipedia article talks mostly about disk fragmentation but the same problem occurs in virtual memory) - in extreme cases, this can make it seem like there's tones of free virtual address space but you still can't load anything because none of the free areas are big enough and things don't get moved out of the way.

What we can say from the numbers is that both versions are running close to the theoretical limit and therefore are likely to be failing to allocate larger chunks of memory (e.g. for textures and buffers). As I said before, I don't understand why just one version would crash with SEHExceptions and not the other. I don't think there's anything in the changes that would make it more. It's a mystery right now. :bigboss:

[ChrisD]

 James Ross, on 09 July 2015 - 11:01 AM, said:

Don't worry, these things often don't make sense initially. You should not consider the numbers directly comparable in most cases because of a thing called memory fragmentation (the Wikipedia article talks mostly about disk fragmentation but the same problem occurs in virtual memory) - in extreme cases, this can make it seem like there's tones of free virtual address space but you still can't load anything because none of the free areas are big enough and things don't get moved out of the way.

This may also explain why X3191-Test RunactivityLAA.exe does not crash all the time.

Isn't there some kind of system call that tidy up the allocated memory?


Something like that seems to be present already. Log shows that allocated memory steadily rises as assets are loaded, but suddenly it drops by more than 100 MegaBytes.


Thanks for Your Time :bigboss:

ChrisD

[Mike B]

Hate to ask (I remember the looooong pauses in early MS BASIC's when string space was exhausted), but how do .Net and/or XNA handle garbage collection? Ultimately, for managed environments like them, that's the remedy for fragmented memory, but if it doesn't happen or isn't done right you can get the situation you describe.

[edwardk]

So do we know if any errors are coming up with the default settings? With the previous discussion of memory fragmentation, I was beginning to wonder if OR was failing because it was unable to get the piece of memory needed. On my system, I have been pushing the limits by changing the max viewing distance to 10,000 meters and slowly increasing the box size settings and I have yet crashed using LAA. In fact, I extended the length to 3000 meters and the width to 1500 meters. With these settings, my favorite is height 175 meters, length 2300 meters and width just under 1000 meters and my system is handling it fine. This of course leads to the usual question of how many background items are running with OR?

Edward K.

[James Ross]

 ChrisD, on 09 July 2015 - 11:48 AM, said:

Isn't there some kind of system call that tidy up the allocated memory?

We simply load and unload based on visible tiles of the world. There is no way to rearrange fragmented virtual memory allocations short of unloading and reloading things (and hoping that results in a better layout).

 Mike B, on 09 July 2015 - 11:50 AM, said:

Hate to ask (I remember the looooong pauses in early MS BASIC's when string space was exhausted), but how do .Net and/or XNA handle garbage collection? Ultimately, for managed environments like them, that's the remedy for fragmented memory, but if it doesn't happen or isn't done right you can get the situation you describe.

The garbage collected memory is done using arena-like behaviour and is compacted (moved to avoid fragmentation) for us. The issue, though, is not that memory - it's the memory used to back the graphic driver stuff, such as vertex buffers and so on, which are native (not managed) allocations.

Notice the large difference here:

 ChrisD, on 09 July 2015 - 09:59 AM, said:

Max memory in use without crashing: 3,701,743,616 Bytes <-- this is all memory in the process
Max managed Memory in use w/o crash: 417,098,184 Bytes <-- this is just the managed memory

The 3+GB of memory which is not managed is the stuff that gets fragmented.

 edwardk, on 10 July 2015 - 12:52 AM, said:

So do we know if any errors are coming up with the default settings? With the previous discussion of memory fragmentation, I was beginning to wonder if OR was failing because it was unable to get the piece of memory needed.
...
This of course leads to the usual question of how many background items are running with OR?

I don't know how or why you've got to "background items", whatever they may be, but fragmentation is something which happens over time (i.e. the longer you play, potentially the worse it gets), and the route chosen is by far the biggest thing in terms of memory usage. I'm pretty sure I have routes where the max viewing distance of 10km will be using less memory than other routes with the default 2km.

I'm going to see if I can find a tool/technique for measuring fragmentation of memory, since it's clearly not something any of us can see right now.

But it won't change the basic issue, which is always that virtual address space is finite and when you run out you get errors. What we might be able to do is pre-emptively skip loading attempts when there is not much space left (the auto tuner may also want to get involved). That might enable us to avoid the fatal SEHException.

[edwardk]

So it took the expansion of the precipitation box to uncover this weakness. I am guessing that we were already close to exploiting this weakness since up to this point, there were no out of memory issues? When you consider that with the time table feature, a number of locomotives can be set up to run and it appears that no problem came from that yet. I say yet because if you take a well detailed route and set up enough trains to come and go, the same thing should happen. Should it?

Edward K.

[James Ross]

 edwardk, on 10 July 2015 - 02:16 AM, said:

So it took the expansion of the precipitation box to uncover this weakness. I am guessing that we were already close to exploiting this weakness since up to this point, there were no out of memory issues? When you consider that with the time table feature, a number of locomotives can be set up to run and it appears that no problem came from that yet. I say yet because if you take a well detailed route and set up enough trains to come and go, the same thing should happen. Should it?

This is absolutely not a new issue. Notice how ChrisD has plenty of OutOfMemoryExceptions before the changes. People have been reporting those errors for years, when they configure OR to load too much. <_<

Any increase is usage can push people over the edge, but like I said: I don't see any reason currently for these fatal SEHExceptions to be related to this code. It could simply be that, because the allocations are slightly different, a different component fails. That's just what happens when things are pushed to the edge/too far.

[disc]

Wouldn't a newer .net framework usage, and 64bit OR would hide these problems?

However i didn't get "out of memory" exeptions (i have 6 gb ram), while i use heavily populated routes, but the cpu limit due to draw calls, and the stutters at tile loads/unloads are always there.