Almost each week I had the option to introduce a new build and progress to Mission-X "Generate Mission" feature. Unfortunately this week I do not think I'll be able to publish a new build, maybe, and the work that was put in involved making the plugin produce a better mission preparation and not on the feature side.
So what went on
One of the things I wanted, and was also suggested at the org is better placement at destination location. At the current state, target coordinates are defined by XPSDK library. Unfortunately it is providing the center point of the target airport.
Therefore, much work has been done to tackle this issue. I wanted the plugin to produce ramp locations according to the plane type defined in the Mission Template.
To do that, I had to work with X-Planes and the Custom Scenery apt.dat files.
My idea was: prepare an optimized apt.dat file that holds "all" Navaids that have "ramp" information. A Navaid without ramp information will be skipped and we will use what Laminar provide us using the XPSDK library.
It is important to understand that preparing the optimized "apt.dat" file is optional and plugin will always fallback to the original data that XPSDK library provides us.
What is the outcome
My first tries were written and tested on Windows 10 on a machine that I believe is above average (I7 8700K + SSD). The parsing of the files took around ~280s which was higher than I anticipated. This is why I decided to thread the whole process.
Here are some results for the parsing of the Custom Scenery:
- (Not Threaded) ~280s, X-Plane was unresponsive during that time (not good, but for POC that was acceptable).
- (Threaded but not optimized) ~280s, X-Plane continue to work, you can fly but not start a mission or generate one (overall file sizes ~270M).
After first tests, I found out that some locations where not available in "Custom Scenery" but in the "default apt.dat" file, which takes around ~300M of file which will be added to the previous files. Overall, we need to read ~500-550M of Navaid information.
The parsing result I received on few machines were:
- (my machine) ~500 seconds.
- (my laptop) ~1600 seconds.
Although you can fly and operate X-Plane, the time to produce the file, was quite long on slower machines.
Performance of generating mission file
My next step was to use the optimized apt.dat file during mission creation.
The optimized file size took: ~7M which is good, but I added an index file that holds the location of the Navaid data in the bigger file. The index file was ~300-400Kb.
My idea was:
- Pick a random airport,
- Search its location in our Index file
- Read optimized apt.dat file (~110k lines)
- Parse relevant information.
- Decide if to use information or not.
The result I received when the plugin generated the mission file where:
- ~200sec for 4 targets.
- ~50sec for 2 targets
Again, all work is done as a Thread, so you could just click the generate mission, and do something else.
And then I tested the implementation in Linux
As part of the code test, I compile and check how it runs on Linux and OSX.
I use same machine for _all_ these tests.
Here are the results I received on Linux:
- 2-4 seconds
I searched the internet and found that IO operations on Windows using the standard libraries (especially iostream) are slow, but I never thought that it will be that slow.
So how to even optimize it more
Once all code was in place and was stable, I decided that I was too caution regarding memory management. I used the file as an external memory instead of the RAM.
So next step, cache to memory.
At first I just stored picked ICAO and fetch airport information that was already looked up during that session, but the hit ratio was low, and performance was still bad for Windows machines.
Now I know that if I had learned computer science I could have learnt how to prepare a better optimized file that I might read faster, but I decided to go brute force. I will:
- Read and store all information to memory.
- Filter out duplicate Navaids.
First we read Custom Scenery
then we read the default airport data file.
- Store only Navaids that have ramp information.
- Create the optimized apt.dat file once all information was loaded to memory for later use.
The write benefit was quite good, less than a second to write to disk (basically I flushed the data), I could do without the index file since all was loaded to memory..
The parse time was also better and consistent:
On Linux and OSX performance is quite good, windows on the other hands is not as good as I would like, but is better.
Last piece of the puzzle
In order to use the cached file data I did not want the simmer to always use the optimized apt.dat menu nor to need to rerun the full process every time he/she starts X-Plane. In my opinion one time should be enough, and every new scenery that you add and has apt.dat you should manually regenerate the cache.
When the cached file will be used?
Only when you generate a random mission the plugin will try and load the cache file. This is quite fast action and should done only once during X-Plane session.
Please do remember, that all operations regarding apt.dat optimization and "mission generation" is running in the background, so you can continue and interact with X-Plane. Performance hit should be minor on decent machines (I won't dare to say what is decent machine, I leave it up to you :-) )
Until Next Time