For many months I was complimenting on the need to have a simpler way to access x-plane navdata to allow Mission-X plugin to easily search navaids, to be more precise, airports and ramps.
How it all starts
After creating the first "random mission" based on set of rules, I wanted to expend the diversity of location options by having a more concrete data taken from X-Plane itself, that would have made the random locations in synch with the program.
This was probably the first bump I encountered. The XPSDK provided functions to get airports relative to a given coordinates, but there was no ramp data to be had.
So now, I had to write my own apt.dat parser that will traverse all sceneries and parse their apt.dat files. This information will be stored in a flat file and loaded in memory only when it is needed.
That meant, long hours of designing and coding this solution.
After few weeks, the challenge was met (well, I have work to do also ), and I head a prototype of the cached data. It worked quite well, and fetching information was never easier, sort of.
In order to have full navigation information for the random location, I needed to merge information from the cached data and use some of the functions in the XPSDK, and what a surprise did I have.
X-Plane XPSDK is not, what programmers call, modernized. Since I'm not a programmer by trade I only know that the function I called are not "thread safe", meaning, you have to call them from the main flight callback code and not from a threaded code.
Think of the main program and threaded code as two parallel roads that move tasks from one end to the other, the traffic from one road should not interfere with the traffic from the other one.
If we will try to deliver a task from the other road it might not be able to finish it correctly since it knows how to deliver its task only on that specific road it is driving.
How to solve it - the delivery approach
The only approach I can think of, and it might be the correct one by others, is to play by the code that is "not thread safe", which means, fetch the data you need from the main plugin callback and use it in the threaded code only when it finished gathering the information.
It is like speaking with someone next to you, but sending the information using a note that you placed in a box and the other side will pick that note and then write something into it and put it back into the same box. The thread will check if the note is ready to be picked and then read the answer.
This makes the code more complex and prone to bugs, if you are not careful, but, in my case I'm the only programmer for this plugin and hence I dictates the rules and how to use the shared data that is being used during this "transaction" between two threads. In no way my methods will be seen as "best practice" but it works and it is stable.
So. the main drawbacks are:
- Complicated code.
- Prone to bugs.
- May cause stutters, and it does, since the load on the main thread is not optimized enough.
In Mission-X case, it asks for airports information in certain radius, and that can take a second or two depends on the covered area.
If we want to do the same task but this time faster, the best approach, in my opinion, would be to run most of the logic in a dedicated thread, outside the main plugin thread, and only when it is done then it will be used by the main thread and to the user.
To achieve that we need all navigation data available to the secondary thread, although the XPSDK does not support it.
To solve this limitation I had to gather all the information into a file and cache it into memory. That worked well, but it is not as flexible as a small and efficient database.
Sqlite to the rescue
Using the sqlite api, I was able to dump all information into few tables. Each table holds specific information and you can join them easily using an "icao_id" that is a unique number for each ICAO (a unique number I added for each ICAO added to the tables).
Over the tables I created two views, so far, which are stored queries that gather all the relevant information into one virtual table (for easier search).
This solution, by itself, considerably reduced the programming logic I had to do inside the plugin.
Before: I had to filter the information using C/C++ code which means redesigning the information inside the plugin.
Example: filter airports by distance and then filter them by ramp types. The more filters I wanted to add the more code and complexity I introduced into the code.
After: I changed the workflow to sqlite centric.
My main data gatherer is the "apt.dat optimization class". All gathering complexities goes into it and written to the sqlite database (as a long term caching).
This means that I need to analyze the information I would like to expose to the plugin, then crosscheck with the "apt.dat" files X-Plane provide us and then implement the code.
Once the data is in the database file, it is very easy to fetch the information from the plugin. I do think that having some former SQL knowledge can make your life much easier, but learning it is quite easy and fun.
So now instead of writing hundreds of line code to filter airports by distance and ramp types and runway types, I just need a view that looks like the following:
What you see is the "view" I created over the airport tables so I'll have one logical table with all information I need. The plugin itself do not need to write such a query, but only to filter the view output by icao.
How that helps
All the code that relate to information gathering is running in the same thread, there is no "back and forth" transaction between two threads, which means less stuttering, if at all, during random mission creation.
This implementation is not used by all types of random missions, it is mainly for medevac and cargo missions, while the rest might only need ramp information which is a much simpler query or their code does need information from thread.
I hope that by implementing this kind of workflow, the plugin will cause less stutter during mission preparation and maybe it will help me expand the options Mission-X provide right now.
As for Laminar Research team, I think they should have provided a utility that does the same so 3rd party developers could just use their own private database to work with. The other option is that LR will have their own database interface that others can use but then LR will have to priorities and handle all requests in a thread safe manner.
Until the next build