Releasing a new rFactor 2 build is typically something we do with a classic bulleted changelog, but this time we felt it deserved a bit more than that.
During our recent rF24 event, technical issues cropped up that left us no alternative but to red flag the race. We decided to immediately regroup and redirect our priorities towards dissecting and fixing what turned out to be long standing issues. It was not a simple task, but we rolled up our collective sleeves and dug in. With a lot of help of the participating teams, we analyzed all the problems and for the first time were able to reproduce most of them. Their cause turned out to be very specific edge cases in ‘rejoins’ and ‘driver swaps’ so we proceeded by focusing on trying to break these features in as many different ways as possible.
So of course, much of the findings relied on intense and focused testing over the past weeks. Having to go back and test multiple scenarios repeatedly and working to find fixes and workarounds required a well-thought-out approach with a solid understanding of the issues from all involved, both on the testing side and the development side. This intense focus has given us an insight into the many ways things can go wrong in the heat of racing. Thankfully, we also had the massive support of the rFactor 2 community through post-race feedback and stories, as well as logging – this was invaluable and an enormous push to help find the root cause of many of these issues people have in online events. As a team, we meticulously went over each of these reports and looked for any specific details that could point us in the right direction.
Car Selection and Upgrades
One of the main areas we focused on was issues related to rejoining after getting disconnected during a session, for example when the network connection goes down for a moment. rFactor 2 has always allowed for a driver to rejoin a race after a computer crash or network issue, but in some cases on rejoining the server, the driver would end up with a DNF (Did not finish), a DQ (disqualification) or their name would show at the bottom of the list as simply ‘pending an open session’. Of course, these outcomes are incorrect and the question for us was: what triggers these scenarios? Our research and testing quickly showed that, in most cases, these issues were related to rejoining and either a) picking a totally different car from the car selection, b) picking the right car with the wrong livery from the car selection, or c) picking the right car and livery but with a different upgrade package.
You might ask, “Why is this a problem, I always pick the correct options”? Whilst that might be true in 99% of the cases, it’s the 1% that hurts us here in the end. It’s hard to be sure a team of multiple drivers always chooses the correct options. Making a mistake, it turns out, causes problems for more people than the driver rejoining, so we needed to make sure this could no longer happen.
To tackle this problem, we first looked at the core code of the rejoin process to make sure all options regarding car and upgrades are inherited and stay with each driver, regardless of disconnections or previous driver swaps. This means when you join with car A and upgrade X, it will be logged in a more robust way that prevents the driver history getting lost. Next, we worked on making this process more user friendly, so that it’s actually impossible to make a mistake on rejoin. We enhanced the network protocol to communicate to your client what car, livery and upgrades were used before, so we can choose the right car for you. For example, if you join with a ‘BMW M8 GTE’ with the ‘Le Mans package’ and ‘my-team-livery’, and you have a network issue during the race and are booted, instead of seeing the whole list of cars, team liveries, and upgrades on rejoin, you only see your BMW M8 GTE, and the option to change upgrade package is no longer available. You simply get your car back!
This brings us to another important point and side effect of rejoin errors. Rejoining with the wrong car or upgrade would often cause lags and freezes for all other drivers already on the server as everyone was forced to load a different car in real-time while on track (instead of the car that got parked in the garage when you disconnected).
“AI Take Over” and Driver Names Stuck in the Pit Menu
A recurring issue we’ve seen is when a driver swap takes place, the AI would suddenly take over the car without warning.
This was caused by trying to hand over the car to a team mate that was no longer a passenger or even on the server at the moment of the pit stop. By default rFactor 2 was then configured to let the AI take over. This turned out to be a bad idea and we altered the code to no longer do this. This means that from now on, if the driver taking over is no longer present, you will retain the car at the end of the pit stop. This will allow you to keep racing and retry a driver swap with your team mate without AI taking over and ruining your race.
When selecting a driver in the pit menu, names of any passengers would stay stuck in the list and would be select-able regardless of whether they had left the server or stopped riding with you. This meant you select your team mate in the pit menu, they leave the server or stop riding with you, but their name stays in the pit menu and can be selected. This caused multiple issues: On disconnection/rejoin you would often end up with a DNF, and if a name of a driver was selected that was no longer riding or had left the server, the AI would take over. We’ve fixed this issue by simply removing any drivers from the pit menu list that are no longer riding with you (as should have been the case all along).
Disconnection/Rejoin with Passenger(s)
Disconnections while another driver is riding along, either waiting on a driver swap or having just completed one, would end in a DNF on rejoin. For example, you’re driving on track, your team mate is riding with you and you get a disconnection. On rejoin, you’re not able to race again and your team mate’s name is now showing in the list as a driver with a DNF. We fixed this by making sure that on disconnect/rejoin only the current driver retains the car, all other teammates simply stay registered as ‘passengers’ and are not considered a driver until an actual driver swap takes place.
Pit Menu Parameters Locked After Rejoin
Yet another issue we looked at and were able to fix was the sudden inability to toggle pit menu options after rejoining. This was particularly a problem if you experienced a disconnect with very little fuel and could not request more fuel in the subsequent pit stop, ultimately you would run out of fuel and end the race with a DNF. All allowed in-car pit menu options should now be open to selection on rejoin.
Steam Integration Improvements
As part of our ongoing profiling process based on logs sent to us by users, we have also discovered that the “real-time” API functions that Steam provides could cause small hiccups. We technically solved that by internalizing the original plugin and making sure we execute such functions on a background thread so they can never interfere with our physics loop. This change is done both client and server-side and it means you will no longer see a SteamPlugin.DLL in your plugins folder (and we’ve made sure that if it is still there by accident, it gets ignored from this build onwards).
Faster Loading Times
Last but not least, we also spent some time profiling and optimizing the track and car loading process. Internal tests have shown improvements in the range of 30-50%, which should help people in general. Faster loading obviously also means you can rejoin quicker, losing less time overall.
Build 1114 is the first of two scheduled releases to address the issues we found. We decided to split the process in two, concentrating on the major bugs first and then addressing the smaller ones. We thought it was important to get an update into everybody’s hands as quickly as possible, but only after making sure we could not break this build anymore. As always we encourage people to update both their dedicated servers and clients and report any issues. We are heavily committed to getting this right and continue to improve the online experience in rFactor 2. We expect to have an update on those issues next month, but again, we’ll take as much time as we need to ensure these minor issues are also completely gone.