Recent Star Wars event, interrupted by honey-mongering BOO pirates, caused a lot of well-documented heat in the DU community. Yep, there was drama inside drama: mysterious “ECM” attack on BOO fleet happened during their skirmish with AC forces (who also presented at the event as hired protection).
After our own NMC investigation (with a bit of lead and a few dead ends), we grasped the opportunity to have a serious talk with Icari, a direct participant of events and veteran game developer himself (currently working on a sandbox title!)
This interview is quite wide -- famous accident, DU tech, game engines and development, dangers, handling of online ecosystems.
* * *
le_souriceau: Hello, Icari! Thank you for joining in our most RAM-intensive investigation in DU history!
Icari: Hello, it is nice to be here!
le_souriceau: So, for readers who are not exactly aware of what happened. Let's do a very quick rehash.
BOO fleet intervened in the Star Wars event, everything was working as usual -- until you encountered AC/HYP ships. One of them had (intentionally or not) bug/exploit “electronic warfare” capability to cause players severe frame drop and dangerous hardware stress by targeting their ships. Is this right?
Icari: Yes, everything was fine, especially when we first engaged in the event there were over 200 targets around us, yet still, FPS was stable around 30 FPS or so. Well, not optimal for a PC game but acceptable for DU I guess.
While we were fighting against Atom and other ships, everything was fine but then HYP ships arrived. There were 7 ships in our fleet. Each ship was on a different voice channel and when the first ship encountered HYP ships they reported lag. When we maneuvered to intercept them as well, suddenly we got hit by a huge lag as well. Even on my PC, I had 10 FPS. We lost all the targets and could not even lock on them. Our pilot (Spicerub) had 1 FPS and could not even control the ship so he had to relog etc. We were dead in the water for almost 5 min.
The same thing has happened with the same ships recently again. While we were fighting with the Atom ships near Alioth, Havoc was fighting against HYP ships 4-5 SU away from us, Havoc was reporting heavy lag spikes so they had to pull back to the safe zone. When Havoc retreated, HYP ships arrived in the area where we are fighting with Atom. The moment they entered my radar my PC started getting slower and slower. We started losing targets again. Even my game dropped below 10 FPS, the pilot of the ship had 1 FPS if not less :D
le_souriceau: Looks like not a coincidence for sure. And, as I know, when this happened, you were playing on your very powerful work PC (something much unlikely to be knocked out by random game shenanigans), can you give exact system specifications for context?
Icari: Yes. I have been working in game studios and simulation companies for 25 years. So on top of my gaming PC, I have a workstation for work as well at home. It has 64 core Xeon CPU, 256 GB RAM and a Titan RTX GPU. I normally don't play games with it, but on large fleet battles I am using it just in case if pilots get disconnected, etc, I can jump on the pilot seat. It is like additional safety solutions against performance issues. I did not think that there was any game that could stress this PC but it seems I was wrong.
le_souriceau: Yep, pretty beefy machine! There is a system status log recorded during the event, let's see data together with our readers…
… some insane spike is noticeable around 02:42.
And if I understood you correctly before, the exact cause of this meltdown can be defined too: a particular construct sending an element data 90 times per millisecond, effectively “DDoSing” player PC. Can we see it too?
Icari: I can provide the log files to whoever wants them. I already provided all the data I have to NQ.
Imortant note: NMC can confirm that NQ acknowledged general technical findings from Icari logs, discussed here. The beforementioned log files will be attached at the end of interview for everyone with technical expertise to examine themselves.
As I said this is a workstation. So by default, there are a lot of things that monitor and record the resource manager and other things working in the background all the time. Thanks to them I was able to gather these things.
le souriceau: I hope NQ figures it out eventually. In the meanwhile… As I know, BOO overall has some serious IT-brains in the roster, have you tried to experimentally reproduce this phenomenon? It's something to do with elements stacking or other dirty tricks so many players can’t resist doing? Or some Lua tricks involved?
Icari: Well to be honest we never even tried, but I do believe this is a side effect of the item stacking and I don't think HYP created this intentionally. This somehow happened as a side effect of all the bug abuses they are using on their cubes, since HYP cubes abuses more than one bug, (element stacking, weapons/engines/adjusters/brakes buried inside the voxels, etc ) it is really hard to figure out which one is causing this, and now they are just taking advantage of it.
Also, it is highly unlikely that they are not aware of it. Even if this isn't intentionally created, if each time I engaged a target and people started getting disconnected, after the 2nd or 3rd fight I would realize it. HYP people are really smart mostly, I don't believe they are not aware of this.
But my educated guess is, some of the elements can not be loaded and our clients start throwing a missing game object instance error multiple times per frame or a radar script bombarding us with data requests. But I am pretty certain that this is caused by an error that happens during runtime, maybe more than once per frame. I experienced a couple of things that happened in simulations. The errors that are related to physics calculations generally pop 3-4 times per frame in the Unreal Engine and Unity3D which can have devastating effects on performance.
le souriceau: What really intrigues me here (as a total IT layman, so forgive me in advance for nonsense) -- if such calculation/request speed is dangerous for game performance, why is it even allowed to reach such insane and, as I understand, totally unnecessary rates for the game to function? Is NQ oversight to not block this somewhere in mysterious Unigine “guts”?
Icari: Well, this is about how game engines operate. Most game engines have tick/update functionality. This is where most of the runtime events happen. The problem with the tick/update is if you are not careful it can cause huge performance issues. Since it is running for each frame and almost all the game objects that are doing something in real-time will have some kind of tick functionality. So most game engines separate their tick/update functionality. For unreal it is pre-physics, during physics, and after physics. For Unity, it is Update, LateUpdate, and FixedUpdate. As I remember Unigine also had the same functionality. This can optimize the game performance BUT it has a pitfall as well.
If something needs to be updated before the physics interaction and causes an error, anything else in that frame using that calculation as a reference will also fail if there are no validation checks which can turn into a chain reaction cluster fuck. As the log files show, for a fight that is less than 5 min my PC generated ~100000 log events, etc.
Also each time a game event fails and returns an error message, it also prints it to server logs as an error message as well if not disabled (it should not be disabled generally because the error logs are the only way to find and track bugs ) this also puts additional stress on the servers. Any game dev can tell you that one of the ultimate goals for development is no error or warning messages during run time. So this is a common thing to happen in games with bugs and errors. A normal PC will reach a limit and you get a “game crashed” message. Because your resources get overloaded and OS forces a shut down for the app that is running. I am able to see these numbers because this workstation is designed to handle these situations.
Unity has tools to reduce these kinds of things with event locking, Unreal has better tools with "fire and forget" events like dispatchers and interfaces that won't return an error message even if it fails. As I remember there was nothing built in the default system for Unigine for that kind of specific situation because in theory Unigine is not designed specifically for games but as an enterprise-grade simulation solution. Of course, no one should expect they have all the tools in-engine for gaming, just like people should not expect that Unity or Unreal would have all the tools for a military-grade simulation.
For this specific reason while coding I prefer an event-driven method and try to avoid using tick functionality as much as possible but this is just my way of coding. No one can tell if a method is better than others in these kinds of situations.
le souriceau: Yet this all in space! I guess planetary PvP (when implemented) will be tenfold more susceptible to any faults of such sort with devastating results to participants (with them literally falling from the sky) with more construct density and the ability to do some cheap lag machines with static constructs (even not intentional)...
Icari: Yes. Imagine this happens in the atmosphere. With these things, the best defense of a base will have heavy lag. Unless you can control your ship, you will just crash and burn. Some games are suffering from this, that is why most game studios have clean rules in the game to prevent this kind of situation because eventually, no matter how well your game is designed or coded, you are limited by the technology.
le_souriceau: While we are in this general territory I can’t resist and not ask about your thoughts (as an industry vet) on Unigine as an engine of choice. It's up to the task? Because some IT people playing DU are really negative about it. What do you think, NQ can put it to good use, or we can expect all kinds of endless painful shenanigans because of it?
Icari: Well, to be honest for the current project I am working on we evaluated some engines to decide which one we should use. Unigine was one of them. Surely by default, it offers some things that other engines are not offering for large open worlds. Like world origin shifting and those are nice changes. The reason we decided to use UE4 was simple. We decided that we can code all these ourselves specifically for our replication system under 2 months and we did it.
Unigine is a nice engine, but I don't believe that it will have a future since UE5 is about to be released soon and I am sure there will be a new Unity3D launch somewhere around next year. So to answer that question, if it were up to me I would choose Unreal. So rather than using a simulation engine to create a game for its large map capabilities, we decided to use a proper game engine and create a large open-world functionality to the engine by ourselves.
le_sourcieau: Returning to our principal story... I feel before we missed a nasty detail of the whole affair -- such shenanigans can potentially damage expensive hardware of players caught in some freak loop? CPU and RAM at risk?
Icari: In theory, no, computers have enough security on the hardware designed to protect themselves, but sadly this is in theory only. The moment someone stops using factory settings, you are leaving that protection zone. Overclocked CPU or GPU for example can easily get damaged. If your fans are also dusty or old, it is another risk. So, in theory, a game should be safe BUT I have seen enough computers and consoles ruined by games (Anthem for example) so I know that practically it is still possible.
le_souriceau: Still, it seems to be part of a much wider issue, that NQ is too liberal toward “dodgy” stuff (up to the point, when game-breaking exploits are “fair game” until fixed). On other hand, to be fair, NQ faces a lot of issues (with apparently very limited) manpower, to handle things as desired in an ideal world. Your thoughts on handling?
Icari: Here we go... To be honest I have been playing online games since the first day of Ultima Online. I have never heard such a thing as "Abusing bugs is fair game till NQ fixes it". It is like the police saying “ Until we find a way to prevent people from driving faster than the speed limit, speeding is allowed”. Don't get me wrong, I am %100 sure that the devs are not happy about it, it is just that a manager decided this and they had to follow. As a dev, I know best that we hate people that are cheating and abusing bugs more than anything else. It haunts you till you fix it if you love the game you are working on. The reason is simple, because of a mistake you made ( since it is a bug ) some prick has the power to ruin your game for every other player.
In Blizzard games, for example, there is no warning. They use the one-strike rule. The moment you get caught abusing a bug or using a cheat or bot, you get a permanent ban. For Ubisoft you get a 24 hours ban on your first offense and the duration of the ban increases with each offense till you receive a permanent ban as well.
DU is a no wipe permanent universe and these bug abuses damage the game A LOT. I honestly can't believe this is happening. EULA that we all agreed gave them enough power over players to police their activity in-game. So if NQ announces that element stacking is forbidden from now on and people who use it might face punishment, that will be enough deterrence. Instead of using EULA to remove the posts from Reddit and forums, I wish they would use EULA to enforce rules where it matters.
Right now that HYP ship is no different than people looting one of the marketplaces. The funny thing is even private Rust and SE servers handle cheaters better than NQ does.
It was during the first week of the early access that NQ realized that people are making so much money that it will ruin the early game economy, they wiped everyone's wallet. So strangely, when it comes to wiping the player progression, admin interference is fine but against bug abusing and/or cheating NQ’s stance is "we don't want to interfere with the players".
Also, another problem is with the players. There are enough players from AC saying that “we did not see any proof yet”. And I know that they know what is going on as well. So they are turning a blind eye because HYP is part of the AC. People should remember that everyone will need justice eventually. Today HYP is utilizing some bugs against us, tomorrow someone can utilize some other bugs against AC.
People should realize that the game is on the edge of turning into The Division 1. Ubisoft ignored cheaters for a long time like NQ is doing right now, at some point rather than a present-day looter shooter, The Division turned into a fantasy ninja game where everyone in Dark Zone teleported around killing each other with a single bullet. Because of it, everyone else who did not use cheats either quit the game or stopped entering the DZ.
le souriceau: Yes, the problem of unfair advantages (big and small, starting from week one) is very painful for the community. I am personally very concerned about the economy… So, last one… no two! Two last ones.
First. In continuation of the “law and order” subject -- is this EQU8 Anti-cheat, is it up to the job? EQU8 was advertised by NQ as mainly a “smart, non-intrusive observing tool”, but it sounds… ultimately not very assuring, to be honest. I have the impression no one actually was banned for messing up with in-game economics. Or am I missing something?
Icari: Well to be honest EQU8, is as good as any anti-cheat goes. All these solutions are designed to monitor your PC process to check if you are using a 3rd party tool. like cheat engine and so, apart from that, it can not do anything and they are very easy to fool. Big companies like Riot Games and Dice have their own anti-cheat solutions. They spent millions of dollars just to develop them and they are not working. People are still able to use 3rd solutions to cheat on those games. That is why I prefer cheat detection to anti-cheat solutions. Because the opposing party has no way to know for sure how I am detecting it. Against an anti-cheat solution, a determined guy try and try and try again till he managed to create a 3rd party solution that can run in-game, and the moment he managed that, he would know for sure that he bypass the anti-cheat. After that developers try to adjust their anti-cheat to prevent that 3rd solution, once they finally managed to block it, cheaters always find another way. This will eventually become an endless fight between two parties. I have seen it happened in a small game named Gunz. There was almost a weekly update both on the game’s anti-cheat system and cheat itself for years.
That’s why I prefer, to create a system that detects and notify me when someone is cheating. Let me give you an example. Let’s say the max speed of a player is 100 m/s. So he can not travel more than 1 km in 10 seconds. You can check a system that checks this 1-2 times per hour. Then the system can notify a GM that there is suspicious behavior on a player’s avatar.
Well, this is a simple example but a cheat detection system is actually a combination of hundreds of this kind of simple examples.
le souriceau: Second. Recently NQ did some visuals changes and there was a bit of discussion about how important the “picture” is for attracting new players (or not?). As for me, personally, DU can even use much simpler/stylized graphics, but smoother performance, fewer demands on PCs. In the context of all we discussed before, what do you think is the golden spot here?
Icari: Well need better graphics ASAP, said no DU player ever. We have much bigger problems. We have performance problems, we have bug abusing problems, we don’t have proper game loops other than mining and selling to the bots. We have big problems with the current PvP state of the game.
People are playing games like Valheim or Minecraft and to be honest, their graphics are pretty bad but no one complains about it because the game itself is actually fun. If improved visuals would make a game better, Crucible, the latest game of Amazon Studios would not be dead and abandoned in a week, or Star Citizen would be the best space game ever created.
So gameplay first. Of course, visuals should be improved over time but it should not be the main focus.
le souriceau: Thank you a lot, Icari! It was a very informative talk, somewhat leaping beyond our initial subject, but, I hope, something our readers appreciate, especially ones who have a soft spot on global, (even philosophical) game development questions!
Icari: Well, thank you for having me, it was actually fun.
Special thanks to Spicerub for proofreading. He stated that everything is fine.