|
Well he's right that untyped programming languages are stupid garbage for idiots
|
#
?
Nov 12, 2019 03:40
|
|
|
|
| # ? May 17, 2024 18:04 |
|
|
Game devs tend to have very negative opinions on automated testing of all sorts, for some good reasons and some bad reasons.
|
|
#
?
Nov 12, 2019 03:59
|
|
|
as much as i love to laugh/grimace at that dude i think that’s pretty much all spot on
|
|
#
?
Nov 12, 2019 04:19
|
|
|
is blow considered a big deal among game devs? and if so, why?
|
|
#
?
Nov 12, 2019 04:28
|
|
|
ironically enough I think it’s the larger and more complex code bases that benefit from unit testing the most. blow strikes me as the type of person that doesn’t work on a lot of sprawling projects.
|
|
#
?
Nov 12, 2019 04:30
|
|
|
It's easy to nurture lovely opinions on testing when you are creating throw away entertainment products or primarily work alone on a closed source code base.
|
|
#
?
Nov 12, 2019 04:35
|
|
|
qsvui posted:is blow considered a big deal among game devs? and if so, why? he was, back when indie games were just becoming a Thing (particularly the breakout of indie games on the xbox 360) but it's basically falloff from that since. he went up a little when the witness turned out to be fantastic and then went right back down when he started mouthing off about  on twitter several times
|
|
#
?
Nov 12, 2019 04:44
|
|
|
uncurable mlady posted:ironically enough I think it’s the larger and more complex code bases that benefit from unit testing the most. blow strikes me as the type of person that doesn’t work on a lot of sprawling projects. The Witness has 5 programmers credited and that seems to be his largest project, so yeah. I suspect he overestimates the scope of the things he's worked on.
|
|
#
?
Nov 12, 2019 04:45
|
|
|
he must be using some very powerful type systems
|
|
#
?
Nov 12, 2019 04:47
|
|
|
I've been flooding my code with unit and integration tests because I have like a dozen lovely web applications I am supposed to support and there's no way I could keep all the dumb corner cases in my head without documenting them. Might as well document them via test code! My successor is hopefully going to have an easier time with my lovely code than I've had with my predecessor's.
|
|
#
?
Nov 12, 2019 04:56
|
|
|
qsvui posted:is blow considered a big deal among game devs? and if so, why? i think it's that thing where people assume someone creating a great game means that they're a genius programmer
|
|
#
?
Nov 12, 2019 05:56
|
|
|
There's huge rifts between gamedevs, webdevs, sciencedevs, enterprisedevs, etc, and that's hardly surprising. Just because they all involve "programming" doesn't mean they all share the same priorities and opinions.
|
|
#
?
Nov 12, 2019 06:19
|
|
|
Could someone with better coding practices than that twitter guy educate me on the way unit tests should fit into game development? For interactive graphics codebases, it seems hard to pin down useful measures to hard pass/fail the entire output. For inner functions leading to that output it seems hard to define tests as well, without waiting for that final result. It's easier to picture for some subsets of a game program of course, like for graph/topology handlers you could detect an inconsistent mesh state. But for a large part of a game/graphics program I only picture a sort of integration test instead. Wherein before each big commit, you run a big guantlet of all your demos (assuming you made one to test each new feature), manually checking that they each load/draw correctly. What kind of additional logic errors could you catch with automatic tests? Next, it sounds like besides checking correctness, automated tests can be useful for avoiding changes to the API. But what if you're in the early stages, still making the API and changing it on purpose whenever you can see a way to make it friendlier? When you DO want to break the API, at best it seems that tests could only tell you that you've done so (if you don't know already). Does that really contribute anything? Are automated tests not useful until an API has stabilized? Once it has stabilized, and you still break it for your users, what do you do? Just manually type up a changelog about it for everyone, and write brand new tests? Happy Thread fucked around with this message at 07:24 on Nov 12, 2019 |
|
#
?
Nov 12, 2019 06:30
|
|
|
https://twitter.com/jonathan_blow/status/1194078776675557377?s=21 this is not what 10x programmer means
|
|
#
?
Nov 12, 2019 06:57
|
|
|
"unit testing takes time and effort so I don't see the point", what a galaxy-brained thought
|
|
#
?
Nov 12, 2019 07:18
|
|
|
unit tests help for things like core math libraries where there is a right answer, but it's very, very difficult to ensure that emergent complex systems (what you *want* in your video game!) have correct behavior under all conditions. unit tests aren't helpful since the behaviors that we want under test with are the interactions of large systems interacting, a lot of which are not guaranteed to be consistent or act similarly in slightly different initial conditions. Even taking away physics and gameplay, writing unit tests for modern renderer higher than just basic math is kinda pointless. everything there to test runs on environment like GPUs that you have little control over, with very little consistency guarantees.
|
|
#
?
Nov 12, 2019 07:23
|
|
|
Probably the best-known example of testing in video games is Riot's test harness for League of Legends, where they record game playback and ensure very high-level things like "when I am this character with this spell with these stats, and fire at this enemy with these stats 5 times, it will eventually die". That's not unit testing, that's super high-level and just about the level that starts becoming helpful for games.
|
|
#
?
Nov 12, 2019 07:24
|
|
|
I'm not sure I'd totally agree with that. The little six-item inventory in that sort of game, for example, is the sort of small component that you can meaningfully unit-test. You can also unit test things like your buff system, to validate that different classes of buff and debuff combine in the intended way and have the intended effect on final stats. In fact, the whole point of unit tests is that you can test various parts of your logic without having to involve unpredictable high-variance bits (like, say, your visual rendering system) at all. That seems even more useful in games than in many other software fields.
|
|
#
?
Nov 12, 2019 07:40
|
|
|
Suspicious Dish posted:Probably the best-known example of testing in video games is Riot's test harness for League of Legends, where they record game playback and ensure very high-level things like "when I am this character with this spell with these stats, and fire at this enemy with these stats 5 times, it will eventually die". That's not unit testing, that's super high-level and just about the level that starts becoming helpful for games. The factorio devs also run a bunch of automated tests on every commit, and have written about it on their dev blog before But yeah those are super high-level non-unit tests also. https://www.youtube.com/watch?v=erYjMMBXy7A Theirs are apparently more in depth than just "this character will eventually die" though. To guarantee that the game logic is deterministic (important for multiplayer, to avoid desync bugs) they simulate every game tick on two separate processes and then checksum the game states to see whether they're identical. RPATDO_LAMD fucked around with this message at 07:49 on Nov 12, 2019 |
|
#
?
Nov 12, 2019 07:44
|
|
|
There genuinely are a lot of things in video games where unit testing either isn't applicable at all or where it has very limited value. The problem that game devs seem to run into is that because they (correctly) aren't in the habit of trying to unit test everything, they sometimes also write it off in places where it actually would be useful. Writing good tests and designing things to make unit tests meaningful is a skill that you don't develop if you never do it.
|
|
#
?
Nov 12, 2019 08:02
|
|
|
And then you have the people who are actually working on something like the chat server and don't realize that they're just working on a normal server thing that happens to be used by a game and none of the special gamedev things are actually applicable.
|
|
#
?
Nov 12, 2019 08:05
|
|
|
I think I lucked out then by mostly having done graphics programming for the last decade, which hasn't been negatively impacted as much by my not learning about CI tools or any streamlined test practices that all these job postings are asking me for now (RIP). I do want to write code that has an API others will want to use so it's important to me if it can help with that. It is occurring to me that there are still certain parts of my gaming/graphics stuff that could benefit from automated tests. I do a lot of mixing interactive graphics into web documents, so maybe some of my code that spits out document elements could be automatically tested even if the 3D graphics code can't. A small portion of my graphics code is also just math definitions, so I could also write a test harness for just those, checking however many math identities I want, even though my implementations are so short and simple that I don't think there is much to be revealed. Happy Thread fucked around with this message at 08:09 on Nov 12, 2019 |
|
#
?
Nov 12, 2019 08:07
|
|
|
Jabor posted:I'm not sure I'd totally agree with that. The little six-item inventory in that sort of game, for example, is the sort of small component that you can meaningfully unit-test. You can also unit test things like your buff system, to validate that different classes of buff and debuff combine in the intended way and have the intended effect on final stats. The simple algorithms for debuffs are cases where unit tests might be helpful, if you can condense it to common code. But it's also code that tends not to be very buggy in my experience, and requires you to have a single debuff system, which is a pretty heavy constraint for a gameplay designer, who typically do not like the constraint that comes with that. e.g. some designer says that debuff A will replace debuff B rather than stack with it, or they might tweak these rules to suit gameplay testing, or radically change them depending on the character class.
|
|
#
?
Nov 12, 2019 08:15
|
|
|
Suspicious Dish posted:e.g. some designer says that debuff A will replace debuff B rather than stack with it, or they might tweak these rules to suit gameplay testing, or radically change them depending on the character class. that sounds like the platonic example of something that would be good to unit test
|
|
#
?
Nov 12, 2019 08:16
|
|
|
Lots of games are based around objects interacting with each other, like the wizard sword should do more damage if you also have the wizard helmet. Very easy to write unit tests for and a very common type of bug. Game balance can also be tested, where having all the wizard gear should give you the same amount of wizard points as troll points if you have all the troll gear. (Can you tell I'm an RPG expert?) The tests seem banal and the first 5000 times they'll run without incident but then you start fiddling with stuff and it's nice to be warned that the princess ring actually causes a deficit of troll points.
|
|
#
?
Nov 12, 2019 08:19
|
|
|
The more often you fiddle with something, the more valuable unit tests are. You might think it's a pain that when you change the design for something, you have to update the tests for that thing in addition to the code - but that's insignificant compared to the security of knowing that you're not changing anything that wasn't part of your intended design change. Although, you can tell a lot of mainstream multiplayer game developers don't do this, because they routinely break interactions that would have been caught by such unit tests, and leave them broken for several patches.
|
|
#
?
Nov 12, 2019 08:25
|
|
|
Jeb Bush 2012 posted:that sounds like the platonic example of something that would be good to unit test To be clear, we do have automated test scenarios from game replays that do test higher level features, and our designers use the system to build interesting combat scenarios and make sure the mechanics work. They are, in no way I would consider, unit tests. They test all components of the combat. This sort of logic is hard to break down in any meaningful way into independent test.
|
|
#
?
Nov 12, 2019 08:29
|
|
|
Ola posted:Lots of games are based around objects interacting with each other, like the wizard sword should do more damage if you also have the wizard helmet. Very easy to write unit tests for and a very common type of bug. Game balance can also be tested, where having all the wizard gear should give you the same amount of wizard points as troll points if you have all the troll gear. (Can you tell I'm an RPG expert?) The tests seem banal and the first 5000 times they'll run without incident but then you start fiddling with stuff and it's nice to be warned that the princess ring actually causes a deficit of troll points. While testing higher level meta-effects like "things with the same tag stack properly" might be useful, this in particular just sounds like data duplication. You're writing the same properties in two places and hand testing if they're out of sync. I'm not gonna say there's no value in that -- it guards against accidental change/deletion -- but more often than not this just means you're now going to change two game files rather than one when you change the effects/stats/tags of certain items. Indeed, people will be so used to changing them they'll probably just change both automatically, breaking the functionality in both the test and the game. You may as well just trigger a QA audit if the git diff for your item list changes. I think the only time something like this may be useful is if you have some sort of combat or level design that's dependent on systemic changes otherwise. E.G. verifying that a jump of X units wide and Y units tall can be made with degree of precision P (so it's not pixel perfect or something) because you know that's the extrema of your level design. Then if you tune your jumping physics you know you didn't just make the jump impossible/too difficult. Then, depending on how ambitious you want your test harness to be and how much manual labeling you want to do, you can even throw warnings if a jump is out of this range. Linear Zoetrope fucked around with this message at 08:45 on Nov 12, 2019 |
|
#
?
Nov 12, 2019 08:36
|
|
|
from what i've heard, a lot of game dev (particularly in the early stages) is very exploratory - you're throwing a bunch of hasty proof-of-concept poo poo at a wall trying to find some fun mechanics. it'd be a waste putting any tests into this throwaway prototype code. and, i guess once you get to the point where your game design has crystallised enough that it wouldn't be a waste of time to unit test, your architecture is hosed to the point where it would be too painful to add tests. c'est la vie i guess
|
|
#
?
Nov 12, 2019 08:46
|
|
|
I have seen Jonathan Blow walking around my neighborhood making gentle martial arts moves with his hands e: i just realized this sounds like a joke. it is not. Doc Hawkins fucked around with this message at 10:26 on Nov 12, 2019 |
|
#
?
Nov 12, 2019 10:03
|
|
|
A lot about games is totally subjective. If it looks pretty when it renders, it doesn't really matter what the unit tests (or any other kind of automated tests) are saying. If the game is no fun to play then it doesn't matter how many of your unit tests are passing or how many of them would need to be broken to fix it. Play testing overrules everything.
|
|
#
?
Nov 12, 2019 10:20
|
|
|
Plorkyeran posted:The Witness has 5 programmers credited and that seems to be his largest project, so yeah. I suspect he overestimates the scope of the things he's worked on. Don't you know that games are the most complex and advanced software? I mean it is harder to write a game than a compiler. (This is something that Blow actually believes)
|
|
#
?
Nov 12, 2019 10:35
|
|
|
Doom Mathematic posted:A lot about games is totally subjective. If it looks pretty when it renders, it doesn't really matter what the unit tests (or any other kind of automated tests) are saying. If the game is no fun to play then it doesn't matter how many of your unit tests are passing or how many of them would need to be broken to fix it. Play testing overrules everything. This is a pretty weak argument and can be used for other types of software too. Tests don't matter as long as the user is happy with the end result! It's the type of argument you hear right before you git checkout some spaghetti unmaintainable garbage. There are portions of all software that aren't great candidates to be covered by unit tests, games probably have more then any other. But it takes some pretty serious arrogance to assume your software won't benefit from a tool used by the literal rest of the discipline.
|
|
#
?
Nov 12, 2019 11:13
|
|
|
Xik posted:This is a pretty weak argument and can be used for other types of software too. Tests don't matter as long as the user is happy with the end result! Nah; most of the systems should have tests in games. The problem is a lack of rigor and experience throughout the industry. You see this in the way the discourse around automated testing has changed over the last five years or so. Fundamentally, neither the problems or the testing methods changed, but you’re hearing more about how automated testing is being used. We just lag the broader industry in engineering practice. I’ve worked with a lot of people who literally don’t know how they could solve any problem without copious interconnected global state — in that environment of course it doesn’t seem worth the effort to test. Testing means rewriting everything; or minimally admitting you did it wrong in the first place.
|
|
#
?
Nov 12, 2019 13:16
|
|
|
Suspicious Dish posted:To be clear, we do have automated test scenarios from game replays that do test higher level features, and our designers use the system to build interesting combat scenarios and make sure the mechanics work. They are, in no way I would consider, unit tests. They test all components of the combat. This sort of logic is hard to break down in any meaningful way into independent test. You could create an instance of the core game engine (so, no graphics integration, just something abstractly that reasons about ticks), have a harness that injects characters and events into it, and then verify its state at particular ticks. I agree that most buff bugs seem to be more like "we never thought about these two scripts both firing on the same tick, and if they do so in this order, the thing that was supposed to remove this debuff doesn't do anything and the debuff becomes permanent", though.
|
|
#
?
Nov 12, 2019 16:57
|
|
|
Xarn posted:Don't you know that games are the most complex and advanced software? I mean it is harder to write a game than a compiler. hasn’t he written two of the former and zero of the latter
|
|
#
?
Nov 12, 2019 17:43
|
|
|
rjmccall posted:You could create an instance of the core game engine (so, no graphics integration, just something abstractly that reasons about ticks), have a harness that injects characters and events into it, and then verify its state at particular ticks. This is effectively what we have, but it requires so much plumbing. e.g. If I trigger an attack, the way that it has been set up is that the artist tags where in the attack the projectile will spawn, so that requires loading animation data and playing it. Still no rendering of graphics, but you're basically doing 99% of the work that it's hard to get under what I consider sufficiently independent test. And yes, we load & run this animation data on the gameserver, too. Lots of intertwined systems due to the workflows of artists and animators and designers.
|
|
#
?
Nov 12, 2019 17:54
|
|
|
JawnV6 posted:hasn’t he written two of the former and zero of the latter He's been working on his own language & compiler for the better part of a few years now. I don't have too much respect for Blow anymore because he seems to play the whole "everything is terrible, I am the only smart person" angle a ton, and all of the weird biotruths junk, but I think he has experience in both areas. I think games are very different from traditional system architecture: 1. Performance matters, like, a ton, you need to run at 60fps these days, so that limits your design space a lot 2. Most of your technical staff are not programmers! It's hard to understate this. You have 1 programmer for every 5 artists/animators/designers, as a rough ratio. These roles are incredibly technical! 3. Timing and weight are very, very central to gamefeel. Systems might apply over time, so it's not as simple as "activate attack, check that enemy dies". It takes time for that projectile to reach the enemy. 4. The workflow that's most natural and welcome for the rest of your staff is to design tools and systems that let them be very flexible, like putting game scripting on specific frames of an animation. 5. Most gameplay designers want the ability to model complex interactions and world states. The single concept of a "debuff system" that has consistent rules and logic seems to be taken for granted in the programming world (why wouldn't your game have consistent rules and logic!), but is already a hard sell to a designer. They want to be able to model pairwise interaction if it comes down to it. 6. Special cases are everywhere. We might disable large parts of gameplay systems when you enter a trigger volume without you ever realizing it. Games are full of fun hacks and tricks. They're not fair, they're not internally consistent, they just appear to be to players. tl;dr the set of states that can affect anything in a game is actually really, really huge, and saying "have less of them, then you might be able to do testing" isn't helpful advice to professional game developers. These are not insurmountable problems, but the benefits do not seem to be worth the tradeoffs in a lot of cases in my experience. Suspicious Dish fucked around with this message at 18:07 on Nov 12, 2019 |
|
#
?
Nov 12, 2019 18:04
|
|
|
JawnV6 posted:hasn’t he written two of the former and zero of the latter That's part of the joke. Suspicious Dish posted:He's been working on his own language & compiler for the better part of a few years now. He said that before he really started, and what he means by compiler is a front-end to LLVM. 
|
|
#
?
Nov 12, 2019 18:17
|
|
|
|
| # ? May 17, 2024 18:04 |
|
|
Also re tests: I am not saying you need unit tests, but you absolutely must have an automated test suite. And if you think that your system is too complex and input dependent for that, you are wrong.
|
|
#
?
Nov 12, 2019 18:20
|
|