|
Like a salmon.
|
#
?
Aug 3, 2015 12:08
|
|
|
|
| # ? May 16, 2024 20:53 |
|
|
Jsor posted:On one hand, I agree it seems a bit excessive. On the other hand, that is exactly the sort of thing where if it stops working for some reason it takes forever to find the bug. We use mocking and this is one of the sorts of things i see sooo much in the code code:Mocking the class you are supposed to be testing......
|
|
#
?
Aug 3, 2015 12:14
|
|
|
Even if you're not using test-driven development I still feel like writing the test that you would have written if you hadn't yet written the implementation (or even had any concept of what it might look like internally) is the best option for robustness. If you actually want to test internal implementation details (like making sure one private method calls another), then I would barely even consider that a unit test. If there is an internal implementation detail that needs to be asserted as part of a unit test, then it should be part of the interface contract (either literally, or at least documented as such). It might make sense to test that the internal quicksort partition function is called log(n) times, but that's because a quicksort function that does not satisfy that property is not quicksort, even if it sorts the array. It does not make sense to test that your mocked model received exactly 1 call to .Validate() and 10 calls to .OnPropertyChanged() over the course of one unit test. Instead, I would test that saving an invalid model results in a validation error, and that observers receive property change notifications. I've heard arguments against using more than one "concrete" class per unit test, or that mixing multiple testable classes into one unit test is not actually unit testing, but more like integration testing, but I feel like that applies only if the dependencies are circular (i.e. don't test two services that depend on each other, within the same unit test -- break the loop using a mock/stub). If you have a Person model class along with a full set of PersonTests that verify it behaves correctly, then I have no problem using Person as a concrete class inside my ModelControllerTests instead of mocking a model class. As long as unit tests use only components that are themselves independently-unit tested (barring the component that is meant to be tested obviously), then it's still a unit test and only tests one thing as long as you don't stupidly write test code for your already-tested components. Yes, if you break the Person model then it may break your ModelControllerTests, but I question the value of being in the situation where some essential code is broken but some other code that depends on it directly is not. The PersonTests should be failing as well, so you will never be masking the source of a test failure. I say there is greater value in the test environment mirroring the real environment as closely as possible, meaning as few ad-hoc testing implementations of things as possible. Obviously that depends on the project size and development strategy (and exactly what the impact of a failing test is), but I don't mind 25% of the tests turning red if one of the core services breaks. It seems like that is kind of a realistic result actually, and you gotta fix that poo poo pronto or else the same number of sweeping faulty interactions will (or could) happen in production. The only time I can think of that I would substitute fake code for real code (i.e. mocks, stubs) is when the real code depends on external services or uncontrollable non-determinism. Even if you have to test a service that depends on random numbers, being able to supply a PRNG and seed to the service and then using a fixed value in the test is better than making a mock service that just returns a predefined list of numbers where randomness would otherwise go. Another option would be to design services that have modular implementations-- your service for downloading from the web could have a DownloaderProvider that, by default, uses some HttpDownloaderProvider that fetches things from the web, but then you could supply a (complete and, most importantly, independently unit-testable) implementation of DownloaderProvider called e.g., LocalDownloaderProvider that fetches things from a local store or generates them. Then the LocalDownloaderProvider is not really a mock but a concrete implementation of an interface provided by your application, rather than just transient/axiomatic test code that exists within your unit testing environment. And then the thing I said earlier about composing already-tested components within a unit test applies: you can use the LocalDownloaderProvider to test the interface of your downloader service without having to create any "fake" test-only code or rely on a mocking library to do the right thing.
|
|
#
?
Aug 3, 2015 14:25
|
|
|
Volte posted:I don't mind 25% of the tests turning red if one of the core services breaks. It seems like that is kind of a realistic result actually, and you gotta fix that poo poo pronto or else the same number of sweeping faulty interactions will (or could) happen in production. Why would you willingly make debugging your tests harder? Digging through 25% of your tests to find which one actually broke seems ridiculous compared to knowing exactly which single test broke.
|
|
#
?
Aug 3, 2015 14:30
|
|
|
Bognar posted:Why would you willingly make debugging your tests harder? Digging through 25% of your tests to find which one actually broke seems ridiculous compared to knowing exactly which single test broke. edit: The other thing is that practically, a fundamental service breaking and causing everything else to break unexpectedly is probably not going to happen enough to make that possibility a fundamental pillar of your design. If you need to alter your fundamental service, then you need to run the tests for that service and make sure they don't break. If you do change a fundamental service and suddenly everything is broken, where's the mystery? Which part of that causes your tests to become hard to debug? Optimizing for the common case is popular in algorithm design, but it seems like optimizing for the worst case is the general modus operandi for software design. Why not assume the common case will hold (i.e., the uber-simple Person model does not magically break overnight) and make it robust enough that the worst case is not a complete catastrophe (i.e., you might have to spend 15 minutes looking through 100 test cases to find out which one is the truly broken one, although looking at the reasons for the secondary failures should also be enough to clue you in) Volte fucked around with this message at 14:53 on Aug 3, 2015 |
|
#
?
Aug 3, 2015 14:45
|
|
|
Even in the worst case of suddenly having 500 failing tests with no idea of what changed, you don't actually need to figure out which is the "real" failure and which are secondary failures caused by a bug outside the thing being tested unless you're working in a language without a functional debugger or something. Just pick a failing test and step through it until you find where the incorrect value that results in the test failure is coming from.
|
|
#
?
Aug 3, 2015 15:06
|
|
|
Except that takes forever since stepping through your "unit test" involves traversing thousands of lines of fairly unrelated components you decided it wasn't worth the effort to mock out.
|
|
#
?
Aug 3, 2015 15:13
|
|
|
Jabor posted:Except that takes forever since stepping through your "unit test" involves traversing thousands of lines of fairly unrelated components you decided it wasn't worth the effort to mock out. And using implementations explicitly may not be the best option -- something like dependency injection could be used at the unit test level. If you have a test that only holds if there is some valid Model object available and allow that object to be injected, then you have a way to construct an acyclic dependency graph. If ModelControllerTests requires a Model and we inject Person, it would be trivial to detect Person-related failures by running PersonTests before even running the ModelControllerTests and bailing if any are found, thus avoiding spurious test failures. If you think about unit tests as proofs of propositions (i.e. the proposition "this class has been tested" NOT "this class is verified correct") then you can get very general tests of the form: Tested(Foo), P1, P2 ⊢ Tested(Bar) or even ∀F⊂Foo(Tested(F), P1, P2 ⊢ Tested(Bar(F))) (in this case P1 and P2 might be something like "adding one should increase total by one" or whatever your normal unit tests would be, the proof of which is given by running the test and returning a passing result). A mock/stub Foo object is a valid proof of Tested(Foo) in this case, but if we already have perfectly reasonable real instances of Tested(Foo) lying around, then why introduce additional (in some cases deeply magical) instances of a potentially complex class into the test hierarchy? Volte fucked around with this message at 15:48 on Aug 3, 2015 |
|
#
?
Aug 3, 2015 15:42
|
|
|
Gazpacho posted:A lack of bugs is desirable, but it doesn't equal a product. If what you've described is really your attitude, god help you when upper management throws you demands that strain the resources of your team(?). If your idea of planning is "show up and work," you can expect to be either late or on death march. I was being a bit flip, but it's how people later in the thread have said it goes. Their implementation of agile involves 20+ person standup meetings that take an hour, organizing works into sprints of 2 weeks, but then only fixing bugs in those sprints and not testing the fixes, meaning that the next sprint is just fixing the bugs they just made fixing their other bugs. Teams with too many open bugs literally aren't permitted to start other projects until their bugs are under a certain threshold, so now they're stuck in a permanent loop of scrambling to fix bugs but just making more work for themselves later by never testing (just reacting to more bug reports being opened). Meanwhile my team's approach has been to say "The software goes out the door 12 months from now. For the first 3 months, let's get functioning versions of our new features ready. The next 3 months are testing those features. After that we'll refactor with a focus on making the code more maintainable and extensible. The last 2 months will be integration testing and fixes related to that." Then we divvy up projects and each developer is left to figure out how to get his work done in those 3 months on his own. Unsurprisingly, we have fewer bugs and release functioning projects because we're aren't scrambling toward arbitrary biweekly deadlines for a piece of software that doesn't need to ship for over a year. Instead we are developing toward getting something we can actually use and test first, then expanding from that base. I'm probably not describing it very well, but basically everyone on the team knows what he or she needs to get done this quarter and is allowed to organize themselves in whatever way works best for them, as long as they can show progress on the project every 2 weeks at the team meeting. Everyone is, of course, encouraged to ask for help as early as possible if they feel like they're starting to fall behind, and there's inevitably always a few folks who have gotten far enough ahead on their projects to jump in and help out. I really don't like the whole artificial deadline that is a sprint. Even if the "ideal" implementation of agile doesn't do this, I feel that in the real world it just encourages developers to rush to get something "done" within this arbitrary deadline, and as a result they never slow down to test things piece by piece, or make fixes to their in-progress code beyond bandaids to "make it work". Inevitably their next sprint is then devoted to figuring out why their project doesn't work. Because they, again, have an arbitrary deadline 2 weeks away, they apply more bandaids. It becomes a death spiral. I suppose if everyone on your team is the type of person who functions at a higher level under time pressure it could work, but it's clearly not working for the folks doing it here and they are blindly sticking to it because the buzzword people are convinced it's better, despite actual results within the company showing the opposite to be true.
|
|
#
?
Aug 3, 2015 15:45
|
|
|
Volte posted:Why would it take stepping through thousands of lines of code? Look at one of your failing tests, determine which assertion is giving an incorrect result. That incorrect result should correspond to another failing test -- find that failing test, and repeat until you find the source of the failure. Again, this is an exceptional case --- you might have to work through lunch. So what happens if the "failing assertion" is due to something like, for example, a callback you expected to get never happened? Are you going to manually dig into wherever the callback was supposed to be called and then walk backwards until you find the part that didn't work? What if you don't actually know anything about that particular component? I also think you're overestimating how much effort it is to create a mock object for a particular test. In my experience, in virtually every interesting case (i.e., every case you especially want to test), it's actually easier to create a mock with the desired behaviour than it is to configure the real object to behave in the desired fashion. Is setting your data storage class up in such a way that it throws a CorruptedDataStoreException honestly easier for you than writing when(mockDataStore.get(any()).throwException(CorruptedDataStoreException)?
|
|
#
?
Aug 3, 2015 16:25
|
|
|
LeftistMuslimObama posted:I really don't like the whole artificial deadline that is a sprint. Even if the "ideal" implementation of agile doesn't do this, I feel that in the real world it just encourages developers to rush to get something "done" within this arbitrary deadline, and as a result they never slow down to test things piece by piece, or make fixes to their in-progress code beyond bandaids to "make it work". Inevitably their next sprint is then devoted to figuring out why their project doesn't work. Because they, again, have an arbitrary deadline 2 weeks away, they apply more bandaids. It becomes a death spiral. You are looking at a broken agile process. They are doing it wrong. If it's being done properly, there is no rushing, because the team commits to no more work than they are actually able to deliver based on previous sprints. This is why it's important to have each story contain tasks for coding, testing, documentation, etc. Those things are part of the estimate, because they are work to be done that requires effort. If any of those things aren't done, the story isn't done. If the team is lovely, no process on earth will be able to save them from being lovely.
|
|
#
?
Aug 3, 2015 17:13
|
|
|
Ithaqua posted:You are looking at a broken agile process. They are doing it wrong. If it's being done properly, there is no rushing, because the team commits to no more work than they are actually able to deliver based on previous sprints. This is why it's important to have each story contain tasks for coding, testing, documentation, etc. Those things are part of the estimate, because they are work to be done that requires effort. If any of those things aren't done, the story isn't done. I agree it's broken. I'm just saying that because Agile is a big buzzword right now, lots of teams that aren't really a great fit for the process are being pressured by management into adopting it anyway. There is no one-size-fits-all for productivity. Whatever you do needs to work well for the type of people who are on your team. That's why it's dumb and bad when people write thousand-paragraph essays about how waterfall is "so 2005" or "agile can boost your productivity by a million%" or whatever. Everyone in programming is to eager to say "Thing X worked for me in this specific scenario, therefore it will work for everyone in every circumstance". That way lies the road to JavaBeanWidgetFactoryHammerFactoryFactoryDeployment.java.
|
|
#
?
Aug 3, 2015 18:30
|
|
|
Jabor posted:So what happens if the "failing assertion" is due to something like, for example, a callback you expected to get never happened? Are you going to manually dig into wherever the callback was supposed to be called and then walk backwards until you find the part that didn't work? What if you don't actually know anything about that particular component? I'm not saying mock objects aren't valuable in their place (whatever their place may be) but I reject the notion that treating components as an island for testing purposes is the only good way to do it. Dependency injection that knows how to associate a component with its test cases (or unit tests that are generic in the underlying implementation) seem like a fairly straightforward way to avoid strong coupling in the unit tests while still letting the components interact with each other more-or-less as they would in production. The more divergence between the testing environment and the production environment, the less effective the tests are going to be at rooting out production-time bugs. Hell, if you wanted to get fancy you could have it attempt a sequence of possible services to inject, choosing one whose tests succeed. If all you care about is "given a bunch of services that have been shown to be tested successfully..." then it doesn't actually matter what concrete implementation gets put in, as long as it behaves itself. And if it does matter, then that's an integration test.
|
|
#
?
Aug 3, 2015 18:41
|
|
|
LeftistMuslimObama posted:I really don't like the whole artificial deadline that is a sprint. Even if the "ideal" implementation of agile doesn't do this, I feel that in the real world it just encourages developers to rush to get something "done" within this arbitrary deadline, and as a result they never slow down to test things piece by piece, or make fixes to their in-progress code beyond bandaids to "make it work". Gazpacho fucked around with this message at 19:01 on Aug 3, 2015 |
|
#
?
Aug 3, 2015 18:58
|
|
|
On another note, we're currently studying network flows in my algorithms class. The textbook and lectures have been kinda handwavy about the actual data structures involved in the algorithms discussed. I know how a general graph is stored, but when we're returning things like maximal flows (which are different from the value of a maximal flow) and residual networks, I would think how this information is represented would have an impact on the running time. Instead it's kinda handwaved over and we're just told "and the running time is polynomial in m*n, here's a proof with no mention of the steps taken to store and return the flow or residual network". Should this concern me, or is that total normal in an undergrad level class? This is the second-to-last lecture in the course, and despite carrying an A through the class so far I feel like I've done far more handwaving than algorithm creating and that seems to be the endorsed way of approaching most problems. I get not wanting to get bogged down in implementation details, but even for homeworks on dynamic programming we were told to avoid describing data structures used in detail. He literally told us "just say you're memoizing it, the data structure would be informed by your algorithm in real life anyway". Except that I had a lot of trouble comming up with dynamic algorithms without first thinking about how I'd represent the memoized data.
|
|
#
?
Aug 3, 2015 19:21
|
|
|
LeftistMuslimObama posted:On another note, we're currently studying network flows in my algorithms class. The textbook and lectures have been kinda handwavy about the actual data structures involved in the algorithms discussed. I know how a general graph is stored, but when we're returning things like maximal flows (which are different from the value of a maximal flow) and residual networks, I would think how this information is represented would have an impact on the running time. Instead it's kinda handwaved over and we're just told "and the running time is polynomial in m*n, here's a proof with no mention of the steps taken to store and return the flow or residual network". Should this concern me, or is that total normal in an undergrad level class? The level of detail you're getting sounds pretty normal. The focus of algorithms classes is on proving that an efficient algorithm exists, not necessarily what kind of code or low level data structures would be used unless they have interesting effects. Unfortunately, it takes a little while to find exactly what basic operations can be done obviously and which need more explanation. Since you sound curious about it, I think residual networks would be stored in a normal graph data structure using adjacency lists which has been modified a bit so each edge object also stores its residual capacity. Whenever an edge goes to 0 residual capacity you could remove it from your data structure as easily as you delete a node from a linked list. You could add new residual edges by just inserting into a vertex's linked list. You could store the flow itself and the algorithm's output in a separate copy of the graph where each edge object is augmented with the current flow value. Of course, the important thing here is that if you give me a pointer to an edge, I can do whatever I want to that edge and that edge alone in constant time. For dynamic programming, pretty much everything is just stored in a multi-dimensional array. You write and lookup the solution to subproblem (x,y,z) in array position subproblem_array[x,y,z]. It really is the dumbest translation of math to data structure you could imagine, and detailing it is therefore kind of redundant. You're encouraged not the think about the data structure so that you keep your focus on the recursive structure of the problem instead of the tedious for loops you'd write when implementing a solution. You should write out the code for something like longest common subsequence once in case you haven't though, just to know you can and in case a dynamic programming problem comes up during a future interview. It will also remind you why nobody wants to make you write that out for every homework and why nobody would want to grade it if you did write it out.
|
|
#
?
Aug 3, 2015 19:55
|
|
|
Gazpacho posted:I agree with this criticism of the sprint-based approach. Developing a complex system from scratch, as a sum of end-user features each developed in two weeks, is bloody stupid. The obsession with completing end-user features means that debt never gets paid down and accumulates to toxic levels. Where I work we try to deliver the smallest, most simple part of a feature possible and iterate from there. So a recent example is on a huge appliance selling website we want to add a "tick to also buy similar product" to an existing list of suggested products; similar to amazon. Our first pass is literally "I see checkboxes now" which is feature-toggled to show and work in our QA + Dev environments but is off in live. Once tested we iterate again; ticking each box will update your subtotal etc. etc. Once its complete the toggle is deleted and its on all the time. There are some things which are really tiny and so our standard "sprint" model falls apart but a lot of stuff can be boiled down to the bare minimum as long as there's a way to hide it and everyone is on board; It works well now, but its taken months of utter pain and fighting PMs/BAs but the QAs love it as they test tiny poo poo now. Cancelbot fucked around with this message at 20:58 on Aug 3, 2015 |
|
#
?
Aug 3, 2015 20:54
|
|
|
Gazpacho posted:I agree with this criticism of the sprint-based approach. Developing a complex system from scratch, as a sum of end-user features each developed in two weeks, is bloody stupid. The obsession with completing end-user features means that debt never gets paid down and accumulates to toxic levels. There are multiple levels. You can have "epic" stories that span multiple iterations. It's the same thing we already do when writing software, just a bit more formalized. "I need to write a web application that frobbles." (epic, might take months). "The first thing I need is a login page." (user story, might take a day or two). "The login page should have two textboxes for username and password and a button I can click" (task, might take a few hours)
|
|
#
?
Aug 3, 2015 21:01
|
|
|
Ithaqua posted:There are multiple levels. You can have "epic" stories that span multiple iterations. And importantly, the customer can see that login page, or that frob, or what have you and have input sooner, so that it's not months down the line and too late to rewrite when they first see the progress and say "no, not frobbles - fribbles and glops"
|
|
#
?
Aug 3, 2015 21:24
|
|
|
Jabor posted:Except that takes forever since stepping through your "unit test" involves traversing thousands of lines of fairly unrelated components you decided it wasn't worth the effort to mock out. As long as you can reliably determine whether or not things have gone wrong yet, n-ary searching tens of thousands of lines of code for the problematic spot in a debugger generally takes a few minutes at most.
|
|
#
?
Aug 3, 2015 21:25
|
|
|
Part of Agile as it's interpreted at my company (and presented in the external training class) is that at the end of the sprint you demonstrate your features to the product team. Complete features, without any scaffolding. This continues to be a source of pain for the reasons I described, but what's the alternative? We can't use the planning meetings to argue about details of implementation and what completion "really" means. There's a tension between the way managers and developers think of development work that I believe Agile aggressively sweeps under the rug.
Gazpacho fucked around with this message at 21:58 on Aug 3, 2015 |
|
#
?
Aug 3, 2015 21:50
|
|
|
Gazpacho posted:Part of Agile as it's interpreted at my company (and presented in the external training class) is that at the end of the sprint you demonstrate your features to the product team. Complete features, without any scaffolding. This continues to be a source of pain for the reasons I described, but what's the alternative? We can't use the planning meetings to argue about details of implementation and what completion "really" means. There's a tension between the way managers and developers think of development work that I believe Agile aggressively sweeps under the rug. Each user story should have clearly defined acceptance criteria. That's what "done" means. That's what you demonstrate.
|
|
#
?
Aug 3, 2015 22:06
|
|
|
Ithaqua posted:Each user story should have clearly defined acceptance criteria. That's what "done" means. That's what you demonstrate. There are some nuances here. Is "done" showing completed AC on a dev box on your own branch (captured by screenshots of course), or deployed into a CI environment with automated E2E tests to ensure AC continue to be met? But yes, you can in fact argue out the AC on every single story. If you find it particularly hard to write AC (and really, it should be product owner/management writing them), then you likely don't have a user story, but something else. Sometimes this is nearly unavoidable (technical refactoring to remove a race condition for example), but usually it means you need to re-evaluate what you consider a story and do more vertical slicing of functionality.
|
|
#
?
Aug 3, 2015 22:16
|
|
|
Ithaqua posted:There are multiple levels. You can have "epic" stories that span multiple iterations. Cool, but a login page implies an authentication system but what type? Is it OAuth? Social auth? Username/password? 2-factor? Each one has its own implementation details that have long-reaching implications across the entire application. What's the official Agile (tm) stance on dealing with that kind of thing?
|
|
#
?
Aug 3, 2015 22:29
|
|
|
Blinkz0rz posted:Cool, but a login page implies an authentication system but what type? Is it OAuth? Social auth? Username/password? 2-factor? Each one has its own implementation details that have long-reaching implications across the entire application. Well, the software engineer stance is to abstract it into a discrete component so you can switch relatively easily in the future and not tie every part of your app deeply into auth code. The Agile stance is then to implement a first-choice and/or least effort Auth component until the product requirement can be better defined.
|
|
#
?
Aug 3, 2015 22:52
|
|
|
The thing I dislike about how Agile is often implemented is that managers will assign inappropriate weights to deadlines. At my last job (which in general was very poorly managed) if a task was supposed to be completed at the end of a sprint then by god it needed to be completed - it didn't matter if people were sick or went on scheduled vacations that the managers forgot about when pointing the sprint, it didn't matter if we underestimated the complexity of the task, and it didn't matter that nobody outside of the dev team (users, customers, or investors) was expecting us to finish the task at the end of the sprint. Not every Agile environment I've been in was this stupid, but the only environments I've been in that were this stupid were Agile environments.
|
|
#
?
Aug 4, 2015 00:45
|
|
|
loinburger posted:The thing I dislike about how Agile is often implemented is that managers will assign inappropriate weights to deadlines. At my last job (which in general was very poorly managed) if a task was supposed to be completed at the end of a sprint then by god it needed to be completed - it didn't matter if people were sick or went on scheduled vacations that the managers forgot about when pointing the sprint, it didn't matter if we underestimated the complexity of the task, and it didn't matter that nobody outside of the dev team (users, customers, or investors) was expecting us to finish the task at the end of the sprint. Not every Agile environment I've been in was this stupid, but the only environments I've been in that were this stupid were Agile environments. I think agile works a lot better in pure web companies. If your poo poo is on the internet, you click a button to deploy, and the market landscape can change at any moment, then you'd better be agile. Lots of companies do not work that way. They have fixed deadlines and fixed features. If both your feature set and your timeline are fixed, then the project planning aspects of scrum are not useful and should not be used.
|
|
#
?
Aug 4, 2015 02:14
|
|
|
loinburger posted:The thing I dislike about how Agile is often implemented is that managers will assign inappropriate weights to deadlines. At my last job (which in general was very poorly managed) if a task was supposed to be completed at the end of a sprint then by god it needed to be completed - it didn't matter if people were sick or went on scheduled vacations that the managers forgot about when pointing the sprint, it didn't matter if we underestimated the complexity of the task, and it didn't matter that nobody outside of the dev team (users, customers, or investors) was expecting us to finish the task at the end of the sprint. Not every Agile environment I've been in was this stupid, but the only environments I've been in that were this stupid were Agile environments. That's just bullshit and not agile at all. At the risk of a No Trust Scottsman fallacy. Agile is about understanding velocity and delivering a consistent amount of work based on that. If it turns out that you can do 15 story points a Sprint, and then when you go on vacation for half the Sprint you still deliver 15 story points by crunching your eyes out it's going to lead planners to say, "Bob can deliver 15 story points while only here for a week, we can schedule him for 30 from now on." And people crunching to meet those goals sprint over sprint are just reinforcing the beliefs of the planners that it's acheivable. Where I am we know how many points teams can pull but we keep team sizes down to 3-4 people which helps. I really feel there should be a process horrors thread.
|
|
#
?
Aug 4, 2015 04:44
|
|
|
Bruegels Fuckbooks posted:Lots of companies do not work that way. They have fixed deadlines and fixed features. If both your feature set and your timeline are fixed, then the project planning aspects of scrum are not useful and should not be used. Yes and no. What if the fixed features cannot be delivered by the fixed deadlines? What if it's actually *most* of the fixed features that can't be delivered? How would you know? How do you respond? Agile at least provides a framework for recognizing and dealing with these problems. Hughlander posted:planners to say, "Bob can deliver 15 story points while only here for a week, we can schedule him for 30 from now on." Oh god, if anyone other than the dev team is determining velocity and story sizing, everyone and everything is hosed. Excessively lazy people will happen and they need to be policed to an extent, but a simple answer of "yeah, we did that with crunch time" should suffice in a healthy organization. Also, that dev team needs to be called out for not accounting for vacation in their planning.
|
|
#
?
Aug 4, 2015 04:53
|
|
|
Volte posted:That seems orthogonal to this concern. If your callback is supposed to be called, there should be a unit test that asserts that fact. If some secondary failure occurs because the callback was never called, you should be able to find the unit test that is failing because of it. "The callback wasn't called" is nowhere near enough information to deduce the underlying reason for that failure. Again, how are you going to figure out where the fault is in a set of components you have no understanding of? Your perception of how difficult various things are to achieve seems completely out of whack with the reality of software development. Apparently writing mocks is hard but writing tests that cover every code path when using a randomly-selected implementation of a dependency is easy? Do you only write tests for the nominal cases and just hope everything works out in the unusual ones?
|
|
#
?
Aug 4, 2015 06:24
|
|
|
Volte's reasoning is actually completely spot-on, I think. I am working on a large system that is split off into many microservices (and in a fairly sane fashion even). I don't have ownership of all of these services but every now and again I may need to go in another team's service to add a tiny feature or fix a bug so that my team can continue working on our own services. If it's a service that has a test for every layer of the service, ideally with mocks for the other layers, it has essentially become a refactor-proof service. *Any* change I make to a method signature will cause a cascade of test failures, missed mock invocations, and so on. Terrible. In the service I'm working on we test from the API inwards (both REST and message queues). Dependencies are replaced, but not necessarily with mocks. The datasource is an in-memory database, we have an embedded FTP server to check our FTP logic, etc. For external services we rely on, we either add a fake class or an actual mock. This works really quite well. I can and have refactored quite a few of the guts of the system without getting any failing tests simply because the output remained correct. It does mean we need pretty exhaustive tests of your API layer. But ultimately, I think that's fine -- it's what on the API of your service that's visible to other systems and users, so that's what should be correct. We use a BDD framework (uh-oh) to properly name all the assertions and scenarios, so if I need to check that a callback is called, I also know *why* it should be called and, if that fails, I should be able to reason out what went wrong. Additionally, if that callback is more than an implementation detail, it's going to be visible on an API somewhere and will have a full suite of tests specifically to that callback.
|
|
#
?
Aug 4, 2015 10:54
|
|
|
Jabor posted:"The callback wasn't called" is nowhere near enough information to deduce the underlying reason for that failure. Again, how are you going to figure out where the fault is in a set of components you have no understanding of? A mock is just an instance of an interface that you have configured to behave in a specific way --- I advocate making your actual services configurable enough to be used that way (where possible/practical), and then using them directly in order to avoid diverging the code path of the test code from the code path of the application code any more than necessary. I didn't think it was that controversial. And yeah obviously if it's a third party component you're going to have to mock it or whatever. But that comes with huge caveats since you're mocking something you don't fully understand and you can definitely miss bugs that way. I can't think of a great way to make unit tests interact nicely with third party components since they are just black boxes. Volte fucked around with this message at 11:39 on Aug 4, 2015 |
|
#
?
Aug 4, 2015 11:26
|
|
|
Sagacity posted:Volte's reasoning is actually completely spot-on, I think. I am working on a large system that is split off into many microservices (and in a fairly sane fashion even). I don't have ownership of all of these services but every now and again I may need to go in another team's service to add a tiny feature or fix a bug so that my team can continue working on our own services. If it's a service that has a test for every layer of the service, ideally with mocks for the other layers, it has essentially become a refactor-proof service. *Any* change I make to a method signature will cause a cascade of test failures, missed mock invocations, and so on. Terrible. I'm not understanding why you would have cascading failure, or how mocking is the real cause here. Could you please elaborate? I have a service A that collaborates with service C. I have some tests for A that mock C. If I stop collaborating with C, why shouldn't my tests fail? Certainly, things become more brittle when you mock, but I don't see how you get around this by using real services (or are you advocating something else?). Now you certainly shouldn't be testing queries to A and asserting on the results of C, but that seems a separate issue to me (command/query separation). edit: Here's an example unit test from our code that has a simple mock. php:<?
public function testValidQueryReturnsGLCode()
{
$this->queryRuleEvaluatorStub
->method('evaluateRuleQuery')
->willReturn(array('glcode' => '1'));
$glRuleEvaluator = new GLRuleEvaluator($this->queryRuleEvaluatorStub, $this->ruleQueryFormatterStub);
$result = $glRuleEvaluator->evaluateGLRule($this->rule, '10');
$this->assertEquals(array('glcode' => '1'), $result);
}
?>IT BEGINS fucked around with this message at 14:31 on Aug 4, 2015 |
|
#
?
Aug 4, 2015 14:06
|
|
|
IT BEGINS posted:
This is a bad test and a bad mock because it's testing "collaboration" as you put it, not functionality.
|
|
#
?
Aug 4, 2015 15:07
|
|
|
IT BEGINS posted:I'm not understanding why you would have cascading failure, or how mocking is the real cause here. Could you please elaborate? It's easy to miss the point using such a simple example. Imagine we're in the real world, and A has a number of methods that you're testing, with a number of tests for each method. A calls a few different methods on C, but in different ways in different methods and different scenarios under test. Now let's imagine you want to refactor. There's a number of different ways you might be refactoring, including eliminating C completely, changing C's interface, changing A's interface, breaking one or both down into smaller components. But for this example let's just assume you are modifying C's interface and leaving A's alone. Scenario 1, with mocks: You change the interface to all of C's methods (and C's tests), and you now must update A to use the new interface. (By the way, in some languages, if you are really prone to loving up, and you don't have solid integration tests, tests might even pass without updating A. Oops!). But then, you have to update A's tests, to reflect C's new interface. This means updating all the mocks, of which there are likely to be a lot more than method call sites in A itself. You might only have to update a handful of calls to the real C in A, but many more mocked method calls in the tests. Scenario 2, with an actual usable (tested) instance of C: You update C's interface (and tests), and you update A to use the new interface, and you're done. If you did everything right. Now imagine that it's not just A that uses C, but 5 other classes. This is how mocking can end up being a massive frustration when refactoring, and is one reason why I only mock when I have good reason to (typically, because I need to verify calls to side-effecting methods, or because some dependency of the class under test would require too much complexity to get in the proper state for the test. most often the former).
|
|
#
?
Aug 4, 2015 15:08
|
|
|
Steve French posted:Now imagine that it's not just A that uses C, but 5 other classes. This is how mocking can end up being a massive frustration when refactoring, and is one reason why I only mock when I have good reason to (typically, because I need to verify calls to side-effecting methods, or because some dependency of the class under test would require too much complexity to get in the proper state for the test. most often the former). Ah, that makes sense. I agree - I suppose some good candidates for mocking would be things like the database layer, particularly if they have side-effects?
|
|
#
?
Aug 4, 2015 15:19
|
|
|
Volte posted:I don't really understand what you're talking about. What I'm talking about doesn't really have anything to do with what specific failures have occurred or why. If your callback is failing to be called, some test somewhere should be failing because of it, or else your unit test coverage is incomplete and missing out on a bug. If you don't have enough understanding of the underlying library to deduce the cause of the failure, that's got nothing to do with unit testing. I am merely advocating using as much of your actual code as possible in the unit tests rather than inventing new ad-hoc code for each unit test, with the (theoretical) justification that you could encode dependency information within the unit tests in order to avoid cascading test failures. (Ignore the stuff about random selection...obviously, you're going to need to know which services were injected in order to compare the outputs of the services to the expected outputs. derp) The key insight to large-scale application development is being able to treat other components of your own code as abstracted black boxes. If diagnosing the root cause of a big failure requires you to have comprehensive knowledge of most of the application, that's indicative of an entirely unsustainable development practice. And are you seriously advocating making your production code configurable to the point of being able to say "this particular request should fail with this particular error"? Do you really think that's useful, and easier than writing the literally one-liner mock for the test that actually exercises that failure?
|
|
#
?
Aug 4, 2015 16:02
|
|
|
I always thought the reason for mocks was you want to test A, which uses C, but C hasn't been implemented yet.
|
|
#
?
Aug 4, 2015 16:12
|
|
|
Jabor, just for reference, how big are the projects you are working on? Are they leaning more towards monolith or more towards micro? I assume the former, because then you will definitely have to start reasoning about separate parts of the same projects as black boxes.
|
|
#
?
Aug 4, 2015 16:14
|
|
|
|
| # ? May 16, 2024 20:53 |
|
|
Jabor posted:The key insight to large-scale application development is being able to treat other components of your own code as abstracted black boxes. If diagnosing the root cause of a big failure requires you to have comprehensive knowledge of most of the application, that's indicative of an entirely unsustainable development practice. edit: With large projects I suppose the test case overlap is a logistical issue. Without built-in assurances that all the components you are using in your test case have already been successfully tested against a predefined set of test cases, it's up to the developers to ensure that the components that are used in each unit test are fully tested, and that the tests are comprehensive enough. But again, mocking in that case comes with a different set of fragilities, like being unfamiliar with the inner workings of the classes you need to mock, and the possibility of refactoring changes that do not propagate properly to mocks and thus allow outdated tests to continue to pass. Volte fucked around with this message at 16:52 on Aug 4, 2015 |
|
#
?
Aug 4, 2015 16:42
|
|