|
Munkeymon posted:Does it maintain those advantages if you either turn off global dirty reads on MySQL or enable them on Postgress? Whenever a big company switches from PostgreSQL to MySQL it's pretty much due to that. Anything else, you can configure away or the performance difference is real minor. Maybe there are other reasons I am blind to. It's not even something wrong with PostgreSQL. Changing how it works is a trade-off. MySQL's way has potential problems. Postgres' way has upfront costs. Instagram uses PostgreSQL as far as I know, and they're massive. Uber could have still used PostgreSQL and done fine. Another interesting bit of information is Uber used MySQL initially, switched to PostgreSQL in late 2012, and then switched back at whatever point in 2016. "pick a random popular SQL database" is not far from the truth. Khorne fucked around with this message at 15:49 on Jul 12, 2018 |
#
?
Jul 12, 2018 15:26
|
|
|
|
| # ? Jun 7, 2024 16:09 |
|
|
What strategies are people using for high availability Postgres? I haven't seen anything as convenient as MySQL with Galera.
|
|
#
?
Jul 12, 2018 17:50
|
|
|
rt4 posted:What strategies are people using for high availability Postgres? I haven't seen anything as convenient as MySQL with Galera. I just tell AWS to do it for me.
|
|
#
?
Jul 12, 2018 18:11
|
|
|
Khorne posted:It's not due to reads. MySQL and PostgreSQL are neck and neck with PostgreSQL even having a few percent advantage when it comes to the majority of tasks, even at very large scales. The real issue is in how PostgreSQL handles storing indices and how it does replication. The most prominent example, and honestly why you see people switch from PostgreSQL to MySQL but not MySQL to PostgreSQL at a large scale, is why uber switched. Here is a postgres expert "rebuttal" of the claims which amounts to "everything they said is true but they got a detail or two wrong" and "it's apples and oranges". Admittedly, one of the details was a big one but it doesn't change the fundamental claim of the Uber engineers. Ah, gotcha. I asked because one of my coworkers was talking up how MySQL is 'better' for web application data because it's configured to allow dirty reads by default, and so faster with fewer deadlocks, which I found kind of absurd since it's all just settings and clearly he knows about that setting, so why the hell is that a concern?
|
|
#
?
Jul 12, 2018 18:49
|
|
|
rt4 posted:What strategies are people using for high availability Postgres? I haven't seen anything as convenient as MySQL with Galera. It's all in Postgres 10, there are a suite of HA options and configurations available.
|
|
#
?
Jul 12, 2018 19:17
|
|
|
"READ UNCOMMITTED is cruise control for cool!" - our CTO.
|
|
#
?
Jul 12, 2018 21:37
|
|
|
Khorne posted:Here is a postgres expert "rebuttal" of the claims which amounts to "everything they said is true but they got a detail or two wrong" and "it's apples and oranges". Admittedly, one of the details was a big one but it doesn't change the fundamental claim of the Uber engineers. That is a pretty poor characterization of the rebuttal. You might notice numerous slides that contain just ¯\_(ツ)_/¯ in the presentation - a lot of the claims by uber that are marked 'mostly true' are 'mostly true' because they are not even wrong.
|
|
#
?
Jul 13, 2018 14:13
|
|
|
Bruegels Fuckbooks posted:That is a pretty poor characterization of the rebuttal. You might notice numerous slides that contain just ¯\_(ツ)_/¯ in the presentation - a lot of the claims by uber that are marked 'mostly true' are 'mostly true' because they are not even wrong. People are going to polarize on whose side they're on because they're either taking the pedantic side or the practical side. Some of the uber complaints weren't true, but that doesn't invalidate the legitimacy of the actual problems using Postgre was causing for them and the differences between Postgre and MySQL that do exist. All of the "apples and oranges", for example, is mostly relevant. If your job needs an apple you don't want an orange, but that doesn't mean there's something wrong with oranges. Dismissing the side effects of something because "it's different than what you're comparing it to" is laughable, because in practice all that matters is the outcome. All of the "but no performance comparisons were provided" seems overly defensive. Khorne fucked around with this message at 16:59 on Jul 13, 2018 |
|
#
?
Jul 13, 2018 16:52
|
|
|
I am kinda-sorta-eventually going to be updating our testing framework. Right now we load multiple sets of test data from excel spreadsheets, and I figure there has to be a better way. I was thinking of using CSV, but would be open to
|
|
#
?
Jul 13, 2018 22:34
|
|
|
AFashionableHat posted:It's not awful, but depending on the level of damage a credential leak can lead to I might call it insufficient. Generally speaking, I try to not leave access tokens for anything in plaintext in my database. (As an example, when I create a session token and give it to a client, I store its hash rather than the token itself.) Just wanted to say thanks for this. Got credstash up and running and it's working like a champ.
|
|
#
?
Jul 14, 2018 00:22
|
|
|
SardonicTyrant posted:I am kinda-sorta-eventually going to be updating our testing framework. Right now we load multiple sets of test data from excel spreadsheets, and I figure there has to be a better way. I was thinking of using CSV, but would be open to CSV is fine. JSON can be good for representing more complex relationships. You can go all the way up to creating an SQL dump of a database and loading that if you want to get really fancy, but I'd guess that's rarely necessary for tests. I guess the real question though is, what problem are you trying to fix? What's wrong with the Excel spreadsheets, just that it's hard to view diffs when they get updated? If that's the only problem, switch to CSV and forget about overengineering it.
|
|
#
?
Jul 14, 2018 00:53
|
|
|
Our current test suite currently uses excel to store data, and it's a mess in ever new and exciting ways. I was just making sure.
|
|
#
?
Jul 14, 2018 01:56
|
|
|
So, in Django-land you write "unit" tests using the Django test client which kind of emulates a browser hitting your views. This is the common, accepted way of doing things. I've been thinking about this and it seems way wrong. For those unfamiliar, you can set up a Django view that lists the books stored in your database using the Django-provided ListView like this: Python code:But, that's just wrong! We don't want to test all the internals of Django. We just want to test our code and our code consists of setting those three class variables so we can just test those! Now, you'll probably want to write integration or functional tests and the test client might be good for that, but you also don't want to not have any unit tests for your view, and thats exactly what everyone is telling you you've got when they have you write tests with the test client. So, my question is: am I wrong here? (asking here instead of the Django thread because I want a wider perspective)
|
|
#
?
Jul 16, 2018 17:00
|
|
|
You could have a bunch of hooks into Django that change the request in ways you forgot about when writing the view, and you could have more hooks that do things to the output of the view. At that point you need to test the full request-handling mechanism.
|
|
#
?
Jul 16, 2018 17:13
|
|
|
nielsm posted:You could have a bunch of hooks into Django that change the request in ways you forgot about when writing the view, and you could have more hooks that do things to the output of the view. At that point you need to test the full request-handling mechanism. Yes, but that's always the case with things you write. You could always forget something, no? In this instance, you should be testing the hooks you've written or configured to be used.
|
|
#
?
Jul 16, 2018 17:17
|
|
|
Your hooks work when tested individually. Your view works when tested under a standard Django environment. Your view breaks when run under the customized Django environment. Yes it's more of an integration test then.
|
|
#
?
Jul 16, 2018 17:22
|
|
|
nielsm posted:Your hooks work when tested individually. Your view works when tested under a standard Django environment. Your view breaks when run under the customized Django environment. Yes it's more of an integration test then. Right, I'm not arguing against integration tests.
|
|
#
?
Jul 16, 2018 17:26
|
|
|
The difference between unit and integration tests are a matter of perspective. A standard unit test also tests the python interpreter, standard library, your OS libc, the compiler that was used to build python, your kernel, your CPU, etc. Does this mean that it's actually an integration test, too?
|
|
#
?
Jul 16, 2018 20:58
|
|
|
Suspicious Dish posted:The difference between unit and integration tests are a matter of perspective. A standard unit test also tests the python interpreter, standard library, your OS libc, the compiler that was used to build python, your kernel, your CPU, etc. I'd say the intent is what matters, i.e a unit tests is for when I'm interested in validating the business logic of my component. The test will run on my build sever (or a container that matches the production environment as much as possible) to remove those variables and give us repeatable builds to prove there are no regressions between versions. However if I'm trying to test how my code interacts with another component in the system (i.e. the database or another web service) then I treat that as integration testing since I now have to orchestrate the setup and tear down of all those components as part of the test.
|
|
#
?
Jul 16, 2018 21:11
|
|
|
Suspicious Dish posted:The difference between unit and integration tests are a matter of perspective. A standard unit test also tests the python interpreter, standard library, your OS libc, the compiler that was used to build python, your kernel, your CPU, etc. Yeah it’s always been kinda handwavy to me. It’s a useful distinction but I define them entirely in the context of a particular project/team/organization/whatever, and will happily ask what someone means if they say "integration test" or "unit test" out of that context.
|
|
#
?
Jul 16, 2018 23:20
|
|
|
From my perspective, it's a unit test because you're assuming that Django already works, just like how Python's unittest module is named assuming that Python already works. But it's more a semantics argument than anything else. Any testing at all, no matter the kind, is good.
|
|
#
?
Jul 17, 2018 00:08
|
|
|
I've definitely seen individual tests that have negative value. To use an example, if you wrote a test that mocked out Book.objects.filter, and tested that Thermopyle's code assigned the return value to the right field, that would be a bad test. It doesn't provide any reassurance that the code is doing the right thing, all it does is inhibit later changes and refactoring. A meaningful test has to do more than just re-state the code being tested.
|
|
#
?
Jul 17, 2018 00:15
|
|
|
A key difference between whatever you want to call unit tests and whatever you want to call integration tests is often that integration tests can takes many times as long to run...and thats exactly the case case with Django's test client. Additionally, tests that use the test client probably have at least 4x many lines of code than a unit test that just directly tests you have the view configured correctly. This means its easier to gently caress up the test so that you're testing the wrong thing. This is probably just me being too sensitive to the issue.
|
|
#
?
Jul 17, 2018 01:49
|
|
|
Jabor posted:I've definitely seen individual tests that have negative value. One of my more recent professional journeys has been trying to give lots of thought and care to what’s worth testing, and what’s not, as opposed to flitting between “test everything!” and “tests suck!”
|
|
#
?
Jul 17, 2018 12:53
|
|
|
raminasi posted:One of my more recent professional journeys has been trying to give lots of thought and care to what’s worth testing, and what’s not, as opposed to flitting between “test everything!” and “tests suck!” 3 of us went out for lunch yesterday with 42 years of collective experience developing - we were having the same conversation. It seems we can solve a lot of complicated problems but not come up with a definitive testing strategy. It always feels like a pendulum for me, the last place didn't have enough tests, the current place has too many tests, repeat.
|
|
#
?
Jul 17, 2018 12:58
|
|
|
Thermopyle posted:A key difference between whatever you want to call unit tests and whatever you want to call integration tests is often that integration tests can takes many times as long to run...and thats exactly the case case with Django's test client. "Unit test" and "integration test" are tough terms because they're used to talk about two different trade spaces (not to mention the confusion caused by the utterly useless distinction drawn by terms like "unit test framework"). The first is test execution time, where unit tests are generally very fast and integration tests are slower. The second is granularity of testing where unit tests are more fine grained and integration tests are more coarse. I generally use "unit test" and "integration test" to refer to the execution time trade space, as I feel that one is the one that is front and center every day in our work. For me, unit tests are ones we can run quickly after every build and they should give us decent coverage. Integration tests, for whatever reason, cannot be quickly run after every build. Either they require some manual setup, need a scarce resource, or they just take a long time to execute. My feelings about fine grained vs coarse grained tests are that the trade space is how easy it is to uncover a particular issue with a test vs maintenance cost of the tests. The more fine grained the tests, the higher their likely maintenance costs (fragile tests) but the more likely it is that a very specific test will break if a regression happens. The more course grained the tests, the lower their likely maintenance costs but the less likely that a very specific test will break if a regression happens. When testing, I start with a more coarse grained test then add more fine grained tests if a component is particularly critical and I want to test more nuanced cases more closely. Generally speaking, I prefer a coarse grained test. We don't normally need to validate every behavior with a test, and such testing becomes very expensive to maintain. However, there are times when setting up the test environment might be burdensome at a coarse grained testing level but easy at a finer grained level such that we can't include the coarse grained tests at build time. In those cases, some fine grained build time tests might be warranted so that we get some coverage at build time. Regarding the Django approach, it sounds like those tests might take a bit long to run. However, they sound like they provide very adequate coverage of the particular methods as well as coverage of possibly unexpected behaviors due to hooks. They are coarse grained but they're providing you with good coverage and they don't sound like they're too fragile. That is, the tests only change when the urls change, not when method names change. On this basis the Django test client approach seems correct to me. Another question to consider with the Django test client is "how hard is it to hook up the debugger and debug a failure to get at the specific cause of the failure using the test?" As I've mentioned, one of the difficulties of a coarse grained test like this is that the particular failure may not be exposed by the test failure. If it's very easy to hook up the debugger, step through, and find out it's the view that's broken and not the hook, then this is another point for Django test client approach. Now, you've also mentioned that it takes 4x as long to run these tests as it would take to run a more standard unit test with unittest and nose or something. Can this be ameliorated by running only the subset of the tests affected by your changes or some other method that doesn't involve writing faster, fine-grained, more fragile tests? If not, you may want to invest in more fine grained tests but beware of the maintenance costs you're incurring by doing so.
|
|
#
?
Jul 17, 2018 14:53
|
|
|
That's a good post. I also find the relative usefulness of different types of tests change depending upon what stage of development we're in. If we're exploring the problem, actively developing a solution for the problem, mostly finished solving the problem, or maintaining old code. For example, if I'm splitting a monster function up into smaller functions, I want fine grained tests. Luckily, django's test runner does make it easy to run specific tests, tests in a specific package, and even tag different tests. I can tag test client tests "integration" or "slow" or both, and slice and dice them that way. I think the real reason I brought the whole thing up is that it just rubs me the wrong way that everywhere these Django test client tests are called "unit" tests. Even though the concept of a unit test is kind of slippery it comes with expectations like "doesn't test a bunch of tangential code" and "is fast to run". Django is a lot of people's first exposure to coding much past Hello World and I'm not sure the community is doing these people any favors here.
|
|
#
?
Jul 17, 2018 15:10
|
|
|
Thermopyle posted:I think the real reason I brought the whole thing up is that it just rubs me the wrong way that everywhere these Django test client tests are called "unit" tests. Even though the concept of a unit test is kind of slippery it comes with expectations like "doesn't test a bunch of tangential code" and "is fast to run". Django is a lot of people's first exposure to coding much past Hello World and I'm not sure the community is doing these people any favors here. "unit test" is one of those terms that unravels the more you look at it and I'm not sure there's anything that can be done to save it at this point. Django, for its part, probably isn't doing anything to make the problem worse than it already is.
|
|
#
?
Jul 17, 2018 15:16
|
|
|
Helicity posted:3 of us went out for lunch yesterday with 42 years of collective experience developing - we were having the same conversation. It seems we can solve a lot of complicated problems but not come up with a definitive testing strategy. It always feels like a pendulum for me, the last place didn't have enough tests, the current place has too many tests, repeat. raminasi posted:One of my more recent professional journeys has been trying to give lots of thought and care to what’s worth testing, and what’s not, as opposed to flitting between “test everything!” and “tests suck!” Testing's a topic that tends toward idealism, eg in TDD. Dominoes fucked around with this message at 16:29 on Jul 17, 2018 |
|
#
?
Jul 17, 2018 16:23
|
|
|
I'm not positive I want to watch a Jonathan Blow video...
|
|
#
?
Jul 17, 2018 16:45
|
|
|
I considered describing it as 'A vid from a controversial, but clever programmer', but thought the current title to be an appropriate warning.
|
|
#
?
Jul 17, 2018 16:51
|
|
|
I think unit testing's more about testing small parts of your code individually, so the tests don't rely on other parts of your code to work, or heavy resources or live data. So when there is a problem, you can identify and fix it quickly You could argue that Django falls under "heavy resources" but ultimately that's part of your core app - testing often involves stuff like testing frameworks, mocking, dependency injection (if your app is built around it) and so on. There's always gonna be some extra machinery running, some stuff your tests depend on - and it's there to make your life easier, at the end of the day, reducing the amount of work you have to do I guess Django calling its own tests "unit tests" kinda blurs the lines a bit if you're also adding some pure Python functions and testing those directly, but the principle can still be the same - it's just the Django tests need to run through that framework, to avoid requiring some kind of mocking setup. So long as the point of them is the same, testing small parts in isolation, "unit test" seems like as good a name as any. Especially if Django is a system where you pretty much write everything as a "Django app" and you're basically testing all those Django wiring functions
|
|
#
?
Jul 17, 2018 17:48
|
|
|
Thermopyle posted:I'm not positive I want to watch a Jonathan Blow video... The comments are pretty funny though, one reply is literally promoting TDD because he cannot think for more than 5 minutes at a time.
|
|
#
?
Jul 17, 2018 19:01
|
|
|
In Unicode, why is a zero width joiner used instead of just another character?
|
|
#
?
Jul 17, 2018 21:18
|
|
|
Thermopyle posted:In Unicode, why is a zero width joiner used instead of just another character? It signifies that the two characters are supposed to be connected. In certain scripts (Arabic, Devanagari) this will change their appearance from what you would get without the zero width joiner. ...the zero width non-joiner does the opposite; breaks apart two characters that would normally be connected.
|
|
#
?
Jul 17, 2018 21:20
|
|
|
This question occurred to me whilst realizing that some emojis are the result of two other emoji's + the zero width joiner. Why do that instead of just creating another emoji? Is it just so that they can create a character that doesn't exist in Unicode? Is what happens when there is not a glyph for the joined result dependent upon the OS or the font? I assume on some systems it would display one of the "base" emojis, and on others it might display whatever you call the character when the font doesn't have the thing to display, but I just have no idea. I'm just curious.
|
|
#
?
Jul 17, 2018 21:33
|
|
|
Thermopyle posted:This question occurred to me whilst realizing that some emojis are the result of two other emoji's + the zero width joiner. Some of it is so that at some point in the future, you could have a "woman walking a dog with brown hair wearing clown makeup" without having an individual codepoint for every single possible combination. The solution for national flags is funnier: there's 26 flag-characters corresponding to a-z, and they use ligatures to turn the two letter ISO country code into the corresponding national flag, which lets the Unicode commitee punt controversial things like "do we add a character for the flag of tibet/islamic state/etc?" to the OS vendors that supply the fonts.
|
|
#
?
Jul 17, 2018 22:04
|
|
|
The unsupported ligature output I've seen is (whatever it can cobble together a glyph for)[extra, emojis, individually, *] Font rendering code must just be  at this point
|
|
#
?
Jul 17, 2018 22:50
|
|
|
b0lt posted:The solution for national flags is funnier: there's 26 flag-characters corresponding to a-z, and they use ligatures to turn the two letter ISO country code into the corresponding national flag, which lets the Unicode commitee punt controversial things like "do we add a character for the flag of tibet/islamic state/etc?" to the OS vendors that supply the fonts.
|
|
#
?
Jul 17, 2018 23:42
|
|
|
|
| # ? Jun 7, 2024 16:09 |
|
|
Munkeymon posted:The unsupported ligature output I've seen is (whatever it can cobble together a glyph for)[extra, emojis, individually, *] Zapfino in particular looks amazing but must be absolutely stupid hard to render because of the sheer number of swoopy ligatures that go all over the place, including non-consecutive characters.
|
|
#
?
Jul 17, 2018 23:50
|
|
