|
Luigi Thirty posted:also there should be dental plan
|
#
?
Apr 26, 2016 17:50
|
|
|
|
| # ? May 25, 2024 21:52 |
|
|
gonadic io posted:This would be very nice. using the old tool past the point where you've outgrown it is how you end up with your company running its expenses out of access.
|
|
#
?
Apr 26, 2016 17:53
|
|
|
if somebody tells you replication is easy you should laugh in their face insert tef post here about people recreating libraries
|
|
#
?
Apr 26, 2016 18:04
|
|
|
Luigi Thirty posted:also there should be dental plan
|
|
#
?
Apr 26, 2016 18:05
|
|
|
Brain Candy posted:insert tef post here if only
|
|
#
?
Apr 26, 2016 18:48
|
|
|
Shaggar posted:nah the first one is better and also should not be one line. i assume you say this as a protection against needing to do more stuff to the list in the future? ditto the braces comment? thats my intuition, but i want to make sure you dont mean something more clever than that.
|
|
#
?
Apr 26, 2016 19:11
|
|
|
yes, but its also more consistent with how linq is used in general. use linq to provide your filters and transforms and then use normal control statements to do stuff to or with that filtered and transformed data.
|
|
#
?
Apr 26, 2016 19:13
|
|
|
regarding braces you should always have braces in all-man style (hard tabs only). braceless control statements are the devil's handiwork
|
|
#
?
Apr 26, 2016 19:14
|
|
|
Shaggar posted:nah the first one is better and also should not be one line. swr
|
|
#
?
Apr 26, 2016 20:43
|
|
|
please don't put side effects in your linq statements unless there's no other option!
|
|
#
?
Apr 26, 2016 20:44
|
|
|
Shaggar posted:regarding braces you should always have braces
|
|
#
?
Apr 26, 2016 20:52
|
|
|
pointsofdata posted:please don't put side effects in your linq statements unless there's no other option! there's always another option
|
|
#
?
Apr 26, 2016 21:07
|
|
|
gonadic io posted:this isn't user facing, this is two different servers needing to reconcile their data. so the idea is to send off a http request from A to B, then B dumps the db into the json body response. what engine does the other server use? mysql? could you do some sort of filtered replication all day instead of at the end of the day?
|
|
#
?
Apr 26, 2016 23:29
|
|
|
Bloody posted:yeah. there's a .ForEach in the PLINQ set but not in the LINQ set and its kind of strange imo that's because there's no parallelforeach language construct i will say that while "don't put side effects in linq" sounds great, seq.iter is really nice to have
|
|
#
?
Apr 27, 2016 00:37
|
|
|
gonadic io posted:i'm trying to reduce the number of fields we pull for each of those rows which seems to speed it up but it's still the bottleneck. gonadic io posted:this is two different servers needing to reconcile their data. so the idea is to send off a http request from A to B, then B dumps the db into the json body response. quote:the response really is just a db dump quote:as long as A sends out an email being all "yep, the data from B matched what's in my db" i don't care if it takes a minute or two. so, you have two databases, and you're wanting to reconcile them let's not ask why you have two databases needing to be reconciled, although that is important taking a dump of the database is probably faster than spitting out json (although db dumps are slow unless you're taking a snapshot of the filesystem, most of the time) (i say dump, i mean copy of some form) pushing the filtering into sql is probably faster too but really you have to ask why you put your data in two different places to begin with
|
|
#
?
Apr 27, 2016 01:09
|
|
|
Brain Candy posted:if somebody tells you replication is easy you should laugh in their face i can't hear you i'm too busy writing yet another raft implementation in go
|
|
#
?
Apr 27, 2016 01:11
|
|
|
i wanna scrape and manipulate and probably dump to sql a buncha json data from a rest api for some reason this feels like itll be unpleasant to do in haskell is that the case and if so what are some other neat languages to do it in i could just do it in c# but idk that doesnt feel like a learning opportunity
|
|
#
?
Apr 27, 2016 03:18
|
|
|
do it in prolog
|
|
#
?
Apr 27, 2016 03:20
|
|
|
have you tried Python op
|
|
#
?
Apr 27, 2016 03:32
|
|
|
Yeah it's really bad
|
|
#
?
Apr 27, 2016 03:32
|
|
|
I'm surprised. Haskell lets you turn a list into a side-effecting calculation with mapM/foldrM/traverse/etc, but I assume that they avoided that in LINQ because there's no type-level annotation saying "here be side effects".
|
|
#
?
Apr 27, 2016 03:58
|
|
|
Bloody posted:Yeah it's really bad dogma is bad
|
|
#
?
Apr 27, 2016 04:02
|
|
|
MeruFM posted:dogma is bad just encapsulate it in a monad
|
|
#
?
Apr 27, 2016 05:59
|
|
|
Bloody posted:i wanna scrape and manipulate and probably dump to sql a buncha json data from a rest api for some reason this feels like itll be unpleasant to do in haskell is that the case and if so what are some other neat languages to do it in This might not be so bad in Haskell. Usually when I want a recommendation for a Haskell library I check here first, and if they don't have anything listed for my problem domain I google 'haskell <problem>' and look at the packages that come up and pick one that looks like it is maintained and has a reasonable API. I've never really done web stuff in Haskell, but I think some people like it. I know Aeson is the standard JSON library that everyone uses, and I think people like Esqueleto for SQL stuff. Not sure what the standard HTTP library is. Maybe HTTP?
|
|
#
?
Apr 27, 2016 06:30
|
|
|
If you wanna see if something is actively maintained, you can check and see if it is on stackage, since they that packages be updated frequently. Not being on Stackage doesn't mean that something is not maintained, but you can count on a package being maintained if it is on Stackage.
|
|
#
?
Apr 27, 2016 06:33
|
|
|
tef posted:so, you have two databases, and you're wanting to reconcile them A contains our view of the world based on user actions, and B gets populated daily by csvs from banks. It's quite important that we check each day that we haven't lost any customer's money, the FCA are sticklers about that. I've moved the filtering into mysql, will suggest that we move from mysql (lol), and if we need it much faster I think the next step is to dump table partitions as binary. Maybe. ty for your help all
|
|
#
?
Apr 27, 2016 08:12
|
|
|
gonadic io posted:banks... FCA... mysql 
|
|
#
?
Apr 27, 2016 08:27
|
|
|
|
|
#
?
Apr 27, 2016 08:36
|
|
|
im leading the push for us to move to mongo, that way it's already in json! 
|
|
#
?
Apr 27, 2016 08:50
|
|
|
gonadic io posted:im leading the push for us to move to mongo, that way it's already in json! Remember to register with Shodan so that you can always find your DB.
|
|
#
?
Apr 27, 2016 09:17
|
|
|
ErIog posted:Remember to register with Shodan so that you can always find your DB. also Canary
|
|
#
?
Apr 27, 2016 10:10
|
|
|
Bloody posted:i wanna scrape and manipulate and probably dump to sql a buncha json data from a rest api for some reason this feels like itll be unpleasant to do in haskell is that the case and if so what are some other neat languages to do it in Gonadic io, is that you?
|
|
#
?
Apr 27, 2016 11:26
|
|
|
Valeyard posted:Gonadic io, is that you? it's the opposite, i'm doing sql -> json -> REST. plus i have no choice of lang.
|
|
#
?
Apr 27, 2016 12:25
|
|
|
i wish json had a schema so i could generate all of my types from it
|
|
#
?
Apr 27, 2016 13:03
|
|
|
Bloody posted:i wish json had a schema so i could generate all of my types from it Just use the types as your json schema I mean the format must be consistent right? Or for the Shaggar answer: are you sure that you can't use xml?
|
|
#
?
Apr 27, 2016 13:15
|
|
|
the api im hitting only spits back json unfortunately
|
|
#
?
Apr 27, 2016 13:17
|
|
|
Bloody posted:i wish json had a schema so i could generate all of my types from it use a json to xml converter and then generate your types
|
|
#
?
Apr 27, 2016 14:56
|
|
|
gonadic io posted:A contains our view of the world based on user actions, and B gets populated daily by csvs from banks. It's quite important that we check each day that we haven't lost any customer's money, the FCA are sticklers about that. the issue isn't mysql, it's that you're trying to compare two databases on two separate servers. assuming your "view of the world" db is prod, you should be backing it up to a separate analytics db along with the fca stuff (and whatever data you want for not streaming analytics) and do your batch jobs there.
|
|
#
?
Apr 27, 2016 14:59
|
|
|
Bloody posted:i wish json had a schema so i could generate all of my types from it there is json schema now, but its external so you basically have to know what schema you want to enforce because you cant read it from the file. its supported in vs 2015+. also maybe 2013. also in vs 15 you can generate rest clients from swagger. goddamn Microsoft is good at this poo poo.
|
|
#
?
Apr 27, 2016 15:01
|
|
|
|
| # ? May 25, 2024 21:52 |
|
|
Bloody posted:i wanna scrape and manipulate and probably dump to sql a buncha json data from a rest api for some reason this feels like itll be unpleasant to do in haskell is that the case and if so what are some other neat languages to do it in Use SSIS.
|
|
#
?
Apr 27, 2016 15:01
|
|