|
AskYourself posted:I'm not sure you were looking for technical advice or not but if yes : Bulk insert, though it could conceivably be improved further by making a file and running COPY. Of course, to do that I'd have to write to disk first, but my disk isn't big enough, so I could get an external hard drive. I tried that once, but it turns out the network copy speed for big files is a little too slow, so instead I wanted to put it on a machine that's on the network, except none of them have drives big enough. That's okay, I can rewrite stuff to produce batches of data and push each incrementally. How do I know I've gotten all the records inserted? That's not hard, I'll just sort them. 
|
#
?
May 12, 2017 18:04
|
|
|
|
| # ? May 10, 2024 13:16 |
|
|
At some point that's the job? Everyone knows what should happen at a high level, there's a thousand little details why the easy way won't work and weeks die to yak shaving BS. If you're manager and team aren't actively complaining about the delays, you might just be dealing with the requisite layers of annoyance. Past a certain point it's your manager's responsibility to get you the resources you need. If you've had several false starts on the same bullet point and the DBA's reaction is "DO BETTER," it's worth asking if somehow this multimillion dollar company could find a way to carve out some of his time to help you. It really sucks to go ask for help when there's "some" blame on you, but if you're facing burnout anyway what's the worst that could happen? Liberation to a new role? Even if it IS 99% your fault (it's not) the org as a whole has a desire to move forward, if that means tasking people with helping you out that's what should happen.
|
|
#
?
May 12, 2017 18:18
|
|
|
Jo posted:Bulk insert, though it could conceivably be improved further by making a file and running COPY. Of course, to do that I'd have to write to disk I'm not sure what DB you're using. "Insert 1 million rows" should absolutely take less than a day, but it requires implementation-specific knowledge and getting it wrong is fairly easy. If this is PostgreSQL, you are making a huge mistake to not use COPY, which was 20x faster than INSERT last time I measured. (The actual difference for you will completely depend on your data and tables, of course.) You can stream the data to the server; you don't need to have a file on the server's disk. But yeah, if you've got a DBA, they should be coming to you saying "I noticed you've got an INSERT job that's been running for the last 10 hours. Please read this documentation of bulk ingestion via COPY and tell me before you try again so that we can watch the performance together." Ask him to help you search for the solution on StackOverflow.
|
|
#
?
May 12, 2017 18:31
|
|
|
Mniot posted:I'm not sure what DB you're using. "Insert 1 million rows" should absolutely take less than a day, but it requires implementation-specific knowledge and getting it wrong is fairly easy. If this is PostgreSQL, you are making a huge mistake to not use COPY, which was 20x faster than INSERT last time I measured. (The actual difference for you will completely depend on your data and tables, of course.) You can stream the data to the server; you don't need to have a file on the server's disk. Yup. Postgres. I'll talk to him about COPY. EDIT: He says I need to make files. No streaming. I'm probably misunderstanding. Jo fucked around with this message at 18:59 on May 12, 2017 |
|
#
?
May 12, 2017 18:52
|
|
|
Jo posted:Yup. Postgres. I'll talk to him about COPY. Your DBA seems pretty lazy. https://www.postgresql.org/docs/9.2/static/sql-copy.html it needs a "file" but that file can be STDIN and then you've got streaming. Boom. It gets kind of 2nd-citizen status in most client libraries, but it should be there. Here's what you get when you search "postgres copy python": http://stackoverflow.com/questions/30050097/copy-data-from-csv-to-postgresql-using-python#30059899
|
|
#
?
May 12, 2017 19:28
|
|
|
 COPY TO is fast. Even with my poo poo connection that's a huge difference in performance time. Took like 30-60 minutes instead of all day.
|
|
#
?
May 12, 2017 21:54
|
|
|
And this is why cranky ducky debugging is a wonderful thing
|
|
#
?
May 12, 2017 22:00
|
|
|
Asking for help did the trick here, please keep doing that even if only in this thread. People who belittle you for asking questions so you can improve should at least point you in the right direction and most will.
|
|
#
?
May 12, 2017 22:01
|
|
|
Jo posted:I think I'm burning out pretty hard and pulling down the rest of my team. Not entirely sure how to broach the subject with my boss. When's the last time you took a real, stress-free vacation?
|
|
#
?
May 12, 2017 22:24
|
|
|
At work we do not have a standard Git branching model and so each project or dev ends up with different branch models. We have decided to standardize on some model. Does anyone have any suggestions as to a model to follow? We are a small organization and probably would only have at most four people working on the same project at the same time. I've looked at GitFlow, OneFlow (which stems from a criticism of GitFlow), and a few others. I'd like to pick an established model so I can just point everyone to it rather than having to come up with something and document it. Any input is appreciated. venutolo fucked around with this message at 02:41 on May 13, 2017 |
|
#
?
May 13, 2017 01:14
|
|
|
When we were looking at branching models, my feeling about the different ones out there was that they all try to be a one-size-fits-all solution. It seemed like they provided solutions for or guards against problems we just didn't have. We didn't need multiple release/hotfix branches, develop and master, or any complicated tagging schemes, and we don't care about history. We ended up using a very condensed version of gitflow adapted to our particular uses and deployment method, enough that it's not really gitflow anymore. My advice is to take only what you need to support what you're already doing. Does it make sense to have multiple release branches? Separate develop and master branches? Figure out which parts you actually need right now, then once everyone is on the same page for the development flow you can build on that foundation to add anything you feel you need.
|
|
#
?
May 13, 2017 02:22
|
|
|
venutolo posted:At work we do not have a standard Git branching model and so each project or dev ends up with different branch models. We have decided to standardize on some model. Does anyone have any suggestions as to a model to follow? We are a small organization and probably would only have at most four people working on the same project at the same time. We've been pretty good with gitflow. OneFlow seems like it would work for us as well. What fucks us up, however, is that QA all has local builds to speed up testing. Which actually works really, really well. No waiting for Jenkins to build and deploy a QA environment. What sucks is my company introduced "Acceptance" after test. Here, our Product Team reviews and makes sure it's what they wanted. So we came up with this whole convoluted branch strategy to have product review. Granted this is more of an organizational problem than git problem, but jesus christ. I'm sick of all the branches.
|
|
#
?
May 13, 2017 03:19
|
|
|
Developers can deal with any git model as long as it's halfway sane, so don't let any weirdos set you on Git Flow just because they read about it and liked the drawing of all the branches. The important thing is who's going to be releasing/deploying things. I did release engineering at my last company for our cloud-based service. It was made of a bunch of micro-services and we had a separate repository for each. We were a small group, we wanted to be able to release quickly (but not "multiple times a day" quickly), and we wanted to easily roll back from a bad deployment. We used something like GitHub Flow: master is always releasable and work gets done on short-lived feature branches. The deviation is that I'd tag master and build a docker image to make a release. Then we could deploy any release to either our staging or production environment. The tags made it easy to make a patch to production if we needed to make a fix but didn't want to take master (usually because there was some large feature that we wanted to test in staging). At my current place, we use essentially the Release Branches variant of GitLab Flow, because our software is deployed on-premises by customers. Release branches tend to be pretty divergent so pushing a bug-fix can be quite painful, but I think that's from our release cadence and not from the git workflow. Our release engineer keeps the automation code in the repository, so even if he changes many parts between releases pushing the "make a build" button works on every release branch. Basically, I'd recommend starting with the most bare-bones (GitHub Flow) and adding whatever complexity you feel you absolutely need to make your release/deployment process work for you.
|
|
#
?
May 13, 2017 03:48
|
|
|
My workplace is on a git branching model that I haven't seen but at least it's something that doesn't have problems so far. Develop branch is where the next release gets commits to, release/$release-version where $release-version =~ /^\d+\.\d+$/. The thing that's odd is that we don't really have multiple versions of our monolithic software active because we're a SaaS company, and we rarely backport changes to old branches, so we'd probably be better off with just develop and master. Now, what's deployed to different environments is another matter but those are complete abortions where people drop in snapshot JARs without getting it tracked anywhere.
|
|
#
?
May 13, 2017 14:09
|
|
|
We just have an always releasable master, short lived feature branches, and a production branch off of master when we do a release. Pretty simple and has worked well for us.
|
|
#
?
May 13, 2017 14:26
|
|
|
smackfu posted:We just have an always releasable master, short lived feature branches, and a production branch off of master when we do a release. Pretty simple and has worked well for us. After a lot of bickering, we finally decided on this system, too, and it's working pretty well. To get a sense of the before and after, we used to have QA, Stage, and Production/Master. Development would happen on short-lived branches and get merged into QA for testing. If approved, QA got merged into staging. Staging got deployed to the preproduction boxes where full regression happened (and emergency bugfixes were tested). Finally, everything was merged to master and deployed. Now: we branch off of master, make changes, and merge into master. QA is always deployed from master. When we prep for a release, we tag the master branch and deploy that to staging and eventually to prod. The biggest advantage to this is just avoiding merge issues. Whatever gets deployed to stage gets deployed to prod.
|
|
#
?
May 13, 2017 15:24
|
|
|
smackfu posted:We just have an always releasable master, short lived feature branches, and a production branch off of master when we do a release. Pretty simple and has worked well for us. Merging into master prior to the production / release branch?
|
|
#
?
May 13, 2017 16:38
|
|
|
Does anyone know of any tools that let you use Github-style pull requests with SVN?
|
|
#
?
May 14, 2017 00:30
|
|
|
redleader posted:Does anyone know of any tools that let you use Github-style pull requests with SVN? git svn clone
|
|
#
?
May 14, 2017 00:44
|
|
|
BabyFur Denny posted:git svn clone That isn't a tool that layers PRs on top of SVN, though.  That would just let you push your SVN code to a Git hosting platform that supports PRs. At which point, you might as well stop using SVN.
|
|
#
?
May 14, 2017 22:35
|
|
|
I think that was the point
|
|
#
?
May 14, 2017 22:37
|
|
|
I mean yeah, that would be great. But "cultural issues" rear their ugly heads, and it would be easier to bolt something on to SVN than to convince the org to go whole hog into some sort of git.
|
|
#
?
May 15, 2017 01:34
|
|
|
But "PR"s aren't a Git thing, they're a GitHub thing. SVN can pass around diffs. Or try Reviewboard (simple and worked great when I used it with Perforce, though I didn't do any of the operations work) or Phabricator (looks complex, never tried it).
|
|
#
?
May 15, 2017 05:03
|
|
|
Mniot posted:But "PR"s aren't a Git thing, they're a GitHub thing. Git has PRs but they're different from github's
|
|
#
?
May 15, 2017 05:18
|
|
|
It is more or less similar to how bitbucket / stash uses this.
|
|
#
?
May 15, 2017 06:42
|
|
|
We released!  It was difficult getting it out, but the release itself went well. I continue to be amazed that we finished on time. I still feel like we could have done better, but hey, the team will be able to move faster now that we've got a chance to refactor. Pollyanna fucked around with this message at 16:58 on May 17, 2017 |
|
#
?
May 17, 2017 14:57
|
|
|
Pollyanna posted:We released! Given how much you bitch about your employer here, do you really want people to know where you work? That seems like a bad idea.
|
|
#
?
May 17, 2017 16:28
|
|
|
New Yorp New Yorp posted:Given how much you bitch about your employer here, do you really want people to know where you work? That seems like a bad idea. Given how often Pollyanna switches jobs, do you think it matters?
|
|
#
?
May 17, 2017 16:47
|
|
|
Pollyanna posted:We released! It's cute you think you'll have a chance to re-factor now, instead of a bunch of new requirements being added.
|
|
#
?
May 18, 2017 09:57
|
|
|
Slash posted:It's cute you think you'll have a chance to re-factor now, instead of a bunch of new requirements being added. Hey it does happen! I work for a small company in the medical sector. Originally they used an Indian firm to write its desktop software in the 90's. Then paid the same company to make it a web application last decade. And then last year we paid them a third time to make it browser independent and use some modicum of modern technology. I started here 2 years ago. Almost a year into the job we went live with that most recent rewrite for Browser Independence and that loving sucked. On top of all new technology none of us were familiar with, it was the company's most challenging year simply in terms of volume of new features/requirements that must be added. This year is the exact opposite; one of the lightest years in terms of workload. I've spent the last 6 months working with static code analyzers and various other utilities trying to refactor as much of this heap of junk as I can. --- On a related note, recently I've been debating if I want to stay at this job long term. I just wanted to get some opinions on what makes you guys change jobs, other than super lovely work environments. I have been working in the field for 5 years, or 4.5 if you don't count my internship. I make $63K -- but I live in one of the cheapest cities in the US. Right now, I'm content with my pay. I make roughly 45% more than the Average Household Income, and more than double the Median Household Income for my area. So despite my ludicrous student debt I can still live pretty comfortably. According to CNN's "Cost of Living Calculator", this would be about the same as $120K in the Bay Area. The benefits are incredible, my health insurance only costs me $130/yr and it's very good coverage. The retirement plan? They don't contribute anything your first two years, but after that they contribute 9% of up to $16.5K and then 12% of anything you make over that. And that's not matching, they just give you that. I'm not sure what the vesting schedule is, though. I get along great a lot of the people who work here, which is a first for me and while I like to keep work and personal life separate, I still think this is important. So what don't I like? There's almost no room for growth. My "team" is basically me and my boss. Another programmer was hired for my team 1 week after me but she was moved onto another project after last year's rewrite. So yea, I work on adding new features, bug fixes etc. and my boss typically does a lot of "speciality" work for specific, larger clients of ours. Above my boss is the CTO. At best, I could hope to be promoted to "Senior Analyst/Programmer" some time in the future. I have no clue what sort of raise that would entail. But my boss is probably at least 10-15 years away from retiring. So there's very little room for growth. Another thing is that due to the volume of the work last year, I was working 60 hour weeks for most of the summer. Our annual "We are 100% royally hosed if we don't make it" deadline is October 1st so summertime can be tough. Especially for vacation requests. The company not very friendly in terms of working remotely. You can definitely do it, but you generally need to have a reason. My last job was the polar opposite; I worked remotely 5 days per week because my boss worked in a different office/state. Overall I think that was bad for me as I'm too easily distracted but it'd be nice if I could work remotely one day per week without needing to give a reason. I'm also afraid that this job can grow stale. We are, by a significant margin, the largest software vendor in our sector. Due to the size of what we do and our small tech team we're kind of stuck being reliant on the aforementioned Indian tech firms for any major projects. So basically I'll probably be stuck using outdated versions of Visual Studio and ancient third party libraries as long as I stay here. And that brings me to my final concern: Location. I relocated 450 miles to a different state to get the gently caress out of my awful first Dev job. I like how affordable this area is, and a very close long-term friend lives here which is why it was on my list of places I'd be willing to relocate to. However, I've been here for a few years now and I don't really think I want to stay. I mean, I've made a few friends and whatnot but I just hold no attachment here. So overall I like my employer, the benefits are excellent, I can't complain about the pay, but there's very little room for growth / increase in earning, stagnant tech, and a mediocre-at-best location. I don't know what I want in life and I don't want to tie myself down to a certain location while I'm young and single, but the health insurance and retirement plans are so good I don't want to leave.
|
|
#
?
May 18, 2017 14:40
|
|
|
Sab669 posted:... There's no pressure to find something immediately, if any new offer doesn't convince you it's a better deal you have no issues continuing another year at your current job. It also makes interviews and the entire application process (and of course salary negotiations) much more relaxing if it's not your entire career that's on the line.
|
|
#
?
May 18, 2017 17:56
|
|
|
BabyFur Denny posted:I think those are very good conditions to go job hunting: I agree. Consider it a long term project of "refactoring your career". You have time to do it right, so figure out what you want, and build towards that.
|
|
#
?
May 18, 2017 18:00
|
|
|
You have it good, but if you have an itch, you got to scratch. Do some interviews for jobs far away and close by, find out what other companies think you are worth and then decide. If you are not tied down by family, please go and see more of the world, your future you will love you for it.
|
|
#
?
May 18, 2017 18:31
|
|
|
Keetron posted:You have it good, but if you have an itch, you got to scratch. Do some interviews for jobs far away and close by, find out what other companies think you are worth and then decide. Far away interviews are super chill because it's easy to schedule them outside normal working hours.
|
|
#
?
May 18, 2017 19:23
|
|
|
Finally getting user feedback 2 weeks before money runs out and the project transitions to sustainment.
|
|
#
?
May 18, 2017 22:17
|
|
|
leper khan posted:Far away interviews are super chill because it's easy to schedule them outside normal working hours. If only there was a way to conduct communication between people tgat are not in the same location.
|
|
#
?
May 19, 2017 05:44
|
|
|
Slash posted:It's cute you think you'll have a chance to re-factor now, instead of a bunch of new requirements being added. I use my own company's inefficiency to make time to refactor! Most of my projects get kicked off with very little planning and get endlessly blocked while we wait for input or decisions from various people. Quite frequently I am blocked on all of my projects. Usually during this period I have nothing assigned to work on outside of meetings and I use it as a chance to refactor/cleanup code and pay off some "technical debt". The sad reality is though, plenty more technical debt will be built up when the projects are no longer blocked weeks later but the original deadline is still observed.
|
|
#
?
May 20, 2017 03:46
|
|
|
ahhhhhhhh our app is officially in production and open to employees and I just watched someone use it and it worked and ahhhhhhh it's nothing special but it's really the first thing I've worked heavily on that's going to be released into the wild and it's just so drat exciting
|
|
#
?
May 24, 2017 13:54
|
|
|
That's cool, I'm usually a nervous wreck when getting things deployed to production. It's my least favorite part of the process.
|
|
#
?
May 24, 2017 18:06
|
|
|
|
| # ? May 10, 2024 13:16 |
|
|
The best is when you release a widely anticipated feature and get all excited for the positive feedback to start rolling in and the customer just goes, "meh." Never quite learned my lesson on that one.
|
|
#
?
May 24, 2017 18:22
|
|
