|
return0 posted:I use Spark extensively at work to process decent sized datasets, and have found it pretty good tbh. One difference is I've used the Scala API exclusively, and haven't touched the java API. I don't recognise your comments about map/flatmap not taking lambda from my experience, for example. I have read anecdotally the Scala API is more consistent. It's still the same rotten codebase running underneath, no matter what API you're using. Small example: Any sane application running on a kerberos secured yarn cluster usually authenticates in the master and distributes the authentication token to all containers. What does spark do? It distributes the keytab to all containers and every single container tries to authenticate with the KDC server.  Imho Spark's main problem is too many people wanting to put Spark contributor on their cv and the committer team letting all that lovely code through.
|
#
?
Sep 8, 2017 08:57
|
|
|
|
| # ? May 10, 2024 19:47 |
|
|
I actually kinda liked working with Spark, well PySpark, but I only made toy programs, didn't have to administer the cluster and so on. My toy ~200 lines of code still managed to run into some stupid problems, like non-existent error reporting. 
|
|
#
?
Sep 8, 2017 13:16
|
|
|
It appears that advertising my work as "Test Automation Engineer" is not clear enough as I still get asked for roles where about 50% of the time I am expected to do manual testing against an application. When responding to a role where they ask for a Test Automation Expert, the conversation often goes like this: Interviewer: For this role, we expect you to do about half of the testing manually Me: Why? In my experience manual testing is a one-time effort as a step towards full automation. I: We found that not everything can be automated. M: How did the developers respond to the request of automation friendly applications? I: We use an old legacy system that is end of life, so we don't do much development on it anymore M: Ok, then why test it in the first place? I: We need it for data generation in the system testing. You see, system A is depending on this system X for data. M: But why don't we pretend the function of system X and generate that same data into system A using a little automated script? I: Then we cannot be the data is the same. We have been doing it like this for a while, trust us, it cannot be done differently! M: Okee, so tell me a bit about the interface between the two systems. Is it SOAP, Rest or something else? A file? I: I am not sure about this, I am not a technical person. Anyway, I think I will advertise my services as "Software engineer in Test", maybe that helps? (I typed quite a bit more, it made me look like an rear end in a top hat because I am)
|
|
#
?
Sep 8, 2017 17:54
|
|
|
Keetron posted:It appears that advertising my work as "Test Automation Engineer" is not clear enough as I still get asked for roles where about 50% of the time I am expected to do manual testing against an application. Don't tease us like that.
|
|
#
?
Sep 8, 2017 18:05
|
|
|
Keetron posted:I: Then we cannot be [sure] the data is the same. We have been doing it like this for a while, trust us, it cannot be done differently! Argh this statement is so frustrating to read!
|
|
#
?
Sep 8, 2017 22:36
|
|
|
Che Delilas posted:Argh this statement is so frustrating to read! Yeah that one jumped off the page and bit me too.
|
|
#
?
Sep 9, 2017 03:12
|
|
|
As you know, this happens. You are brought in as the expert to resolve their issues they cannot resolve themselves, only to be told that your solution is poo poo.Skandranon posted:Don't tease us like that. edit a few hours later: This took a very different turn but is a fair reflection of my sentiment on the topic "As a test automation expert we expect you to do manual testing" http://thenewcareeradventures.blogspot.nl/2017/09/from-tester-to-software-engineer-in-test.html On linkedin I changed my title to "Software Engineer in Test", I think it will save everyone a lot of time and frustration during the interview process. Keetron fucked around with this message at 14:44 on Sep 9, 2017 |
|
#
?
Sep 9, 2017 07:38
|
|
|
Keetron posted:On linkedin I changed my title to "Software Engineer in Test", I think it will save everyone a lot of time and frustration during the interview process. I hate to break your bubble, but as a SDET in a previous life... - SDET? Isn't that the same as a automation engineer? Go write tests for a legacy application in WinRunner or QTP. - SDET? Wow, great! Our engineers don't have enough time to write to write their own unit tests, you can totally write unit tests written by people who don't understand how to write against interfaces instead of implementation or what dependency injection is. - SDET? Well we're Agile (if you read between the lines this means scrum whenever it's mentioned in an interview). Can you write automated tests for features that don't exist at the beginning of the sprint within 2 weeks?
|
|
#
?
Sep 9, 2017 18:40
|
|
|
Other than the first, it beats "we expect you to do about 50% manual testing" but indeed, most jobs suck one way or the other.
|
|
#
?
Sep 9, 2017 19:08
|
|
|
There's gotta be some terminology you can use to separate manual testing drones from skilled test engineers similar to what's happened between "system administrator" and "devops engineer." Maybe software reliability engineer to co-opt the site reliability engineer acronym?
|
|
#
?
Sep 9, 2017 19:11
|
|
|
What is the relationship between developers and stuff like not knowingly violating CAL licensing when it comes to ethical workplace behavior when your bosses thought process is essentially "oh yeah we know that's bad we'll get to it later when we're not as busy working on actual projects." Would you just look the other way and let the company worry about it? Asking for a friend.
|
|
#
?
Sep 9, 2017 22:06
|
|
|
Portland Sucks posted:What is the relationship between developers and stuff like not knowingly violating CAL licensing when it comes to ethical workplace behavior when your bosses thought process is essentially "oh yeah we know that's bad we'll get to it later when we're not as busy working on actual projects." Would you just look the other way and let the company worry about it? Asking for a friend. This aint legal advice blah blah. The gendarmes over at the BSA would like you to rat out your company forthwith, but seeing as you've (apparently) already raised concerns to your bosses, a sudden raid from the BSA at this point might result in you coincidentally getting fired for "poor performance" or some other vague yet legally rear end-covered reason. Tough call though. For me I'd think it'd depend on how directly it affected me and how blatant it was. I would have trouble running like obviously cracked versions of core tools (visual studio) to do my day-to-day work, but I might not care so much if it was some piece of software installed on a server. At minimum I'd keep raising the concern, find articles about the "true cost" of being non-compliant and send them to your bosses, and generally cover your rear end in paper. I would personally also flat-out refuse to run any cracks or keygens on my work machine if asked; if they want to install software on my box while I'm not there, they can, but I'm not going to do that particular dirty job for them. It's not my job at the end of the day to police the company for software license compliance (unless, you know, I was hired to do that). I highly doubt, though I can't be certain, that I would face legal or financial repercussions for NOT instantly blowing the whistle (and conversely I'm very sure that I would lose my job if the company found out I did blow said whistle). You'll have to figure out where you land on the morality scale. Also you could actually talk to a lawyer about your obligations and risks if it's bugging you enough, a short consultation won't be that expensive for piece of mind, and you might even find a free one.
|
|
#
?
Sep 10, 2017 01:16
|
|
|
Che Delilas posted:This aint legal advice blah blah. Nothing like cracks or keygens. More like "its too expensive to pay per core for server licenses and there's no way we want to manage individual workstation licenses even though all of our workstations should have one so we'll just pretend its no big deal - keep developing more enterprise apps for us that access licensed data with no CAL"
|
|
#
?
Sep 10, 2017 02:36
|
|
|
Do we all need to learn how to use Python? https://insights.stackoverflow.com/...e-growth-python https://stackoverflow.blog/2017/09/06/incredible-growth-python/ Just when I was getting the hang of this java thing...
|
|
#
?
Sep 10, 2017 12:27
|
|
|
Python is growing because it's the main language used for big data/data analytics/machine learning, and that stuff is basically a fad. I wouldn't freak out over a bunch of companies thinking they can throw machine learning at any problem to fix it. It's not a bad language, though it's not my favorite.
|
|
#
?
Sep 10, 2017 13:23
|
|
|
Keetron posted:Do we all need to learn how to use Python? Python has a good number of science/math libraries, it's relatively easy to learn, and, IMO, it's better than Ruby & Perl. It wouldn't hurt to give it a look. Ither fucked around with this message at 15:09 on Sep 10, 2017 |
|
#
?
Sep 10, 2017 15:07
|
|
|
Ither posted:Python has a good number of science/math libraries, it's relatively easy to learn, and, IMO, it's better than Ruby & Perl. The advantage of Python is pretty much that you can write science/math/analytics as well as web applications and basic programming in the same language. Which to be fair, is quite major. I just don't really have a reason to do science/math/analytics work and I've found other languages to have better web/app/systems programming capabilities and libraries. 
|
|
#
?
Sep 10, 2017 15:15
|
|
|
python is often used by people who are not software engineers. They tend to have less experience at software engineering and therefore might be more likely to ask for help on the internet. I wouldn't use the number of people who need help with a particular language as a proxy for how popular or good that language is.
|
|
#
?
Sep 10, 2017 15:24
|
|
|
Python has a few advantages when it comes to data science: it's easy to pick up enough of it to write short scripts, the performance is acceptable, and the scipy stack has all of the popular machine learning algorithms. On the other hand, if you're developing large software systems, need better than acceptable performance, or have to use a model that's not in ML for Dummies, it's a bad choice.
|
|
#
?
Sep 10, 2017 18:06
|
|
|
Keetron posted:Do we all need to learn how to use Python? Pollyanna hit it pretty much on the head. Python can do a lot of things acceptably, but a few things well. "But should I learn Python?" If you... 1) Do a lot of numerical or scientific computation, then yes. 2) Need more functionality than a standard calculator or spreadsheet, but not enough to merit building an entire application, then yes. 3) Have spoken with your doctor and decided Python is right for you, then yes. 4) Need to be able to quickly prototype algorithms and play with them interactively, then yes. If you... 1) Are learning a programming language for the first time, then no. 2) Need to be able to ship a standalone project with minimal effort that people can just run, then no. 3) Are doing embedded work or CPU critical work, then no. 4) Are building a large project which you expect to keep maintained for many years to come, then no. IMHO, 90% of Python's strength is numpy. You can build some prototype applications from it very quickly (I still have a good amount of respect for Django and its ORM), but once you have to refactor you're almost better burning your app to the ground and rebuilding it. The REPL/IPython/Jupyter, though, are probably some of my favorite things. For data-munging and modeling and general machine learning, I've not found anything I like better. Some use cases I run into and the language I tend to use: Games: I like Java and libGDX for these. It has about the right level of hands-free-ness that I can pay attention to gameplay. If I need to test an equation, though, I'll use the IPython shell and quickly type out an algorithm to see if it works and interact with it. Databases: I had to run a sorta' complicated query that kept getting killed by our DBA, so I used Psycopg2 to connect to the database, run the query in smaller batches (offset+limit), detect when it was disconnected, wait, reconnect, and continue. (Before anyone gets up in arms: I did tell them I was doing this. I maintain it's bullshit I get kicked off our dev box while doing dev work on the weekend.) Visualizations: If you want to do stuff like histograms or pick out subtle things in images, PIL/Pillow are very useful. It's handy to be able to quickly open an image, normalize contrast, and save it out. Text munging: One of the few places dynamic typing comes into handy is when munging CSVs or TSVs. Pandas can technically be used, but I've not found it easier in general. Python fills the hole in my heart that Perl and Matlab left so long ago. It's the benefits they provide with less insanity and fewer of their weaknesses.
|
|
#
?
Sep 10, 2017 20:37
|
|
|
Help I got a Scala job after having gotten familiar with Java and I cannot wrap my head around functional programming at all yet. Especially in combination with Akka and the other frameworks they use.
|
|
#
?
Sep 10, 2017 20:56
|
|
|
For scientific computing, Python is ideal for people who do enough of it that they need to have some high-performance libraries, but not enough that it's worth investing in code written in Fortran/C++/etc. If that's you, you should seriously look at Julia, which is Matlab's programming language with all the crazy removed.Carbon dioxide posted:Help I got a Scala job after having gotten familiar with Java and I cannot wrap my head around functional programming at all yet. Especially in combination with Akka and the other frameworks they use. Take a look at Martin Odersky's courses on Coursera. The first one, which introduces the basics of functional programming, is very good. The second one is terrible, but its first week used to be part of the first one, and is pretty essential. The third and fourth are a bit more of a mixed bag, from what I've heard, but they still might be worthwhile. ultrafilter fucked around with this message at 21:20 on Sep 10, 2017 |
|
#
?
Sep 10, 2017 21:16
|
|
|
Carbon dioxide posted:Help I got a Scala job after having gotten familiar with Java and I cannot wrap my head around functional programming at all yet. Especially in combination with Akka and the other frameworks they use. This recent article suggests the main concept is that the functional style is most strongly characterized by explicating the dependencies of your code.
|
|
#
?
Sep 10, 2017 21:23
|
|
|
It'll take a couple months to really click, but once it does there's no going back. I hated Scala at first but after 6 months I was in love. As long as you stay focused on writing your code the idiomatic way (immutable, functional, etc.) you'll be fine. I inherited a Scala code base written by contracted Java devs who couldn't be bothered and it's a hot mess. If you're not using Intellij I highly recommend checking it out.
|
|
#
?
Sep 10, 2017 22:01
|
|
|
Jo posted:Pollyanna hit it pretty much on the head. Python can do a lot of things acceptably, but a few things well. Why shouldn't Python be someone's first language?
|
|
#
?
Sep 10, 2017 22:20
|
|
|
Ither posted:Why shouldn't Python be someone's first language? I believe it hides too many of the important details of the underlying system and induces some bad habits. I feel like if you first are forced to feel for yourself how to do an array copy or check for duplicates in a list it will give you an understanding of the hardware underneath that's very badly needed. You can teach program flow and logic with most any language, but you can't teach someone about how computer memory works if you don't have pointers. It's harder to drive home _why_ doing a single lookup in a 1D array (to which a 2d space is mapped) incurs less overhead than an array of arrays that requires two lookups. It feels to me like teaching a drawing class and using a perspective distortion tool instead of learning how perspective works. Or maybe I'm an old fart. :rotor: EDIT: Is the rotor emoticon no longer a thing? Jesus. Maybe I am getting there.
|
|
#
?
Sep 11, 2017 00:25
|
|
|
ultrafilter posted:need better than acceptable performance Hmm.
|
|
#
?
Sep 11, 2017 00:38
|
|
|
Every first programming language will teach you some bad habits. It's only a problem if you don't let your second programming language correct any of them.
|
|
#
?
Sep 11, 2017 00:50
|
|
|
Jo posted:I believe it hides too many of the important details of the underlying system and induces some bad habits. I feel like if you first are forced to feel for yourself how to do an array copy or check for duplicates in a list it will give you an understanding of the hardware underneath that's very badly needed. You can teach program flow and logic with most any language, but you can't teach someone about how computer memory works if you don't have pointers. It's harder to drive home _why_ doing a single lookup in a 1D array (to which a 2d space is mapped) incurs less overhead than an array of arrays that requires two lookups. My first programming class was a semester of Python, followed by C++ for all the pointer stuff and data structures. You're not wrong about how learning how languages interact with the underlying architecture is valuable, but I do think it's a fine babbys-first-programming language, and great for folks doing more mathy-sciencey things who don't have time for learning the awkward syntax and gotchas of traditional languages. It's also a lot of fun to program in IMHO, and I'm somewhat sad that I live in Javaland now, but at least our project owner made the call that our scripting should be in Python instead of Ruby.
|
|
#
?
Sep 11, 2017 02:12
|
|
|
ultrafilter posted:Python has a few advantages when it comes to data science: it's easy to pick up enough of it to write short scripts, the performance is acceptable, and the scipy stack has all of the popular machine learning algorithms. On the other hand, if you're developing large software systems, need better than acceptable performance, or have to use a model that's not in ML for Dummies, it's a bad choice. And if you're mad at Python performance, most of the data science libraries rest on C/Fortran code, or are written in Cython. There's also Numba. Performance in Python is kind of meh (and that loving GIL!!!), but numeric code doesn't really live in Python-land, and you should do your best to vectorize your data code to operate on numpy arrays. I mean, at the end of the day, just use what the best tool for the job is, and in a lot of data cases that's Python (sometimes it's R! maybe someday it'll be Julia!). SO if you're interested in the data science aspects of Python, it would behoove you to learn R as well. Ghost of Reagan Past fucked around with this message at 03:43 on Sep 11, 2017 |
|
#
?
Sep 11, 2017 03:39
|
|
|
Before anyone gets the wrong idea: don't think for a moment I'm making GBS threads on Python. It's part of my daily work and I'd not have it any other way. I do, however, want to be realistic with it's flaws and imperfections.
|
|
#
?
Sep 11, 2017 03:53
|
|
|
The best thing about python for data science is that python is a normal language that developed some really nice data science tools instead of a language for data science that got normal programming bolted on later.
|
|
#
?
Sep 11, 2017 07:51
|
|
|
Bongo Bill posted:This recent article suggests the main concept is that the functional style is most strongly characterized by explicating the dependencies of your code. This is a pro read, good article. I actually really like R, but I only have to do really simple things with it. I think the most difficult thing so far was "take these two folders containing ~1000 xml messages, match them with each other, find the differences and e-mail these". No idea how difficult this would be in another language, but R has tons of support and packages for stuff like that. Most of the things I do though could also have been done in excel. (I'm not a software developer.)
|
|
#
?
Sep 11, 2017 08:48
|
|
|
Jo posted:"But should I learn Python?" Even here though, LLDB scripts are written in Python. Sure it's a bad choice to run on the target, but there's enough work that looks enough like text munging (e.g. parsing console spew, turning repetitive documentation into repetitive code) where it's really handy to know something outside of C. I, too, graduated from perl. I still miss regexes being 2 characters away instead of 20 or whatever.
|
|
#
?
Sep 11, 2017 19:01
|
|
|
I have no-poo poo reached the point where job hunting is less stressful and tiring than my actual job. I don't know if it's possible to get sick off of a job, but I've developed some weird allergy to the work I do. My brain just shuts down when I have to figure out how to wire something up in our behemoth of a codebase. It's almost like writer's block or something. And our team lead has been out for days without contact from him. We don't know what the gently caress is going on. What a shitshow.
|
|
#
?
Sep 12, 2017 14:54
|
|
|
Pollyanna posted:I don't know if it's possible to get sick off of a job It is. This is what burnout feels like. If you have some sick time you should take it.
|
|
#
?
Sep 12, 2017 15:45
|
|
|
Munkeymon posted:It is. This is what burnout feels like. If you have some sick time you should take it. I spent all my sick time for the year recovering from my surgery cause they didn't approve sick leave for it  I'll have to take vacation time, but at that point why not just wait for the holidays?I'm gonna spend an entire week working remotely and see if anyone even notices.
|
|
#
?
Sep 12, 2017 15:49
|
|
|
Pollyanna posted:I have no-poo poo reached the point where job hunting is less stressful and tiring than my actual job. I don't know if it's possible to get sick off of a job, but I've developed some weird allergy to the work I do. My brain just shuts down when I have to figure out how to wire something up in our behemoth of a codebase. It's almost like writer's block or something.
|
|
#
?
Sep 12, 2017 16:07
|
|
|
Pollyanna posted:I spent all my sick time for the year recovering from my surgery cause they didn't approve sick leave for it Have you considered just not working at all and focus on finding a new job? See how long you can tread water, I bet it's awhile, given how your team lead is acting.
|
|
#
?
Sep 12, 2017 16:21
|
|
|
|
| # ? May 10, 2024 19:47 |
|
|
Vulture Culture posted:Can you remind me what tech stack you work with? Currently React+Redux with a minimum of Node, about 1 year's worth. Prior to that was 2~3 years of Rails. Not sure I like front end, but there's more jobs and recruiters out there looking for React people these days, so I take what I can get. Skandranon posted:Have you considered just not working at all and focus on finding a new job? See how long you can tread water, I bet it's awhile, given how your team lead is acting. Been treading water since March, buddy  drat me for being so picky about whatever new position I find.
|
|
#
?
Sep 12, 2017 16:23
|
|