|
Storysmith posted:You're assuming a Java developer knows poo poo about Java operationally, though. And cares. There's a reason companies tend to have to be dragged kicking and screaming to devops if their culture is walled this way. So, I'm able to get the java proxy testing button to work after setting the keystore options in the jenkins arguments. If I remove that string, that proxy test button fails when I connect to their Jira, so I know that setting is being honored. I figure that if I don't define a proxy, that button just does a curl or something from that url. On wireshark, I see the cert setup happen correctly, so that's good. The JIRA still fails. I'm supposing that the JIRA call loads a different JDK version since there's a ton on the server. The JAVA_HOME variable defaults to jre_1.7, but when I dig around the Jenkins settings, I see it loading JDK 1.8. The cacert in jdk1.8/jre/lib/security/cacert does not include the CA I need, so I added it and I'm restarting Jenkins now and running the build (which takes twenty minutes to fail). So I'll just be sitting here watching dota2 all day until I die.
|
#
?
Jun 22, 2016 15:10
|
|
|
|
| # ? May 27, 2024 21:48 |
|
|
If it's Windows, it bundles a copy of Java with Jenkins if they used the all-in-1 setup, as far as I remember. It's been about 10 months since I last did Jenkins on Windows though. What I'm saying is there's a high likelihood it's not using the system Java. If that's the case, I think the JAVA_HOME that Jenkins is running under is the JENKINS_HOME or some path under it.
|
|
#
?
Jun 22, 2016 15:19
|
|
|
Urit posted:If it's Windows, it bundles a copy of Java with Jenkins if they used the all-in-1 setup, as far as I remember. It's been about 10 months since I last did Jenkins on Windows though. What I'm saying is there's a high likelihood it's not using the system Java. If that's the case, I think the JAVA_HOME that Jenkins is running under is the JENKINS_HOME or some path under it. Yeah, that makes sense. It's working now, though. What I found is that jenkins itself uses JRE 1.7, so when you open configure jenkins->advanced->proxy and enter a url (even without proxy configured) you need to make sure that the jenkins.xml contains arguments that point it to the a keystore containing the CA you need, but in this case is defaulting to ~ c:\program files\jre_1.7.0\jre\lib\security\cacerts, so the url works there, if I specifically added the CA to that keystore. HOWEVER, since Java is developed by satanists, when you do a build in jenkins, and that build connects to JIRA, it somehow uses JDK 1.8. So, you need to either call the custom keystore when you call that action, which I guess is just jenkins firing off "java -jar some_jira_thing.jar", or do what I did, and add the CA cert to what I guess is the default keystore for jre1.8, which is c:\program files\jdk_1.8.0\jre\lib\security\cacerts Once I did that, it all worked. Note that I had to do it for both jre1.7 and jdk1.8, because reasons. Also, the dev that told me this ABSOLUTELY critical to finish has not responded to emails or messages for an hour, you know it's only 1030, who'd expect you to be on desk, right? I've typed keystore twenty times in this message and I want to hang myself. What I'm most made about is when I finally talk to them, she's going to smugly say "see I told you to add it to the keystore, you should have listened to me".
|
|
#
?
Jun 22, 2016 15:35
|
|
|
Collateral Damage posted:Why can't the JRE provide an abstraction layer to the OS keystore instead of having its own special snowflake keystore? Well, on Linux/Unix, the answer is 'what OS keystore?'.
|
|
#
?
Jun 22, 2016 16:27
|
|
|
/etc/pki of course!
|
|
#
?
Jun 22, 2016 16:31
|
|
|
Jerk McJerkface posted:Yeah, that makes sense. It's working now, though. What I found is that jenkins itself uses JRE 1.7, so when you open configure jenkins->advanced->proxy and enter a url (even without proxy configured) you need to make sure that the jenkins.xml contains arguments that point it to the a keystore containing the CA you need, but in this case is defaulting to ~ c:\program files\jre_1.7.0\jre\lib\security\cacerts, so the url works there, if I specifically added the CA to that keystore. Wait wait, this is a build? Like, compile-test-deploy build? Yeah, those will run whatever jvm you configure the job with so you can test things like "will the new release of Java break my code" (answer: yes). Jobs are incredibly flexible and powerful things; having them all locked to the same JVM the Jenkins master is running would be dumb (and impossible across OSes). But then, so is developers making a sysadmin fix problems in a failing Jenkins build until they've actually tried. Why is the Jenkins front end server running builds at all? There's build executors for a reason. That reason is "to be able to throw them away and recreate them when things inevitably get hosed in your build environment".
|
|
#
?
Jun 22, 2016 17:07
|
|
|
Storysmith posted:
Because setting four build slots on the master is the default config.  Jenkins is a cool thing but it's a giant house of cards.
|
|
#
?
Jun 22, 2016 17:20
|
|
|
Storysmith posted:Wait wait, this is a build? Like, compile-test-deploy build? Yeah, those will run whatever jvm you configure the job with so you can test things like "will the new release of Java break my code" (answer: yes). Jobs are incredibly flexible and powerful things; having them all locked to the same JVM the Jenkins master is running would be dumb (and impossible across OSes). But then, so is developers making a sysadmin fix problems in a failing Jenkins build until they've actually tried. My job title is Global System Support Engineer, and I specialize in Linux, but primary my job is to make sure our devs, the guys that billing hourly on client projects can do their jobs. Typically that mean making sure the client servers are online, backed up, etc. When this lady called, the first thing I did was kick it to her PM and my manager, and they told me to help her. My manager didn't like it, since he didn't have an idea of how long it'd take, but ultimately, my job is to make sure they are making money. I'm paid way more than them, though, so go figure. Also, I have no idea about why they are doing what they are doing, but I can certainly google a jenkins error and sort it out. So could they, but whatever.
|
|
#
?
Jun 22, 2016 17:21
|
|
|
I ended up with a decent gig, saddled with maintaining Jenkins and a bunch of build slaves, but I only support the service. If a user blows up their build and can't figure out why, that gets dumped on an actual developer who got pegged for supporting projects. Saves me some headache, but not all of it. The support developer and I don't really get along because I need to keep things stable and secure and they're one of those "ughhhhhh your silly security policy is preventing me from launching builds from anonymous sources why can't we leave the api open to the world?" type of people. Also they love installing plugins. Like if there's a new checkbox on the plugins page they can't resist clicking it.
|
|
#
?
Jun 22, 2016 17:39
|
|
|
xzzy posted:I ended up with a decent gig, saddled with maintaining Jenkins and a bunch of build slaves, but I only support the service. If a user blows up their build and can't figure out why, that gets dumped on an actual developer who got pegged for supporting projects. I couldn't find the plugin for Jira integration. I'm not sure how that is supposed to work but just that I needed to fix the keystore.
|
|
#
?
Jun 22, 2016 17:41
|
|
|
More poo poo that pisses you off: Keystore Keystore KEYSTORE
|
|
#
?
Jun 22, 2016 17:59
|
|
|
Jerk McJerkface posted:keystore. The use of this word has started to make ME angry, I propose a ban.
|
|
#
?
Jun 22, 2016 18:13
|
|
|
Just checking in, it's been 4 hours since I fixed this critical issue, and haven't heard back from the dev I chatted/emailed to tell it was done.
|
|
#
?
Jun 22, 2016 18:25
|
|
|
That means the problem is fixed. SOP for high maintenance users, in your grill when they need something but ghosts when they don't.
|
|
#
?
Jun 22, 2016 18:30
|
|
|
MF_James posted:The use of this word has started to make ME angry, I propose a ban. NAh, can we do wordfilters? or does that require admins? Keystore = "COCK IN YOUR rear end"
|
|
#
?
Jun 22, 2016 18:57
|
|
|
xzzy posted:Because setting four build slots on the master is the default config. I think it just defaults to the number of cores on the machine. But yeah, it should always be 0.
|
|
#
?
Jun 22, 2016 19:03
|
|
|
Gounads posted:More poo poo that pisses you off: Keystore Keystore KEYSTORE Please do the needful.
|
|
#
?
Jun 22, 2016 19:13
|
|
|
Gounads posted:More poo poo that pisses you off: Keystore Keystore KEYSTORE I'm sorry to bring this on all of you, but it's good to share the pain.
|
|
#
?
Jun 22, 2016 19:30
|
|
|
Jerk McJerkface posted:I'm sorry to bring this on all of you, but it's good to share the pain. Share with me your pain: https://www.youtube.com/watch?v=gJGwEP7AZHg&t=66s
|
|
#
?
Jun 22, 2016 19:44
|
|
|
Gounads posted:More poo poo that pisses you off: Keystore Keystore KEYSTORE More poo poo that pisses you off: Featuring the Keystore Ops!
|
|
#
?
Jun 22, 2016 21:24
|
|
|
Getting told to put a bug-fix or worse, new feature, in "this release" two-three hours before code cutoff. If I do it, people testing are pissed because it's last minute unplanned work and poorly tested. If I can even get a PR. If I don't do it, my immediate team is up my rear end because the product is supporting our everyday work. Lose lose. 
|
|
#
?
Jun 22, 2016 21:27
|
|
|
Zamboni Apocalypse posted:More poo poo that pisses you off: Featuring the Keystore Ops! Beautiful. 
|
|
#
?
Jun 22, 2016 22:43
|
|
|
99% of the customer services at my job can be built and troubleshot automatically. We paid to have a tool developed to do just that and it has been one shitshow after another getting this thing to work. Putting aside that 5% of the time it's not setting customers up correctly in the first place, one of the most basic functions for troubleshooting are a "Check" button and a "Rebuild" button. For the simpler of the two, "Check," one out of five times you get lovely errors like these: Index and length must refer to a location within the string. Parameter name: length Or, my favorite: Error retrieving info: Object reference not set to an instance of an object Are you loving kidding me? We are several months into production and we still aren't handling basic exceptions? I know that behind the scenes, the work itself actually is not trivial, but still, everytime it doesn't work, I have to do all the updates manually, many times after hours. Then, once I make the update, the Check button still won't work and I get more complaints "If you built it why is there still an error?." Fine, whatever, systems have bugs to work out (I guess.) The thing I just can't get over is the complete lack of competence from the dev team. There's a lot of behind the scenes stuff they are constantly doing that blows my mind, but the icing on this poo poo cake are those errors. Why is the system throwing unhelpful exception text verbatim into user output? The front line techs are not the smartest people and it confuses them. If you aren't going to add a debug ID or something (WHY NOT!?!?) at least give a general, "$SYSTEMNAME hosed up and can't do what you're asking." So they don't keep escalating the same issue over and over. Just needed to vent a little.
|
|
#
?
Jun 23, 2016 00:12
|
|
|
the real blah posted:99% of the customer services at my job can be built and troubleshot automatically. We paid to have a tool developed to do just that and it has been one shitshow after another getting this thing to work. Putting aside that 5% of the time it's not setting customers up correctly in the first place, one of the most basic functions for troubleshooting are a "Check" button and a "Rebuild" button. For the simpler of the two, "Check," one out of five times you get lovely errors like these: Just two things I'd like to say: 1: You absolutely want those exceptions bubbled up as high as you can go. I've seen a few codebases that try catch every exception and silently fail (aka the exception black hole anti-pattern); and then all of a sudden billing hasn't been working for 6 months and we have no money and aaaaaaaaahhhhhhhhhh. To be honest though, log4{language} all that poo poo. I know what you are getting at but make absofuckinglutely sure you capture every last exception. 2: Unit tests. more tests more tests. Something broke? A: Write a test that replicates hosed state. B: Write a happy path state for expected behavior if you don't have one already. C: Refactor into unfuckedness.
|
|
#
?
Jun 23, 2016 00:28
|
|
|
Wozbo posted:Just two things I'd like to say: Yeah, that's why I don't get why they are not logging stuff like that and showing a generic message with an ID to the entry in the log. Right now they just shrug and tell the user they're hosed, but not in a language they understand. Unit tests: I wish. All I can do is pass on bug reports as a user. Granted, as a higher level engineering user, but a user all the same. I would make a report for every single time, but that's too passive aggressive, even for me. That's another thing that blows my mind. There's practically no test environment now, they have a dev and they push into production without telling anyone. That this continues is very much a management issue. I wish they would grow a pair and tell the devs to knock that poo poo off.
|
|
#
?
Jun 23, 2016 01:11
|
|
|
nitrogen posted:Seriously, though, not all OS's have such a thing. Think java for embedded devices, cars, refrigerators, etc, so you have to create your java VM to the lowest common denominator (which is also why Java does its own timezone thing, for instance)
|
|
#
?
Jun 23, 2016 11:49
|
|
|
Jerk McJerkface posted:Just checking in, it's been 4 hours since I fixed this critical issue, and haven't heard back from the dev I chatted/emailed to tell it was done. Keystore update: So yesterday she came into the office (typically works at client site) and I asked her if it worked, and she said "what? No" and walked past me. Later when she came by again I asked her to clarify what didn't work. And she told me it did, but that she wanted to know what I fix. I told her it was the keystore, and she was like "ah, see I told you it was the keystore." I told her that it was more complicated than that, but if it's working, then great. She then asks me to setup Jenkins on all the PCs of the dev team (there's 6), even though some of the users don't have admin access to their computers. I tell her that's a terrible idea. If they need Jenkins they should put it on the dev server, which I can do for them, assuming they createa project for me and I can bill time to it. I can't just keep billing time against their project with out approval. She's says "well, that's not my problem so whatever, we just need it so deal with that." I ask her if she can just tell her PM the request so he can engage my manager and get my time booked, and she was like "lol no that's not my problem I just need jenkins, if you don't want to do it on the server then just run it locally on all our PCs." No no no, that won't work, it's an awful idea, since the client owns those PCs, and they can be lost at any time. You need to back up Jenkins and make sure whatever you are doing is done right." Her: "Well, I don't care about the design or whatever, I'll do it on my PC since you refuse to help." Then she walked away. I almost jumped out the window. Fake edit; Also a coworker told me the following story about her on Wednesday (the day she told me it was critical to fix, and when I started early to get it working, she didn't respond all day). Tuesday, she leaves the office early, 4:00pm. My friend worked until 8:00pm. Right before he left, he emailed her with a question. After he left, she emailed him at 8:30, just restating his question, but directing back to him. He sees the email and shakes his head. She emails again, the same email, just correcting her name in the signature. Then she emails him the same email a THIRD TIME, but CC's the manager. Wednesday, they get in at 10am, and she leaves for lunch and doesn't come back until 3pm. When she gets back, the PM asks here where she's been, and she says well "I emailed $COWORKER THREE TIMES last night, and he never responded, so I didn't have any work to do, so I left." The PM flipped out, "HE SITS RIGHT NEXT TO YOU. WHY DIDN'T YOU ASK HIM AT 10 WHEN YOU TWO WERE SITTING THERE?" He answer was basically "don't get mad at me that $COWORKER didn't answer my email." The PM just gave up, since the lady is impossible to talk to.
|
|
#
?
Jun 24, 2016 19:13
|
|
|
Sounds like one of those game players, knows how to exploit the rules to run around and do whatever the gently caress they want and blame other people for problems. Does she have ties to someone further up the power chain? In my experience that's usually how dead wood gets to stick around. Or they're close to retirement.
|
|
#
?
Jun 24, 2016 19:23
|
|
|
xzzy posted:Sounds like one of those game players, knows how to exploit the rules to run around and do whatever the gently caress they want and blame other people for problems. It's really odd, the company isn't that big, and she's a relatively new hire (6 months). What makes it even more complicated is that she's an older lady, so you can't really get upset at her with out being extremely disrespectful. Apparently she does really good work and is a skilled developer, but her communication skills are lacking.
|
|
#
?
Jun 24, 2016 19:46
|
|
|
Jerk McJerkface posted:... she's an older lady, so you can't really get upset at her with out being extremely disrespectful. Yes, you can. You can also tell her "no" if she's making an end-run around process.
|
|
#
?
Jun 24, 2016 21:22
|
|
|
Don't get mad at your elders you son of a bitch (Wait her out, she'll be dead soon)
|
|
#
?
Jun 24, 2016 21:23
|
|
|
No (see how easy that was?) Special thanks to FruitStripe for helping me enact this short After School Special called "Don't Be A Doormat"
|
|
#
?
Jun 24, 2016 21:24
|
|
|
Well, there goes my evening I guess: (Sorry for German UI)
|
|
#
?
Jun 24, 2016 21:33
|
|
|
hazzlebarth posted:Well, there goes my evening I guess: (Sorry for German UI) Did you just lose 3 drives in a RIAD6?
|
|
#
?
Jun 24, 2016 21:38
|
|
|
pixaal posted:Did you just lose 3 drives in a RIAD6? In 2 and a half hours, yes. First disk gone at 19:04 CEST, second 71 minutes later, last at 21:29. Someone up there must really really hate me.
|
|
#
?
Jun 24, 2016 21:45
|
|
|
We had that happen a week ago.. server was barely 9 months old, between 5pm one evening and 8am the next morning it barfed up three drives. Fortunately our disk guru was able to get things back online with some magic. We used it as an opportunity to bug the end user about not letting us allocate any spares. "but we need all that space and we're already losing a bunch of capacity to this dumb raid 6 you're making us use." 
|
|
#
?
Jun 24, 2016 21:48
|
|
|
hazzlebarth posted:Well, there goes my evening I guess: (Sorry for German UI) Scheiße.
|
|
#
?
Jun 24, 2016 22:04
|
|
|
I thought of this thread today. I was reminded of all the talk that has been in this thread(and the ticket thread) over keeping legacy systems alive, and how much industrial machinery are still running on 486-era systems. What reminded me of this? I picked up a box with 15 Western Digital Caviar 280s. 85.3 MEGABTYE drives. There are some sold auctions on ebay for 50 bucks on these. Guess I'm spending my weekend running CHKDSK on these things! (also several 286/386/486 boards/cpus/ram and expansion cards)
|
|
#
?
Jun 25, 2016 04:34
|
|
|
RAID failures are ALWAYS fun! Check those controller logs for punctures! (as I see it's a MegaRAID)...  Even if this is not the case (and I hope it's something that simple), something I've learned from supporting other sysadmins: never miss the hard drive firmware updates if they're available. Sure, you can probably wait for a scheduled outage since it's proactive updates but make sure to at least be aware of the releases. I've seen a RAID failure caused by a firmware bug that was long since fixed but drive firmware was the only thing not getting updated. Fortunately I was able to fix the RAID stability and have no data loss in my most recent case. I looked up both the Seagate and Hitachi drives in the image and on our vendor versions of the firmware has some significant updates. For example: "Resolved rare issue that could cause drive to initiate a self-initiated reset" on the Hitachis I'm sure Seagate and Hitachi released their own bundled versions we just repackaged but something to think about in general.
|
|
#
?
Jun 25, 2016 12:26
|
|
|
|
| # ? May 27, 2024 21:48 |
|
|
flosofl posted:Yes, you can. I stick to my guns. My manager is great and always has my back. The PM is a cool guy too, so it's all good. I'm 100% onboard to hello my coworkers male money but I'm was too busy to help someone that won't even meet me halfway and treat me like a human being.
|
|
#
?
Jun 25, 2016 14:54
|
|