Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
evol262
Nov 30, 2010
#!/usr/bin/perl

hackedaccount posted:

Perfect, thanks.

Just to clarify (though I think you already got it), you won't have a bad time if you semanage then chcon, only if you chcon with semanage.

Adbot
ADBOT LOVES YOU

YouTuber
Jul 31, 2004

by FactsAreUseless

tonberrytoby posted:

Are you sure your settings in pulse audio control are correct? Did you check if it muted itself? My pulse sometimes does that.
Remember "Monitor of whatever device" is the buildin loopback for system recording. "Whatever device" is the device's microphone.
You recording tab should look like this:



:suicide: You have to be loving kidding me. Just switching to Monitor of [Device] gave me audio of programs again. I'll assume that my voice will carry through either Mumble or Steam's voice components. Nobody particularly wants to hear my banshee voice anyway.

Well, at least I learned a thing or two about ffmpeg while doing all that googling.

YouTuber fucked around with this message at 20:28 on May 9, 2013

hackedaccount
Sep 28, 2009
I might be confused unless you meant to say "only if you chcon WITHOUT semanage."

SELinux "database" - Holds that default / correct context settings for all files and directories. Read during a re-label or a restorecon, modified by semanage
semanage = Updates the SELinux database, tells it what the correct / default context of a file or directory should be, does not change the context of any files or directories
restorecon = Restores file or directory to the correct / default context by reading settings in the SELinux database
chcon = Change context permanently, but the change will be revered during a re-label or a restorecon according to the files or directories settings in the SELinux Database
Re-label = Restores ALL files and directories to the default / correct settings by reading the SELinux database and applying changes

Is that right?

Goblin Cock
Mar 18, 2013

Kaluza-Klein posted:

I really wish I knew if this whole thing was a freak accident or if it relates at all to the new SSD I am using...

It could be related to the fact that btrfs isn't really ready for production. Don't use it on your work system if stability is a requirement.

Goon Matchmaker
Oct 23, 2003

I play too much EVE-Online

Goblin Cock posted:

It could be related to the fact that btrfs isn't really ready for production. Don't use it on your work system if stability is a requirement.

I learned the btrfs lesson too. I'm now back on trusty XFS where my rear end will be staying for a long long while.

evol262
Nov 30, 2010
#!/usr/bin/perl

hackedaccount posted:

I might be confused unless you meant to say "only if you chcon WITHOUT semanage."

SELinux "database" - Holds that default / correct context settings for all files and directories. Read during a re-label or a restorecon, modified by semanage
semanage = Updates the SELinux database, tells it what the correct / default context of a file or directory should be, does not change the context of any files or directories
restorecon = Restores file or directory to the correct / default context by reading settings in the SELinux database
chcon = Change context permanently, but the change will be revered during a re-label or a restorecon according to the files or directories settings in the SELinux Database
Re-label = Restores ALL files and directories to the default / correct settings by reading the SELinux database and applying changes

Is that right?

/etc/selinux/targeted/contexts/files/* -> SELinux "database" that keeps track of default contexts. You can edit/create/view these like any other file. Create a file in /tmp, add its name to file_contexts with come context flags and restorecon to be amazed.
semanage -> updates default contexts, users, login mappings booleans, policies, and a lot more -- SELinux is a fairly fast moving target, but there are reasonably good books. It will update
rectorecon -> checks SELinux "database". Will set to default (file_t, though you can change thsi with fstab flags) if it doesn't find anything
chcon/relabel -> your responses are pretty much accurate

evol262 fucked around with this message at 21:39 on May 9, 2013

hackedaccount
Sep 28, 2009
Great, that works for me.

I'm reviewing poo poo for my RHCE so I just need enough to pass the test and I don't work with it so most of it's been forgotten. The booleans are pretty straightforward but all the fart_butt_rw_t stuff is a bit much - guess I'll have to skim the book and memorize the major ones.

Now to decide if I want to gamble if chcon is enough and hope they don't re-label during grading or memorize semanage + restorecon...

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

evol262 posted:

/etc/selinux/targeted/contexts/files/* -> SELinux "database" that keeps track of default contexts. You can edit/create/view these like any other file. Create a file in /tmp, add its name to file_contexts with come context flags and restorecon to be amazed.
semanage -> updates default contexts, users, login mappings booleans, policies, and a lot more -- SELinux is a fairly fast moving target, but there are reasonably good books. It will update
rectorecon -> checks SELinux "database". Will set to default (file_t, though you can change thsi with fstab flags) if it doesn't find anything
chcon/relabel -> your responses are pretty much accurate
One other thing to note is that while chcon is "permanent," if your file is modified by creating a new file and doing an atomic replacement (i.e. mv) rather than truncating the file and rewriting, you will lose your assigned context. FTP and friends will all clobber you.

Goon Matchmaker
Oct 23, 2003

I play too much EVE-Online

Misogynist posted:

One other thing to note is that while chcon is "permanent," if your file is modified by creating a new file and doing an atomic replacement (i.e. mv) rather than truncating the file and rewriting, you will lose your assigned context. FTP and friends will all clobber you.

Isn't restorecond specifically for that situation?

hackedaccount
Sep 28, 2009
Today I learned that even though NFSv4 only needs port 2049 TCP you still need to start portmap and other associated crap on RHEL / CentOS :argh:

evol262
Nov 30, 2010
#!/usr/bin/perl

Goon Matchmaker posted:

Isn't restorecond specifically for that situation?

Yes but no. The recurring theme here is that if you don't add it to /etc/selinux/targeted/contexts/files/* somehow (manually, semanage, whatever), restorecon and restorecond look at a new file and say "I don't know what to make of this: file_t" (or whatever your default is). chcon doesn't add it to the list of known contexts, so restorecond will happily restore it to whatever the default is, not what you chconned it to.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Goon Matchmaker posted:

Isn't restorecond specifically for that situation?
restorecond is inotify-based and only watches files you specifically tell it to.

Vin BioEthanol
Jan 18, 2002

by Ralp
I have a headless/no-X fedora 16 system running on an old optiplex gx280. Capacitors are starting to bulge and leak on it, noticed when I was blowing it out.

I have a slightly less-old optiplex 745 I think it is that's not being used. Any way I can just pop the drive in there? Or is it going to just poo poo itself like windows would?

Vin BioEthanol fucked around with this message at 04:36 on May 12, 2013

pseudorandom name
May 6, 2007

It should just work.

waffle iron
Jan 16, 2004

Wagonburner posted:

I have a headless/no-X fedora 16 system running on an old optiplex gx280. Capacitors are starting to bulge and leak on it, noticed when I was blowing it out.

I have a slightly less-old optiplex 745 I think it is that's not being used. Any way I can just pop the drive in there? Or is it going to just poo poo itself like windows would?

Fedora 16 might still use the udev for persistent naming of network interfaces, so you might have to take a look in /etc/udev/rules.d/ and delete a file and then reboot the machine (or restart the service that generates it). That's the only caveat I can think of.

Edit: And that is unlikely you'll even have to do that.

waffle iron fucked around with this message at 07:37 on May 12, 2013

hackedaccount
Sep 28, 2009
Might want to also double check to make sure your MAC addresses aren't hard coded into to your /etc/sysconfig/network-scripts/ifcfg-ethX files

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

waffle iron posted:

Fedora 16 might still use the udev for persistent naming of network interfaces, so you might have to take a look in /etc/udev/rules.d/ and delete a file and then reboot the machine (or restart the service that generates it). That's the only caveat I can think of.

Edit: And that is unlikely you'll even have to do that.
On the other hand, if the PCI slot assignment changed, the device name will have changed regardless of whether there's a persistent-net name for it.

Vin BioEthanol
Jan 18, 2002

by Ralp
Thanks you all. Gonna give it a try Tues or weds night.

Pixotic
Jan 14, 2008

He could be in this very room!
He could be
you!
He could be
me!
He could even b:commissar:
At work we host several dozen of our clients across a handful of Debian VPSs, it's all unmanaged hosting and of course, hardly any of them have updated their CMS solutions and a whole mess of them using Joomla have been infected and are spamming.

Our server guy resigned recently and it's fallen to a fellow developer and myself, somehow, to try and unfuck this entire situation. I'm learning linux on the go here, so hopefully this is something a more experienced goon can offer some insight into:

Are there any programs around that can monitor what, exactly, certain processes are doing? Specifically I want to know what's happening when users are running php5-cgi, to see if I can identify problem accounts more easily.

The infection seems to be characterised by both new files being generated which trigger obfuscated code that does the spamming, or inserting obfuscated lines blindly into existing files to achieve the same effect. It's typically eval(base64_decode(foo)) but recently whoever's doing this has been getting crafty and using other methods such as gzip_deflate, in case any of this rings a bell.

So far I've been using Neopi to try to root out affected files, but am only achieving limited success with that. Mostly I'm just monitoring top for php processes that spike up above 15-20% cpu, noting down the user associated with it and inspecting their files semi-manually, and it's taking forever.

Any tips or tools that could be suggested would be greatly appreciated, thanks!

nitrogen
May 21, 2004

Oh, what's a 217°C difference between friends?
Nevermind, misread what I was replying to.

fatherdog
Feb 16, 2005

Pixotic posted:

At work we host several dozen of our clients across a handful of Debian VPSs, it's all unmanaged hosting and of course, hardly any of them have updated their CMS solutions and a whole mess of them using Joomla have been infected and are spamming.

Our server guy resigned recently and it's fallen to a fellow developer and myself, somehow, to try and unfuck this entire situation. I'm learning linux on the go here, so hopefully this is something a more experienced goon can offer some insight into:

Are there any programs around that can monitor what, exactly, certain processes are doing? Specifically I want to know what's happening when users are running php5-cgi, to see if I can identify problem accounts more easily.

See if strace or truss are installed.

evol262
Nov 30, 2010
#!/usr/bin/perl

fatherdog posted:

See if strace or truss are installed.

It's not Solaris. You won't find truss. strace and ltrace are ok, but it's going to be a huge headache to try to troubleshoot things by reading each and every system/library (respectively) call php5-cgi makes.

Pixotic: when you say "what, exactly", what do you mean? You probably don't want or need to know exactly what they're doing each time. Do you want to track who's sending mail? Who's doing database accesses? Narrowing down what you're looking for would help immensely.

SYSV Fanfic
Sep 9, 2003

by Pragmatica
I don't know anything about the software that you are running, but it seems like adding basic logging to the application every time it calls specific functions would let you identify problem accounts...

Bob Morales
Aug 18, 2006


Just wear the fucking mask, Bob

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

keyvin posted:

I don't know anything about the software that you are running, but it seems like adding basic logging to the application every time it calls specific functions would let you identify problem accounts...

You'd think that there's some logging in PHP or whatever server you're using that would give you that info, and offer it at configurable levels (errors, warnings, debug, etc)

12.34.56.78 GET whatever.com/index.php
12.34.56.78 POST {asasdfasdfasfdasfd} to /login.php
304 whatever.com/login.php

etc

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Pixotic posted:

At work we host several dozen of our clients across a handful of Debian VPSs, it's all unmanaged hosting and of course, hardly any of them have updated their CMS solutions and a whole mess of them using Joomla have been infected and are spamming.

Our server guy resigned recently and it's fallen to a fellow developer and myself, somehow, to try and unfuck this entire situation. I'm learning linux on the go here, so hopefully this is something a more experienced goon can offer some insight into:

Are there any programs around that can monitor what, exactly, certain processes are doing? Specifically I want to know what's happening when users are running php5-cgi, to see if I can identify problem accounts more easily.

The infection seems to be characterised by both new files being generated which trigger obfuscated code that does the spamming, or inserting obfuscated lines blindly into existing files to achieve the same effect. It's typically eval(base64_decode(foo)) but recently whoever's doing this has been getting crafty and using other methods such as gzip_deflate, in case any of this rings a bell.

So far I've been using Neopi to try to root out affected files, but am only achieving limited success with that. Mostly I'm just monitoring top for php processes that spike up above 15-20% cpu, noting down the user associated with it and inspecting their files semi-manually, and it's taking forever.

Any tips or tools that could be suggested would be greatly appreciated, thanks!
Unhelpful rant about how your company sucks at this:

It's not your job, or the job of anyone at your company, to try to clean this poo poo up. Ask your customers to upload clean files and patch their install. Unless the customers are paying you $100/month, or unless you spend most of your day idle and your company is just itching to find something to actually do with the body in your chair, it's costing you more money in lost productivity to fix this than to just sandbox off these accounts and tell the customers to fix their poo poo or find another host.

No hosting service worth using has an "our server guy" anyway.

Bob Morales
Aug 18, 2006


Just wear the fucking mask, Bob

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

Anyone have a basic troubleshooting checklist for 'my apache server went down for 20 minutes for what appears to be no reason at all. Load didn't go up, it didn't start paging, CPU and memory never went up above 40%...' ?

Our Rackspace tech did say there were 492 Apache processes running when normally there's ~ 100

evol262
Nov 30, 2010
#!/usr/bin/perl

Bob Morales posted:

Anyone have a basic troubleshooting checklist for 'my apache server went down for 20 minutes for what appears to be no reason at all. Load didn't go up, it didn't start paging, CPU and memory never went up above 40%...' ?

Our Rackspace tech did say there were 492 Apache processes running when normally there's ~ 100

Are you sure CPU and memory never went above 40%?

Are you really sure?

Are you positive the OOM killer didn't get its grubby hands on the Apache parent process?

If not, usual culprits: dmesg, /var/log/messages|syslog, Apache logs, look for core files.

If Apache didn't actually go down, and it simply became unresponsive instead (and "done for 20 minutes" implies that, unless you have something kicking Apache when it dies, which indicates that you have a whole other level of problems), start looking at network. If there were 5 times as many Apache processes as normal, what was going on? Was somebody mirroring your webpage? Did you make the frontpage of Reddit, Slashdot, Hackernews, or whatever? Was traffic up? Where was it coming from? Were other services (SSH, whatever) available? Was your monitoring working normally? Do you have monitoring? Why is a Rackspace tech investigating your webserver problems?

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.
I highly recommend having collectd or Munin watch your scoreboard.

Bob Morales
Aug 18, 2006


Just wear the fucking mask, Bob

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

Another dev called Rackspace in case we needed them to reboot the box. And to make sure they didn't do something dumb like restart MySQL in the middle of the day for no good reason.

It was weird because if we get a traffic spike the load average goes sky high and ssh sessions become very very lagggy or even unusable, but none of that happened. We just started getting pingdom alerts at 14:38. The huge dip in network I/O is the outage.



About 10 minutes later it all came back up. Sure enough, the # of Apache processes went through the roof. Pegged at 500 which is what MaxClients its set at.



Looking thought the logs with commands like
code:
egrep -rn '13\/May\/2013:14:3[5-9]' *_log | cut -c1-8 |sort|uniq -c|sort -nr|more
   1273 t2-acc
   1098 tr-acc
   1088 access_l
    952 1k-acce
    893 walk-
The second one in that list, 'tr' should be about 1/5 of the traffic. It's a new app and it is likely spawning some re-direct or getting stuck somewhere and is most likely the problem. But we have 3x the apps all crammed on one server than we did 2 years ago and nobody wants to upgrade the hardware (Dual Xeon 5520's and 16GB) or even upgrade the RAM....and don't have any load-balancing or failover setup and nobody wants to do jack poo poo about it.

theperminator
Sep 16, 2009

by Smythe
Fun Shoe

Misogynist posted:

Unhelpful rant about how your company sucks at this:

It's not your job, or the job of anyone at your company, to try to clean this poo poo up. Ask your customers to upload clean files and patch their install. Unless the customers are paying you $100/month, or unless you spend most of your day idle and your company is just itching to find something to actually do with the body in your chair, it's costing you more money in lost productivity to fix this than to just sandbox off these accounts and tell the customers to fix their poo poo or find another host.

No hosting service worth using has an "our server guy" anyway.

Yeah the problem is that if they only have "a server guy" they obviously can't afford to piss off too many customers or I'd suggest they give 3-strikes and terminate after they're up.

However, I can say that Maldet is pretty good for what he wants.
It has inotify support and so it'll scan new files/modified files and is designed to take care of the PHP injection attacks people are seeing and it can even do a cpanel suspension or lock a shell account if you want it to.

From my experience these PHP injections you see are rarely from your customer and are more often than not the result of malware being used to steal FTP details or from wordpress/joomla etc exploits.

evol262
Nov 30, 2010
#!/usr/bin/perl

Bob Morales posted:

Looking thought the logs with commands like
code:
egrep -rn '13\/May\/2013:14:3[5-9]' *_log | cut -c1-8 |sort|uniq -c|sort -nr|more
   1273 t2-acc
   1098 tr-acc
   1088 access_l
    952 1k-acce
    893 walk-
The second one in that list, 'tr' should be about 1/5 of the traffic. It's a new app and it is likely spawning some re-direct or getting stuck somewhere and is most likely the problem.
Stop. It's irrelevant how much traffic it created, or "should" create. There are too many "likely"s in this statement. You know it's almost certainly the new website because it's new, and you didn't have this problem before. It's not an Apache problem. It's a problem with your Sinatra/Rails/whatever mod_passenger app. Look at the access logs, find out what users were doing with that application that spun up httpd processes which did nothing, replicate it in dev/QA (or in production if you don't have a dev environment), and find/fix the bug.

Bob Morales posted:

But we have 3x the apps all crammed on one server than we did 2 years ago and nobody wants to upgrade the hardware (Dual Xeon 5520's and 16GB) or even upgrade the RAM....and don't have any load-balancing or failover setup and nobody wants to do jack poo poo about it.
This isn't really relevant unless you just want to generally bitch.

insidius
Jul 21, 2009

What a guy!

theperminator posted:

Yeah the problem is that if they only have "a server guy" they obviously can't afford to piss off too many customers or I'd suggest they give 3-strikes and terminate after they're up.

However, I can say that Maldet is pretty good for what he wants.
It has inotify support and so it'll scan new files/modified files and is designed to take care of the PHP injection attacks people are seeing and it can even do a cpanel suspension or lock a shell account if you want it to.

From my experience these PHP injections you see are rarely from your customer and are more often than not the result of malware being used to steal FTP details or from wordpress/joomla etc exploits.

CXS can also be good for after the fact infections, though neither solution is fool proof.

I have found using the auto updating ASL mod_security rules does a solid job at minimising such infections in the first place.

Bob Morales
Aug 18, 2006


Just wear the fucking mask, Bob

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

evol262 posted:

Stop. It's irrelevant how much traffic it created, or "should" create. There are too many "likely"s in this statement. You know it's almost certainly the new website because it's new, and you didn't have this problem before. It's not an Apache problem. It's a problem with your Sinatra/Rails/whatever mod_passenger app. Look at the access logs, find out what users were doing with that application that spun up httpd processes which did nothing, replicate it in dev/QA (or in production if you don't have a dev environment), and find/fix the bug.

This isn't really relevant unless you just want to generally bitch.

This is just step 1/2 of the process. Caaalm down.

WarauInu
Jul 29, 2003
In case anyone hasn't seen yet.

https://news.ycombinator.com/item?id=5703758

Looks like an exploit was backported into kernel

https://www.centos.org/modules/newbb/viewtopic.php?topic_id=42827&forum=59

covener
Jan 10, 2004

You know, for kids!

WarauInu posted:

Looks like an exploit was backported into kernel

That's a really weird way to phrase it.

fatherdog
Feb 16, 2005

FYI on a vanilla Fedora 17 install this won't give root, but it will crash the box.

It flat out won't run on any of our Oracle 5/6 servers, but our setups are far from vanilla and probably won't be representative of most.

Adult Sword Owner
Jun 19, 2011

u deserve diploma for sublime comedy expertise
This is probably another one of my Dumb Easy questions, but I have a box that won't boot because I fatfingered an entry somewhere, so it gets stuck on NFS. I haven't let it go very long so I don't know if it's going to time out but I thought I read that service does not actually have a time out. I don't know how to boot interactive and tell it "no do not load that service" because pressing i doesn't do it. I've tried booting into Knoppix but something is wrong with either the drives or my burn because after the prompt and crappy ASCII logo it just spins and goes nowhere, so I need some way to get in that's not going to load the nfs service, or at least a way to specify not to load it.

SYSV Fanfic
Sep 9, 2003

by Pragmatica
Add init=/bin/bash to your kernel options at boot, then remount / rw.

Adult Sword Owner
Jun 19, 2011

u deserve diploma for sublime comedy expertise

keyvin posted:

Add init=/bin/bash to your kernel options at boot, then remount / rw.

Thanks, that'd probably also work, but apparently single user mode was not working because I fatfingered "single" in the arguments. Got in and was able to fix it. Thanks!

Adbot
ADBOT LOVES YOU

Vedder
Jun 20, 2006

This is probably a really simple one but I thought I would I ask, I have a Ubuntu desktop machine at work that has a Nagios server for monitoring our (mostly) Windows environment. We have moved our webhosting and I am looking after the hosted Ubuntu server box. I have done some reading into getting the Nagios server to provide reports on the Ubuntu server and I have found by creating a ssh key and using the check_by_ssl plugin I am able to get feedback on CPU/memory usage etc. The problem is that I need to enter a password in the command line everytime. I have two user accounts on the box that controls the Nagios server and I get different results, they are:

If I am logged in as the admin (username: ubuntu) then the command ssh nagiosadmin@host.ip logs straight into the Ubuntu server which is what I want.

If I log in as my nagiosadmin account then use the same command I am prompted for the password, if I enter this correctly it works.

I have a user called Nagiosadmin on both boxes and it has full control on the Nagios plugin folders, I am guessing that I am missing some sort of account priviledge to allow me to connect straight through without using a password?

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply