The Linux Questions Thread: a bunch of pitfalls, but technically it's possible

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Linux Questions Thread: a bunch of pitfalls, but technically it's possible

«‹›1036 »

Major Ryan: May 11, 2008; Completely blank

xzzy posted:

How big is the database? Is anything running some potentially goofy queries against it? In my experience such queries reveal themselves by filling up /tmp first but maybe your config is a bit different with a relatively big /tmp.

Various databases on the server which vary in size. Biggest is about 200G, total is about 500G on the server.

/tmp is 20G although we configure temporary tables into a specific database rather than going to /tmp, so most things go there. Disk is 750Gish - we've had space cause problems with /tmp filling up before, but it's not he issue here at the moment.

Someone is almost certainly running goofy queries against it - that's a different battle to fight but "It's always worked before" means this is my problem not theirs for the moment.

# ? Dec 15, 2020 11:11

Adbot: ADBOT LOVES YOU

# ? May 29, 2024 15:33

BlankSystemDaemon: Mar 13, 2009

Not to go off on a tanned gent too much, but do people still choose MariaDB over postgres on new deployments?

# ? Dec 15, 2020 12:16

Truga: May 4, 2014; Lipstick Apathy

BlankSystemDaemon posted:

Not to go off on a tanned gent too much, but do people still choose MariaDB over postgres on new deployments?

unfortunately bad software doesn't support postgres :v:

# ? Dec 15, 2020 12:26

Hadlock: Nov 9, 2004

BlankSystemDaemon posted:

Not to go off on a tanned gent too much, but do people still choose MariaDB over postgres on new deployments?

There are still people spinning up new LAMP stacks every day. If you've been writing cake/php apps against mysql for 15+ years, I'd imagine it would be hard to switch tech at this point

I personally haven't heard of anyone using anything but postgres in the last 10 years. Postgres 9.4 (2012?) was a really fantastic release, 9.6 got native json support, 10.0(?) got logical replication and now they're up to v13.something and it's running pretty much every startup company at this point

Pre-containers maybe the hassle of installing/configuring the db from scratch was a reason to pick one over the other but now it's a literal one liner to "install postgres" as a container

# ? Dec 15, 2020 12:33

Bob Morales: Aug 18, 2006; ~~Just wear the fucking mask, Bob~~

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

Major Ryan posted:

I don't know to a more specific databases thread, so if anyone here could help, it'd really be appreciated since this is doing my head in.

I'm running MariaDB (10.3.27) on a fully updated version of CentOS 8 VM with 8 CPU/32G RAM. For a while now I've been seeing increase large numbers of oom events, which is obviously annoying.

Here's the specific question though:

I've got innodb_buffer_pool_size set to 12G on this machine. Recommendation is to set it at about 70% of the memory available to the machine, so that the database can run and there's overhead for the other things the db service needs and the server itself. So with this set at more like 40%, how is it possible that server runs out of memory? As I understand it, it should be swapping out past this limit, which isn't ideal but at least the service wouldn't crash.

Anyone familiar enough with database memory usage to offer me a suggestion? I've tweaked lots of variables but everything seems to be minor performance bits rather than what I need to get fixed which is 20G of memory usage out of nowhere.

Do you have a web server running on there, or is it just a database server?

Show some top output
Post your my.cnf
Post your mysql.err

You'd actually want to make the buffer_pool_size smaller if you're running out of memory.

Usually when I've seen this happen, it's because of some query that's returning a billion rows of data. And usually chucking another 32GB of memory in the server isn't very expensive.

# ? Dec 15, 2020 13:52

Major Ryan: May 11, 2008; Completely blank

Bob Morales posted:

Do you have a web server running on there, or is it just a database server?

Show some top output
Post your my.cnf
Post your mysql.err

You'd actually want to make the buffer_pool_size smaller if you're running out of memory.

Usually when I've seen this happen, it's because of some query that's returning a billion rows of data. And usually chucking another 32GB of memory in the server isn't very expensive.

I'll have to grab some bits. Top is fine at the moment because it's not busy right now, so if I can catch something when it crashes next.

It's just the db server, there's nothing else on here apart from the OS basics and the Maria service.

We've been tweaking the buffer_pool_size for a while, it would have been 24G, then 15G, 10G, now we've tried 5G...

More memory is an option and we can do that to make the problem go away (which is probably the answer we'll take to be honest). I'm just confused from a technical point about the buffer_pool_size - it's obviously not the hard memory usage limit that I thought it was.

my.cnf posted:

[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
datadir = /var/lib/mysql/

innodb_buffer_pool_size = 5G # 20201203 15 to 10 after oom. 20201204 down to 5G

skip-external-locking
#sort_buffer_size = 4M
table_definition_cache = 1000
max_connections = 500
wait_timeout = 7200
innodb_adaptive_hash_index = 0 # 20201209 commented, then uncommented to try and solve hung semaphore bug.

log-error = /var/log/mariadb/mariadb.log
log_warnings = 4
character-set-server = utf8
open-files-limit = 65535
sql-mode = "STRICT_ALL_TABLES,NO_AUTO_CREATE_USER"
slow_query_log = 1

innodb_flush_method = O_DIRECT
innodb_file_per_table
innodb_print_all_deadlocks = ON
innodb_strict_mode = ON
innodb_log_buffer_size = 1G # 20201209 reporting nonzero Innodb_log_waits, raising default to 1G

userstat = 1 # Enables user stat recording in pmm

# ? Dec 15, 2020 14:29

Bob Morales: Aug 18, 2006; ~~Just wear the fucking mask, Bob~~

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

Major Ryan posted:

More memory is an option and we can do that to make the problem go away (which is probably the answer we'll take to be honest). I'm just confused from a technical point about the buffer_pool_size - it's obviously not the hard memory usage limit that I thought it was.

There are a million other things that mysql will use memory for, the buffer pool size just limits one thing

https://www.percona.com/blog/2018/06/28/what-to-do-when-mysql-runs-out-of-memory-troubleshooting-guide/

Are you monitoring all your mysql stats with something? You can get a lot of metrics out of it.

# ? Dec 15, 2020 15:04

Major Ryan: May 11, 2008; Completely blank

Bob Morales posted:

There are a million other things that mysql will use memory for, the buffer pool size just limits one thing

Are you monitoring all your mysql stats with something? You can get a lot of metrics out of it.

Yeah, the gap isn't in the monitoring, it's in the lack of understanding of what it means :-)

There's someone with more SQL knowledge that me looking into it too, so hopefully they'll be able to offer some suggestions based off the monitoring. The user-stats line in the my.cnf is for the percona monitoring and management tools that we've just set up, so we've got some nice grafana graphs to go through and try to spot something.

Thanks for the help; it's enough to know that the buffer pool size isn't the limit I thought it was. At least confirms there's memory usage somewhere else in Maria, just need to find out where...!

# ? Dec 15, 2020 16:43

Antigravitas: Dec 8, 2019; Die Rettung fuer die Landwirte:

I'd log queries (especially slow ones) to see what is happening before the kernel goes on a killing spree.

Without knowing what it's actually doing it will be hard to figure out the problem. It could be creating in-memory tables and filling them or something similarly crazy.

# ? Dec 15, 2020 18:55

xzzy: Mar 5, 2009

Major Ryan posted:

Yeah, the gap isn't in the monitoring, it's in the lack of understanding of what it means :-)

It means the same thing every sysadmin problem means: users are always the problem and this job would be so much easier if they stopped using their computers so drat much.

Or in this case quit running queries that result in enormous joins.

# ? Dec 15, 2020 19:06

Diametunim: Oct 26, 2010

Bob Morales posted:

Does it not increment the .1 because you have the date in the file which changes for each file?

Can you post the actual file names and redact the IP�s?

That's correct, the real file names increment by date. This is handled by syslog-ng automatically at the start of each day (midnight). Below is an example of actual filenames from my systems.

code:

ls -la /var/log/remote/stacktaces/192.168.1.100/
-rw-r----- 1 root splunk 312709966 Nov 28 23:59 2020-11-28-a_stacktrace.log.1
-rw-r----- 1 root splunk   6437439 Nov 28 22:47 2020-11-28-p_stacktrace.log.1
-rw-r----- 1 root splunk 263691195 Nov 29 23:59 2020-11-29-a_stacktrace.log.1
-rw-r----- 1 root splunk   1699023 Nov 29 18:06 2020-11-29-p_stacktrace.log.1
-rw-r----- 1 root splunk 314861156 Nov 30 23:59 2020-11-30-a_stacktrace.log.1
-rw-r----- 1 root splunk  12953713 Nov 30 23:56 2020-11-30-p_stacktrace.log.1
-rw-r----- 1 root splunk 225326350 Dec  1 23:59 2020-12-01-a_stacktrace.log.1
-rw-r----- 1 root splunk   5627112 Dec  1 23:05 2020-12-01-p_stacktrace.log.1
-rw-r----- 1 root splunk 204430336 Dec  2 23:59 2020-12-02-a_stacktrace.log.1
-rw-r----- 1 root splunk   4568128 Dec  2 23:38 2020-12-02-p_stacktrace.log.1
-rw-r----- 1 root splunk   1437852 Dec  3 04:23 2020-12-03-a_stacktrace.log.1
-rw-r----- 1 root splunk    898105 Dec  4 04:10 2020-12-04-a_stacktrace.log.1
-rw-r----- 1 root splunk   1416604 Dec  5 05:11 2020-12-05-a_stacktrace.log.1
-rw-r----- 1 root splunk     35874 Dec  5 04:53 2020-12-05-p_stacktrace.log.1
-rw-r----- 1 root splunk   1302955 Dec  6 05:37 2020-12-06-a_stacktrace.log.1
-rw-r----- 1 root splunk 190678316 Dec  7 23:59 2020-12-07-a_stacktrace.log.1
-rw-r----- 1 root splunk   4916366 Dec  7 23:00 2020-12-07-p_stacktrace.log.1
-rw-r----- 1 root splunk   3180144 Dec  8 04:53 2020-12-08-a_stacktrace.log.1
-rw-r----- 1 root splunk   1194672 Dec  9 03:18 2020-12-09-a_stacktrace.log.1
-rw-r----- 1 root splunk    215379 Dec 10 04:19 2020-12-10-a_stacktrace.log.1
-rw-r----- 1 root splunk 179245131 Dec 11 23:59 2020-12-11-a_stacktrace.log.1
-rw-r----- 1 root splunk   3249459 Dec 11 22:33 2020-12-11-p_stacktrace.log.1
-rw-r----- 1 root splunk 216049636 Dec 12 23:58 2020-12-12-a_stacktrace.log.1
-rw-r----- 1 root splunk   1913410 Dec 12 18:56 2020-12-12-p_stacktrace.log.1
-rw-r----- 1 root splunk 188304564 Dec 13 23:59 2020-12-13-a_stacktrace.log.1
-rw-r----- 1 root splunk     19624 Dec 13 14:40 2020-12-13-p_stacktrace.log.1
-rw-r----- 1 root splunk 186729816 Dec 14 23:59 2020-12-14-a_stacktrace.log.1
-rw-r----- 1 root splunk   4888699 Dec 14 23:24 2020-12-14-p_stacktrace.log.1
-rw-r----- 1 root splunk  91903059 Dec 15 14:07 2020-12-15-a_stacktrace.log
-rw-r----- 1 root splunk   2269751 Dec 15 14:07 2020-12-15-p_stacktrace.log

xzzy posted:

logrotate has limitations dealing with files that have different file names. If you match them with a glob it runs the config against each match as a separate task, not as different versions of the same log file.

Thanks for the advice, I had read a similar response on Stack Overflow. I just wanted to confirm with the masses before I start looking for a new method to rotate these files. I'm honestly looking to keep 7 days worth of log files on these systems. Anything else can be thrown away. Any advice on the specific approach I should use?

From my perspective I could:
1) Create a cronjob in cron.daily and leverage the find command to search for logs older than 7 days, then delete. I don't like managing cron jobs though.
2) Create a hacky log-rotate script that leverages the post-rotate method, to call the find command, search for logs older than 7 days, then delete.
3) Use syslog-ng natively to handle deletion of old log files. (Not even sure if this is possible TBH).

# ? Dec 15, 2020 21:16

VictualSquid: Feb 29, 2012; Gently enveloping the target with indiscriminate love.

I think I used to have a setup where either logrotate or something else compatible with it was rotating logs like that for me.
But I don't remember how those settings worked. It might just be an exotic option in the logrotate config.

# ? Dec 15, 2020 21:31

xzzy: Mar 5, 2009

Unless the business logic required dates in the filename, I'd alter my syslog config to name the log "stacktrace.log" and use the mtime of the files if someone wanted to know what day the log was from.

Logrotate has an option to append the date if it's really a requirement (dateext I think, but it has a few "date" options).

# ? Dec 15, 2020 22:00

Bob Morales: Aug 18, 2006; ~~Just wear the fucking mask, Bob~~

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

I would just include the date in the entry in the file

# ? Dec 15, 2020 22:59

Antigravitas: Dec 8, 2019; Die Rettung fuer die Landwirte:

Reading the release announcement for the next OpenSuSe leap:

quote:

openSUSE Leap 15.3 is based on the Jump concept that was developed over the past several months, which makes it and SUSE Linux Enterprise compatible. openSUSE Leap aligns with SLE and its Service Packs (SP), which keeps the system updated, stable and patched. Upon General Availability of this release, there will be a whole new level of harmony between Leap 15.3 and SUSE Linux Enterprise 15 SP3.

The death of the CentOS announced last week aligns well for those users who are ready to move away from RedHat�s community enterprise release to a better release model with openSUSE Leap, which has a life cycle of about 18 months of maintenance and security updates per minor release.

So they are doing literally the opposite of RH and appear to be kind of smug about it. I guess I'll try it out anyway, I kind of want to play more with OBS.

# ? Dec 16, 2020 16:16

xzzy: Mar 5, 2009

I've brought up opensuse in my work group as a "haha just kidding but not really, anyone wanna try it?" when discussing the centos stuff and there's zero interest. Users are the same way, all they care about is which free redhat derivative will give them the longest shelf life between updates.

Which makes no sense to me, if you're so angry at redhat for killing your distro that you want to jump ship from centos, why the gently caress would you continue to use anything redhat based? What stops them from gobbling up the "new savior" Rocky Linux in a few years and killing it too?

I don't understand people.

# ? Dec 16, 2020 16:23

RFC2324: Jun 7, 2012; http 418

xzzy posted:

I've brought up opensuse in my work group as a "haha just kidding but not really, anyone wanna try it?" when discussing the centos stuff and there's zero interest. Users are the same way, all they care about is which free redhat derivative will give them the longest shelf life between updates.

Which makes no sense to me, if you're so angry at redhat for killing your distro that you want to jump ship from centos, why the gently caress would you continue to use anything redhat based? What stops them from gobbling up the "new savior" Rocky Linux in a few years and killing it too?

I don't understand people.

Non linux people don't realize that redhat != Linux, so if you suggest a different distro they react as if was a change to AIX or BSD or some unix that is arcane and bizzare.

This includes alot of windows IT people

# ? Dec 16, 2020 16:33

NihilCredo: Jun 6, 2011; iram omni possibili modo preme:
plus una illa te diffamabit, quam multæ virtutes commendabunt

I just remembered that Fedora Server exists.

Who are its target users, exactly? People who run pet servers and can afford to update them both carefully and frequently? People who for some reason need to keep package versions synchronized between their workstation(s) and their server(s)?

# ? Dec 16, 2020 17:28

mystes: May 31, 2006

NihilCredo posted:

I just remembered that Fedora Server exists.

Who are its target users, exactly? People who run pet servers and can afford to update them both carefully and frequently? People who for some reason need to keep package versions synchronized between their workstation(s) and their server(s)?

Psychics who foresaw the demise of centos?

# ? Dec 16, 2020 17:30

RFC2324: Jun 7, 2012; http 418

NihilCredo posted:

I just remembered that Fedora Server exists.

Who are its target users, exactly? People who run pet servers and can afford to update them both carefully and frequently? People who for some reason need to keep package versions synchronized between their workstation(s) and their server(s)?

When I worked for a telecom in their attempt at a cloud solution, they used a customized fedora 21 as their server OS for running xen on top of, so them?

# ? Dec 16, 2020 17:31

hifi: Jul 25, 2012

NihilCredo posted:

I just remembered that Fedora Server exists.

Who are its target users, exactly? People who run pet servers and can afford to update them both carefully and frequently? People who for some reason need to keep package versions synchronized between their workstation(s) and their server(s)?

i decided i wasn't larping an enterprise setup at home with regards to uptime percentage or support. it works great, i don't know what you mean by updating carefully and frequently

# ? Dec 16, 2020 17:50

Bob Morales: Aug 18, 2006; ~~Just wear the fucking mask, Bob~~

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

NihilCredo posted:

I just remembered that Fedora Server exists.

Who are its target users, exactly? People who run pet servers and can afford to update them both carefully and frequently? People who for some reason need to keep package versions synchronized between their workstation(s) and their server(s)?

According to the IRC channel like 20% of their downloads are server

# ? Dec 16, 2020 18:03

xtal: Jan 9, 2011; by Fluffdaddy

Is it also 20% of their ISOs because that's probably just mirrors or something, unlike actual usage data

# ? Dec 16, 2020 18:11

xzzy: Mar 5, 2009

hifi posted:

i decided i wasn't larping an enterprise setup at home with regards to uptime percentage or support. it works great, i don't know what you mean by updating carefully and frequently

My enterprise larp is flatcar, gently caress the OS, put it all in containers.

I've been really happy with it, if I want to try something new pull the image. When I decide I hate it, delete it and it was never there. Never get that impulse to reinstall because I got too many extra packages or dead config installed.

# ? Dec 16, 2020 18:28

Chilled Milk: Jun 22, 2003; No one here is alone,
satellites in every home

I use Fedora Server in VMs/for messing with poo poo, because yes I use Fedora Workstation and it's easier. Nothing production-ish but it would probably be fine enough :shrug:

. I do plan on switching my docker host to Fedora CoreOS once it's a little more baked.

# ? Dec 16, 2020 18:48

Saukkis: May 16, 2003; Unless I'm on the inside curve pointing straight at oncoming traffic the high beams stay on and I laugh at your puny protest flashes.
I am Most Important Man. Most Important Man in the World.

xzzy posted:

I've brought up opensuse in my work group as a "haha just kidding but not really, anyone wanna try it?" when discussing the centos stuff and there's zero interest. Users are the same way, all they care about is which free redhat derivative will give them the longest shelf life between updates.

Which makes no sense to me, if you're so angry at redhat for killing your distro that you want to jump ship from centos, why the gently caress would you continue to use anything redhat based? What stops them from gobbling up the "new savior" Rocky Linux in a few years and killing it too?

I don't understand people.

I'd say it's mainly familiarity and software compatibility. Apt, yum and zypper are very similar, but still annoyingly different from each other, not to mention any other components. Especially if you want to do automated installs. And sometimes you need to install software outside the distribution and you better hope it's packaged for the distro you are using. Red Hat has always been a popular packaging target by upstream and you were almost certain it would work in CentOS. And then there is the research group which wants a new server and gives a list of software they need. Software A is packaged for Red Hat by upstream, software B for Ubuntu. Good luck choosing the distro.

It's very trivial to prevent the same from happening to Rocky Linux. Don't give Red Hat the trademark. The only reason Red Hat could make this announcement is because they own the CentOS trademark, and they own it because the people behind CentOS donated the trademark with Red Hat took it under their umbrella.

# ? Dec 16, 2020 19:06

Butter Activities: May 4, 2018

I still don�t understand why they don�t standardize package managers. I mean I only use Debian based distros so far but it�s odd that there are so many variants to do the exact same thing.

gently caress snap though.

# ? Dec 18, 2020 07:04

RFC2324: Jun 7, 2012; http 418

SpaceSDoorGunner posted:

I still don�t understand why they don�t standardize package managers. I mean I only use Debian based distros so far but it�s odd that there are so many variants to do the exact same thing.

gently caress snap though.

who is "they"? package managers tend to be the same or very similar within distro families, tho the gui may vary wildly. apt is apt, yum is yum, pacman is pacman

one reason I always do my updates from the cli

# ? Dec 18, 2020 07:09

Mr Shiny Pants: Nov 12, 2012

RFC2324 posted:

who is "they"? package managers tend to be the same or very similar within distro families, tho the gui may vary wildly. apt is apt, yum is yum, pacman is pacman

one reason I always do my updates from the cli

Well, KDE uses discover (I think) and I think Gnome uses something else.

I just use the CLI, because that always just works.

# ? Dec 18, 2020 07:42

BlankSystemDaemon: Mar 13, 2009

It's most likely a combination of NIH syndrome and insufficient knowledge about other options that're available.

# ? Dec 18, 2020 12:24

Butter Activities: May 4, 2018

What�s the least headache inducing GUI for base Debian? I�ve tried Gnome and Xfce and both has the disappearing desktop/taskbar issue that I�ve seen referenced online and in books and had to rely on using the win key to pull up anything from the GUI.

I know Kali and Kali variants I�ve used use xfce and I never have this problem. Any good fixes? One of my projects was to create an up to date buscador off of base Debian rather than use Ubuntu like Bazzell recommends now for ease of use. I tried a month ago and I ended up giving up just because there were too many small headaches but it did teach me a lot about Linux.

# ? Dec 18, 2020 14:10

Truga: May 4, 2014; Lipstick Apathy

never not KDE

e: this goes for any distro not just debian, really. out of the box, kde acts more or less like a good version of windows would, and pretty much everything can be configured to work the way you want

Truga fucked around with this message at 14:31 on Dec 18, 2020

# ? Dec 18, 2020 14:21

xtal: Jan 9, 2011; by Fluffdaddy

SpaceSDoorGunner posted:

What’s the least headache inducing GUI for base Debian? I’ve tried Gnome and Xfce and both has the disappearing desktop/taskbar issue that I’ve seen referenced online and in books and had to rely on using the win key to pull up anything from the GUI.

I know Kali and Kali variants I’ve used use xfce and I never have this problem. Any good fixes? One of my projects was to create an up to date buscador off of base Debian rather than use Ubuntu like Bazzell recommends now for ease of use. I tried a month ago and I ended up giving up just because there were too many small headaches but it did teach me a lot about Linux.

This is a personal choice, I would always keep using Xfce

# ? Dec 18, 2020 15:42

RFC2324: Jun 7, 2012; http 418

Truga posted:

never not KDE

e: this goes for any distro not just debian, really. out of the box, kde acts more or less like a good version of windows would, and pretty much everything can be configured to work the way you want

I'm not a debian person, but KDE is always thr answer, unless the question is "which de do I install on this potato?"

# ? Dec 18, 2020 16:35

xzzy: Mar 5, 2009

I always end up having to install both because users are very fickle.

(mostly on vnc servers, which is about the only case where you have multiple people running desktops on one machine)

# ? Dec 18, 2020 16:36

Yaoi Gagarin: Feb 20, 2014

I never, ever, ever use the gui package managers anymore. In my experience they always gently caress up eventually.

# ? Dec 18, 2020 17:54

RFC2324: Jun 7, 2012; http 418

VostokProgram posted:

I never, ever, ever use the gui package managers anymore. In my experience they always gently caress up eventually.

I stopped using them years ago because of a habit of loving up the package database, making the whole system fucky. The worst case left me only able to install tarballs the old fashioned way because the rpm database had so completely poo poo itself

# ? Dec 18, 2020 18:03

xzzy: Mar 5, 2009

VostokProgram posted:

I never, ever, ever use the gui package managers anymore. In my experience they always gently caress up eventually.

So do the command line ones. On our compute servers that are running full tilt 24/7 yum corrupts the rpmdb on steady percentage of nodes every week. It's single digit percentages, but still enough to be annoying.

# ? Dec 18, 2020 18:04

RFC2324: Jun 7, 2012; http 418

xzzy posted:

So do the command line ones. On our compute servers that are running full tilt 24/7 yum corrupts the rpmdb on steady percentage of nodes every week. It's single digit percentages, but still enough to be annoying.

It never corrupts the rpmdb badly enough to be unrecoverable tho, right?

My experience with cli utils is that when they do gently caress up you can use the backend to them(rpm or whatever the equivalent is on deb systems) to fix the problem. Mildly annoying vs requiring that you flatten and reinstall.

Everything fucks up eventually on linux

# ? Dec 18, 2020 18:09

Adbot: ADBOT LOVES YOU

# ? May 29, 2024 15:33

xzzy: Mar 5, 2009

Yeah, rpm --rebuilddb fixes it.

We never should have invented computers.

# ? Dec 18, 2020 18:22

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Linux Questions Thread: a bunch of pitfalls, but technically it's possible

«‹›1036 »