The Linux Questions Thread: a bunch of pitfalls, but technically it's possible

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Linux Questions Thread: a bunch of pitfalls, but technically it's possible

«‹›1040 »

RFC2324: Jun 7, 2012; http 418

Does the string work if they aren't variable?

Like if you just type the full, expanded command, does it perform as expected? Because of so the problem is your variables, not the & in sed.

# ? Dec 1, 2021 17:51

Adbot: ADBOT LOVES YOU

# ? Jun 6, 2024 22:33

Combat Pretzel: Jun 23, 2004; No, seriously... what kurds?!

What are the things I need to look at to drop the power draw of my NAS? It's an old Haswell Xeon E3-1230v3 with four 5400rpm hard drives and it's currently using 55W. According to random stuff I googled, it ought to run at lower than that. Are there any guides on how to tune for lower power usage?

--edit:
According to turbostat, the CPU is like 99.8% in C7s state. I guess I have to tear this thing apart to see what's going on.

--edit:
That RAPL stuff from Intel says the CPU only uses 4W. This is gonna be fun figuring out.

Combat Pretzel fucked around with this message at 18:09 on Dec 1, 2021

# ? Dec 1, 2021 17:53

JLaw: Feb 10, 2008; - harmless -

LochNessMonster posted:

They're actual variables. It reads from a file (template) and replaces a placeholder value before writing back to file.
Shell session code:
orginal_value="PLACEHOLDER"
new_value="stringwith&init"
file="file.txt"

sed -i "s|${original_value}|${new_value}|g" "${file}"
which results in stringwithPLACEHOLDERinit

(BTW you have a typo "orginal" in your example there.)

FWIW a single backslash works for me:

Shell session code:

joel@lazarus:~/Downloads$ cat file.txt 
PLACEHOLDER
joel@lazarus:~/Downloads$ original_value="PLACEHOLDER"
joel@lazarus:~/Downloads$ new_value="stringwith\&init"
joel@lazarus:~/Downloads$ sed -i "s|${original_value}|${new_value}|g" "${file}"
joel@lazarus:~/Downloads$ cat file.txt 
stringwith&init

e: If the original/replacement strings might have unknown values, you'll probably want to use some non-sed solution that only does simple literal string replacement. Or you could get a bit fancier and try to auto-backslash those values as necessary e.g. the discussion at https://stackoverflow.com/questions/407523/escape-a-string-for-a-sed-replace-pattern

JLaw fucked around with this message at 18:08 on Dec 1, 2021

# ? Dec 1, 2021 18:03

LochNessMonster: Feb 3, 2005; I need about three fitty

RFC2324 posted:

Does the string work if they aren't variable?

Like if you just type the full, expanded command, does it perform as expected? Because of so the problem is your variables, not the & in sed.

The behaviour is exactly the same if I use the values instead of the variables. The behaviour comes from the & being a special character in sed.

JLaw posted:

(BTW you have a typo "orginal" in your example there.)

FWIW a single backslash works for me:
Shell session code:
joel@lazarus:~/Downloads$ cat file.txt 
PLACEHOLDER
joel@lazarus:~/Downloads$ original_value="PLACEHOLDER"
joel@lazarus:~/Downloads$ new_value="stringwith\&init"
joel@lazarus:~/Downloads$ sed -i "s|${original_value}|${new_value}|g" "${file}"
joel@lazarus:~/Downloads$ cat file.txt 
stringwith&init
e: If the original/replacement strings might have unknown values, you'll probably want to use some non-sed solution that only does simple literal string replacement. Or you could get a bit fancier and try to auto-backslash those values as necessary e.g. the discussion at https://stackoverflow.com/questions/407523/escape-a-string-for-a-sed-replace-pattern

ah that typo is from phoneposting, the original code does not have it.

I've tried to come up with escaping the & with something like fixed_value="$(echo $new_value | sed -e 's/[&]/\\&/g')" but that didn't seem to work either. Tried with both single and double backslashes as escape character.

Might have to go with a different solution instead of sed I guess.

# ? Dec 2, 2021 11:09

MrPablo: Mar 21, 2003

LochNessMonster posted:

The behaviour is exactly the same if I use the values instead of the variables. The behaviour comes from the & being a special character in sed.

ah that typo is from phoneposting, the original code does not have it.

I've tried to come up with escaping the & with something like fixed_value="$(echo $new_value | sed -e 's/[&]/\\&/g')" but that didn't seem to work either. Tried with both single and double backslashes as escape character.

Might have to go with a different solution instead of sed I guess.

My suggestion would be to go with a general purpose scripting language (Perl, Ruby, Python, etc) that doesn't have complicated and counterintuitive string interpolation rules, because you're going to be fighting with both bash and sed to handle all the special cases, and you can still accomplish the same thing with a Perl or Ruby one-liner:

code:

> echo -e 'hello\nthere' | ruby -lpe '$_.gsub!(/hello/, "foo&bar")'
foo&bar
there
> echo -e 'hello\nthere' | perl -lpe 's/hello/foo&bar/'
foo&bar
there

You can also provide a file as the first argument and these commands will work as expected with the "-i" flag:

code:

> cat > foo.txt
hello
there
> perl -i -lpe 's/hello/foo&bar/' foo.txt 
> cat foo.txt
foo&bar
there
> cat > foo.txt
hello
there
> ruby -i -lpe '$_.gsub!(/hello/, "foo&bar")' foo.txt
> cat foo.txt
foo&bar
there
>

Python is a bit more verbose, but still easier than dealing with bash and sed:

code:

#!/usr/bin/python3

import re
import os
import sys

# compile regex
src_re = re.compile(r'hello')

# read lines from standard input, replace matches, and print result to
# standard output
for line in sys.stdin:
  print(src_re.sub('foo&bar', line).rstrip())

# ? Dec 2, 2021 18:04

RFC2324: Jun 7, 2012; http 418

MrPablo posted:

My suggestion would be to go with a general purpose scripting language (Perl, Ruby, Python, etc) that doesn't have complicated and counterintuitive string interpolation rules, because you're going to be fighting with both bash and sed to handle all the special cases, and you can still accomplish the same thing with a Perl or Ruby one-liner:
code:
> echo -e 'hello\nthere' | ruby -lpe '$_.gsub!(/hello/, "foo&bar")'
foo&bar
there
> echo -e 'hello\nthere' | perl -lpe 's/hello/foo&bar/'
foo&bar
there
You can also provide a file as the first argument and these commands will work as expected with the "-i" flag:
code:
> cat > foo.txt
hello
there
> perl -i -lpe 's/hello/foo&bar/' foo.txt 
> cat foo.txt
foo&bar
there
> cat > foo.txt
hello
there
> ruby -i -lpe '$_.gsub!(/hello/, "foo&bar")' foo.txt
> cat foo.txt
foo&bar
there
>
Python is a bit more verbose, but still easier than dealing with bash and sed:
code:
#!/usr/bin/python3

import re
import os
import sys

# compile regex
src_re = re.compile(r'hello')

# read lines from standard input, replace matches, and print result to
# standard output
for line in sys.stdin:
  print(src_re.sub('foo&bar', line).rstrip())

This is good post.

And yeah, I was aware of the special behavior of & in sed, I'm also aware that bash has weird funky rules that can gently caress with special behaviors, especially in regards to passing them through a variable. Basic troubleshooting sometimes involves thinking basic thoughts.

Do the(ugh) perl route, its gonna be the easiest

# ? Dec 2, 2021 18:11

xzzy: Mar 5, 2009

That's been my approach for years. Shell scripts are my default choice when approaching an issue but the instant I spend more than 30 seconds wondering how to arrange single and double quotes to get the result I want I give up and switch to python. At that point it's just not worth the effort of coaxing shell into doing what I want.

# ? Dec 2, 2021 18:12

Armauk: Jun 23, 2021

xzzy posted:

the instant I spend more than 30 seconds wondering how to arrange single and double quotes to get the result I want I give up and switch to python.

What about Perl?

# ? Dec 2, 2021 18:15

RFC2324: Jun 7, 2012; http 418

xzzy posted:

That's been my approach for years. Shell scripts are my default choice when approaching an issue but the instant I spend more than 30 seconds wondering how to arrange single and double quotes to get the result I want I give up and switch to python. At that point it's just not worth the effort of coaxing shell into doing what I want.

I have trouble wrapping my head around needing to define functions and poo poo in advance, so struggle with the need to do some poo poo in an actual programming language. That said, I'm glad I have friends who do programming and are happy to help me

# ? Dec 2, 2021 18:17

hifi: Jul 25, 2012

Combat Pretzel posted:

What are the things I need to look at to drop the power draw of my NAS? It's an old Haswell Xeon E3-1230v3 with four 5400rpm hard drives and it's currently using 55W. According to random stuff I googled, it ought to run at lower than that. Are there any guides on how to tune for lower power usage?

--edit:
According to turbostat, the CPU is like 99.8% in C7s state. I guess I have to tear this thing apart to see what's going on.

--edit:
That RAPL stuff from Intel says the CPU only uses 4W. This is gonna be fun figuring out.

The answer is probably to set the hard drives to spin down, but 55 watts is low enough that things like case fans matter and you are in the portion of your power supply's output where 1) it's unregulated by 80plus standards and 2) it isn't very efficient as a result

maybe something with the graphics hw too

# ? Dec 2, 2021 18:19

xzzy: Mar 5, 2009

Armauk posted:

What about Perl?

I got rid of my last perl script in 2020. It was 15 years old and every time I had to update it I got mad once more at how readily perl lets you shoot yourself in the foot with bad coding practices.

Rest in agony, perl.

Python sucks too but at least it forces some coherence. But searching for python solutions goes down that same dark road because everyone tries to come up with the goofiest one liner possible or builds a ridiculous test setup to determine which implementation is 0.1 seconds faster.

# ? Dec 2, 2021 18:45

BlankSystemDaemon: Mar 13, 2009

I still have some perl scripts around from doing FreeBSD-specific VM monitoring that date back to 2003 - and they still work.

Serendipitously, I recently found that someone has created an API compatible version of it.

# ? Dec 2, 2021 20:12

MrPablo: Mar 21, 2003

RFC2324 posted:

This is good post.

Thanks!

xzzy posted:

I got rid of my last perl script in 2020. It was 15 years old and every time I had to update it I got mad once more at how readily perl lets you shoot yourself in the foot with bad coding practices.

Rest in agony, perl.

Python sucks too but at least it forces some coherence. But searching for python solutions goes down that same dark road because everyone tries to come up with the goofiest one liner possible or builds a ridiculous test setup to determine which implementation is 0.1 seconds faster.

I maintain a handful of ~16 year old Perl scripts at work. We've been gradually replacing the Perl scripts with Ruby and Python scripts as time permits, but it's hard to come up with a business justification for the replacement work because the existing Perl scripts are pretty solid (other than pissing off our static analyzer).

# ? Dec 3, 2021 04:41

LochNessMonster: Feb 3, 2005; I need about three fitty

MrPablo posted:

My suggestion would be to go with a general purpose scripting language (Perl, Ruby, Python, etc) that doesn't have complicated and counterintuitive string interpolation rules, because you're going to be fighting with both bash and sed to handle all the special cases, and you can still accomplish the same thing with a Perl or Ruby one-liner:
code:
> echo -e 'hello\nthere' | ruby -lpe '$_.gsub!(/hello/, "foo&bar")'
foo&bar
there
> echo -e 'hello\nthere' | perl -lpe 's/hello/foo&bar/'
foo&bar
there
You can also provide a file as the first argument and these commands will work as expected with the "-i" flag:
code:
> cat > foo.txt
hello
there
> perl -i -lpe 's/hello/foo&bar/' foo.txt 
> cat foo.txt
foo&bar
there
> cat > foo.txt
hello
there
> ruby -i -lpe '$_.gsub!(/hello/, "foo&bar")' foo.txt
> cat foo.txt
foo&bar
there
>
Python is a bit more verbose, but still easier than dealing with bash and sed:
code:
#!/usr/bin/python3

import re
import os
import sys

# compile regex
src_re = re.compile(r'hello')

# read lines from standard input, replace matches, and print result to
# standard output
for line in sys.stdin:
  print(src_re.sub('foo&bar', line).rstrip())

Thanks, I�m ok with python but really was trying to get it working with bash. I try to keep my codebase clean and try to avoid mixing languages where possible. This seemed like a scenario that should be doable from shell so I tried getting it to work and use perl/python as a last resort.

The earlier suggestion of scripting escape chars into the replacement string DID work, but I made an error while testing.

xzzy posted:

That's been my approach for years. Shell scripts are my default choice when approaching an issue but the instant I spend more than 30 seconds wondering how to arrange single and double quotes to get the result I want I give up and switch to python. At that point it's just not worth the effort of coaxing shell into doing what I want.

This is a good suggestion and I should probably stop wasting hours making bash do what I want while it�d take me minutes to do the same in python.

# ? Dec 3, 2021 07:41

Bob Morales: Aug 18, 2006; ~~Just wear the fucking mask, Bob~~

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

Bash is easier for me to read and less "what does this command option mean", easier for others to read etc, and hopefully avoids cute one-liners

# ? Dec 3, 2021 12:46

BlankSystemDaemon: Mar 13, 2009

Why are people so afraid of oneliners?

# ? Dec 3, 2021 12:54

Bob Morales: Aug 18, 2006; ~~Just wear the fucking mask, Bob~~

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

BlankSystemDaemon posted:

Why are people so afraid of oneliners?

Because they often result from people seeing how many "tricks" they can stick in one expression

# ? Dec 3, 2021 13:12

Sheep: Jul 24, 2003

I've seen newer people have trouble parsing longer stacks of commands, especially when stuff like awk gets involved. Seems easier to break it up and add comments so the person coming behind you doesn't suffer unnecessarily.

There's that saying that goes "code like your successor is a psychopath who knows where you live" or something like that.

Sheep fucked around with this message at 13:41 on Dec 3, 2021

# ? Dec 3, 2021 13:36

Grey Area: Sep 9, 2000; Battle Without Honor or Humanity

(The successor is probably you three years from now and that psycho definitely knows where you live.)

# ? Dec 3, 2021 15:46

RFC2324: Jun 7, 2012; http 418

Nothing worse than wondering what rear end in a top hat wrote this indecipherable script, and seeing one of you own commands in there

# ? Dec 3, 2021 16:56

rufius: Feb 27, 2011; Clear alcohols are for rich women on diets.

RFC2324 posted:

Nothing worse than wondering what rear end in a top hat wrote this indecipherable script, and seeing one of you own commands in there

Programming: a murder mystery in which you are the detective, victim, and killer.

# ? Dec 3, 2021 18:20

ExcessBLarg!: Sep 1, 2001

Armauk posted:

What about Perl?

Dropped like a bad habit in 2005. Ruby is better for every use case of PERL aside from "needs to work with some weird CPAN thing" which, surprisingly, hasn't come up recently.

# ? Dec 3, 2021 19:27

BlankSystemDaemon: Mar 13, 2009

Bob Morales posted:

Because they often result from people seeing how many "tricks" they can stick in one expression

I'm not very familiar with this sysadmin stunting you speak of, so I'll have to take your word for it.

I think maybe the people who do sysadmin stunting are an advanced form of kind of people who spend way too much time continually messing with their UX to look a certain way with special attention paid to spacing between windows, window decorations, wallpaper choice, compositing, and all those other things.
As opposed to, you know, using tools to do something specific like a task or a job.

Sheep posted:

I've seen newer people have trouble parsing longer stacks of commands, especially when stuff like awk gets involved. Seems easier to break it up and add comments so the person coming behind you doesn't suffer unnecessarily.

There's that saying that goes "code like your successor is a psychopath who knows where you live" or something like that.

RFC2324 posted:

Nothing worse than wondering what rear end in a top hat wrote this indecipherable script, and seeing one of you own commands in there

What you both appear to be indirectly saying is that "inline documentation is good".

I wouldn't know what 95% of my scripts did if I hadn't documented them when writing them, because even if I hadn't had chemobrain, as soon as something is paged out of my memory, it's unlikely that I can just look at it again and know what it means in an instant.

# ? Dec 3, 2021 23:50

RFC2324: Jun 7, 2012; http 418

BlankSystemDaemon posted:

What you both appear to be indirectly saying is that "inline documentation is good".

I wouldn't know what 95% of my scripts did if I hadn't documented them when writing them, because even if I hadn't had chemobrain, as soon as something is paged out of my memory, it's unlikely that I can just look at it again and know what it means in an instant.

yeah, it really is. I comment the gently caress out of everything I write, if only so I know I am the one who committed the atrocity when I find it again later

# ? Dec 4, 2021 00:11

Computer viking: May 30, 2011; Now with less breakage.

Oneliners also tend to use absolutely minimal variable names - but reasonable variable names are arguably a form of inline documentation, so I guess that's already covered.

# ? Dec 4, 2021 00:32

MrPablo: Mar 21, 2003

LochNessMonster posted:

This is a good suggestion and I should probably stop wasting hours making bash do what I want while it�d take me minutes to do the same in python.

For what it's worth, I do this too: I'll start with a bash script (with the obligatory "set -euo pipefile" to maintain my sanity), but I switch to a scripting language the instant I start fighting with string interpolation, need logging, or need functions, or have to add anything more than trivial logic.

In addition to being simpler to work with, scripting languages can actually execute faster than a shell script in some circumstances. For example, a few years ago I replaced a shell script that did small operations (sed, mv, rm, etc) on a large number of files with a Ruby script. The Ruby script was noticeably faster because I was able to replace mv with File.rename and rm with File.unlink, so the Ruby version didn't need to fork/exec nearly as often as the shell script.

That said, a slow shell script that works and is consistent with the rest of your system will always be better than a script written in a random language that either doesn't work or is inconsistent with the rest of your system. I inherited a few csh scripts from someone else and I'm still bitter about it 15 years later

# ? Dec 4, 2021 01:16

RFC2324: Jun 7, 2012; http 418

for me they were ksh and perl scripts

I hate perl so much

# ? Dec 4, 2021 01:25

Computer viking: May 30, 2011; Now with less breakage.

Windows program that generated php code for a MySQL frontend thing. Scripts would have been nicer, even in weird languages.

# ? Dec 4, 2021 01:38

BlankSystemDaemon: Mar 13, 2009

Computer viking posted:

Oneliners also tend to use absolutely minimal variable names - but reasonable variable names are arguably a form of inline documentation, so I guess that's already covered.

Variable names as a form of documentation can't stand on its own, though.

I feel as if I've done enough documentation when a script is two thirds documentation and one-third logic.

# ? Dec 4, 2021 03:31

Bob Morales: Aug 18, 2006; ~~Just wear the fucking mask, Bob~~

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

We were talking about bugs in cron job that was a bash script, it did a lot of string manipulations and stuff that would have been more readable in python so I suggested we just re-write it. I had found this to be the cause of some issues and found other things the script wasn't doing quite right but we hadn't ran into issues with production, yet.

My boss replied with "I'm not a bash expert..."

Okay that right there shows you shouldn't be doing fancy poo poo with it

# ? Dec 4, 2021 04:06

RFC2324: Jun 7, 2012; http 418

Bob Morales posted:

We were talking about bugs in cron job that was a bash script, it did a lot of string manipulations and stuff that would have been more readable in python so I suggested we just re-write it. I had found this to be the cause of some issues and found other things the script wasn't doing quite right but we hadn't ran into issues with production, yet.

My boss replied with "I'm not a bash expert..."

Okay that right there shows you shouldn't be doing fancy poo poo with it

people tell me I'm a bash expert, and I can tell you that I shouldn't be doing fancy poo poo with it

# ? Dec 4, 2021 04:22

DerekSmartymans: Feb 14, 2005; The
Copacetic
Ascetic

RFC2324 posted:

people tell me I'm a bash expert, and I can tell you that I shouldn't be doing fancy poo poo with it

Ah, but is this because you are a bash expert and know this, or does not doing the fancy poo poo make you the bash expert, hmmm?

Linux

# ? Dec 4, 2021 04:33

Perplx: Jun 26, 2004; Best viewed on Orgasma Plasma; Lipstick Apathy

The only good reason to make something complex in bash is if you need compatibility with every *nix made in the last 30 years.

# ? Dec 4, 2021 05:01

Voodoo Cafe: Jul 19, 2004; "You got, uhh, Holden Caulfield in there, man?"

ExcessBLarg! posted:

Ruby is better for every use case of PERL

almost unanimously true, though it is less likely to be part of a base configuration for most distros in my experience while Perl usually tends to be present in some capacity, though widely displaced by Python these days. It's a shame; Ruby is a language that i find both fun to write and without so many of Perl's more 'esoteric' choices

# ? Dec 4, 2021 05:41

RFC2324: Jun 7, 2012; http 418

Perplx posted:

The only good reason to make something complex in bash is if you need compatibility with every *nix made in the last 30 years.

half my bash scripts won't work on the 10 year old bash on osx because it lacks things like longopts, so this isn't even true.

# ? Dec 4, 2021 06:29

Saukkis: May 16, 2003; Unless I'm on the inside curve pointing straight at oncoming traffic the high beams stay on and I laugh at your puny protest flashes.
I am Most Important Man. Most Important Man in the World.

For the Perl vs Python vs Ruby question, do we have any assurances that we don't have to go through the migration mess with Python 4? Have we had to go through the same with Perl or Ruby? Probably not with Perl, since we've had Perl 5 for close to three decades.

# ? Dec 4, 2021 09:22

Voodoo Cafe: Jul 19, 2004; "You got, uhh, Holden Caulfield in there, man?"

Saukkis posted:

For the Perl vs Python vs Ruby question, do we have any assurances that we don't have to go through the migration mess with Python 4? Have we had to go through the same with Perl or Ruby? Probably not with Perl, since we've had Perl 5 for close to three decades.

I doubt Python 3->4 will be as much of an upheaval as 2->3 has been (hopefully).

Perl 6 was planned for nearly 20 years, ultimately it became own beast entirely called Raku and Perl 7 will be a more iterative upgrade to 5.32

# ? Dec 4, 2021 15:00

rufius: Feb 27, 2011; Clear alcohols are for rich women on diets.

Saukkis posted:

For the Perl vs Python vs Ruby question, do we have any assurances that we don't have to go through the migration mess with Python 4? Have we had to go through the same with Perl or Ruby? Probably not with Perl, since we've had Perl 5 for close to three decades.

Hot take: �This thing will run forever without crappy migration� is a bad bet to make. All code/collateral/hardware is debt and has to be serviced.

Some of that might be cheaper to service. When I run my dev teams, I build in time in our schedule to do debt work every 8-12 weeks. It�s just part of the job.

For my personal stuff, I just update periodically for the stuff I most often use.

# ? Dec 4, 2021 16:19

ExcessBLarg!: Sep 1, 2001

Saukkis posted:

Have we had to go through the same with Perl or Ruby?

There was a pretty big migration from Ruby 1.8 to 1.9 over a decade ago, with the 1.8 series getting long-term support for a number of years after the initial release of 1.9.

It wasn't quite the same as Python 2 to 3 as there wasn't as many deprecation of language features. The main change was swapping out the interpreter with a VM, which had some impacts on the threading model, and a small number of incompatible syntax changes that were much necessary to resolve ambiguities in the parser.

Most Ruby 1.8 scripts "just worked" on 1.9. I had to change a few colons to semicolons in case/where clauses (known migration issue) and that was it.

# ? Dec 4, 2021 16:52

Adbot: ADBOT LOVES YOU

# ? Jun 6, 2024 22:33

Computer viking: May 30, 2011; Now with less breakage.

Having just spent some time reading about it, the Raku regex syntax looks rather nice - Larry Wall was apparently a bit unhappy about how write-only the perl5 regex style had become. The big downside is that much of the new simplicity and power comes from more deeply integrating with the surrounding perl6/raku language, so I doubt it'll show up in any other language anytime soon.

# ? Dec 4, 2021 17:57

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Linux Questions Thread: a bunch of pitfalls, but technically it's possible

«‹›1040 »