Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Saukkis
May 16, 2003

Unless I'm on the inside curve pointing straight at oncoming traffic the high beams stay on and I laugh at your puny protest flashes.
I am Most Important Man. Most Important Man in the World.

BrainDance posted:

It's a whole event and I wouldn't be surprised if that process broke something, or if the data just had those weird character codes. Is there anything I could do with sed or something that would just strip it of any weird character codes? I'm not sure how to word that, but you know, stuff that isn't just... normal?

I once had to use find to search any files with unusual characters, I suspect grep or sed would support similar character set searches.

code:
find . -maxdepth 1 -regextype posix-egrep -not -regex '[[:alnum:][:punct:][:space:]]*'

Adbot
ADBOT LOVES YOU

BrainDance
May 8, 2007

Disco all night long!

Saukkis posted:

I once had to use find to search any files with unusual characters, I suspect grep or sed would support similar character set searches.

code:
find . -maxdepth 1 -regextype posix-egrep -not -regex '[[:alnum:][:punct:][:space:]]*'

Thanks, this got me started on the right path, managed to do it with sed
code:
sed 's/[^[:print:][:space:][:punct:]]//g' file.txt > output.txt
And then it still didn't work, lol. But by that point I was like "ok, the text is clean, this makes no sense"
So, then I just took the working Shakespeare example and appended the daodejing to the bottom of it and tried that, and it started training.
Then I deleted all the Shakespeare stuff and left the daodejing... and it trained. Which left only the validation file.

I was using one example for each for the validation file. So one verse of the daodejing or one message from the chat logs. The Shakespeare validation file is longer because Shakespeare just is longer than a verse from the very short daodejing. And, in the end, basically it turns out there is a minimum length for the validation file. This is not documented anywhere, because the documentation for this is all kinda garbage, but yeah that seems to be what it was.

Computer viking
May 30, 2011
Now with less breakage.

BrainDance posted:

That actually seems like it might be something. The other text, it's ripped from chat logs. The chat logs are from a Chinese chat program though that sometimes formats things weird (like weird apostrophes.) I've built a whole process for cleaning them up, like, for the chat logs
awk '/username/ {f=NR} f && NR==f+1' file > file.txt to extract my messages
grep -vE "^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}" file.txt > output.txt to try to filter out any loose screennames (they have timestamps, sometimes the previous awk would grab someone mentioning me and then copy the next line which would be a username)
grep -P -v '[\x{4e00}-\x{9fa5}]' file.txt > output.txt to remove Chinese characters (the Chinese chat program, it saves stickers posted as a little Chinese text flag thing)
sed 's/http[^ ]*//g' file.txt > output.txt to strip it of urls
grep -v '^[[:space:]]*$' file.txt > output.txt to empty lines
sed -i 's/^[ ]*//g' file.txt to get rid of some leading white space
awk '{print "responseuser1: " $0;}' file.txt > output.txt to add the trigger word for the AI
sed -e 's/$/<|endoftext|>/' -i file.txt to add the terminator (though, this is gonna need to be reworked)

It's a whole event and I wouldn't be surprised if that process broke something, or if the data just had those weird character codes. Is there anything I could do with sed or something that would just strip it of any weird character codes? I'm not sure how to word that, but you know, stuff that isn't just... normal?

For later, you could also try piping it through iconv and converting it to something minimal - ASCII//IGNORE should keep the US-ASCII characters and just drop the rest, or you can try //TRANSLIT and see how that interprets things. (I have no idea what translit would do with chinese characters.)

ziasquinn
Jan 1, 2006

Fallen Rib
I'm having a super annoying issue I've not been able to fix consistently since it started happening.

I have an Elgato XLR interface with audio output and (obviously) an XLR-microphone input. However, the microphone works intermittently. It won't even act like it's receiving sometimes. I usually don't notice it isn't working until I'm under time pressure to talk to someone on Discord -- the microphone won't work. It shows up in Alsamixer and is unmuted, and the only "consistent" way I can resolve the issue is by opening pavucontrol and switching the XLR Usb Audio Configuration from Digital Out + Mono Input to Analog Out + Mono Input back and forth like 10x-20x, then sometimes, sometimes, the microphone will spontaneously begin to function normally.

I've tried so many things. I've tried adding snd-usb-audio to alsa-config, One time opening pulse meter just fixed it, like it woke it up from some sleep state? One time, unlocking and locking the input channels fixed it. One time, I couldn't fix it without restarting my machine three times.

It works fine in Windows and MacOS.

I'm so desperate I started asking ChatGPT what other poo poo it could think of to try and resolve it. I just disabled usb power management, I think? I don't even know man.

I will literally try anything at this point.

(I really don't want to replace it because the microphone really wants 60db lift and the interface provides it and it sounds like anything else I'd get I'd have to pair it with a cloudlifter pre-amp or something).

I guess I could just tear it all out and sell it or something (the microphone and interface),,

Oh it's happening in Ubuntu and Arch but seems to be mostly present under Gnome de, and it doesn't care if I'm using pulseaudio or pipewire (I assume because it's using pipewire-pulse) (just using ALSA doesn't fix it either). I haven't tried running a JACK only environment, and I don't... really want to especially for literally no reason.

F_Shit_Fitzgerald
Feb 2, 2017



Thanks for all the previous help.

Why in hell is it that if, say, I want to delete all the pdf files in a directory, I can simply invoke

code:
if [ -f *.pdf ];
then
  rm *.pdf
fi
...but if I want to do the same thing with mp4 files, I can't simply do that but something more along the lines of

code:
for i in *.mp4 do
  rm *.mp4
done
Is there an easier way to automatically detect any mp4 files in a directory and delete them (or do nothing if there aren't any)?

Pablo Bluth
Sep 7, 2007

I've made a huge mistake.
find . -type f -iname "*.mp4" -delete

Leave off the delete flag flag first time if you want to check what will be remove. Add a -maxdepth 0 if you don't want to search subfolders. The find command is massively powerful and all linux admins should master its ways.

Keito
Jul 21, 2005

WHAT DO I CHOOSE ?

F_Shit_Fitzgerald posted:

Thanks for all the previous help.

Why in hell is it that if, say, I want to delete all the pdf files in a directory, I can simply invoke

code:
if [ -f *.pdf ];
then
  rm *.pdf
fi
...but if I want to do the same thing with mp4 files, I can't simply do that but something more along the lines of

code:
for i in *.mp4 do
  rm *.mp4
done
Is there an easier way to automatically detect any mp4 files in a directory and delete them (or do nothing if there aren't any)?

Neither of those snippets are working the way you are expecting them to, actually. In the first example you are doing shell expansion for all files "*.pdf" in the directory, and then running "rm *.pdf" to delete everything if the test returns true ("[ -f FILE]" is to check if a file exists and is a regular file).

I'm not sure why your second example wouldn't have the same result to be honest, but why even set up a for loop for this when you are just running "rm *.mp4"? In your example you're trying to delete everything matching "*.mp4" for each element you iterate over in the loop of files matching the pattern "*.mp4".

When setting up a loop you should use the variable you have assigned to operate on the current element:

code:
for f in *.mp4; do
  rm -v "$f"
done

VictualSquid
Feb 29, 2012

Gently enveloping the target with indiscriminate love.

F_Shit_Fitzgerald posted:

Thanks for all the previous help.

Why in hell is it that if, say, I want to delete all the pdf files in a directory, I can simply invoke

code:
if [ -f *.pdf ];
then
  rm *.pdf
fi
...but if I want to do the same thing with mp4 files, I can't simply do that but something more along the lines of

code:
for i in *.mp4 do
  rm *.mp4
done
Is there an easier way to automatically detect any mp4 files in a directory and delete them (or do nothing if there aren't any)?

What do you want that isn't done by "rm -f *.mp4" ?

F_Shit_Fitzgerald
Feb 2, 2017



rm -f *.mp4 is perfect! I didn't realize it could be that simple; I thought you had to set up a conditional for each file type. Thanks! Still a lot I have to learn about Bash.

F_Shit_Fitzgerald fucked around with this message at 20:56 on Feb 26, 2023

Pablo Bluth
Sep 7, 2007

I've made a huge mistake.
I'd leave off the -f flag. Most of the time it's not required, such as here, and it increases the risk of pain should an rm command ever be typed wrong. Generally speaking I only ever use it to delete a non-empty directory.

F_Shit_Fitzgerald posted:

rm -f *.mp4 is perfect! I didn't realize it could be that simple; I thought you had to set up a conditional for each file type. Thanks! Still a lot I have to learn about Bash.
The only reason to use a conditional is to avoid a "No such file or directory" error. If you're writing a long-lasting script that will be used repeatedly, it's probably a good idea to clean up the output but for quick scripts or directly typed commands it's unnecessary.

Pablo Bluth fucked around with this message at 21:03 on Feb 26, 2023

Mr. Crow
May 22, 2008

Snap City mayor for life
To save yourself looking into the abyss use a real programming language for scripts if they get longer than several lines of bash. Its great for quick one off stuff, not so much for medium-large scripts.

F_Shit_Fitzgerald
Feb 2, 2017



If that's the case, I might keep it, actually. I do volunteer work that uses a variety of different file types - docx, pdf, mp4 - and this script is intended to only clean out old files of the specific type while preserving other files (e.g: text files, subdirectories, etc).

Mr. Crow
May 22, 2008

Snap City mayor for life

F_Shit_Fitzgerald posted:

If that's the case, I might keep it, actually. I do volunteer work that uses a variety of different file types - docx, pdf, mp4 - and this script is intended to only clean out old files of the specific type while preserving other files (e.g: text files, subdirectories, etc).

Delete files 7 days old,
https://unix.stackexchange.com/a/447363


code:

find /var/log/butts -mtime +7 -type f -name '*.pdf' -delete


Throw in in a systemd unit timer to really piss off grey beards

Nitrousoxide
May 30, 2011

do not buy a oneplus phone



This is exactly the kind of thing chatGPT will spit out for you by the way if you ask it.

quote:

"write a bash script to remove all files in a directory older than 7 days"

Certainly, here's a bash script that will remove all files in a directory older than 7 days:

bash
Copy code
code:
#!/bin/bash

# Navigate to the directory where the files are located
cd /path/to/directory

# Remove all files older than 7 days
find . -type f -mtime +7 -exec rm {} \;
Just replace /path/to/directory with the actual path to the directory where the files are located. This script uses the find command to locate all files in the directory that are older than 7 days (-mtime +7) and then removes them using the rm command. The -exec option tells find to execute the rm command for each file that matches the search criteria, and the {} is a placeholder for the name of the file.

You can save this script with any name (for example, remove_old_files.sh) and then execute it in a terminal by navigating to the directory where the script is saved and typing ./remove_old_files.sh.

You can even ask it to tweak it.

quote:

"change the script so it asks the user what type of file to be removed"

Certainly! Here's a modified version of the script that will ask the user what type of file to remove:

bash
Copy code
code:
#!/bin/bash

# Ask the user what type of file to remove
read -p "Enter the file extension (e.g. txt, pdf): " file_ext

# Navigate to the directory where the files are located
cd /path/to/directory

# Remove all files of the specified type older than 7 days
find . -type f -name "*.$file_ext" -mtime +7 -exec rm {} \;
This version of the script uses the read command to prompt the user to enter the file extension of the files they want to remove. The find command has been modified to include the -name option, which specifies that only files with the specified extension should be removed.

Just replace /path/to/directory with the actual path to the directory where the files are located. You can save this script with any name (for example, remove_old_files_by_extension.sh) and then execute it in a terminal by navigating to the directory where the script is saved and typing ./remove_old_files_by_extension.sh.

hazzlebarth
May 13, 2013

Pablo Bluth posted:

I'd leave off the -f flag. Most of the time it's not required, such as here, and it increases the risk of pain should an rm command ever be typed wrong. Generally speaking I only ever use it to delete a non-empty directory.

rm without "-f" will throw an error if there is no matching file. And to delete a directory you would use "-r", just "-f" doesn't work.

Pablo Bluth
Sep 7, 2007

I've made a huge mistake.

hazzlebarth posted:

rm without "-f" will throw an error if there is no matching file. And to delete a directory you would use "-r", just "-f" doesn't work.
Quite. I didn't mean that I use the f flag exclusively when deleting a directory, but in combination with r. If I know I want to blitz a directory I use rf by habit to not have to fight with any prompts/errors (write protected, etc). But it's a dangerous flag as there's always the chance you'll do a rm -rf /. That said, I wrote all of that as I thought I'd seen a "rm -rf" in a post I now can't. So it's unnecessary rambling.

Nitrousoxide
May 30, 2011

do not buy a oneplus phone



hazzlebarth posted:

rm without "-f" will throw an error if there is no matching file. And to delete a directory you would use "-r", just "-f" doesn't work.

Isn't it better to wrap it up with an
code:
if [ "$(ls -A /path/to/directory/*.pdf)" ]; then

find /path/to/directory -mtime +7 -type f -name '*.pdf' -delete
fi
This *should* only run if it finds pdfs in the directory and would only delete the ones older than 7 days. It shouldn't error out if there's no appropriate files in the directory and also won't potentially go crazy deleting stuff with the force flag.

Tesseraction
Apr 5, 2009

You can't accidentally rm -rf /

It specifically detects that and forces you to add a flag saying "I meant to do that"

Nitrousoxide
May 30, 2011

do not buy a oneplus phone



Tesseraction posted:

You can't accidentally rm -rf /

It specifically detects that and forces you to add a flag saying "I meant to do that"



I think
code:
rm -rf /*
ignores that failsafe.

Tesseraction
Apr 5, 2009

You're correct, although I'd argue that's a weirder error to make.

Also I need to reinstall my Debian WSL.

Pablo Bluth
Sep 7, 2007

I've made a huge mistake.
rm -rf ${path variable_i_forgot_to_set}/* would thus be a command of doom... (or you type a path with a space and forget to quote/escape it)

Pablo Bluth fucked around with this message at 20:12 on Feb 27, 2023

BlankSystemDaemon
Mar 13, 2009



Nitrousoxide posted:

I think
code:
rm -rf /*
ignores that failsafe.
Well, that depends on what shell you use; tcsh has a rmstar variable that'll ask you to confirm if you're using global wildcards like that.

Volguus
Mar 3, 2009
rm -rf /* is at most a nuisance. Nothing that can't be fixed with a reinstall. rm -rf somedir * while in <home>, now that's scary poo poo. Because in <home> the important files are located. Of course, <home> also is most likely to be backed up, so if you did your homework, that too can be just a minor nuisance.

Yaoi Gagarin
Feb 20, 2014

Volguus posted:

rm -rf /* is at most a nuisance. Nothing that can't be fixed with a reinstall. rm -rf somedir * while in <home>, now that's scary poo poo. Because in <home> the important files are located. Of course, <home> also is most likely to be backed up, so if you did your homework, that too can be just a minor nuisance.

Well the expansion of /* will include /home so it'll eventually delete the same stuff as doing rm -rf * in your home directory would. It's just a question of how long it runs before you stop it

E: on another note, will `cp /dev/sda /dev/sdb` clone a drive?

BlankSystemDaemon
Mar 13, 2009



/home/ is a symlink.

/usr/home/ is the correct directory. :colbert:

Volguus
Mar 3, 2009

VostokProgram posted:

Well the expansion of /* will include /home so it'll eventually delete the same stuff as doing rm -rf * in your home directory would. It's just a question of how long it runs before you stop it

E: on another note, will `cp /dev/sda /dev/sdb` clone a drive?

Yes, the key word in here is "eventually". Depending on how fast your fingers are and how fast the drive is ... you may get lucky and <home> may escape untouched. When you're already in <home>, using a nvme drive, after you had a few beers, the odds are very much not in your favour. Don't ask how I know.

Pablo Bluth
Sep 7, 2007

I've made a huge mistake.
I once did the opposite and filled a directory with so many zero size files (a horribly broken script written in a rush and left running on a server overnight at work. I recall it got somewhere past the 100 million mark), attempting to do anything bought the system to a halt, despite being a high performance Panasas system. Find's delete was the only thing that would touch it without the process immediately freezing. Find has been my goto tool ever since....

Tesseraction
Apr 5, 2009

Pablo Bluth posted:

I once did the opposite and filled a directory with so many zero size files (a horribly broken script written in a rush and left running on a server overnight at work. I recall it got somewhere past the 100 million mark), attempting to do anything bought the system to a halt, despite being a high performance Panasas system. Find's delete was the only thing that would touch it without the process immediately freezing. Find has been my goto tool ever since....

It's almost funny how quickly Panasas shits the bed if a directory has "too many" files. All these specific optimisations for throughput and 65536 is enough to bring stuff to a crawl.

Saukkis
May 16, 2003

Unless I'm on the inside curve pointing straight at oncoming traffic the high beams stay on and I laugh at your puny protest flashes.
I am Most Important Man. Most Important Man in the World.

Pablo Bluth posted:

rm -rf ${path variable_i_forgot_to_set}/* would thus be a command of doom... (or you type a path with a space and forget to quote/escape it)

The Moodle service at my university got destroyed, because the admin accidentally right-clicked a PuTTY window and pasted a bash script to a rooted terminal with 'rm -rf $variable/' in it.

Nitrousoxide
May 30, 2011

do not buy a oneplus phone



VostokProgram posted:

Well the expansion of /* will include /home so it'll eventually delete the same stuff as doing rm -rf * in your home directory would. It's just a question of how long it runs before you stop it

E: on another note, will `cp /dev/sda /dev/sdb` clone a drive?

I think the real question is whether it would nuke the binary for rm or some other mission critical system dependency before it gets to /home

Voodoo Cafe
Jul 19, 2004
"You got, uhh, Holden Caulfield in there, man?"

Nitrousoxide posted:

I think the real question is whether it would nuke the binary for rm or some other mission critical system dependency before it gets to /home

as long as the rm command is still running, the binary still referenced in /proc/$PID/exe and so still resident in memory and will still happily run until completion, as far as i know

xzzy
Mar 5, 2009

The first sign of problems is usually "ls not found? what the gently caress, how??"

hazzlebarth
May 13, 2013

Saukkis posted:

The Moodle service at my university got destroyed, because the admin accidentally right-clicked a PuTTY window and pasted a bash script to a rooted terminal with 'rm -rf $variable/' in it.

Ugh, that is a stupid setting in PuTTY then. Other ssh clients at least have a confirmation window when pasting multi-line text.

AlexDeGruven
Jun 29, 2007

Watch me pull my dongle out of this tiny box


I stopped using putty once Windows Terminal + WSL2 became solidly mature. It just works so well, and WinTerm does the multi-line paste warning.

ExcessBLarg!
Sep 1, 2001

VostokProgram posted:

E: on another note, will `cp /dev/sda /dev/sdb` clone a drive?
No, that will overwrite the device node for /dev/sdb with a copy of /dev/sda. Or error because /dev/sdb already exists.

Depending on the kind of drive you probably want some flavor of:
code:
dd if=/dev/sda of=/dev/sdb bs=1M
If /dev/sdb is some kind of flash/SSD media and you've just ran TRIM on it (hdparm --security-erase or blkdiscard even) you might prefer:
code:
dd if=/dev/sda of=/dev/sdb conv=sparse bs=256k
since that won't write erase-block-sized zero blocks which helps out the wear leveling. For mechanical drives go with the first one though.

Computer viking
May 30, 2011
Now with less breakage.

AlexDeGruven posted:

I stopped using putty once Windows Terminal + WSL2 became solidly mature. It just works so well, and WinTerm does the multi-line paste warning.

It still doesn't handle something about deleting or overwriting right when connected to FreeBSD, so I keep putty around just to save me the frustration of trying to edit a line from history while the rendering on screen is a couple of characters off.

AlexDeGruven
Jun 29, 2007

Watch me pull my dongle out of this tiny box


Computer viking posted:

It still doesn't handle something about deleting or overwriting right when connected to FreeBSD, so I keep putty around just to save me the frustration of trying to edit a line from history while the rendering on screen is a couple of characters off.

That's fair. I only ever connect to Linux, so it's worked perfectly for me for a while.

cruft
Oct 25, 2007

This is a question about ffmpeg. I don't know where else in the forums to ask it.

I've got this HEVC mkv file and I'm trying to convert it to H.264 using the h264_v4l2m2m codec. The following works fine:

code:
ffmpeg -i in.mkv -map 0:v -vf "scale=-2:'min(200,ih)',format=yuv420p" -b:v 3000k -c:v h264_v4l2m2m interim.mp4
ffmpeg -i in.mkv -c:a copy interim.ac3
ffmpeg -i in.mkv -c:s copy interim.srt
ffmpeg -i interim.mp4 -i interim.ac3 -i interim.srt -map 0:v -c:v copy -map 1:a -c:a copy -map 2:s -c:s copy out.mkv
This, however, fails:

code:
ffmpeg -i in.mkv -map 0:v -vf "scale=-2:'min(200,ih)',format=yuv420p" -b:v 3000k -c:v h264_v4l2m2m -map 0:a -c:a copy -map 0:s -c:s copy -num_capture_buffers 16 -num_output_buffers 32 out.mkv
Could not write header for output file #0 (incorrect codec parameters ?): Invalid data found when processing input
Error initializing output stream 0:0 --
This starts working, then spews a bunch of errors saying "All capture buffers returned to userspace. Increase num_capture_buffers to prevent device deadlock or dropped packets/frames", and eventually hangs. I can give it 128 capture buffers and it does the same thing :

code:
ffmpeg -y -i in.mkv -map 0:v -vf "scale=-2:'min(200,ih)',format=yuv420p" -b:v 3000k -c:v h264_v4l2m2m -map 0:a -c:a copy -map 0:s -c:s mov_text -num_capture_buffers 16 -num_output_buffers 32 out.mp4
This works fine:

code:
ffmpeg -y -i in.mkv -map 0:v -vf "scale=-2:'min(200,ih)',format=yuv420p" -b:v 3000k -c:v h264_v4l2m2m -map 0:a -c:a copy -num_capture_buffers 16 -num_output_buffers 32 out.mp4
This fails with the same "Invalid data found when processing input":

code:
ffmpeg -y -i in.mkv -map 0:v -vf "scale=-2:'min(200,ih)',format=yuv420p" -b:v 3000k -c:v h264_v4l2m2m -map 0:a -c:a copy -num_capture_buffers 16 -num_output_buffers 32 out.mkv
I know it's a long shot asking such a specific software question on SA but maybe there's an ffmpeg nerd here...

wash bucket
Feb 21, 2006

I've been messing around with Pop OS and just like people said Steam games run pretty well with Proton enabled. However, I have noticed that the games have a bad habit of "hitching" every second or so. Any suggestions on what to look into first? I have an Nvidia GPU and I'm using the driver that was suggested in the built in Pop OS app store.

Also, if there's a better place to ask about this please let me know.

Adbot
ADBOT LOVES YOU

Klyith
Aug 3, 2007

GBS Pledge Week

McCracAttack posted:

I've been messing around with Pop OS and just like people said Steam games run pretty well with Proton enabled. However, I have noticed that the games have a bad habit of "hitching" every second or so. Any suggestions on what to look into first? I have an Nvidia GPU and I'm using the driver that was suggested in the built in Pop OS app store.

Also, if there's a better place to ask about this please let me know.

Start with a look at the system journal with journalctl to see if anything obvious is happening. Helpful options:
-b -p4 = events since last reboot, priority 4 or higher (weeds out unimportant junk)
--follow = monitor the journal live, if you have multi-monitor you could watch a console window while playing a game

You could also try to narrow down whether it's steam or the system in general by trying out lutris, or finding a linux-native game.


there's a linux gaming thread

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply