Powershell Megathread: Verb-Noun Scripting is the future

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Powershell Megathread: Verb-Noun Scripting is the future

«‹›94 »

anthonypants: May 6, 2007; by Nyc_Tattoo; Dinosaur Gum

Zaepho posted:

[xml] $foo = '<xml><thing/></xml>'
$foo.Save('test.xml')

That's...great. But it's more like this

code:

[xml] $foo = '<xml>"thing's"</xml>'
$foo.Save('test.xml')

# ? Aug 4, 2018 01:20

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 08:11

Zaepho: Oct 31, 2013

anthonypants posted:

That's...great. But it's more like this
code:
[xml] $foo = '<xml>"thing's"</xml>'
$foo.Save('test.xml')

both are perfectly valid XML. There's also a .Net class you can use to pretty print it which I've done before but can;t recall the syntax off the top of my head.

# ? Aug 4, 2018 01:33

Pile Of Garbage: May 28, 2007

Zaepho posted:

[xml] $foo = '<xml><thing/></xml>'
$foo.Save('test.xml')

Read before you post, the question wasn't about how to do it and you just regurgitated the exact method which anthonypants already mentioned in their own post.

To expand on what nielsm said, standards-wise anything that isn't markup is just character data which can be anything really: https://www.w3.org/TR/xml/#syntax

# ? Aug 4, 2018 11:50

peak debt: Mar 11, 2001; b& :(; Nap Ghost

The Fool posted:

Sanity check.

I am using VSTS to automate a bunch of stuff. I have some scripts that require having credentials to service accounts.

You can store a password in DPAPI like this:

code:

'p@ssw0rd' | ConvertTo-SecureString -AsPlainText -Force | ConvertFrom-SecureString | Out-File "d:\secretpassword.txt"

And then recover it like this:

code:

    $encrypted_password = gc "d:\secretpassword.txt" | ConvertTo-SecureString
    $credentials = New-Object System.Management.Automation.PSCredential -ArgumentList "domain\dummy", $encrypted_password
    $password = $credentials.GetNetworkCredential().Password

The second script has to run under the same account the first one did.

# ? Aug 4, 2018 16:11

Pile Of Garbage: May 28, 2007

Is the thread gimmick not reading posts?

# ? Aug 4, 2018 16:43

Nth Doctor: Sep 7, 2010; Darkrai used Dream Eater!
It's super effective!

cheese-cube posted:

Is the thread gimmick not reading posts?

My Read-Thread implementation is flaky, okay?

# ? Aug 4, 2018 17:18

Pile Of Garbage: May 28, 2007

Nth Doctor posted:

My Read-Thread implementation is flaky, okay?

Nah you're good, I'm pissing in the direction of Zaepho and peak debt.

# ? Aug 4, 2018 18:15

The Fool: Oct 16, 2003

New Yorp New Yorp posted:

Why can't you just store them as encrypted values in the build/release definition and pass them into the script when running?

Because VSTS doesn't pass encrypted variables as enviroment variables, I would have to pass them as command line arguments which means they would end up getting logged in plain text.

peak debt posted:

You can store a password in DPAPI like this:

DPAPI is encrypted per account and per machine. The entire point is to be able to store credentials in a way that doesn't require me pregenerating and storing them for a dozen servers.

# ? Aug 5, 2018 03:33

FISHMANPET: Mar 3, 2007; Sweet 'N Sour
Can't
Melt
Steel Beams

Does anyone know of a good resources for "best practices" around error handling? I understand the mechanics of the try-catch block but at work we've just got so much code that catches the error and throws "error: error" or some nonsense like that and I'm sure other people have solved this problem beyond "throw it all in a try-catch block and shrug your shoulders"

# ? Aug 7, 2018 23:38

MF_James: May 8, 2008; I CANNOT HANDLE BEING CALLED OUT ON MY DUMBASS OPINIONS ABOUT ANTI-VIRUS AND SECURITY. I REALLY LIKE TO THINK THAT I KNOW THINGS HERE

INSTEAD I AM GOING TO WHINE ABOUT IT IN OTHER THREADS SO MY OPINION CAN FEEL VALIDATED IN AN ECHO CHAMBER I LIKE

FISHMANPET posted:

Does anyone know of a good resources for "best practices" around error handling? I understand the mechanics of the try-catch block but at work we've just got so much code that catches the error and throws "error: error" or some nonsense like that and I'm sure other people have solved this problem beyond "throw it all in a try-catch block and shrug your shoulders"

On my own scripts I usually throw an error regarding where the error happened so I don't have to step through the code every time, also I do logging of 'milestones' in the script, not sure what a standard is. I've also seen some scripts from our programmers that make .NET calls and they can pull back the error codes from that into a variable (or perhaps it's natively dumped into $error) and they'll give you that in the message.

MF_James fucked around with this message at 23:45 on Aug 7, 2018

# ? Aug 7, 2018 23:43

anthonypants: May 6, 2007; by Nyc_Tattoo; Dinosaur Gum

It depends a lot on what the script's doing, too. Like maybe a script silently failing is okay, or maybe it needs to throw an error and stop processing immediately, or maybe it needs to return a different value, or maybe it needs to pass an error along to something else

# ? Aug 7, 2018 23:44

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

FISHMANPET posted:

Does anyone know of a good resources for "best practices" around error handling? I understand the mechanics of the try-catch block but at work we've just got so much code that catches the error and throws "error: error" or some nonsense like that and I'm sure other people have solved this problem beyond "throw it all in a try-catch block and shrug your shoulders"

It depends a bit�does a mechanism for recovery exist? If it�s an unrecoverable error there�s not much more you can do then stop and get the environmental variables to review what may have gone wrong. Some programs may require restaging or moving invalid files, etc so you can do cleanup work.

You can break your errors out further by type and decide on actions based on specific known issues. Try/Catch can get as simple or complicated as you want to make it, it�s essentially an if/else for �does this code work�.

# ? Aug 7, 2018 23:46

FISHMANPET: Mar 3, 2007; Sweet 'N Sour
Can't
Melt
Steel Beams

This is all just in scripts we run, and I'm absolutely tired of seeing poo poo like this:

code:

    try                               
        {        
            #do stuff  in a loop
                Try
                    {
                           # do more stuff
                    }
                Catch {Write-Warning "Failed to send mail message to $target"} 
        }
    Catch {Write-Warning "Unable to email to $department"}

And I'd really like to write some standards, like, if you're gonna use a try-catch block this is what you should do with the error etc. Stuff like this just obfuscates it and makes it 100 times harder to debug.

# ? Aug 8, 2018 00:37

Dirt Road Junglist: Oct 8, 2010; We will be cruel
And through our cruelty
They will know who we are

FISHMANPET posted:

This is all just in scripts we run, and I'm absolutely tired of seeing poo poo like this:
code:
    try {        
            #do stuff  in a loop
                Try {
                           # do more stuff
                    }
                Catch {Write-Warning "Failed to send mail message to $target"} 
        }
    Catch {Write-Warning "Unable to email to $department"}
And I'd really like to write some standards, like, if you're gonna use a try-catch block this is what you should do with the error etc. Stuff like this just obfuscates it and makes it 100 times harder to debug.

I tend to leave my try/catch efforts out until everything else is solidly coded. I'll pepper my code with comments and send error messages directly to the console. Unless they're script-stopping, then I'll chuck a try/catch around them just to keep the script flowing until I've debugged.

It also depends on what I'm being asked to return. Sometimes I'll throw a try/catch around something that could error out with nothing inside the catch {}, but sometimes I'll have multi-nested crap going on because someone at some point is going to be pissed if they can't see something in a log on the regular.

But I'm a bit of a feral coder who likes to iterate rather than pseudo-code and refine, so YMMV.

# ? Aug 8, 2018 20:53

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

If you want to standardize your exception catching, you can write a function to rule over evaluating fatal/non-fatal errors and return values your script can handle in some sort of switch or other evaluation method:

code:

Function EvalException {
	param([System.Exception]$Exception,[int]$LineNum,[boolean]$PrintStack)
	Write-Host Exception caught ($Exception.Source) at line number $LineNum
	Write-Host Reason: $Exception.Message
	If ($PrintStack) {Write-Host Stack trace: $Exception.StackTrace}
	#Do some evaluation of exception type, return value that can be acted on in future
	#Example returns recoverable error 3
	return 3
}

$DefaultValue = 0
Try {
	Get-Content test -ErrorAction Stop
} Catch {
	Switch (EvalException -Exception $_.Exception -LineNum $_.InvocationInfo.ScriptLineNumber -PrintStack $True) {
		0 {Write-Host Everthing is great!}
		1 {Write-Host Non-fatal error handling type 1}
		2 {Write-Host Non-fatal error handling type 2}
		3 {Write-Host Non-fatal error handling type 3}
		default {Write-Host Oh poo poo its fatal error time}
	}
}

Try/Catching can get pretty specific to the situation at hand, though, so this type of approach doesn't cover everything (at least not without becoming a giant mess on its own). At the very least this helps to standardize how your team might approach exceptions.

# ? Aug 8, 2018 21:32

Scikar: Nov 20, 2005; 5? Seriously?

I'm not doing anything particularly crazy with Powershell scripts so I can't say it would work for everyone, but I don't use unfiltered try/catch blocks. I use a try/catch if a) what I'm doing has a non-negligible chance of directly throwing an exception, AND b) that exception is something that either is recoverable or requires me to do some cleanup. And then the catch block is scoped to the type of exception that I expect. If your script is deleting a list of files then absolutely catch System.IO.FileNotFoundException (and maybe whatever permissions errors evaluate to), log that something weird happened with that file, then continue and try to delete the rest. But if that action throws an OutOfMemoryException or something then the last thing you need is a try/catch swallowing it and just writing "Something went wrong!" to a log file. So in terms of standards I would start with never write catch without a list of types that you expect and will deal with in the catch block.

# ? Aug 8, 2018 22:53

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

Mega agreed, generally avoid blanket catching in final code unless you are a glutton for punishment. Like all things, there are exceptions ( :haw:

) but always keep types in mind.

# ? Aug 8, 2018 23:06

Happiness Commando: Feb 1, 2002; $$ joy at gunpoint $$

Can someone point me to a VSCode and Git/VSTS for dummies blog post?My team has no version control whatsoever and this should be an easy thing for me to set up and get brownie points for. I've barely managed to get VSCode set up with a couple plugins for Powershell and VSTS. I tried setting up a Git repository in VSTS and I think I've managed to get VSTS to commit to my local repo but I can't get the changes to sync to VSTS.

# ? Aug 10, 2018 02:48

The Fool: Oct 16, 2003

Your VSTS repo needs to be the remote origin for your local repo, then you commit and push and your changes will show up.

# ? Aug 10, 2018 06:04

slartibartfast: Nov 13, 2002

The folks behind dbatools have a guide for how to contribute to their project using VSCode and Github. You'll have to swap out the URLs for your own repos, but it's a pretty clear step-by-step guide.

# ? Aug 10, 2018 16:40

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

For the purposes of testing a different file load mechanism someone wants me to split a 7.5GB text file into six equal portions.

Am I overthinking it or is this all it really needs:

code:

$Reader = New-Object $System.IO.StreamReader($SourceFile)
$SplitFileRoot = $SourceFile.Replace(".txt","")
$FileCount = 1
While(($Line = $Reader.ReadLine()) -NE $null) {
   $Path = $SplitFileRoot + "_" + $FileCount + ".txt"
   Add-Content -Path $Path -Value $Line
   If ($FileCount -GT 6) {$FileCount = 1}
}
$Reader.Close()

It's a fairly simple task but seems like it will take forever. I'm not sure if there would be an appreciable difference putting this to another language, but I'd consider it if the gains were significant and they wanted to make this a standard process.

# ? Aug 21, 2018 19:50

Scikar: Nov 20, 2005; 5? Seriously?

I don't know if it was ever fixed but Powershell loops are hella slow, and some variations are like 10x slower than others (piping to ForEach-Object is particularly bad if I remember right). If you don't have to process the actual text then you just need FileStreams. Open the source file with one stream, create a new destination text file and open as a second stream, then just feed the first stream into the second in big chunks until you hit ~1.2GB. Then close that destination stream, open a new one and continue. If you loop line by line in PS it might take days.

# ? Aug 21, 2018 22:50

nielsm: Jun 1, 2009

At the very least don't use Add-Content inside the loop like that. You should open all the output files before the loop and keep them open throughout.

# ? Aug 21, 2018 23:27

Judge Schnoopy: Nov 2, 2005; dont even TRY it, pal

Scikar posted:

I don't know if it was ever fixed but Powershell loops are hella slow, and some variations are like 10x slower than others (piping to ForEach-Object is particularly bad if I remember right). If you don't have to process the actual text then you just need FileStreams. Open the source file with one stream, create a new destination text file and open as a second stream, then just feed the first stream into the second in big chunks until you hit ~1.2GB. Then close that destination stream, open a new one and continue. If you loop line by line in PS it might take days.

Is this why my stupid compare function takes forever? I mean, I'm doing a few hundred thousand loops (possibly a million+?) but I've never had a powershell script take 45 minutes to complete.

# ? Aug 21, 2018 23:34

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

Yeah for whatever reason I didn�t think to do outgoing streams too, which is obviously now. It took 2.5 hours but I�ll make the update and see what the difference is.

# ? Aug 22, 2018 00:29

Scikar: Nov 20, 2005; 5? Seriously?

Judge Schnoopy posted:

Is this why my stupid compare function takes forever? I mean, I'm doing a few hundred thousand loops (possibly a million+?) but I've never had a powershell script take 45 minutes to complete.

I think this is what I was thinking of: https://blogs.technet.microsoft.com/heyscriptingguy/2014/07/08/getting-to-know-foreach-and-foreach-object/

# ? Aug 22, 2018 00:33

Toshimo: Aug 23, 2012; He's outta line...

But he's right!

PierreTheMime posted:

Yeah for whatever reason I didn�t think to do outgoing streams too, which is obviously now. It took 2.5 hours but I�ll make the update and see what the difference is.

I'm doing ~2.5gb in ~15sec with:

code:

function split($path)
{
    $chunkSize=[math]::Ceiling((Get-Item $path).Length/6)
    $fileName = [System.IO.Path]::GetFileNameWithoutExtension($path)
    $directory = [System.IO.Path]::GetDirectoryName($path)
    $extension = [System.IO.Path]::GetExtension($path)

    $file = New-Object System.IO.FileInfo($path)
    $totalChunks = [int]($file.Length / $chunkSize) + 1
    $digitCount = [int][System.Math]::Log10($totalChunks) + 1

    $reader = [System.IO.File]::OpenRead($path)
    $count = 0
    $buffer = New-Object Byte[] $chunkSize
    $hasMore = $true
    while($hasMore)
    {
        $bytesRead = $reader.Read($buffer, 0, $buffer.Length)
        $chunkFileName = "$directory\$fileName$extension.{0:D$digitCount}.part"
        $chunkFileName = $chunkFileName -f $count
        $output = $buffer
        if ($bytesRead -ne $buffer.Length)
        {
            $hasMore = $false
            $output = New-Object Byte[] $bytesRead
            [System.Array]::Copy($buffer, $output, $bytesRead)
        }
        [System.IO.File]::WriteAllBytes($chunkFileName, $output)
        ++$count
    }

    $reader.Close()
}

split C:\Path\File.txt

# ? Aug 22, 2018 01:01

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

I reworked it using a StreamWriter and it is so much faster it's laughable (1.5GB in about 10 seconds) but now I'm getting an interesting behavior where it's deleting my header from files 2-6, but writing it on 1. I would think if the stream was overwriting the entirely of the file it would overwrite all: Creating a WriteStream to a preexisting filepath overwrites the original file and there�s no append option, so I�m just having it loop all streams on the first line to set the header before proceeding with normal distribution.

Edit: Semi-final code seems to work, probably a little bit more verbose than necessary, but let me know if anything pops out as "bad" since I'm always up for learning better methods.

code:

Param([Parameter(Mandatory=$True)][string]$SourceFile, [Parameter(Mandatory=$True)][int]$SplitCount, [Parameter(Mandatory=$True)][bool]$HasHeader)

$StreamArray = @()
$SplitFileRoot = $SourceFile.Replace(".txt","")
$Reader = New-Object System.IO.StreamReader($SourceFile)
For($i=1;$i -LT ($SplitCount+1);$i++) {
	$Path = $SplitFileRoot + "_" + $i + ".txt"
	$StreamArray += [System.IO.StreamWriter] "$Path"
}

$FileCount = 0
While(($Line = $Reader.ReadLine()) -NE $null) {
	If ($HasHeader) {
		ForEach ($Stream in $StreamArray) { $Stream.WriteLine($Line) }
		$HasHeader = $False
	} Else {
		$StreamArray[$FileCount].WriteLine($Line)
	}
	$FileCount++
	If ($FileCount -EQ $StreamArray.Count) { $FileCount = 0 }
}
ForEach ($Stream in $StreamArray) { $Stream.Close() }
$Reader.Close()

PierreTheMime fucked around with this message at 15:54 on Aug 22, 2018

# ? Aug 22, 2018 02:42

Judge Schnoopy: Nov 2, 2005; dont even TRY it, pal

Scikar posted:

I think this is what I was thinking of: https://blogs.technet.microsoft.com/heyscriptingguy/2014/07/08/getting-to-know-foreach-and-foreach-object/

Ahh, I exclusively used foreach statements so it wouldn't have impacted my script.

Turns out a million loops just takes a long loving time to do!

# ? Aug 22, 2018 03:03

CLAM DOWN: Feb 13, 2007

How do I read the HTTP response headers that I get back from a successful Invoke-RestMethod request? This stupid API has put the ID value that I need inside the header, rather than in the json.

For example:

code:

$stuff = Invoke-RestMethod -Uri $apiuri -ContentType $contenttype -Headers $headers -Method $method -Body $body

$stuff contains the json but I need a value in the headers

I tried $stuff.responseHeaders which I recall working for maybe Invoke-WebRequest, but it doesn't exist/doesn't work in this case.

# ? Aug 23, 2018 21:48

anthonypants: May 6, 2007; by Nyc_Tattoo; Dinosaur Gum

I think you either get to use PowerShell Core, where this might be fixed, or to use something else.

# ? Aug 23, 2018 22:23

sloshmonger: Mar 21, 2013

I don't know if it helps this situation at all, but the Powershell Core 6.x version of Invoke-RestMethod has a ResponseHeadersVariable outputs the return headers to a variable.

If it's a only-need-it-once type thing, maybe Fiddler?

Otherwise, you're stuck using Invoke-WebRequest and piping the data to a json.

# ? Aug 23, 2018 22:36

anthonypants: May 6, 2007; by Nyc_Tattoo; Dinosaur Gum

Fiddler is okay but Postman is super good at writing and testing REST statements.

# ? Aug 23, 2018 22:39

CLAM DOWN: Feb 13, 2007

anthonypants posted:

I think you either get to use PowerShell Core, where this might be fixed, or to use something else.

sloshmonger posted:

I don't know if it helps this situation at all, but the Powershell Core 6.x version of Invoke-RestMethod has a ResponseHeadersVariable outputs the return headers to a variable.

If it's a only-need-it-once type thing, maybe Fiddler?

Otherwise, you're stuck using Invoke-WebRequest and piping the data to a json.

gently caress me. Thanks.

My script has to run on a VSTS hosted agent, so I don't think I can go the core route. drat, I was so close, why can't this stupid API return the id value that I need in the json rather than loving putting it in the header. So dumb.

# ? Aug 23, 2018 22:44

anthonypants: May 6, 2007; by Nyc_Tattoo; Dinosaur Gum

You could probably use PowerShell to make the REST calls via .NET?

# ? Aug 23, 2018 22:52

CLAM DOWN: Feb 13, 2007

anthonypants posted:

You could probably use PowerShell to make the REST calls via .NET?

Haha yup, that's literally what I'm starting to test right now. Going to have to do this via .NET objects instead of that nice cmdlet. Sigh.

# ? Aug 23, 2018 22:56

PBS: Sep 21, 2015

CLAM DOWN posted:

Haha yup, that's literally what I'm starting to test right now. Going to have to do this via .NET objects instead of that nice cmdlet. Sigh.

Why not use Invoke-WebRequest? If the response is json just do, $stuff.Content | ConvertFrom-Json

# ? Aug 24, 2018 01:20

anthonypants: May 6, 2007; by Nyc_Tattoo; Dinosaur Gum

PBS posted:

Why not use Invoke-WebRequest? If the response is json just do, $stuff.Content | ConvertFrom-Json

Invoke-WebRequest isn't available in certain versions of PowerShell.

# ? Aug 24, 2018 01:24

CLAM DOWN: Feb 13, 2007

PBS posted:

Why not use Invoke-WebRequest? If the response is json just do, $stuff.Content | ConvertFrom-Json

I forgot about ConvertTo/From-Json. I'll try that. I'm still annoyed that Invoke-RestMethod is lacking this.

# ? Aug 24, 2018 01:36

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 08:11

PBS: Sep 21, 2015

anthonypants posted:

Invoke-WebRequest isn't available in certain versions of PowerShell.

Ah, that makes sense.

CLAM DOWN posted:

I forgot about ConvertTo/From-Json. I'll try that. I'm still annoyed that Invoke-RestMethod is lacking this.

Yeah, it's dumb.

# ? Aug 24, 2018 01:41

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Powershell Megathread: Verb-Noun Scripting is the future

«‹›94 »