Powershell Megathread: Verb-Noun Scripting is the future

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Powershell Megathread: Verb-Noun Scripting is the future

«‹›94 »

thebigcow: Jan 3, 2001; Bully!

Warbird posted:

Is there a simple way to install a standalone version of the SQL Server Module that I'm missing? MS docs say you have to download it from their servers via Install-Module, but all that's blocked on our network. We've attempted extracting the module from a SQL full install and manually doing it, but it becomes a nightmare of needing random .dlls and other files.

I think it ships with SSMS if that's an option.

# ? Mar 23, 2019 01:37

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 17:19

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

I'm having a hell of a time converting curl commands to Invoke-RestMethod when using an authorization token--could someone look this over and tell me what I'm missing?

Works:

code:

$AuthBearer = "Authorization: Bearer " + $AuthToken.access_token
.\curl -s "$Gateway/admin/provisioning/app/v1/${AppName}" -H "Accept: application/json" -H "$AuthBearer" -H "cache-control: no-cache"

Doesn't work (server rejects access with a 404):

code:

$AuthBearer = "Bearer " + $AuthToken.access_token
Invoke-RestMethod "$Gateway/admin/provisioning/app/v1/${AppName}" -Method 'Get' -Headers @{"Cache-Control"="no-cache";"Authorization"="$AuthBearer";"Accept"="application/json"}

# ? Mar 23, 2019 03:58

FISHMANPET: Mar 3, 2007; Sweet 'N Sour
Can't
Melt
Steel Beams

This works for us, but looks like you're doing the same thing.

code:

Invoke-RestMethod -Method Get -Uri $uri -Headers @{"Authorization"="Bearer $accessToken"} -OutFile $outFilePath

https://github.com/umn-microsoft-automation/UMN-Google/blob/master/UMN-Google.psm1#L328

You maybe able to get more details about the error with this: https://github.com/umn-microsoft-automation/UMN-Common/blob/master/UMN-Common.psm1#L380

But if it's a 404 that sounds like a URL issue not an auth issue.

# ? Mar 23, 2019 05:53

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

FISHMANPET posted:

But if it's a 404 that sounds like a URL issue not an auth issue.

The URL definitely works, as it's the exact one going to curl as well. The server in question rejects traffic with a 404 if it wasn't provided the correct headers, but I'm really puzzled by how these are any different. I'll try digging into the error when I get back, but knowing the team that supports the server I doubt it will have any actionable information.

# ? Mar 23, 2019 13:10

Judge Schnoopy: Nov 2, 2005; dont even TRY it, pal

PierreTheMime posted:

I'm having a hell of a time converting curl commands to Invoke-RestMethod when using an authorization token--could someone look this over and tell me what I'm missing?

Works:
code:
$AuthBearer = "Authorization: Bearer " + $AuthToken.access_token
.\curl -s "$Gateway/admin/provisioning/app/v1/${AppName}" -H "Accept: application/json" -H "$AuthBearer" -H "cache-control: no-cache"
Doesn't work (server rejects access with a 404):
code:
$AuthBearer = "Bearer " + $AuthToken.access_token
Invoke-RestMethod "$Gateway/admin/provisioning/app/v1/${AppName}" -Method 'Get' -Headers @{"Cache-Control"="no-cache";"Authorization"="$AuthBearer";"Accept"="application/json"}

As a matter of trying to make things easier, I grab all my rest method parameters in a hash table and splat them into invoke-restmethod.

code:


$params = {
    "Headers" = {
        "Authorization" = $authbearer
        "Accept" = "application/json"
    }
    "URI" = "$Gateway/admin/provisioning/app/v1/${AppName}"
}
Invoke-restmethod @params

With all your other stuff in there too. Might make it easier to organize your parameters.

Also, do -verbose, it will give you the exact URI it's hitting with some other good troubleshooting details.

# ? Mar 23, 2019 13:55

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

Judge Schnoopy posted:

As a matter of trying to make things easier, I grab all my rest method parameters in a hash table and splat them into invoke-restmethod.
code:
$params = {
    "Headers" = {
        "Authorization" = $authbearer
        "Accept" = "application/json"
    }
    "URI" = "$Gateway/admin/provisioning/app/v1/${AppName}"
}
Invoke-restmethod @params
With all your other stuff in there too. Might make it easier to organize your parameters.

I appreciate the suggestion. I did try this as well, mostly to confirm it was parsing them as expected.

Judge Schnoopy posted:

Also, do -verbose, it will give you the exact URI it's hitting with some other good troubleshooting details.

I�ll try this out, thanks!

# ? Mar 23, 2019 14:04

The Fool: Oct 16, 2003

If you are still having issues you can use Fiddler to snoop your requests and compare them.

# ? Mar 23, 2019 14:45

New Yorp New Yorp: Jul 18, 2003; Only in Kenya.; Pillbug

Whoever decided to make a trailing space after a line continuation in PowerShell be an error is a monster.

[edit] Okay, okay, it's not an error, it's just causes behavior that is 100% not what you'd expect. Another PowerShell footgun.

# ? Mar 25, 2019 15:00

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

The Fool posted:

If you are still having issues you can use Fiddler to snoop your requests and compare them.

I've tried every permutation of headers I can think of at this point. Fiddler reminded me that since it's a redirect for HTTPS, it's likely stripping the auth token in transit. Apparently PS6 offers -PreserveAuthorizationOnRedirect, which might do the trick but I'll need to find the time to update my dev env and mess around with that. The weird thing is I would think curl wouldn't keep it either, but it works so I'm not going to question it. At this point I'm just going to stick with curl until I have more time to waste on this annoying item, but it sure would be great to no longer rely on a third-party application.

# ? Mar 25, 2019 15:54

FISHMANPET: Mar 3, 2007; Sweet 'N Sour
Can't
Melt
Steel Beams

From my formal computer science education where I learned C and Python etc, I've always hated doing for loops with ForEach-Object, though I warmed to the foreach ($i in $list) construct. But now more ammunition against ForEach-Object. You can't break out of a loop with the break statement the way you can in a foreach loop.

edit: Apparently return return will do it in ForEach-Object but I'm still very suspicious...

FISHMANPET fucked around with this message at 16:09 on Mar 25, 2019

# ? Mar 25, 2019 16:01

Judge Schnoopy: Nov 2, 2005; dont even TRY it, pal

A coworker asked me if I ever got confused which object $_ was referring to in my nested foreach-object loops. I told him no because I use the superior ForEach loops like a sane person, and my variables are declared by the loop at each level to avoid ambiguity.

He still thinks doing a | % {} is slick and won't stop, even in scripts.

# ? Mar 25, 2019 16:09

FISHMANPET: Mar 3, 2007; Sweet 'N Sour
Can't
Melt
Steel Beams

My favorite part of coming across a ForEach-Object loop is when the first command in it is $_ = $usefulName

I get that there are performance benefits to using ForEach-Object vs foreach, but that only matters when your array is so huge that loading it into memory would be burdensome. For my 80 element array a foreach loop should be just fine.

# ? Mar 25, 2019 16:14

xsf421: Feb 17, 2011

Judge Schnoopy posted:

A coworker asked me if I ever got confused which object $_ was referring to in my nested foreach-object loops. I told him no because I use the superior ForEach loops like a sane person, and my variables are declared by the loop at each level to avoid ambiguity.

He still thinks doing a | % {} is slick and won't stop, even in scripts.

$array.foreach() :heysexy:

# ? Mar 25, 2019 16:16

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

FISHMANPET posted:

From my formal computer science education where I learned C and Python etc, I've always hated doing for loops with ForEach-Object, though I warmed to the foreach ($i in $list) construct. But now more ammunition against ForEach-Object. You can't break out of a loop with the break statement the way you can in a foreach loop.

edit: Apparently return return will do it in ForEach-Object but I'm still very suspicious...

You should be suspicious, as it only returns on the "break" condition but continues the loop since Powershell returns work in a less "final" way than other languages and any return gets added to an array of values.

Example:

code:

$Array = @("Item1","Item2","Item3","Item4")
$Test = $Array | ForEach-Object {
	If ($_ -Match "Item2" -Or $_ -Match "Item3") { Return $_ }
	Write-Host $_
}

Results in:

code:

Item1
Item4
PS C:\Users\PierreTheMime> $Test
Item2
Item3

# ? Mar 25, 2019 16:21

FISHMANPET: Mar 3, 2007; Sweet 'N Sour
Can't
Melt
Steel Beams

In this particular case, what's actually returned doesn't matter as this isn't something that's passing into a pipeline, and continuing the loop is actually what I want (I think I'm trying to emulate continue, not break upon further reflection).

Also your demonstration is way nicer than what I puked into a single line.

code:

1..10 | ForEach-Object {$_; if ($_ -eq 5) {write-host "break";return;write-host "whoops"}}; write-host "i got here"

In my case I want to see "break" written out but not "whoops" and I still want to see all numbers 1 through 10, plus the final execution at the end.

FISHMANPET fucked around with this message at 16:31 on Mar 25, 2019

# ? Mar 25, 2019 16:29

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

FISHMANPET posted:

In this particular case, what's actually returned doesn't matter as this isn't something that's passing into a pipeline, and continuing the loop is actually what I want (I think I'm trying to emulate continue, not break upon further reflection).

Also your demonstration is way nicer than what I puked into a single line.
code:
1..10 | ForEach-Object {$_; if ($_ -eq 5) {write-host "break";return;write-host "whoops"}}; write-host "i got here"

Oh if you actually want to continue the loop and simply modify behavior based on a value, that's perfect. I thought you actually wanted a break.

# ? Mar 25, 2019 16:32

FISHMANPET: Mar 3, 2007; Sweet 'N Sour
Can't
Melt
Steel Beams

Yeah every time I use break or continue I have to spend 5 minutes re-learning what each of them mean.

I put break in my production code that's using a ForEach-Object loop and it didn't do what I wanted, so I tried to solve the problem of "replicate break in ForEach-Object" because I assumed past me had used the correct keyword but I should know better than to trust that jerk.

# ? Mar 25, 2019 16:41

nielsm: Jun 1, 2009

ForEach-Object is a replacement for writing a one-off filter function to insert in a pipeline. foreach statement is a way to repeat a block of statements not connected to a pipeline as such. In my day -to-day use, ForEach-Object is more useful most of the time.

# ? Mar 25, 2019 20:04

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

I just created a Class to manage my auth token and refresh it if it's expired prior to making a call, saved it off to a .psm1, and imported it as a module. Having Powershell as a babby OOP is fun and occasionally useful. :feelsgood:

I would feel bad about my methods except I'm replacing 40+ python scripts doing essentially the same thing but worse.

# ? Mar 26, 2019 16:10

Dirt Road Junglist: Oct 8, 2010; We will be cruel
And through our cruelty
They will know who we are

The project to unblock Powershell at work ran into a snag because Security apparently wants to make my life as hard as possible :v:

Now on indefinite hold!

# ? Mar 26, 2019 22:54

slartibartfast: Nov 13, 2002

Dirt Road Junglist posted:

The project to unblock Powershell at work ran into a snag because Security apparently wants to make my life as hard as possible Now on indefinite hold!

There's a Powershell module called dbatools which put up a post about how to convince your security team to trust them. A lot of their points are generalized to Powershell and may be helpful to you.

# ? Mar 26, 2019 23:32

Dirt Road Junglist: Oct 8, 2010; We will be cruel
And through our cruelty
They will know who we are

slartibartfast posted:

There's a Powershell module called dbatools which put up a post about how to convince your security team to trust them. A lot of their points are generalized to Powershell and may be helpful to you.

If this works, where do I send the your prizes for getting the enterprise to stop being idiots?

# ? Mar 27, 2019 00:03

The Fool: Oct 16, 2003

Because I like to share when I do terrible things:

code:

    # Magic values for Exchange
    $adUser.msExchRemoteRecipientType = "4"
    $adUser.mailNickname = $adUser.EmailAddress.Split("@")[0]
    $adUser.msExchProvisioningFlags = "0"
    $adUser.msExchModerationFlags = "6"
    $adUser.msExchAddressBookFlags = "1"
    $adUser.targetaddress $adUser.EmailAddress
    $adUser.msExchRecipientDisplayType = "-2147483642"
    $adUser.msExchRecipientTypeDetails = "2147483648"

# ? Mar 27, 2019 00:48

slartibartfast: Nov 13, 2002

Dirt Road Junglist posted:

If this works, where do I send the your prizes for getting the enterprise to stop being idiots?

If you're ever at a conference and bump into a dbatools contributor, buy 'em a beer.

# ? Mar 27, 2019 02:09

Pile Of Garbage: May 28, 2007

The Fool posted:

Because I like to share when I do terrible things:

code:

    # Magic values for Exchange
    $adUser.msExchRemoteRecipientType = "4"
    $adUser.mailNickname = $adUser.EmailAddress.Split("@")[0]
    $adUser.msExchProvisioningFlags = "0"
    $adUser.msExchModerationFlags = "6"
    $adUser.msExchAddressBookFlags = "1"
    $adUser.targetaddress $adUser.EmailAddress
    $adUser.msExchRecipientDisplayType = "-2147483642"
    $adUser.msExchRecipientTypeDetails = "2147483648"

Why are you setting these attribute values manually? If you're migrating mailboxes to EXO via Remove Move Requests then it will set the attributes automatically when the move requests are finalised.

That aside, you might want to consider clearing the homeMDB attribute as well on the users whose mailboxes have been migrated. A year or two ago I encountered an issue where Autodiscover would start failing when you enable archiving for a remote user mailbox. Turns out that when attempting to discover the archive mailbox it would read the homeMDB attribute which makes it think that it's on-prem. NFI why it does that, only affects archive mailbox as well, primary mailboxes are fine.

# ? Mar 27, 2019 06:49

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

PierreTheMime posted:

I've tried every permutation of headers I can think of at this point. Fiddler reminded me that since it's a redirect for HTTPS, it's likely stripping the auth token in transit. Apparently PS6 offers -PreserveAuthorizationOnRedirect, which might do the trick but I'll need to find the time to update my dev env and mess around with that. The weird thing is I would think curl wouldn't keep it either, but it works so I'm not going to question it. At this point I'm just going to stick with curl until I have more time to waste on this annoying item, but it sure would be great to no longer rely on a third-party application.

AHA! I knew it! I installed 6.1.3 on a Dev box and used the -PreserveAuthorizationOnRedirect and the call worked fine. I still don't know why curl does this without prompting, because apparently it's not supposed to, but I'll need to get our prod servers patched for me to use all PowerShell without external dependencies.

Edit: Arg, since PS Core is separate from the default shell I can't use it from my automation tool without some rework. Anyone know the easiest way to invoke Core and pass in commands? I'm not a huge fan of piping the original commands into a script and running them, compared to just continuing to use curl. It seems like a large change for a minor benefit.

Edit edit: Ok, so you can invoke Core from normal PowerShell by just running the &"(pwsh.exe filepath)" and then running the commands. Hoo boy what an adventure this has been.

PierreTheMime fucked around with this message at 02:01 on Mar 28, 2019

# ? Mar 27, 2019 22:22

The Fool: Oct 16, 2003

Pile Of Garbage posted:

Why are you setting these attribute values manually? If you're migrating mailboxes to EXO via Remove Move Requests then it will set the attributes automatically when the move requests are finalised.

That aside, you might want to consider clearing the homeMDB attribute as well on the users whose mailboxes have been migrated. A year or two ago I encountered an issue where Autodiscover would start failing when you enable archiving for a remote user mailbox. Turns out that when attempting to discover the archive mailbox it would read the homeMDB attribute which makes it think that it's on-prem. NFI why it does that, only affects archive mailbox as well, primary mailboxes are fine.

These attributes are only set on new users, and assuming testing doesn't expose any issues, moves me towards removing my on-prem exchange server.

# ? Mar 27, 2019 22:37

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

My long, agonizing API journey has come to an end. Apparently my automation tool does not play well with Core, so I have to invoke a single command and handle both success and failure as separate JSON returns:

code:

$Command = "Try { Return (Invoke-RestMethod -Uri ""$Gateway/admin/provisioning/app/v1/${AppName}"" -Method 'Get' -PreserveAuthorizationOnRedirect -Headers @{'Accept'='application/json';'Authorization'=""Bearer $($Keycloak.GetAuthToken())"";'Cache-Control'='no-cache'}) | ConvertFrom-JSON } Catch { Return (`$Error[0].ErrorDetails.Message | ConvertFrom-JSON) }"
$Response = (pwsh -Command { Invoke-Expression $Args[0] } -Args @($Command))

It's not pretty, but it works. All this because of a SSO redirect.

# ? Mar 28, 2019 20:14

Pile Of Garbage: May 28, 2007

The Fool posted:

These attributes are only set on new users, and assuming testing doesn't expose any issues, moves me towards removing my on-prem exchange server.

Are you using AADC with Exchange Hybrid enabled? Because if so then you only need to assign an EXO license to the user and the attributes get set automatically.

Edit: also if this is for new users who have mailboxes provisioned in EXO then msExchRemoteRecipientType should be set to 1 (ProvisionMailbox) instead of 4 (Migrated).

Pile Of Garbage fucked around with this message at 06:14 on Mar 29, 2019

# ? Mar 29, 2019 02:30

Happiness Commando: Feb 1, 2002; $$ joy at gunpoint $$

slartibartfast posted:

If you're ever at a conference and bump into a dbatools contributor, buy 'em a beer.

Not emptyquoting. Dbatools is loving awesome.

# ? Mar 29, 2019 03:11

The Fool: Oct 16, 2003

Pile Of Garbage posted:

Are you using AADC with Exchange Hybrid enabled?

Yes.

quote:

Because if so then you only need to assign an EXO license to the user and the attributes get set automatically.

This doesn't work, maybe requires AAD P1?

quote:

Edit: also if this is for new users who have mailboxes provisioned in EXO then msExchRemoteRecipientType should be set to 1 (ProvisionMailbox) instead of 4 (Migrated).

Fixed, thank you.

# ? Apr 1, 2019 18:52

CLAM DOWN: Feb 13, 2007

Anyone know if it's possible to use Add-AzTableRow against a storage context rather than loading it into the existing session with Add-AzAccount? I need to use a connection string in my case.

# ? Apr 5, 2019 23:32

SnatchRabbit: Feb 23, 2006; by sebmojo

I some help on the following command. I want to replace some strings in a file.
I need to work above command which should replace a keyword E:\logs\appserv\* to E:\logs\appserv\psfdev1\$hostname

Its checking * as part of the keyword. Can anyone tell what I'm doing wrong?

code:

(Get-Content D:\app\fscm\pt\ps_cfg_home\appserv\PSFDEV1\psappsrv.cfg) | foreach {$_.replace("Log Directory=E:\logs\appserv\*","Log Directory=E:\logs\appserv\psfdev1\$hostname")} | Set-Content D:\app\fscm\pt\ps_cfg_home\appserv\PSFDEV1\psappsrv.cfg

# ? Apr 15, 2019 18:33

The Fool: Oct 16, 2003

AFAIK, .replace() doesn't do any matching, try it with -replace which will do regex matching.

# ? Apr 15, 2019 20:04

CLAM DOWN: Feb 13, 2007

Piping to Out-Null should just send stdout to $null essentially, correct? Should it be changing the behaviour of the command/line that I'm piping into it? I feel like I'm doing something wrong.

# ? Apr 16, 2019 18:52

The Fool: Oct 16, 2003

Piping to Out-Null will be sending your entire pipeline to $null, not just stdout.

# ? Apr 16, 2019 19:12

CLAM DOWN: Feb 13, 2007

The Fool posted:

Piping to Out-Null will be sending your entire pipeline to $null, not just stdout.

Hmm weird, okay. Does -erroraction still work? I need to mess around with this some more.

Basically I'm using Add-AzTableRow and Update-AzTableRow to write data to Azure table storage, and want to suppress the stdout 204 response from the console. When I pipe it to Out-Null, the command fails. My code is messy atm so I feel very self-conscious posting it

CLAM DOWN fucked around with this message at 19:23 on Apr 16, 2019

# ? Apr 16, 2019 19:16

The Fool: Oct 16, 2003

I'm not familiar with the module, but unless you're doing other things with the pipeline, out-null should work the way that you want.

The module cmdlet could be doing something weird itself.

Try just dumping your output into a variable: $result = Add-AzTableRow $args

Then you can use $result for error handling.

# ? Apr 16, 2019 19:28

CLAM DOWN: Feb 13, 2007

Figured it out. The response from table storage for that cmdlet is an HTTP 204 and that was getting sent to null haha, I took your idea of putting the results in a variable and was able to handle it better that way including error handling. Thank you!!

# ? Apr 16, 2019 19:44

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 17:19

FISHMANPET: Mar 3, 2007; Sweet 'N Sour
Can't
Melt
Steel Beams

I'm writing an Azure runbook that's going to validate a spreadsheet of customer data to make sure the data meets various conditions (make sure every email address is actually an email address, make sure the data matches up with our inventory, etc). It's a series of for each loops that loop through testing every row for whatever the condition is. I'd really like to pesterize this but I can't figure out any way to do it. I can break each test into a function, and I can write a pester test against that function, but I have no idea what the "best" way to actually run the tests is. The functions aren't exported, they only exist within the script. I don't want to put them in a module or something like that, because they serve no purpose outside of this runbook. I don't really want to have the functions external to the runbook in anyway because I don't want to have extra runbooks hanging out that only exist to be a source of functions.

It seems like there's no way for Pester to extract the functions from a script to test them without also executing the rest of the script, so the question becomes what's the best hacky work around.

# ? Apr 19, 2019 03:36

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Powershell Megathread: Verb-Noun Scripting is the future

«‹›94 »