The Web Design & Development Megathread

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > The Web Design & Development Megathread

«‹›361 »

Warbird: May 23, 2012; America's Favorite Dumbass

I've been playing around with some Python web scraping and I'm hoping someone can point me in the right direction for something I'm running up against right now.

I'm trying to get the price of a given item at Lowe's via lxml and requests. I can get "to" the item just fine by

code:

https://www.lowes.com/search?searchTerm={}'.format(x)

with X as a SKU, but I'm not sure how to pass or spoof a store location which is needed to get the price for a given place. I've tried passing a handful of other arguments along with the SKU to no success. Any suggestions?

# ? Apr 4, 2017 22:18

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 18:07

Lumpy: Apr 26, 2002; La! La! La! Laaaa!; College Slice

Warbird posted:

I've been playing around with some Python web scraping and I'm hoping someone can point me in the right direction for something I'm running up against right now.

I'm trying to get the price of a given item at Lowe's via lxml and requests. I can get "to" the item just fine by
code:
https://www.lowes.com/search?searchTerm={}'.format(x)
with X as a SKU, but I'm not sure how to pass or spoof a store location which is needed to get the price for a given place. I've tried passing a handful of other arguments along with the SKU to no success. Any suggestions?

Check your cookies / headers while browsing the site to see how they are passing the location, then copy!

# ? Apr 5, 2017 04:35

-JS-: Jun 1, 2004

Warbird posted:

I've tried passing a handful of other arguments along with the SKU to no success. Any suggestions?

When you enter a zip code into the site, it sets a cookie called sn containing the ID for the closest store (where I assume sn means Store Number).

e.g. Zip code: 54101, gets you Buchanan Lowe's: https://www.lowes.com/store/WI-Appleton/2486

And the sn cookie contains the same ID as the URL - 2486

-JS- fucked around with this message at 10:49 on Apr 5, 2017

# ? Apr 5, 2017 10:46

Warbird: May 23, 2012; America's Favorite Dumbass

-JS- posted:

When you enter a zip code into the site, it sets a cookie called sn containing the ID for the closest store (where I assume sn means Store Number).

e.g. Zip code: 54101, gets you Buchanan Lowe's: https://www.lowes.com/store/WI-Appleton/2486

And the sn cookie contains the same ID as the URL - 2486

I figured it was cookies based, but was hoping I could override it via a proper URL call. There is a proper API it seems, but I can't get it to play along either. I've accomplished what I wanted to do via other means, but would would be a sane way to approach spoofing cookies and such?

# ? Apr 5, 2017 13:38

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Warbird posted:

I figured it was cookies based, but was hoping I could override it via a proper URL call. There is a proper API it seems, but I can't get it to play along either. I've accomplished what I wanted to do via other means, but would would be a sane way to approach spoofing cookies and such?

You just use whatever facility your python http library uses for sending cookies.

For example if you're using the requests module, you can do this:

code:

 >>> url = 'http://httpbin.org/cookies' 
>>> cookies = dict(cookies_are='working')
 >>> r = requests.get(url, cookies=cookies) 
>>> r.text '{"cookies": {"cookies_are": "working"}}'

# ? Apr 5, 2017 13:46

Warbird: May 23, 2012; America's Favorite Dumbass

Oh neat, I didn't know it was able to do that! I'll play around with it and see what I can do. Thanks!

Edit - That worked perfectly! There's a minor hiccup where items searched return more than one option instead of proceeding straight to the SKU's item page, but that's no big deal.

Warbird fucked around with this message at 14:27 on Apr 5, 2017

# ? Apr 5, 2017 14:05

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Warbird posted:

Oh neat, I didn't know it was able to do that! I'll play around with it and see what I can do. Thanks!

Edit - That worked perfectly! There's a minor hiccup where items searched return more than one option instead of proceeding straight to the SKU's item page, but that's no big deal.

I'm not sure if its useful in your case but look into using sessions with requests.

# ? Apr 5, 2017 15:11

Warbird: May 23, 2012; America's Favorite Dumbass

I don't think that would have too much play for this experiment, but I could definitely see me using that down the line for something else. Thanks for the heads up!

So I've fired up VSCode again and tried playing around with the Chrome debugger a bit more and I think I see what the issue is: IIS (and also massive massive user error).

I was only vaguely aware this existed, but I'm guessing I'd want to make a site to run locally within it, point it to my local working directory for the file source, and tell it to use port ###. At that point I go into the Chrome debugger configs and tell it to look there. Does this sound about right?

# ? Apr 5, 2017 16:40

Space Kablooey: May 6, 2009

Warbird posted:

I figured it was cookies based, but was hoping I could override it via a proper URL call. There is a proper API it seems, but I can't get it to play along either. I've accomplished what I wanted to do via other means, but would would be a sane way to approach spoofing cookies and such?

FWIW, I just tried the API link and it seemed to work fine for me. :confused:

What is the issue with the API, exactly?

Space Kablooey fucked around with this message at 20:07 on Apr 5, 2017

# ? Apr 5, 2017 20:02

Warbird: May 23, 2012; America's Favorite Dumbass

Most likely me missing something I'm sure. I could never get it to narrow down to a specific item, or return anything other than that data set I linked. I'm not quite sure what it's pulling as that is by no means the full selection of that store. It appears to just be a selection of washing machines as best I can tell.

# ? Apr 5, 2017 20:44

Space Kablooey: May 6, 2009

Warbird posted:

Most likely me missing something I'm sure. I could never get it to narrow down to a specific item, or return anything other than that data set I linked. I'm not quite sure what it's pulling as that is by no means the full selection of that store. It appears to just be a selection of washing machines as best I can tell.

Gotcha, that API looks like a pain to use. Besides, apparently it is not filtering at all, and while it shows how many pages of products, it doesn't show how to paginate the results.

What I got from that URL is that the last part defines a catalog of products and it then returns the whole thing unfiltered but paginated. It also returns a list of related catalogs, but it doesn't feel like it is what you want.

# ? Apr 5, 2017 21:26

MrMoo: Sep 14, 2000

kedo posted:

What's the current hotness in terms of CSS and/or canvas animation?

Just use WebAnimations and the polyfill? I've seen a lot of random stuff for Canvas elements.

# ? Apr 6, 2017 01:34

teen phone cutie: Jun 18, 2012; last year i rewrote something awful from scratch because i hate myself

Just updated to the new version of VSCode and now I'm getting this error:

Extension host terminated unexpectedly. Please reload the window to recover.

Anyone else?

e: Yes, apparently:

https://github.com/Microsoft/vscode/issues/24028

# ? Apr 6, 2017 17:38

skeptic22: Aug 13, 2004; Immaculate

I'd like to require https sitewide on a Drupal 7 site and have our developer make this change (surprised it was not in place already).

Could someone provide a sense of whether this is something that can be done by obtaining the certificate, changing a couple of configuration settings and having these changes apply sitewide? Or if it will be a project that could require configuring individual modules or pages and take many hours?

I know this is not a lot of info, but just trying to get the broadest strokes on project scope here so I can evaluate their bid without getting bilked. Can provide more information if needed to make a call.

# ? Apr 7, 2017 18:21

Scaramouche: Mar 26, 2001; SPACE FACE! SPACE FACE!

I'm not super Drupal familiar, but theoretically it could be as easy as one line in .htaccess. This assumes relative references and a lack of hardcoded URLs.

# ? Apr 7, 2017 18:27

Warbird: May 23, 2012; America's Favorite Dumbass

Got IIS to show my site on localhost after giving a couple of IIS related user accounts full access to the local webroot folders. Now the Chrome debugger actually launches the page when started, but times out at 10 seconds because reasons. I'm calling it a win.

This all seems incredibly esoteric. Is this just the nature of webdev? Is it because I'm on Windows? Was I spoiled by IDEs like Unity and Visual Studio being one stop shops? Is there a solution for me to make a website (static even) that doesn't require 4 different platforms and tons of config? Stuff like VSCode/Atom/Sublime seem close to that, but require what is becoming a silly amount of tweaking and prior knowledge.

Am I just bringing this on myself by choosing to us VSCode? Should I just use something like Brackets instead?

/rant

# ? Apr 7, 2017 20:02

Maed: Aug 23, 2006

Warbird posted:

Got IIS to show my site on localhost after giving a couple of IIS related user accounts full access to the local webroot folders. Now the Chrome debugger actually launches the page when started, but times out at 10 seconds because reasons. I'm calling it a win.

This all seems incredibly esoteric. Is this just the nature of webdev? Is it because I'm on Windows? Was I spoiled by IDEs like Unity and Visual Studio being one stop shops? Is there a solution for me to make a website (static even) that doesn't require 4 different platforms and tons of config? Stuff like VSCode/Atom/Sublime seem close to that, but require what is becoming a silly amount of tweaking and prior knowledge.

Am I just bringing this on myself by choosing to us VSCode? Should I just use something like Brackets instead?

/rant

https://www.npmjs.com/package/live-reload

# ? Apr 7, 2017 20:04

Warbird: May 23, 2012; America's Favorite Dumbass

Maed posted:

https://www.npmjs.com/package/live-reload

So if I'm reading this right, I start up the live-reload server and toss that script tag in and it'll auto refresh anything I have open in the browser? That's pretty good, but comes back to what I'm talking about. VSCode and other similar editors seem the be the vi/emacs of the web development IDEs. Incredibly useful and powerful if you know what you're doing, but the learning curve (paired with the fractured nature of web development) is intense. I need the nano equivalent. Brackets seems like that might be that? (Which makes Dreamweaver the Microsoft Word of this poorly formed metaphor?)

# ? Apr 7, 2017 20:12

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

If you want a one stop shop you might look into the Webstorm IDE. It's pretty great.

That being said web dev is pretty poo poo when it comes to tooling. That's largely because you're developing for a different environment (the browser) than the one you're working in, and the browser is just a terrible environment...at least for developers.

# ? Apr 7, 2017 20:23

NewForumSoftware: Oct 8, 2016; by Lowtax

Warbird posted:

Got IIS to show my site on localhost after giving a couple of IIS related user accounts full access to the local webroot folders. Now the Chrome debugger actually launches the page when started, but times out at 10 seconds because reasons. I'm calling it a win.

This all seems incredibly esoteric. Is this just the nature of webdev? Is it because I'm on Windows? Was I spoiled by IDEs like Unity and Visual Studio being one stop shops? Is there a solution for me to make a website (static even) that doesn't require 4 different platforms and tons of config? Stuff like VSCode/Atom/Sublime seem close to that, but require what is becoming a silly amount of tweaking and prior knowledge.

Am I just bringing this on myself by choosing to us VSCode? Should I just use something like Brackets instead?

/rant

create a file on your computer called index.html and open it with chrome

thats as basic as it gets

# ? Apr 7, 2017 21:09

grenada: Apr 20, 2013; Relax.

skeptic22 posted:

I'd like to require https sitewide on a Drupal 7 site and have our developer make this change (surprised it was not in place already).

Could someone provide a sense of whether this is something that can be done by obtaining the certificate, changing a couple of configuration settings and having these changes apply sitewide? Or if it will be a project that could require configuring individual modules or pages and take many hours?

I know this is not a lot of info, but just trying to get the broadest strokes on project scope here so I can evaluate their bid without getting bilked. Can provide more information if needed to make a call.

edit: It was really simple on Wordpress. Just buy the certificate and put the hashes into Cpanel https wizard, and then force your traffic to https in .htaccess. Will let someone else chime in on if it is the same in Drupal.

grenada fucked around with this message at 13:59 on Apr 8, 2017

# ? Apr 8, 2017 13:51

Love Stole the Day: Nov 4, 2012; Please give me free quality professional advice so I can be a baby about it and insult you

Really comprehensive ebook about Redux just got released and it's doing a free until april 14th special thing: https://leanpub.com/redux-book

# ? Apr 9, 2017 05:46

karms: Jan 22, 2006; by Nyc_Tattoo; Yam Slacker

Love Stole the Day posted:

Really comprehensive ebook about Redux just got released and it's doing a free until april 14th special thing: https://leanpub.com/redux-book

Cool! thanks for the heads up.

# ? Apr 9, 2017 12:22

SurgicalOntologist: Jun 17, 2004

I've been banging my head against the wall with this, maybe someone has an idea.

I have a simple flask app and I'm trying to inline css styles on emails using python premailer, implemented as a jinja2 filter. However, premailer hangs indefinitely (left it running overnight to see if it was just slow--nope). If I empty out the relevant css file, it still hangs (so it's not an issue of bad css). If I make the filter function just return its input, it works (so it's not an issue of getting the filter wrong in jinja2).

The weird thing is, if I run premailer from the command line entry point, it works! And quickly. l tried to look through the source code to see if there are different defaults, but nothing I tried made a difference.

Any ideas?

Edit: It was hanging when premailer requested the css file, so the problem was that flask couldn't handle one request in the middle of handling another. Increasing the gunicorn workers fixed it. Not sure if there's a solution on the development server.

SurgicalOntologist fucked around with this message at 17:27 on Apr 10, 2017

# ? Apr 10, 2017 17:09

Space Kablooey: May 6, 2009

SurgicalOntologist posted:

Edit: It was hanging when premailer requested the css file, so the problem was that flask couldn't handle one request in the middle of handling another. Increasing the gunicorn workers fixed it. Not sure if there's a solution on the development server.

Yup, the dev server can't handle one request on the middle of another.

# ? Apr 10, 2017 18:13

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

SurgicalOntologist posted:

I've been banging my head against the wall with this, maybe someone has an idea.

I have a simple flask app and I'm trying to inline css styles on emails using python premailer, implemented as a jinja2 filter. However, premailer hangs indefinitely (left it running overnight to see if it was just slow--nope). If I empty out the relevant css file, it still hangs (so it's not an issue of bad css). If I make the filter function just return its input, it works (so it's not an issue of getting the filter wrong in jinja2).

The weird thing is, if I run premailer from the command line entry point, it works! And quickly. l tried to look through the source code to see if there are different defaults, but nothing I tried made a difference.

Any ideas?

Edit: It was hanging when premailer requested the css file, so the problem was that flask couldn't handle one request in the middle of handling another. Increasing the gunicorn workers fixed it. Not sure if there's a solution on the development server.

The best practices way of handling this is avoiding Flask/Django/whatever dev servers and using a Procfile with honcho. Then you can run in the same environment locally as you do in staging and production.

For example you can have a Procfile that looks like:

code:

web: gunicorn rapp_proc.wsgi --log-file -
# you'd also have the other processes your application may require like background workers, but it's not required:
worker: rqworker high default low

Whether you're developing locally or running in production you run honcho start to start up your app. In production gunicorn is probably behind nginx, locally thats not required. The key is that you're developing in the same environment that your production is running in.

# ? Apr 10, 2017 20:15

SurgicalOntologist: Jun 17, 2004

Is there any way to get Flask debug mode back with this setup (auto-reload, tracebacks, console in browser)?

# ? Apr 10, 2017 20:24

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

No. It might work if you set gunicorn to use gevent workers, but I wouldn't hold my breath.

# ? Apr 10, 2017 21:12

Dominoes: Sep 20, 2007

Hey dudes - running into issues with node modules on Heroku. In order to get the node buildpack to use npm/yarn to download the libs I want, package.json needs to be in the same directory as the gitrepo, per Heroku docs. It's not there, since it's in my static directory (ie a subfolder), along with all the other front-end files. How should I handle this? (Other than committing my node_modules folder)

# ? Apr 10, 2017 21:46

darthbob88: Oct 13, 2011; YOSPOS

Is there a good, responsive, way to get an element with a background-image overflow its element, specifically full-browser-window-width? I've been using this method, with margin and padding set to 250px, but on browsers smaller than 1920x1080, that added 250px of padding causes the element to overflow the window. There's also this CSS3 method, but that runs into the same problem. Should I just use jQuery to calculate the necessary size of the margins?

# ? Apr 10, 2017 22:02

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Dominoes posted:

Hey dudes - running into issues with node modules on Heroku. In order to get the node buildpack to use npm/yarn to download the libs I want, package.json needs to be in the same directory as the gitrepo, per Heroku docs. It's not there, since it's in my static directory (ie a subfolder), along with all the other front-end files. How should I handle this? (Other than committing my node_modules folder)

Add another package.json at the top level with something like this in it:

JavaScript code:

{
  "scripts": {
    "postinstall": "cd static && yarn install"
  }
}

edit: fixed my bad formatting from phone-posting

Thermopyle fucked around with this message at 15:06 on Apr 11, 2017

# ? Apr 10, 2017 23:19

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

I'm starting a new project that will only be used in our company for at least a year before eventually being released into the wild and making me billions and billions of :tenbux:.

I know all of our in-house devices are using browsers that support CSS Grid. Anyone have any thoughts about whether I can expect browser support in the general public to be good enough in a year when we go public with the product? Obviously that depends on the target audience (in our case that'd be small business owners), but I'm interested in any general thoughts on the matter too.

# ? Apr 11, 2017 15:05

Munkeymon: Aug 14, 2003; Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.

Thermopyle posted:

I'm starting a new project that will only be used in our company for at least a year before eventually being released into the wild and making me billions and billions of :tenbux:.

I know all of our in-house devices are using browsers that support CSS Grid. Anyone have any thoughts about whether I can expect browser support in the general public to be good enough in a year when we go public with the product? Obviously that depends on the target audience (in our case that'd be small business owners), but I'm interested in any general thoughts on the matter too.

Doubt it very much. Small business are probably less likely to upgrade hardware and/or operating systems than regular people.

# ? Apr 11, 2017 15:45

kedo: Nov 27, 2007

Thermopyle posted:

I'm starting a new project that will only be used in our company for at least a year before eventually being released into the wild and making me billions and billions of :tenbux:.

I know all of our in-house devices are using browsers that support CSS Grid. Anyone have any thoughts about whether I can expect browser support in the general public to be good enough in a year when we go public with the product? Obviously that depends on the target audience (in our case that'd be small business owners), but I'm interested in any general thoughts on the matter too.

Doubtful imo. If you're supporting anything produced by Microsoft, even Edge, you have at least a few years before you'll be able to use it without fallbacks.

Heck, if you're coding for IE even flexbox isn't fully supported.

# ? Apr 11, 2017 16:26

SurgicalOntologist: Jun 17, 2004

Another weird Flask deployment bug.

I'm using flask-user, and I'm overriding the default email templates by supplying my own on the same paths. This works fine in development, but in production (Heroku) it is falling back to the templates provided by flask-user. I've definitely got the same versions of all the packages, and there are no config differences that could be in any way relevant (just DB and redis settings).

It's also weird because I have some views for testing out the look of the emails, and they work fine:

Python code:

@mail_examples.route('/confirm_email')
def confirm_email():
    return render_template(
        current_app.config['USER_CONFIRM_EMAIL_EMAIL_TEMPLATE'] + '_message.html',
        confirm_email_link='EXAMPLE_CONFIRM_EMAIL_LINK',
    )

How could it be that this is working but Flask-User is using the wrong templates to send emails?

# ? Apr 11, 2017 17:49

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

SurgicalOntologist posted:

Another weird Flask deployment bug.

I'm using flask-user, and I'm overriding the default email templates by supplying my own on the same paths. This works fine in development, but in production (Heroku) it is falling back to the templates provided by flask-user. I've definitely got the same versions of all the packages, and there are no config differences that could be in any way relevant (just DB and redis settings).

It's also weird because I have some views for testing out the look of the emails, and they work fine:
Python code:
@mail_examples.route('/confirm_email')
def confirm_email():
    return render_template(
        current_app.config['USER_CONFIRM_EMAIL_EMAIL_TEMPLATE'] + '_message.html',
        confirm_email_link='EXAMPLE_CONFIRM_EMAIL_LINK',
    )
How could it be that this is working but Flask-User is using the wrong templates to send emails?

I don't really know, but my first suspicion is something with Heroku and static files. I don't recall deploying Flask to Heroku before, but that class of errors is something you can encounter if you're not familiar with how to do static files on Heroku.

# ? Apr 11, 2017 18:20

SurgicalOntologist: Jun 17, 2004

Oh yeah, I've used Flask-S3 in the past but I didn't bother this time. I should probably set that up. Thanks for the tip.

# ? Apr 11, 2017 18:50

Dominoes: Sep 20, 2007

Thermopyle posted:

Add another package.json at the top level with something like this in it:
JavaScript code:
{
  "scripts": {
    "postinstall": "cd static && yarn install"
  }
}
edit: fixed my bad formatting from phone-posting

I was unable to get that working, but sorted it out by moving my JS config files (webpack.config.js, tsconfig.json, and package.json) and node_modules folder to my top level project directory, and pointed to the dist and src folders being subfolders. (ie tsconfig's outdir as '"outDir": "./myapp/static/dist", include as ""./myapp/static/src/**/*", and the same idea for webpack.config.js's entry and output sections.)

I'm 50/50 on weather I like this setup more. (Keeping it because it works; just commentary here) It puts the config at the proj's top level with related backend config files (like requirements.txt), but it loses having all the JS stuff together.

edit: While yarn/npm's working on Heroku, I can't get heroku to run webpack: 'Error: EROFS: read-only file system, mkdir '/app.' Note that I had to move the dev-only type dependencies to normal ones to get the build to start, which is probably not a best practice... although I'm sure committing the dist directory isn't either.

edit2: If I add '"postinstall": "webpack",' under scripts in my package.json, it runs, but fails; unsure why. Heroku might be detecting Typescript errors as reasons to fail the build? Seems to get around the read-only issue.

Dominoes fucked around with this message at 19:25 on Apr 11, 2017

# ? Apr 11, 2017 19:11

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

SurgicalOntologist posted:

Oh yeah, I've used Flask-S3 in the past but I didn't bother this time. I should probably set that up. Thanks for the tip.

You'll probably be fine with whitenoise instead and it doesn't require setting up S3 or anything else.

# ? Apr 11, 2017 19:39

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 18:07

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Dominoes posted:

I was unable to get that working, but sorted it out by moving my JS config files (webpack.config.js, tsconfig.json, and package.json) and node_modules folder to my top level project directory, and pointed to the dist and src folders being subfolders. (ie tsconfig's outdir as '"outDir": "./myapp/static/dist", include as ""./myapp/static/src/**/*", and the same idea for webpack.config.js's entry and output sections.)

I'm 50/50 on weather I like this setup more. (Keeping it because it works; just commentary here) It puts the config at the proj's top level with related backend config files (like requirements.txt), but it loses having all the JS stuff together.

edit: While yarn/npm's working on Heroku, I can't get heroku to run webpack: 'Error: EROFS: read-only file system, mkdir '/app.' Note that I had to move the dev-only type dependencies to normal ones to get the build to start, which is probably not a best practice... although I'm sure committing the dist directory isn't either.

edit2: If I add '"postinstall": "webpack",' under scripts in my package.json, it runs, but fails; unsure why. Heroku might be detecting Typescript errors as reasons to fail the build? Seems to get around the read-only issue.

Not that this helps you exactly with your current setup, but the way I'd be doing this is that all the webpack stuff would happen as a build step on my machine (well really my CI server) before I pushed to heroku and all heroku would know about is that I have a 'bundle.js' file in my static directory referenced in my HTML files.

# ? Apr 11, 2017 19:44

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > The Web Design & Development Megathread

«‹›361 »