Ruby on Rails Love-In

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ruby on Rails Love-In

«‹›178 »

kayakyakr: Feb 16, 2004; Kayak is true

KoRMaK posted:

How would I do automated testing for javascript? I need to verify that bootstrap is working right after an upgrade and do regression testing, so that means making sure things like modals open and respond to input correctly.

Is there a way to do this in ruby or rails via automated testing? I can understand checking inputs/outputs on models and controllers and even views, but javascript seem like a can of worms. How do you describe what is acceptable passing criteria? Looking around, it seems like there are some options but nothing that is an internet favorite. I've spotted cucumber and jasmine. Are either of these great options, and if not is there something else that is?

Your best bet is Capybara with a JS driver (webkit or the newer poltergeist, which wasn't around last time I did JS testing). You can use Capybara with Test::Unit and Rspec, so it's ready to go into your test suite as-is.

https://github.com/jnicklas/capybara#drivers

# ? Sep 3, 2014 15:50

Adbot: ADBOT LOVES YOU

# ? May 29, 2024 13:34

plasticbugs: Dec 13, 2006; Special Batman and Robin

I'm running out of memory with a long-running process on my Heroku app (error code R14 & R15). I'm converting MP4s to MP3s with a worker -- I'm downloading the MP4 to the Heroku server, running the conversion with FFMPEG (this is what the worker is doing) and uploading the MP3 to Amazon S3. I have one instance where a user has 2000+ videos to convert to MP3. Every 50 or so files, I'd like to clear out the memory of temporary files by restarting the process. Can I just restart the worker and have it clear out the memory, or do I have to restart the whole app?

I'm planning to do something like this:

code:

heroku = Heroku::Client.new(ENV['APP_USERNAME'], ENV['APP_PASSWORD'])
heroku.ps_restart(ENV['APP_NAME'], :ps =>'worker.1')

# ? Sep 3, 2014 21:55

kayakyakr: Feb 16, 2004; Kayak is true

plasticbugs posted:

I'm running out of memory with a long-running process on my Heroku app (error code R14 & R15). I'm converting MP4s to MP3s with a worker -- I'm downloading the MP4 to the Heroku server, running the conversion with FFMPEG (this is what the worker is doing) and uploading the MP3 to Amazon S3. I have one instance where a user has 2000+ videos to convert to MP3. Every 50 or so files, I'd like to clear out the memory of temporary files by restarting the process. Can I just restart the worker and have it clear out the memory, or do I have to restart the whole app?

I'm planning to do something like this:
code:
heroku = Heroku::Client.new(ENV['APP_USERNAME'], ENV['APP_PASSWORD'])
heroku.ps_restart(ENV['APP_NAME'], :ps =>'worker.1')

Save money, switch to a standalone VPS (A $20 DO VPS will have a lot more power and space. A $10 DO VPS will have more space and similar power), add a large swapfile, and build better queuing behaviors.

# ? Sep 3, 2014 22:09

EVGA Longoria: Dec 25, 2005; Let's go exploring!

Can anyone comment on rails-api vs the full Rails experience?

I'm pulling out a Rails built API to be a smaller service. Because we're using Devise and Doorkeeper, it's a lot simpler to keep it on Rails vs splitting to Sinatra/Grape. Any benefits to paring it down to rails-api?

# ? Sep 3, 2014 23:40

plasticbugs: Dec 13, 2006; Special Batman and Robin

kayakyakr posted:

Save money, switch to a standalone VPS (A $20 DO VPS will have a lot more power and space. A $10 DO VPS will have more space and similar power), add a large swapfile, and build better queuing behaviors.

This is for a one-off site I built for free for a friend - he wants his YouTube channel as an RSS'd MP3 podcast feed. I might do a workaround by manually restarting the server to get it to churn through his ridiculous 2,000 YouTube videos -- next time, I'll probably experiment with VPS, but I do enjoy Heroku because I'm used to it and don't have to worry too much about setup.

That said, does anyone know if restarting the worker on Heroku clears out the memory or do I need to restart the whole app?

plasticbugs fucked around with this message at 00:24 on Sep 4, 2014

# ? Sep 4, 2014 00:10

kayakyakr: Feb 16, 2004; Kayak is true

plasticbugs posted:

That said, does anyone know if restarting the worker on Heroku clears out the memory or do I need to restart the whole app?

Why don't you try it and report back?

# ? Sep 4, 2014 00:17

Smol: Jun 1, 2011; Stat rosa pristina nomine, nomina nuda tenemus.

KoRMaK posted:

How would I do automated testing for javascript? I need to verify that bootstrap is working right after an upgrade and do regression testing, so that means making sure things like modals open and respond to input correctly.

Is there a way to do this in ruby or rails via automated testing? I can understand checking inputs/outputs on models and controllers and even views, but javascript seem like a can of worms. How do you describe what is acceptable passing criteria? Looking around, it seems like there are some options but nothing that is an internet favorite. I've spotted cucumber and jasmine. Are either of these great options, and if not is there something else that is?

What does your front-end stack look like? If you're no using some sort of data-binding there (i.e, your business logic is intertwined with DOM manipulation code), forget about JavaScript tests and go all in on browser level integration testing with some sort of selenium-based tool.

# ? Sep 4, 2014 00:52

MrDoDo: Jun 27, 2004; You better remember quick before we haul your sweet ass down to the precinct.

EVGA Longoria posted:

Can anyone comment on rails-api vs the full Rails experience?

I'm pulling out a Rails built API to be a smaller service. Because we're using Devise and Doorkeeper, it's a lot simpler to keep it on Rails vs splitting to Sinatra/Grape. Any benefits to paring it down to rails-api?

I like using it for APIs because out of the box it won't generate a bunch of view crap I won't use. Also just generally feeling like its a bit more speedy, though thats just anecdotal.

If you are building a straight JSON API I think its a pretty good way to go. It seems to cut out a lot of bloat that would be necessary in a full stack Rails app, while still keeping a lot of the baked in nice pieces of Rails that you would otherwise have to add to a Sinatra/Grape app (i.e Security)

# ? Sep 4, 2014 02:14

KoRMaK: Jul 31, 2012

Smol posted:

What does your front-end stack look like? If you're no using some sort of data-binding there (i.e, your business logic is intertwined with DOM manipulation code), forget about JavaScript tests and go all in on browser level integration testing with some sort of selenium-based tool.

Cool this looks good.

I think I can pull something off with jasmine too. I can conceivably enter the tests via the chrome console as a client and be able to watch most of the stuff I need I think, like events and data input.

I might give jasmine a try first and see how good or bad that goes and then fallback to selenium. Thanks for the tip.

# ? Sep 4, 2014 03:36

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

KoRMaK posted:

Cool this looks good.

I think I can pull something off with jasmine too. I can conceivably enter the tests via the chrome console as a client and be able to watch most of the stuff I need I think, like events and data input.

I might give jasmine a try first and see how good or bad that goes and then fallback to selenium. Thanks for the tip.

You'll probably find DalekJS a lot simpler than Selenium, for what it's worth.

# ? Sep 4, 2014 03:43

plasticbugs: Dec 13, 2006; Special Batman and Robin

kayakyakr posted:

Why don't you try it and report back?

Restarting the worker after x number of items have been transcoded seems to be freeing up memory. I'm not monitoring file sizes, so occasionally, I'll see a memory warning. But after the worker restarts, the warnings disappear.

I'm going to start playing around with deploying to a VPS and trying to get comfortable with it so I have options down the road. Thanks for the kick in the rear end!

# ? Sep 4, 2014 06:32

Molten Llama: Sep 20, 2006

While a VPS is often better suited to this sort of task than Heroku tends to be, it also wouldn't hurt to take a good look at your worker code.

R14 doesn't occur on a 1X dyno until you've broken 512MB. R15 shouldn't occur until you've bloated a 1X dyno all the way to 2.5GB.

ffmpeg doesn't demand much memory overhead for the kinds of conversions you're doing. If the worker's not running a full Rails stack, Ruby shouldn't be horrifyingly hungry either. Unless you're trying to run way, way too many conversions in parallel, you should be getting an R15 approximately never if you're cleaning up your file handles properly and not leaking memory elsewhere.

If the problem's in your code, it will only get worse on a VPS because nothing will ever step in to kill The Worker That Ate Manhattan (barring configuring some kind of monitoring and killing tool to do just that).

Molten Llama fucked around with this message at 08:23 on Sep 4, 2014

# ? Sep 4, 2014 08:20

Jaded Burnout: Jul 10, 2004

Molten Llama posted:

you should be getting an R15 approximately never if you're cleaning up your file handles properly and not leaking memory elsewhere.

This is my question too. If you're restarting the workers to clear memory after conversions are done that would surely indicate you're keeping around a reference to something post-conversion so it isn't being GC'd.

# ? Sep 4, 2014 08:54

plasticbugs: Dec 13, 2006; Special Batman and Robin

Arachnamus posted:

This is my question too. If you're restarting the workers to clear memory after conversions are done that would surely indicate you're keeping around a reference to something post-conversion so it isn't being GC'd.

Thanks for the additional info. I never did actually get an R15. After 2000 files, it may have eventually exceeded 2.5 GB, but I didn't let it run that long.

Restarting the process helped, but caused other problems because the app would forget where is was in the encoding queue.

Because this is a one-off for a friend, I created a simple work around for that and pray that my one user doesn't attempt to record 2000 more youtube videos and upload them all at once.

# ? Sep 4, 2014 17:22

Lexicon: Jul 29, 2003; I had a beer with Stephen Harper once and now I like him.

This is an embarrassing question for how long I've messed around with Rails, but here goes anyway: how does a particular rails instance know what its environment is (i.e. the response to Rails.env)? I assume it's in an environment variable, but what are the conventions around how this is set?

# ? Sep 4, 2014 17:31

Jaded Burnout: Jul 10, 2004

Lexicon posted:

This is an embarrassing question for how long I've messed around with Rails, but here goes anyway: how does a particular rails instance know what its environment is (i.e. the response to Rails.env)? I assume it's in an environment variable, but what are the conventions around how this is set?

The environment variable RAILS_ENV is the most common route. This is usually set in Nginx, Passenger, etc.

# ? Sep 4, 2014 17:58

Lexicon: Jul 29, 2003; I had a beer with Stephen Harper once and now I like him.

Arachnamus posted:

The environment variable RAILS_ENV is the most common route. This is usually set in Nginx, Passenger, etc.

Ah ok. And so locally the absence of that being set defaults to development then I'm guessing?

# ? Sep 4, 2014 17:59

Molten Llama: Sep 20, 2006

Lexicon posted:

Ah ok. And so locally the absence of that being set defaults to development then I'm guessing?

Yep. I can't immediately find the code, but Rails basically loads with a ENV['RAILS_ENV'] ||= 'development'.

Edit: Duh, RACK_ENV cuts through the clutter. Here's everywhere some piece of Rails initializes the environment. As you might have already figured out, this is why the first line of your test_helper or spec_helper explicitly sets the environment to test.

Molten Llama fucked around with this message at 19:37 on Sep 4, 2014

# ? Sep 4, 2014 18:43

Lexicon: Jul 29, 2003; I had a beer with Stephen Harper once and now I like him.

Perfect, thanks all!

# ? Sep 4, 2014 19:26

raej: Sep 25, 2003; "Being drunk is the worst feeling of all. Except for all those other feelings."

Well, after months of bothering everyone in this thread with off the wall questions, I've finally put my app out there:

http://www.alespace.com

There are still a few rough spots, but I'm still continually improving it.

It's a site where you can keep track of beers in your cellar. On top of that, you can compile a list of "In Search Of" or ISO beers. There is a system for setting up trades with people as well, which matches up beers in their cellar with your ISO beers, and vice versa, making it easier to figure out what to offer and trade for.

If anyone is interested in helping out with this application, send me a PM. It would be much easier to move it forward with someone who has more experience with Rails than I do

# ? Sep 9, 2014 16:29

KoRMaK: Jul 31, 2012

I wanted to add some methods to a model that is coming from a gem. I'd like to keep the file in the models directory with the other models. Is there a way I can do that? I'd rather not have it in the lib or do the .class_eval do methodology (like this http://guides.rubyonrails.org/plugins.html#extending-core-classes)

# ? Sep 19, 2014 14:36

kayakyakr: Feb 16, 2004; Kayak is true

KoRMaK posted:

I wanted to add some methods to a model that is coming from a gem. I'd like to keep the file in the models directory with the other models. Is there a way I can do that? I'd rather not have it in the lib or do the .class_eval do methodology (like this http://guides.rubyonrails.org/plugins.html#extending-core-classes)

possible to derive the class that you actually use from the class coming from the gem?

# ? Sep 19, 2014 14:53

KoRMaK: Jul 31, 2012

kayakyakr posted:

possible to derive the class that you actually use from the class coming from the gem?

Maybe, what do you mean by derive?

It's this class that I want to add some stuff to https://github.com/airblade/paper_trail/blob/master/lib/paper_trail/frameworks/active_record/models/paper_trail/version.rb

code:

require 'paper_trail/version_concern'

module PaperTrail
  class Version < ::ActiveRecord::Base
    include PaperTrail::VersionConcern
  end
end

# ? Sep 19, 2014 15:05

kayakyakr: Feb 16, 2004; Kayak is true

If you were using the model directly inside your app, you just use a subclass of what you were wanting. But you're not doing that.

You could try monkeypatching the class inside your models and see if it loads, but rails/models works off autoload (meaning it'll try to load it the first time it's needed), so it might not ever load that particular class.

Something else you should consider is, if it's a feature or fix you're adding to the papertrail gem, maybe you should fork it and implement the feature in your own version of papertrail?

# ? Sep 19, 2014 15:31

xenilk: Apr 17, 2004; ERRYDAY I BE SPLIT-TONING! Honestly, its the only skill I got other than shooting the back of women and calling it "Editorial".

raej posted:

Well, after months of bothering everyone in this thread with off the wall questions, I've finally put my app out there:

http://www.alespace.com

There are still a few rough spots, but I'm still continually improving it.

It's a site where you can keep track of beers in your cellar. On top of that, you can compile a list of "In Search Of" or ISO beers. There is a system for setting up trades with people as well, which matches up beers in their cellar with your ISO beers, and vice versa, making it easier to figure out what to offer and trade for.

If anyone is interested in helping out with this application, send me a PM. It would be much easier to move it forward with someone who has more experience with Rails than I do

I get

code:

{
   "error": {
      "message": "Invalid redirect_uri: Given URL is not allowed by the Application configuration.",
      "type": "OAuthException",
      "code": 191
   }
}

When I sign in using facebook

# ? Sep 19, 2014 15:45

Jaded Burnout: Jul 10, 2004

KoRMaK posted:

I wanted to add some methods to a model that is coming from a gem. I'd like to keep the file in the models directory with the other models. Is there a way I can do that? I'd rather not have it in the lib or do the .class_eval do methodology (like this http://guides.rubyonrails.org/plugins.html#extending-core-classes)

Recommend you don't do that. Do this:

kayakyakr posted:

If you were using the model directly inside your app, you just use a subclass of what you were wanting.

If you can't do that, do this:

kayakyakr posted:

Something else you should consider is, if it's a feature or fix you're adding to the papertrail gem, maybe you should fork it and implement the feature in your own version of papertrail?

# ? Sep 19, 2014 20:19

KoRMaK: Jul 31, 2012

I monkey patched it. Wrote a lib, then added a call at the bottom to that did PaperTrail::Version.send(:include, MyExtensionStuff)

I am a bad person

# ? Sep 19, 2014 20:56

Jaded Burnout: Jul 10, 2004

KoRMaK posted:

I monkey patched it. Wrote a lib, then added a call at the bottom to that did PaperTrail::Version.send(:include, MyExtensionStuff)

I am a bad person

head straight to programmer jail do not write Go do not collect $200/day

# ? Sep 19, 2014 21:08

KoRMaK: Jul 31, 2012

kayakyakr posted:

Something else you should consider is, if it's a feature or fix you're adding to the papertrail gem, maybe you should fork it and implement the feature in your own version of papertrail?

I'm a software developer, not a god damned software engineer!

# ? Sep 19, 2014 21:11

Sil: Jan 4, 2007

KoRMaK posted:

I'm a software developer, not a god damned software engineer!

It's actually quite easy to do, though I found it super intimidating.

First time I ever did it was yesterday when I forked rails_admin. I wanted to run it with Rails 4.2 beta and needed to change a dependency version and 1 or 2 methods that were causing crashes. It's super straightforward, plus it lets you make much better use of bundle open, now that you can edit the gem and actually save the changes to github.

Question

Are there any good gems to help build CSV(or whatever format) downloads out of DB data? Currently I'm dealing with a legacy system that provides a custom filtering+checkboxes interface. It lets users select what columns they want to download and filter with a few hardcoded search boxes(ie, no introspection on the DB to dynamically generate filters).

The CSV can be as large as 80ish columns and 15k rows. It's using a few cache tables, but the whole process can still take 30-60 seconds. I would like to use something like the RailsAdmin CSV interface but that thing is slooooooooowwwww. Every attribute/cell goes through some ruby methods to make sure it wasn't decorated beyond the raw result of the sql query, this takes forever. It would probably take upwards of an hour to generate some of the larger download files. Ideally I'd like to be able to rapidly generate the CSV with a single sql statement. Then modify only some columns based on Ruby code. Not sure how I'd do that.

The benefit of it is that it lets me configure the download with ruby code. Right now the CSV columns are all defined using SQL SELECT AS statements. I'm talking like 200 LOC of just DB column to CSV column mapping and join logic all done in SQL. It's not as intimidating as it was the first time I read it, but I still think(hope?) that there's a better way to do it.

Any tips and/or recommendations or is this one of those grown up programmer 'figure it out yourself' moments?

# ? Sep 20, 2014 15:27

kayakyakr: Feb 16, 2004; Kayak is true

Sil posted:

It's actually quite easy to do, though I found it super intimidating.

First time I ever did it was yesterday when I forked rails_admin. I wanted to run it with Rails 4.2 beta and needed to change a dependency version and 1 or 2 methods that were causing crashes. It's super straightforward, plus it lets you make much better use of bundle open, now that you can edit the gem and actually save the changes to github.

Question

Are there any good gems to help build CSV(or whatever format) downloads out of DB data? Currently I'm dealing with a legacy system that provides a custom filtering+checkboxes interface. It lets users select what columns they want to download and filter with a few hardcoded search boxes(ie, no introspection on the DB to dynamically generate filters).

The CSV can be as large as 80ish columns and 15k rows. It's using a few cache tables, but the whole process can still take 30-60 seconds. I would like to use something like the RailsAdmin CSV interface but that thing is slooooooooowwwww. Every attribute/cell goes through some ruby methods to make sure it wasn't decorated beyond the raw result of the sql query, this takes forever. It would probably take upwards of an hour to generate some of the larger download files. Ideally I'd like to be able to rapidly generate the CSV with a single sql statement. Then modify only some columns based on Ruby code. Not sure how I'd do that.

The benefit of it is that it lets me configure the download with ruby code. Right now the CSV columns are all defined using SQL SELECT AS statements. I'm talking like 200 LOC of just DB column to CSV column mapping and join logic all done in SQL. It's not as intimidating as it was the first time I read it, but I still think(hope?) that there's a better way to do it.

Any tips and/or recommendations or is this one of those grown up programmer 'figure it out yourself' moments?

If you can, see if you can do any data transforms straight from the database so rails just has to pass through the csv...

Otherwise...

Here's an interesting one: use Rails 4's streaming responses so that you can process the data coming from the database in batches and write the result directly to the response stream as you go. Dunno how that would affect caching. It'd have the strength of limiting memory needed while you role through the db.

# ? Sep 20, 2014 17:32

Sil: Jan 4, 2007

kayakyakr posted:

If you can, see if you can do any data transforms straight from the database so rails just has to pass through the csv...

Otherwise...

Here's an interesting one: use Rails 4's streaming responses so that you can process the data coming from the database in batches and write the result directly to the response stream as you go. Dunno how that would affect caching. It'd have the strength of limiting memory needed while you role through the db.

Would having a super cache model/table dedicated to this exclusively make sense? Something like a SpreadsheetRow model that I just make sure gets updated by the relevant models in callbacks whenever they change? The download process right now still joins like half a dozen tables before it can even start filtering.

Part of the problem is that the code to generate the spreadsheet comes in at about 450 LOC and is totally entangled inside the Project model. Working to eliminate the God class problem at which point a more rational solution might reveal itself.

Also we don't use caching at all for this part of the application even though we obviously should

Probably just figuring out how to implement that would make the whole thing nice and quick.

Most of the data is transformed in the DB. The Ruby side just assembles the returned objects(Project instances extended with additional columns from other tables) into a CSV. I should probably also try getting the DB to output CSV directly to /tmp and just send that, rather than building the CSV in Ruby.

I'd like to use streaming to at least solve the occasional out of memory error I get on the tiny staging server we use for the app. Unfortunately the apps I work with are currently on 3.2. I'm planning on moving them to 4.2 sometime before the end of this year, but it's not a huge priority right now.

# ? Sep 20, 2014 19:49

Pollyanna: Mar 5, 2005; Milk's on them.

Generic Ruby question: I'm trying to make a Menu class to make putting together CLI menus significantly less of a pain in the rear end. The basic idea is that I pass in a hash of hashes, with the choice symbol associated with a hash of { :choice_text => "Choice text", :choice_action => choice_action() }. Right now, it looks something like this: