Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Pollyanna
Mar 5, 2005

Milk's on them.


Is it worth testing Rake tasks? Mine's small enough that it's more easily tested via Rails console, but I don't wanna skirt the unit testing requirement on my team.

Adbot
ADBOT LOVES YOU

xtal
Jan 9, 2011

by Fluffdaddy

Pollyanna posted:

Is it worth testing Rake tasks? Mine's small enough that it's more easily tested via Rails console, but I don't wanna skirt the unit testing requirement on my team.

No, because the rake task should just be the entry point into the PORO that you do test.

Pollyanna
Mar 5, 2005

Milk's on them.


xtal posted:

No, because the rake task should just be the entry point into the PORO that you do test.

That makes sense. Where should these live? lib/tasks/helpers/?

xtal
Jan 9, 2011

by Fluffdaddy

Pollyanna posted:

That makes sense. Where should these live? lib/tasks/helpers/?

There's no convention here that I'm aware of. I would keep it in app/ somewhere.

Peristalsis
Apr 5, 2004
Move along.
I'm using the clockwork gem, connected to an ActiveRecord model. One requirement is to add method frequency to the AR model, "...returning the how frequently (in seconds) the database event should be run." This doesn't make any sense to me. I want the code to run exactly once - if I wanted to run multiple times, shouldn't I just have clock.rb send it multiple times? Isn't this mixing clockwork's scheduling responsibility with the application's responsibility to run actual code? Is there a frequency setting I can use just to run it one time total, or do I misunderstand this completely?

necrotic
Aug 2, 2005
I owe my brother big time for this!
Clockwork is for repeating tasks, not one offs. It's a cron system as a gem.

Use sidekiq or something similar for one-offs.

xtal
Jan 9, 2011

by Fluffdaddy
Clockwork also sucks, just use cron when you need cron

Gmaz
Apr 3, 2011

New DLC for Aoe2 is out: Dynasties of India
I usually use this gem when dealing with cron jobs:

https://github.com/javan/whenever

Makes it a little bit simpler than pure cron IMO

Peristalsis
Apr 5, 2004
Move along.
I think I'm going to take a whack at rufus scheduler. It seems simpler, and may just work, if the web server doesn't screw it up.

necrotic
Aug 2, 2005
I owe my brother big time for this!
Seriously, just use cron (through whenever if you want).

xtal
Jan 9, 2011

by Fluffdaddy

Peristalsis posted:

I think I'm going to take a whack at rufus scheduler. It seems simpler, and may just work, if the web server doesn't screw it up.

Can you tell us what you're trying to do? This is an XY problem

8ender
Sep 24, 2003

clown is watching you sleep
Yeah what the others said. Use cron, or if you want a Ruby DSL for cron, use the whenever gem

Peristalsis
Apr 5, 2004
Move along.

xtal posted:

Can you tell us what you're trying to do? This is an XY problem

Sorry - I guess I haven't posted anything about the problem itself in a whle. I'm writing an app to allow users to schedule text messages to be sent as part of a research study.

8ender posted:

Yeah what the others said. Use cron, or if you want a Ruby DSL for cron, use the whenever gem

necrotic posted:

Seriously, just use cron (through whenever if you want).

Cron and whenever (and clockwork) seem to be designed for repeated tasks. The actual text messages can't be assumed to be on a regular schedule, so I'm treating each message as a one-off event. I'm storing the message details in a database table, and I just need to schedule a time to send each message to the appropriate URL (and mark the message row as sent in the database).

I'd also prefer to keep this app as self-contained as possible. I don't know if I'll always be in charge of (or even involved with) deploying and running the app, and the simpler I can keep the instructions to get it up and running, the better. I wasn't very happy with clockwork and delayed_job requiring additional commands at the terminal prompt, and cron could potentially have a similar issue. I don't think anyone will try to run the app on Windows, but since I don't know for sure what OS or configuration will be on the server running this, and since I may not control the permissions of the user the app is running under, avoiding cron seems safer.

8ender
Sep 24, 2003

clown is watching you sleep

Peristalsis posted:

Cron and whenever (and clockwork) seem to be designed for repeated tasks. The actual text messages can't be assumed to be on a regular schedule, so I'm treating each message as a one-off event. I'm storing the message details in a database table, and I just need to schedule a time to send each message to the appropriate URL (and mark the message row as sent in the database).

I think you might be overthinking it, as that still seems like a cron like task, especially if you want this to scale properly. You're going to want to store those messages and their timestamps, then run something at an interval (Every hour, every 10 minutes, depends on how vital timeliness is) that will batch send the messages that need to go out.

If you need to avoid cron then Clockwork (or maybe Recurrent) are probably your only options. However if someone at your company is going to try to run a production Rails app on something that isn't *nix based then they have a much larger problem beyond the existence of cron.

EVGA Longoria
Dec 25, 2005

Let's go exploring!

Peristalsis posted:

Sorry - I guess I haven't posted anything about the problem itself in a whle. I'm writing an app to allow users to schedule text messages to be sent as part of a research study.



Cron and whenever (and clockwork) seem to be designed for repeated tasks. The actual text messages can't be assumed to be on a regular schedule, so I'm treating each message as a one-off event. I'm storing the message details in a database table, and I just need to schedule a time to send each message to the appropriate URL (and mark the message row as sent in the database).

I'd also prefer to keep this app as self-contained as possible. I don't know if I'll always be in charge of (or even involved with) deploying and running the app, and the simpler I can keep the instructions to get it up and running, the better. I wasn't very happy with clockwork and delayed_job requiring additional commands at the terminal prompt, and cron could potentially have a similar issue. I don't think anyone will try to run the app on Windows, but since I don't know for sure what OS or configuration will be on the server running this, and since I may not control the permissions of the user the app is running under, avoiding cron seems safer.

Store the events in something like redis or a sql server with a time stamp including when they should go out.

Write a rake task that will run through that db and send the messages.

Use cron to run that task every 5 minutes or whatever, depending on your window.

Windows has scheduled tasks which can do the same.

Triggering an event x time later where x is variable is brittle within a single process so don't try it.

This isn't a new problem, this is the only reasonable way to do it. Having a scheduled task to go with your service is not some horrendous thing to be avoided. Make sure it's documented, write up a .sh and a .bat file for Linux/Windows that runs the rake task, and wash your hands.

EVGA Longoria fucked around with this message at 13:17 on Oct 24, 2016

xtal
Jan 9, 2011

by Fluffdaddy
Or use Sidekiq instead of building a worse version of it yourself. If you're using Rails (as opposed to Ruby) ActiveJob is intended for this.

xtal fucked around with this message at 13:51 on Oct 24, 2016

kayakyakr
Feb 16, 2004

Kayak is true
Sorry to not have actually read your initial post. You don't want a scheduler, you want a queue/worker. Sidekiq is the dominant queue right now. Write it as an ActiveJob job and make it back-end independent so you can start it via suckerpunch during development and switch to something real for scale.

EVGA Longoria
Dec 25, 2005

Let's go exploring!

xtal posted:

Or use Sidekiq instead of building a worse version of it yourself. If you're using Rails (as opposed to Ruby) ActiveJob is intended for this.

Had never actually used the delayed part of Sidekiq so I forgot about it. Yes, use Sidekiq.

Peristalsis
Apr 5, 2004
Move along.
I really appreciate all the input, and I promise I'm taking it seriously, not just arguing with you all. I'll look into the paths you suggested, but I do want to understand this a little better.

If I were to use cron for the scheduling, I'll need to have some API in the system for cron to interact with, right? So there would be a cron job that pings https://myserver/myapp/ping every minute, and that route tells the app to check for any messages that need to be sent (and send them)? -- Answered below.

8ender posted:

I think you might be overthinking it,

That's a specialty of mine.

8ender posted:

as that still seems like a cron like task, especially if you want this to scale properly. You're going to want to store those messages and their timestamps, then run something at an interval (Every hour, every 10 minutes, depends on how vital timeliness is) that will batch send the messages that need to go out.

If you need to avoid cron then Clockwork (or maybe Recurrent) are probably your only options. However if someone at your company is going to try to run a production Rails app on something that isn't *nix based then they have a much larger problem beyond the existence of cron.

EVGA Longoria posted:

Store the events in something like redis or a sql server with a time stamp including when they should go out.

Write a rake task that will run through that db and send the messages.

I'm not familiar with redis - what's the advantage of using that, instead of just storing the texts in an ActiveRecord model?

EVGA Longoria posted:

Use cron to run that task every 5 minutes or whatever, depending on your window.

Windows has scheduled tasks which can do the same.

Triggering an event x time later where x is variable is brittle within a single process so don't try it.

I intended to schedule things to run at a specific time, not after x minutes - is that also brittle? Also, is the brittleness due to the fact that the single process may be busy with other things, and get to the scheduled event late, or something else I haven't considered? Many of the gems/approaches I've looked at use distinct threads for each scheduled task - does that reduce the brittleness, or just make it harder to debug when it does break down?


To summarize, you're all saying that trying to schedule each task internally with rufus is just too complicated/brittle, and I should, instead, use cron (or clockwork if necessary) to periodically, externally trigger the system to do a check for what needs to be sent and send it. That seems reasonable. What's the smallest time interval I can reasonably set cron for? The users have said they want the texts sent within a minute of the scheduled time, but I don't know if they have a reason for that, or just pulled a number out of thin air.

xtal posted:

Or use Sidekiq instead of building a worse version of it yourself. If you're using Rails (as opposed to Ruby) ActiveJob is intended for this.

I am using Rails, and I discounted Sidekiq because it seemed like overkill* - much more involved than the combination of clockwork and delayed_job I implemented. It also wasn't clear that it offered the kind of scheduling I thought I wanted.

ActiveJob is interesting, though. It looks like it's just a standardized front-end for a bunch of different queuing systems. Is the point to be able to switch between queuing systems painlessly, or does it give a better abstraction and/or add functionality?

kayakyakr posted:

You don't want a scheduler, you want a queue/worker. Sidekiq is the dominant queue right now. Write it as an ActiveJob job and make it back-end independent so you can start it via suckerpunch during development and switch to something real for scale.

I sort of agree, but I do need a way to send the texts on a schedule. That's why I currently have a combination of delayed_job and clockwork (though I found clockwork expects to have recurring jobs, rather than a bunch of single jobs). Does Sidekiq offer some way to schedule jobs on the queue? It's likely that I'll have a lot of jobs entered at once, for texts to be sent over several weeks, so I don't think I want a queue that has to loop over every job every minute, looking for any that happen to be ready to send. If I'm going to do that, I might as well just use cron with a rake task. Also, does Sidekiq expect recurring jobs, or is it okay with a bunch of one-off tasks? For a small app, does it offer any real advantages over delayed_job?


* This is a small application for a small research lab that just needs to send and track some text messages for a study, using an API from TextMarks. It probably could have been done with a set of CLI tools and maybe some cron jobs, but it's a good use case for a simple web app. I also suspect the PI has overestimated the ability and willingness of his students and staff to interact with the Linux command line effectively, so I talked them into a web interface for the whole thing. I'm just trying to do a good job on it so I can use it as part of a portfolio if/when I end up looking for a new job. It doesn't need to be all things to all people, but I would like it to demonstrate good programing practices.

Peristalsis
Apr 5, 2004
Move along.
I didn't see this post until after I posted my reply.

EVGA Longoria posted:

Had never actually used the delayed part of Sidekiq so I forgot about it. Yes, use Sidekiq.

Okay, I'll definitely look at Sidekiq, since it sounds like it has the scheduling bit I need, and almost seems like the consensus at this point. I trust this somehow avoids the single-process brittleness you described in your previous post?

xtal
Jan 9, 2011

by Fluffdaddy
Redis is used because it tends to be faster than relational databases for small, temporary messages. delayed_job is one queue adapter that uses ActiveRecord but it's fallen out of favour since Redis caught on. I think there are arguments not to use Redis, but they are arguments in favour of RabbitMQ or a more robust broker.

There are a bunch of reasons why it's a bad idea to use background threads in your web processes. That's not even on the table. Worker processes are what you want.

xtal fucked around with this message at 17:20 on Oct 24, 2016

Chilled Milk
Jun 22, 2003

No one here is alone,
satellites in every home

Peristalsis posted:

ActiveJob is interesting, though. It looks like it's just a standardized front-end for a bunch of different queuing systems. Is the point to be able to switch between queuing systems painlessly, or does it give a better abstraction and/or add functionality?
Yeah, both actually.

If amount of jobs is pretty low it doesn't really matter which backend you use, I default to delayed_job just to avoid having to set up redis right off the bat when it might not be needed. ActiveJob will make it easy to step up to Sidekiq if it turns out you need it, since your Job classes won't change.

And yes, you do want a separate worker process instead of some background thread. Maybe I missed something in my skimming but I don't see why this has to be more complicated than

SendTextJob.set(wait_until: @message.delivery_time).perform_later(@message)

Aside from the standard caveats about scheduling future jobs. But again, this seems like a simple use case?

edit: I'd also look at the service you're using to send the SMS and make sure they don't have a scheduling system in their API, that might be something they can do on their end.

Chilled Milk fucked around with this message at 19:37 on Oct 24, 2016

prom candy
Dec 16, 2005

Only I may dance
What are you guys using for end-to-end/acceptance testing? I'm finally working on more mission critical applications and need to get my poo poo together vis a vis testing and deployment. I usually do TDD unit testing because it's just a faster way to code for me but I mostly don't test anything besides the POROs that do the heavy lifting business logic in my apps. I need some to end-to-end tests to make sure that changing Feature B didn't break Feature J. Is rspec + capybara + capybara-webkit still a good choice? Is bitbucket pipelines worth looking into? Can/should I even run my end-to-end tests via the Rails stack if my javascript is being compiled and served with webpack-dev-server or do I need to set up a an actual web server and then use a different stack to test against it?

xtal
Jan 9, 2011

by Fluffdaddy
What you say is correct and stuff but I think if you're using webpack (or not the asset pipeline) you're at the point where you can fully separate your server and client. Test the Rails side by simulating the requests and responses. In an entirely separate repo build your web app and test it in a way that is recommended in that ecosystem, treating the API as a black box.

What I'm saying is that your app is no longer a Rails app. Capybara is a good approach but there might be better approaches if you consider this a JavaScript project.

What's so mission critical about it?

xtal fucked around with this message at 17:45 on Oct 25, 2016

prom candy
Dec 16, 2005

Only I may dance
My Rails app isn't completely separate from my web front end. Rails is still serving views, just some views are very very React heavy so using the asset pipeline went out the window. At some point I'd like to do as you say and just have Rails responsible for API only but we're not there yet, my team has way more Rails experience than Javascript experience and working at an agency often means using the tool you know how to use even if it's not necessarily the most correct one anymore.

Even if it was running as two separate projects there must be some way to do end-to-end testing that tests the front-end and the API at the same time? Otherwise you run the risk of having a passing test on the front-end, a passing test on the back-end, and some kind of stupid uncaught error that happens in communication (eg. you change a parameter name on the back-end, update the back-end tests, but don't change it on the front-end or update the front-end tests).

As far as what's mission critical about it, nothing exciting, it's just a business product that's very important to a small number of people.

EVGA Longoria
Dec 25, 2005

Let's go exploring!

prom candy posted:

What are you guys using for end-to-end/acceptance testing? I'm finally working on more mission critical applications and need to get my poo poo together vis a vis testing and deployment. I usually do TDD unit testing because it's just a faster way to code for me but I mostly don't test anything besides the POROs that do the heavy lifting business logic in my apps. I need some to end-to-end tests to make sure that changing Feature B didn't break Feature J. Is rspec + capybara + capybara-webkit still a good choice? Is bitbucket pipelines worth looking into? Can/should I even run my end-to-end tests via the Rails stack if my javascript is being compiled and served with webpack-dev-server or do I need to set up a an actual web server and then use a different stack to test against it?

If you can't get the whole shebang via rails s, don't try to test it through rails.

If you're going to use rails, rspec + capybara + poltergeist is what we run with and it's pretty solid since they upgraded to support phantomjs 2. Capybara-webkit requires an X server and the Qt library so it's a bit more to setup.

prom candy
Dec 16, 2005

Only I may dance
I'm using a bit of a frankenstein setup. In development some assets are served via the asset pipeline (eg. things I might have to pull in from a gem) and then the rest is served through webpack-dev-server. In production webpack builds its assets and then Rails reads those in during rake assets:precompile and makes the usual Rails bundle. It's kind of cumbersome but the dev experience with hot reloading is pretty nice.

Chilled Milk
Jun 22, 2003

No one here is alone,
satellites in every home
Seems like you're using Webpack for more than this, but I can say we use this method along with sassc-rails to get fast-ish live reloads within the asset pipeline, and it works well enough. A little clunky but I've automated it as part of our Rails project bootstrapping.

prom candy
Dec 16, 2005

Only I may dance
That's pretty cool, but yeah we're using webpack for building JavaScript with es6 and a proper module system and everything. I should really just embrace react-router and get Rails out of my front end but I have 10 years of rails helpers and conveniences and little tricks up my sleeve so it's hard to give up the development pace. I did a couple projects with Ember as the front end and Rails as API only and holy cow you forget how easy Rails makes certain tasks.

EVGA Longoria
Dec 25, 2005

Let's go exploring!

prom candy posted:

I'm using a bit of a frankenstein setup. In development some assets are served via the asset pipeline (eg. things I might have to pull in from a gem) and then the rest is served through webpack-dev-server. In production webpack builds its assets and then Rails reads those in during rake assets:precompile and makes the usual Rails bundle. It's kind of cumbersome but the dev experience with hot reloading is pretty nice.

Yeah, that makes me say your testing is gonna be a bad time.

Do you hit any APIs from Rails, or is it all ActiveRecord? If it's the former, I'd say do your testing in Ruby because it's easier to inject Webmocks so you don't depend on external services. If the latter, I'd say write it in whatever you're most comfortable with as a standalone process that starts rails in test mode.

prom candy
Dec 16, 2005

Only I may dance
Like third party APIs? Yeah, we're integrated with quite a few. I guess maybe testing the back-end and the front-end separately is the only real way this will work. The good news is I'll have some good blog post fodder if I get this figured out.

I'm hoping the rumours are true and 5.1 is going to bring more official integration with alternate asset building tools. Webpack is just so far beyond the asset pipeline now, and something will probably make webpack look like a pile of puke in the next year or two.

EVGA Longoria
Dec 25, 2005

Let's go exploring!

prom candy posted:

Like third party APIs? Yeah, we're integrated with quite a few. I guess maybe testing the back-end and the front-end separately is the only real way this will work. The good news is I'll have some good blog post fodder if I get this figured out.

I'm hoping the rumours are true and 5.1 is going to bring more official integration with alternate asset building tools. Webpack is just so far beyond the asset pipeline now, and something will probably make webpack look like a pile of puke in the next year or two.

I mean, you can totally precompile assets in tests. Honestly, I'd recommend it for speed purposes.

Ruby code:
RSpec.configure do |config|
  unless ENV['DONT_PRECOMPILE_TEST_ASSETS']
    config.before(:suite) { `bundle exec rake assets:precompile > /dev/null 2>&1` }
    config.after(:suite) { `bundle exec rake assets:clobber > /dev/null 2>&1` }
  end
end
is from our test suite. You could easily add in the step that does the webpack compilation first.

(I just looked at that for the first time, ick. Gonna add some cleanup for that to the todo list)

prom candy
Dec 16, 2005

Only I may dance
Hey, that's a pretty great idea! Our production asset compile is pretty slow (need to tweak webpack, still a big noob with it) but ideally the end-to-end suite would just run on a CI server anyway.

8ender
Sep 24, 2003

clown is watching you sleep

The Milkman posted:

Seems like you're using Webpack for more than this, but I can say we use this method along with sassc-rails to get fast-ish live reloads within the asset pipeline, and it works well enough. A little clunky but I've automated it as part of our Rails project bootstrapping.

We had this running for a while but a lot of our devs didn't like it for some reason so for the ones who did we fell back to using the Livereload app for Mac, which seems to do the same thing without needing all the gems. That said our assets use @import a lot and the pipeline is slow under docker to refresh, but it works.

8ender
Sep 24, 2003

clown is watching you sleep
---

Peristalsis
Apr 5, 2004
Move along.
Our main application at work has some custom logging functionality. Specifically, we have a singleton logger instance that logs to our_app_name.log. It isn't used much, but I'm troubleshooting another logging issue today, and I'm curious about something. When the application is running in our staging environment, it has an instance of this singleton logger for this custom log file. When I run rails console in this environment, it creates its own singleton logger instance, but to the same log file. Is it a problem to have two processes writing to the same log file like this? If so, is there a nice, clean way to differentiate between application input and rails console input in the logging code? I don't mind redirecting rails console logging to a different file, if there's a good way to do so.

Pollyanna
Mar 5, 2005

Milk's on them.


I'm in charge of scoping out an upgrade from JRuby 1.7 to 9.1.5.0 for our Rails app, and I'm wondering - what I should be looking for in terms of regressions/upgrade breakage, and what do I do to find stuff like that outside of clicking around and making sure tests still pass/gems are up to date? Since Rails is a pretty predictable architecture, I don't really expect it to break much by upgrading JRuby, and so far it hasn't.

Pollyanna
Mar 5, 2005

Milk's on them.


Sorry to double post, but I've also got another question. We're gearing up to run some database cleaning/changes on our production DB, and we need a backout plan. We test-drove our initial plan on staging, by bringing the server down, dumping the database, making the changes, then wiping and recreating with the SQL dump and making sure the database was in the expected state. Now we're getting intermittent JDBC connection errors with no useful traces to speak of. :shepface:

How do you guys back up/restore your databases for situations like these? What are your backout plans?

xenilk
Apr 17, 2004

ERRYDAY I BE SPLIT-TONING! Honestly, its the only skill I got other than shooting the back of women and calling it "Editorial".

Pollyanna posted:

Sorry to double post, but I've also got another question. We're gearing up to run some database cleaning/changes on our production DB, and we need a backout plan. We test-drove our initial plan on staging, by bringing the server down, dumping the database, making the changes, then wiping and recreating with the SQL dump and making sure the database was in the expected state. Now we're getting intermittent JDBC connection errors with no useful traces to speak of. :shepface:

How do you guys back up/restore your databases for situations like these? What are your backout plans?

have you updated something other than JRuby? Seems like it might be a problem with your DB and not JRuby, but that's my shot in the dark haha

Adbot
ADBOT LOVES YOU

Pardot
Jul 25, 2001




Pollyanna posted:

How do you guys back up/restore your databases for situations like these? What are your backout plans?

Depends a lot on how much data you have, and how much downtime you can tolerate.

Simplest is pg_dump/pg_restore with a dump file. Be sure to use -Fc (almost always). This will take time proportional to data size, so might not be fast enough. For a faster but more complicated solution, create a replica, and pause or break replication right before the changes. If needed you can failover, with nearly no downtime. Caches will be cold, but that’s probably not the end of the world.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply