|
Is it worth testing Rake tasks? Mine's small enough that it's more easily tested via Rails console, but I don't wanna skirt the unit testing requirement on my team.
|
# ? Oct 17, 2016 18:54 |
|
|
# ? May 30, 2024 14:01 |
|
Pollyanna posted:Is it worth testing Rake tasks? Mine's small enough that it's more easily tested via Rails console, but I don't wanna skirt the unit testing requirement on my team. No, because the rake task should just be the entry point into the PORO that you do test.
|
# ? Oct 17, 2016 19:04 |
|
xtal posted:No, because the rake task should just be the entry point into the PORO that you do test. That makes sense. Where should these live? lib/tasks/helpers/?
|
# ? Oct 17, 2016 19:39 |
|
Pollyanna posted:That makes sense. Where should these live? lib/tasks/helpers/? There's no convention here that I'm aware of. I would keep it in app/ somewhere.
|
# ? Oct 17, 2016 19:55 |
|
I'm using the clockwork gem, connected to an ActiveRecord model. One requirement is to add method frequency to the AR model, "...returning the how frequently (in seconds) the database event should be run." This doesn't make any sense to me. I want the code to run exactly once - if I wanted to run multiple times, shouldn't I just have clock.rb send it multiple times? Isn't this mixing clockwork's scheduling responsibility with the application's responsibility to run actual code? Is there a frequency setting I can use just to run it one time total, or do I misunderstand this completely?
|
# ? Oct 23, 2016 04:35 |
|
Clockwork is for repeating tasks, not one offs. It's a cron system as a gem. Use sidekiq or something similar for one-offs.
|
# ? Oct 23, 2016 05:19 |
|
Clockwork also sucks, just use cron when you need cron
|
# ? Oct 23, 2016 13:53 |
|
I usually use this gem when dealing with cron jobs: https://github.com/javan/whenever Makes it a little bit simpler than pure cron IMO
|
# ? Oct 23, 2016 14:29 |
|
I think I'm going to take a whack at rufus scheduler. It seems simpler, and may just work, if the web server doesn't screw it up.
|
# ? Oct 23, 2016 17:40 |
|
Seriously, just use cron (through whenever if you want).
|
# ? Oct 23, 2016 18:33 |
|
Peristalsis posted:I think I'm going to take a whack at rufus scheduler. It seems simpler, and may just work, if the web server doesn't screw it up. Can you tell us what you're trying to do? This is an XY problem
|
# ? Oct 23, 2016 19:08 |
|
Yeah what the others said. Use cron, or if you want a Ruby DSL for cron, use the whenever gem
|
# ? Oct 24, 2016 03:23 |
|
xtal posted:Can you tell us what you're trying to do? This is an XY problem Sorry - I guess I haven't posted anything about the problem itself in a whle. I'm writing an app to allow users to schedule text messages to be sent as part of a research study. 8ender posted:Yeah what the others said. Use cron, or if you want a Ruby DSL for cron, use the whenever gem necrotic posted:Seriously, just use cron (through whenever if you want). Cron and whenever (and clockwork) seem to be designed for repeated tasks. The actual text messages can't be assumed to be on a regular schedule, so I'm treating each message as a one-off event. I'm storing the message details in a database table, and I just need to schedule a time to send each message to the appropriate URL (and mark the message row as sent in the database). I'd also prefer to keep this app as self-contained as possible. I don't know if I'll always be in charge of (or even involved with) deploying and running the app, and the simpler I can keep the instructions to get it up and running, the better. I wasn't very happy with clockwork and delayed_job requiring additional commands at the terminal prompt, and cron could potentially have a similar issue. I don't think anyone will try to run the app on Windows, but since I don't know for sure what OS or configuration will be on the server running this, and since I may not control the permissions of the user the app is running under, avoiding cron seems safer.
|
# ? Oct 24, 2016 05:37 |
|
Peristalsis posted:Cron and whenever (and clockwork) seem to be designed for repeated tasks. The actual text messages can't be assumed to be on a regular schedule, so I'm treating each message as a one-off event. I'm storing the message details in a database table, and I just need to schedule a time to send each message to the appropriate URL (and mark the message row as sent in the database). I think you might be overthinking it, as that still seems like a cron like task, especially if you want this to scale properly. You're going to want to store those messages and their timestamps, then run something at an interval (Every hour, every 10 minutes, depends on how vital timeliness is) that will batch send the messages that need to go out. If you need to avoid cron then Clockwork (or maybe Recurrent) are probably your only options. However if someone at your company is going to try to run a production Rails app on something that isn't *nix based then they have a much larger problem beyond the existence of cron.
|
# ? Oct 24, 2016 06:40 |
|
Peristalsis posted:Sorry - I guess I haven't posted anything about the problem itself in a whle. I'm writing an app to allow users to schedule text messages to be sent as part of a research study. Store the events in something like redis or a sql server with a time stamp including when they should go out. Write a rake task that will run through that db and send the messages. Use cron to run that task every 5 minutes or whatever, depending on your window. Windows has scheduled tasks which can do the same. Triggering an event x time later where x is variable is brittle within a single process so don't try it. This isn't a new problem, this is the only reasonable way to do it. Having a scheduled task to go with your service is not some horrendous thing to be avoided. Make sure it's documented, write up a .sh and a .bat file for Linux/Windows that runs the rake task, and wash your hands. EVGA Longoria fucked around with this message at 13:17 on Oct 24, 2016 |
# ? Oct 24, 2016 13:13 |
|
Or use Sidekiq instead of building a worse version of it yourself. If you're using Rails (as opposed to Ruby) ActiveJob is intended for this.
xtal fucked around with this message at 13:51 on Oct 24, 2016 |
# ? Oct 24, 2016 13:49 |
|
Sorry to not have actually read your initial post. You don't want a scheduler, you want a queue/worker. Sidekiq is the dominant queue right now. Write it as an ActiveJob job and make it back-end independent so you can start it via suckerpunch during development and switch to something real for scale.
|
# ? Oct 24, 2016 15:24 |
|
xtal posted:Or use Sidekiq instead of building a worse version of it yourself. If you're using Rails (as opposed to Ruby) ActiveJob is intended for this. Had never actually used the delayed part of Sidekiq so I forgot about it. Yes, use Sidekiq.
|
# ? Oct 24, 2016 15:57 |
|
I really appreciate all the input, and I promise I'm taking it seriously, not just arguing with you all. I'll look into the paths you suggested, but I do want to understand this a little better.8ender posted:I think you might be overthinking it, That's a specialty of mine. 8ender posted:as that still seems like a cron like task, especially if you want this to scale properly. You're going to want to store those messages and their timestamps, then run something at an interval (Every hour, every 10 minutes, depends on how vital timeliness is) that will batch send the messages that need to go out. EVGA Longoria posted:Store the events in something like redis or a sql server with a time stamp including when they should go out. I'm not familiar with redis - what's the advantage of using that, instead of just storing the texts in an ActiveRecord model? EVGA Longoria posted:Use cron to run that task every 5 minutes or whatever, depending on your window. I intended to schedule things to run at a specific time, not after x minutes - is that also brittle? Also, is the brittleness due to the fact that the single process may be busy with other things, and get to the scheduled event late, or something else I haven't considered? Many of the gems/approaches I've looked at use distinct threads for each scheduled task - does that reduce the brittleness, or just make it harder to debug when it does break down? To summarize, you're all saying that trying to schedule each task internally with rufus is just too complicated/brittle, and I should, instead, use cron (or clockwork if necessary) to periodically, externally trigger the system to do a check for what needs to be sent and send it. That seems reasonable. What's the smallest time interval I can reasonably set cron for? The users have said they want the texts sent within a minute of the scheduled time, but I don't know if they have a reason for that, or just pulled a number out of thin air. xtal posted:Or use Sidekiq instead of building a worse version of it yourself. If you're using Rails (as opposed to Ruby) ActiveJob is intended for this. I am using Rails, and I discounted Sidekiq because it seemed like overkill* - much more involved than the combination of clockwork and delayed_job I implemented. It also wasn't clear that it offered the kind of scheduling I thought I wanted. ActiveJob is interesting, though. It looks like it's just a standardized front-end for a bunch of different queuing systems. Is the point to be able to switch between queuing systems painlessly, or does it give a better abstraction and/or add functionality? kayakyakr posted:You don't want a scheduler, you want a queue/worker. Sidekiq is the dominant queue right now. Write it as an ActiveJob job and make it back-end independent so you can start it via suckerpunch during development and switch to something real for scale. I sort of agree, but I do need a way to send the texts on a schedule. That's why I currently have a combination of delayed_job and clockwork (though I found clockwork expects to have recurring jobs, rather than a bunch of single jobs). Does Sidekiq offer some way to schedule jobs on the queue? It's likely that I'll have a lot of jobs entered at once, for texts to be sent over several weeks, so I don't think I want a queue that has to loop over every job every minute, looking for any that happen to be ready to send. If I'm going to do that, I might as well just use cron with a rake task. Also, does Sidekiq expect recurring jobs, or is it okay with a bunch of one-off tasks? For a small app, does it offer any real advantages over delayed_job? * This is a small application for a small research lab that just needs to send and track some text messages for a study, using an API from TextMarks. It probably could have been done with a set of CLI tools and maybe some cron jobs, but it's a good use case for a simple web app. I also suspect the PI has overestimated the ability and willingness of his students and staff to interact with the Linux command line effectively, so I talked them into a web interface for the whole thing. I'm just trying to do a good job on it so I can use it as part of a portfolio if/when I end up looking for a new job. It doesn't need to be all things to all people, but I would like it to demonstrate good programing practices.
|
# ? Oct 24, 2016 16:35 |
|
I didn't see this post until after I posted my reply.EVGA Longoria posted:Had never actually used the delayed part of Sidekiq so I forgot about it. Yes, use Sidekiq. Okay, I'll definitely look at Sidekiq, since it sounds like it has the scheduling bit I need, and almost seems like the consensus at this point. I trust this somehow avoids the single-process brittleness you described in your previous post?
|
# ? Oct 24, 2016 16:48 |
|
Redis is used because it tends to be faster than relational databases for small, temporary messages. delayed_job is one queue adapter that uses ActiveRecord but it's fallen out of favour since Redis caught on. I think there are arguments not to use Redis, but they are arguments in favour of RabbitMQ or a more robust broker. There are a bunch of reasons why it's a bad idea to use background threads in your web processes. That's not even on the table. Worker processes are what you want. xtal fucked around with this message at 17:20 on Oct 24, 2016 |
# ? Oct 24, 2016 17:17 |
|
Peristalsis posted:ActiveJob is interesting, though. It looks like it's just a standardized front-end for a bunch of different queuing systems. Is the point to be able to switch between queuing systems painlessly, or does it give a better abstraction and/or add functionality? If amount of jobs is pretty low it doesn't really matter which backend you use, I default to delayed_job just to avoid having to set up redis right off the bat when it might not be needed. ActiveJob will make it easy to step up to Sidekiq if it turns out you need it, since your Job classes won't change. And yes, you do want a separate worker process instead of some background thread. Maybe I missed something in my skimming but I don't see why this has to be more complicated than SendTextJob.set(wait_until: @message.delivery_time).perform_later(@message) Aside from the standard caveats about scheduling future jobs. But again, this seems like a simple use case? edit: I'd also look at the service you're using to send the SMS and make sure they don't have a scheduling system in their API, that might be something they can do on their end. Chilled Milk fucked around with this message at 19:37 on Oct 24, 2016 |
# ? Oct 24, 2016 19:32 |
|
What are you guys using for end-to-end/acceptance testing? I'm finally working on more mission critical applications and need to get my poo poo together vis a vis testing and deployment. I usually do TDD unit testing because it's just a faster way to code for me but I mostly don't test anything besides the POROs that do the heavy lifting business logic in my apps. I need some to end-to-end tests to make sure that changing Feature B didn't break Feature J. Is rspec + capybara + capybara-webkit still a good choice? Is bitbucket pipelines worth looking into? Can/should I even run my end-to-end tests via the Rails stack if my javascript is being compiled and served with webpack-dev-server or do I need to set up a an actual web server and then use a different stack to test against it?
|
# ? Oct 25, 2016 14:33 |
|
What you say is correct and stuff but I think if you're using webpack (or not the asset pipeline) you're at the point where you can fully separate your server and client. Test the Rails side by simulating the requests and responses. In an entirely separate repo build your web app and test it in a way that is recommended in that ecosystem, treating the API as a black box. What I'm saying is that your app is no longer a Rails app. Capybara is a good approach but there might be better approaches if you consider this a JavaScript project. What's so mission critical about it? xtal fucked around with this message at 17:45 on Oct 25, 2016 |
# ? Oct 25, 2016 17:39 |
|
My Rails app isn't completely separate from my web front end. Rails is still serving views, just some views are very very React heavy so using the asset pipeline went out the window. At some point I'd like to do as you say and just have Rails responsible for API only but we're not there yet, my team has way more Rails experience than Javascript experience and working at an agency often means using the tool you know how to use even if it's not necessarily the most correct one anymore. Even if it was running as two separate projects there must be some way to do end-to-end testing that tests the front-end and the API at the same time? Otherwise you run the risk of having a passing test on the front-end, a passing test on the back-end, and some kind of stupid uncaught error that happens in communication (eg. you change a parameter name on the back-end, update the back-end tests, but don't change it on the front-end or update the front-end tests). As far as what's mission critical about it, nothing exciting, it's just a business product that's very important to a small number of people.
|
# ? Oct 25, 2016 19:32 |
|
prom candy posted:What are you guys using for end-to-end/acceptance testing? I'm finally working on more mission critical applications and need to get my poo poo together vis a vis testing and deployment. I usually do TDD unit testing because it's just a faster way to code for me but I mostly don't test anything besides the POROs that do the heavy lifting business logic in my apps. I need some to end-to-end tests to make sure that changing Feature B didn't break Feature J. Is rspec + capybara + capybara-webkit still a good choice? Is bitbucket pipelines worth looking into? Can/should I even run my end-to-end tests via the Rails stack if my javascript is being compiled and served with webpack-dev-server or do I need to set up a an actual web server and then use a different stack to test against it? If you can't get the whole shebang via rails s, don't try to test it through rails. If you're going to use rails, rspec + capybara + poltergeist is what we run with and it's pretty solid since they upgraded to support phantomjs 2. Capybara-webkit requires an X server and the Qt library so it's a bit more to setup.
|
# ? Oct 25, 2016 21:24 |
|
I'm using a bit of a frankenstein setup. In development some assets are served via the asset pipeline (eg. things I might have to pull in from a gem) and then the rest is served through webpack-dev-server. In production webpack builds its assets and then Rails reads those in during rake assets:precompile and makes the usual Rails bundle. It's kind of cumbersome but the dev experience with hot reloading is pretty nice.
|
# ? Oct 26, 2016 01:58 |
|
Seems like you're using Webpack for more than this, but I can say we use this method along with sassc-rails to get fast-ish live reloads within the asset pipeline, and it works well enough. A little clunky but I've automated it as part of our Rails project bootstrapping.
|
# ? Oct 26, 2016 03:59 |
|
That's pretty cool, but yeah we're using webpack for building JavaScript with es6 and a proper module system and everything. I should really just embrace react-router and get Rails out of my front end but I have 10 years of rails helpers and conveniences and little tricks up my sleeve so it's hard to give up the development pace. I did a couple projects with Ember as the front end and Rails as API only and holy cow you forget how easy Rails makes certain tasks.
|
# ? Oct 26, 2016 04:41 |
|
prom candy posted:I'm using a bit of a frankenstein setup. In development some assets are served via the asset pipeline (eg. things I might have to pull in from a gem) and then the rest is served through webpack-dev-server. In production webpack builds its assets and then Rails reads those in during rake assets:precompile and makes the usual Rails bundle. It's kind of cumbersome but the dev experience with hot reloading is pretty nice. Yeah, that makes me say your testing is gonna be a bad time. Do you hit any APIs from Rails, or is it all ActiveRecord? If it's the former, I'd say do your testing in Ruby because it's easier to inject Webmocks so you don't depend on external services. If the latter, I'd say write it in whatever you're most comfortable with as a standalone process that starts rails in test mode.
|
# ? Oct 26, 2016 13:12 |
|
Like third party APIs? Yeah, we're integrated with quite a few. I guess maybe testing the back-end and the front-end separately is the only real way this will work. The good news is I'll have some good blog post fodder if I get this figured out. I'm hoping the rumours are true and 5.1 is going to bring more official integration with alternate asset building tools. Webpack is just so far beyond the asset pipeline now, and something will probably make webpack look like a pile of puke in the next year or two.
|
# ? Oct 26, 2016 13:53 |
|
prom candy posted:Like third party APIs? Yeah, we're integrated with quite a few. I guess maybe testing the back-end and the front-end separately is the only real way this will work. The good news is I'll have some good blog post fodder if I get this figured out. I mean, you can totally precompile assets in tests. Honestly, I'd recommend it for speed purposes. Ruby code:
(I just looked at that for the first time, ick. Gonna add some cleanup for that to the todo list)
|
# ? Oct 26, 2016 17:31 |
|
Hey, that's a pretty great idea! Our production asset compile is pretty slow (need to tweak webpack, still a big noob with it) but ideally the end-to-end suite would just run on a CI server anyway.
|
# ? Oct 27, 2016 00:36 |
|
The Milkman posted:Seems like you're using Webpack for more than this, but I can say we use this method along with sassc-rails to get fast-ish live reloads within the asset pipeline, and it works well enough. A little clunky but I've automated it as part of our Rails project bootstrapping. We had this running for a while but a lot of our devs didn't like it for some reason so for the ones who did we fell back to using the Livereload app for Mac, which seems to do the same thing without needing all the gems. That said our assets use @import a lot and the pipeline is slow under docker to refresh, but it works.
|
# ? Oct 27, 2016 06:36 |
|
---
|
# ? Oct 27, 2016 06:44 |
|
Our main application at work has some custom logging functionality. Specifically, we have a singleton logger instance that logs to our_app_name.log. It isn't used much, but I'm troubleshooting another logging issue today, and I'm curious about something. When the application is running in our staging environment, it has an instance of this singleton logger for this custom log file. When I run rails console in this environment, it creates its own singleton logger instance, but to the same log file. Is it a problem to have two processes writing to the same log file like this? If so, is there a nice, clean way to differentiate between application input and rails console input in the logging code? I don't mind redirecting rails console logging to a different file, if there's a good way to do so.
|
# ? Oct 27, 2016 17:52 |
|
I'm in charge of scoping out an upgrade from JRuby 1.7 to 9.1.5.0 for our Rails app, and I'm wondering - what I should be looking for in terms of regressions/upgrade breakage, and what do I do to find stuff like that outside of clicking around and making sure tests still pass/gems are up to date? Since Rails is a pretty predictable architecture, I don't really expect it to break much by upgrading JRuby, and so far it hasn't.
|
# ? Nov 1, 2016 19:45 |
|
Sorry to double post, but I've also got another question. We're gearing up to run some database cleaning/changes on our production DB, and we need a backout plan. We test-drove our initial plan on staging, by bringing the server down, dumping the database, making the changes, then wiping and recreating with the SQL dump and making sure the database was in the expected state. Now we're getting intermittent JDBC connection errors with no useful traces to speak of. How do you guys back up/restore your databases for situations like these? What are your backout plans?
|
# ? Nov 3, 2016 15:31 |
|
Pollyanna posted:Sorry to double post, but I've also got another question. We're gearing up to run some database cleaning/changes on our production DB, and we need a backout plan. We test-drove our initial plan on staging, by bringing the server down, dumping the database, making the changes, then wiping and recreating with the SQL dump and making sure the database was in the expected state. Now we're getting intermittent JDBC connection errors with no useful traces to speak of. have you updated something other than JRuby? Seems like it might be a problem with your DB and not JRuby, but that's my shot in the dark haha
|
# ? Nov 3, 2016 23:00 |
|
|
# ? May 30, 2024 14:01 |
|
Pollyanna posted:How do you guys back up/restore your databases for situations like these? What are your backout plans? Depends a lot on how much data you have, and how much downtime you can tolerate. Simplest is pg_dump/pg_restore with a dump file. Be sure to use -Fc (almost always). This will take time proportional to data size, so might not be fast enough. For a faster but more complicated solution, create a replica, and pause or break replication right before the changes. If needed you can failover, with nearly no downtime. Caches will be cold, but that’s probably not the end of the world.
|
# ? Nov 4, 2016 11:27 |