Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Peristalsis
Apr 5, 2004
Move along.

The Milkman posted:

On the Datafile class, so that the console output is less verbose


Smol posted:

Just add "def inspect; "foo"; end" in DataFile and whatever other AR models you were having troubles with. It sounds like the REPL somehow hangs up when it tries to print the results of inspecting the object graph.

Big #inspect messages can be pretty broken especially when something throws a NameError.


necrotic posted:

Exactly what I meant with "It could be an issue "rendering" with the deep nesting". The fact that it runs fine in the script is indicative of this as well.

Okay, that makes sense. I added this to the Datafile class:
code:
def inspect
  return self.id
end
but executing Datafile.includes(:datafile_permissions) still freezes if not limited to less than 5400 rows.

HOWEVER, another developer suggested using Datafile.includes(:datafile_permissions).find_each, which iterates over each Datafile object (with its linked permissions) in batches of 1000, and at least allows me to build up the array I want. It's not ideal, since I have to manually build the resulting array in the for_each loop instead of just doing an array subtraction in one line, but it's still far better than the n+1 query issue we had before.

It really seems like a memory issue of some sort, it's just weird that increasing the available memory in the VM doesn't affect it, while batch loading and limiting the query both do avoid the problem. Does rails have some memory constraint beyond just using what's available in the VM? I'm also going to ask the other developer to load the same data into his dev environment, to see if he can reproduce the problem with it. For that matter, maybe I'll try to recreate it in a different VM image. I'm still open to ideas and suggestions, but for now I'm going to go with the find_each workaround (unless that's a horrible idea for some reason).

I appreciate all your time and effort on this.


Edit: I imported everything in to my VMWare image and ran the problematic includes command there on the same data without problems. So, there's something about my main dev environment's configuration that's causing the issue. I really doubt it's worth pursuing any more - isolating whether it's Vagrant, the VM image, or weirdness about some version of some part of what I have installed probably won't teach me enough to be worth spending the time to track it down.

Edit 2: If anyone is interested, the problem seems to be with the cmd.exe in Windows. I installed ConEmu, connected to Vagrant, fired up Rails Console, and Datafile.includes(:datafile_permissions) runs through without an execption. There must just be some limitation with the default console window that won't accept the large MySQL return.

Peristalsis fucked around with this message at 21:19 on Jan 12, 2016

Adbot
ADBOT LOVES YOU

Hadlock
Nov 9, 2004

I feel like I'm going about this the hard way. I want to spin up a ruby on rails app (ruby 2.x/rails 4.x) that accepts an API post, saves it to a database (sqllite3 is fine thanks) and then outputs the most recent api result to result.html. I have a sensor device that's going to push meeting room occupied status (room name, occupied status) back to the server. Perferably using some sort of manually configured auth token.

What project/gems am I looking at? I feel like this should be maybe easier than I'm making it out to be. I'm looking for either a rails for dummies type tutorial, or the 20 or so lines needed to accomplish this (so I can learn from it and build on top of it, rather than install and forget it)

aunt jenkins
Jan 12, 2001

I would use Sinatra with Sequel (since I'm presuming you aren't familiar with ActiveRecord yet) for this. Rails is way too much framework for what you're trying to do and you will spend more time trying to find which of the 45 auto-generated files you need to modify then actually programming. This should be about 20-30 lines tops with Sinatra, split across a main app.rb file and a view :)

Even if your overall objective is 'I wanna learn Rails' you'll do better by learning some Ruby syntax first and working with these gems, both of which are well-renowned for their elegant simplicity, will help set you up well to not have to learn both Ruby and Rails at once, which is a tough thing to do.

Buckhead
Aug 12, 2005

___ days until the 2010 trade deadline :(
I have two models, Users and Books. A user can have many books. I want to generate a count of the numbers of users that have at least five books.

What is the easiest way to do this with ActiveRecord? I'm sure there is an easy one liner, but right now I'm running a loop to get this number.

fantastic in plastic
Jun 15, 2007

The Socialist Workers Party's newspaper proved to be a tough sell to downtown businessmen.
code:
User.joins(:books).group("users.id").having("count(books.id) > 4").length
should do it.

KoRMaK
Jul 31, 2012



My xml and json builder files look exactly the same. How do I combine them into one file?

They both look like this, except swap out xml with json

code:
xml.my_model do
 fields_to_show.each do |f|
      xml.tag!(f.field_name) do 
        xml.tag!("field_name", t("my_model." + "#{f.field_name}"))
        xml.tag!("data", data)
    end
  end
end

Pollyanna
Mar 5, 2005

Milk's on them.


I've got kind of a weird use case I've been called in to fix that I'm having trouble fitting to ActiveRecord.

A major feature of the application I work on is the ability to define tables of arbitrary column length and data types. For example, we have one kind of table, which is referred to as a "tile", that's three columns long and has, in order: text, text, and datetime data. Much of the predefined features in our application are variations on this "schema" setup, e.g. announcements, project statuses, and the like. Data types range from text, to numerics, to pre-defined groups of options (enums), to datetimes, to URLs, to images. We also want the ability to perform operations on these tables of data, e.g. getting the sum total of a numeric type table column and changing the rendering of the column if it's above a certain limit. So, they have to be strongly typed and have to be cast to various formats.

Users can also make their own custom versions of these tables, and often do (I think, I don't actually know how much this feature is used). So, we also need to have an interface to define a schema of this kind and store it in the database with an associated table owner (the Board that table is on). Users can therefore make their own tables and schema, like in Excel.

I guess the question I have is: how do you record strongly typed, customizable and user-defined tables of arbitrary column length in ActiveRecord while still being able to perform operations on the data the table holds depending on each table column's data type? Basically, how do you recreate Excel in ActiveRecord?

kayakyakr
Feb 16, 2004

Kayak is true

Pollyanna posted:

I've got kind of a weird use case I've been called in to fix that I'm having trouble fitting to ActiveRecord.

A major feature of the application I work on is the ability to define tables of arbitrary column length and data types. For example, we have one kind of table, which is referred to as a "tile", that's three columns long and has, in order: text, text, and datetime data. Much of the predefined features in our application are variations on this "schema" setup, e.g. announcements, project statuses, and the like. Data types range from text, to numerics, to pre-defined groups of options (enums), to datetimes, to URLs, to images. We also want the ability to perform operations on these tables of data, e.g. getting the sum total of a numeric type table column and changing the rendering of the column if it's above a certain limit. So, they have to be strongly typed and have to be cast to various formats.

Users can also make their own custom versions of these tables, and often do (I think, I don't actually know how much this feature is used). So, we also need to have an interface to define a schema of this kind and store it in the database with an associated table owner (the Board that table is on). Users can therefore make their own tables and schema, like in Excel.

I guess the question I have is: how do you record strongly typed, customizable and user-defined tables of arbitrary column length in ActiveRecord while still being able to perform operations on the data the table holds depending on each table column's data type? Basically, how do you recreate Excel in ActiveRecord?

Postgresql's JSON implementation is quite good these days. Give it a look.

EVGA Longoria
Dec 25, 2005

Let's go exploring!

KoRMaK posted:

My xml and json builder files look exactly the same. How do I combine them into one file?

They both look like this, except swap out xml with json

code:

xml.my_model do
 fields_to_show.each do |f|
      xml.tag!(f.field_name) do 
        xml.tag!("field_name", t("my_model." + "#{f.field_name}"))
        xml.tag!("data", data)
    end
  end
end

You don't. Sprockets doesn't really work for one file being multiple templates.

Part of convention over configuration is that a new dev can pick up an app and immediately know where to look to change something. I get that it's duplicate code, but sometimes the disadvantages of deduping the code are higher than a little extra code to maintain.

It's a pet peeve of mine because my team is happy to jump the meta programming and some serious abstraction in the name of slavishly following code climate.

KoRMaK
Jul 31, 2012



EVGA Longoria posted:

It's a pet peeve of mine because my team is happy to jump the meta programming and some serious abstraction in the name of slavishly following code climate.
I can't follow this last part. Your team is happy to dump, as is - not do, meta programming and abstraction?

And can you explain what you mean by code climate?


I can't tell which side of this complaint I fall on :)

EVGA Longoria
Dec 25, 2005

Let's go exploring!

KoRMaK posted:

I can't follow this last part. Your team is happy to dump, as is - not do, meta programming and abstraction?

And can you explain what you mean by code climate?


I can't tell which side of this complaint I fall on :)

Jump to meta programming, as in do it. Code climate is a site that runs rubocop and other style tools on your code, and looks for supposed "code smells". One of the big ones is duplicate code, like having really similar code for two templates like your case.

My advice to you is just "don't worry about two templates being similar, it's part of builder templates". If I came into a rails project and the xml or json code was not in a template to avoid some minor duplication, I'd bash my head against the wall.

Chilled Milk
Jun 22, 2003

No one here is alone,
satellites in every home
Yeah, I'd say the cost of abstracting that out is much greater than living with the perceived duplication (especially for only two cases). I say perceived because just because something has a similar structure doesn't mean it violates DRY.

KoRMaK
Jul 31, 2012



EVGA Longoria posted:

Jump to meta programming, as in do it. Code climate is a site that runs rubocop and other style tools on your code, and looks for supposed "code smells". One of the big ones is duplicate code, like having really similar code for two templates like your case.

My advice to you is just "don't worry about two templates being similar, it's part of builder templates". If I came into a rails project and the xml or json code was not in a template to avoid some minor duplication, I'd bash my head against the wall.

Ohhhhh code climate the website. Ok, gotcha.


It's just so drat similar that I know its vulnerable to a mistake because one will get changed and the other won't get updated. But also, the tags are slightly different so they aren't exactly the same, so two templates it is.


When people quote code climate results at me I secretly roll my eyes because how big or complex of an app has that thing been run on? A twitter clone is way diff than some complicated business app. Like, is code climate actually useful or is it just a buzzword sorta silicon valley-ish thing?


The Milkman posted:

Yeah, I'd say the cost of abstracting that out is much greater than living with the perceived duplication (especially for only two cases). I say perceived because just because something has a similar structure doesn't mean it violates DRY.
What I'm doing is two cases times some_number_of_controllers. But they can all share those same two templates so I guess I'm good.

EVGA Longoria
Dec 25, 2005

Let's go exploring!

KoRMaK posted:

Ohhhhh code climate the website. Ok, gotcha.


It's just so drat similar that I know its vulnerable to a mistake because one will get changed and the other won't get updated. But also, the tags are slightly different so they aren't exactly the same, so two templates it is.


When people quote code climate results at me I secretly roll my eyes because how big or complex of an app has that thing been run on? A twitter clone is way diff than some complicated business app. Like, is code climate actually useful or is it just a buzzword sorta silicon valley-ish thing?

What I'm doing is two cases times some_number_of_controllers. But they can all share those same two templates so I guess I'm good.

Codeclimate is a legit idea, and you can definitely run it on big apps without problem. I don't have LoC or anything offhand but we run it on all of our repos for a fairly big site.

It's useful as a tool, but it's not very smart. For instance, I had it go off on one of my PRs recently for similar code in two files. The similar code was 2 'require' statements.

Give it a shot, just don't become slavishly devoted to it. A lot of my coworkers treat it like a religious authority. But a lot of them have bought into the cult of Ruby anyway.

KoRMaK
Jul 31, 2012



EVGA Longoria posted:

Codeclimate is a legit idea, and you can definitely run it on big apps without problem. I don't have LoC or anything offhand but we run it on all of our repos for a fairly big site.

It's useful as a tool, but it's not very smart. For instance, I had it go off on one of my PRs recently for similar code in two files. The similar code was 2 'require' statements.

Give it a shot, just don't become slavishly devoted to it. A lot of my coworkers treat it like a religious authority. But a lot of them have bought into the cult of Ruby anyway.
This


fe: whats LoC?

Hadlock
Nov 9, 2004

Line(s) of Code
:goonsay:

Read this:
https://en.wikipedia.org/wiki/The_Mythical_Man-Month

KoRMaK
Jul 31, 2012



Holy poo poo I'm a paragraph into the wiki page and I'm in love

Thanks, gonna order and expense this poo poo tomorrow

quote:

The Mythical Man-Month: Essays on Software Engineering is a book on software engineering and project management by Fred Brooks, whose central theme is that "adding manpower to a late software project makes it later".

The tendency for managers to repeat such errors in project development led Brooks to quip that his book is called "The Bible of Software Engineering", because "everybody quotes it, some people read it, and a few people go by it"

Hadlock
Nov 9, 2004

So I had a new problem crop up recently. The product originally started off as RoR on Postgres on a traditional ubuntu LTS machine. Now we're containerizing it.

We have our primary RoR app running in one docker container, and a delayed job worker running in a different container. This is a problem as the system was designed for the delayed job worker to build the massive PDF report and drop it on the file system for the primary app to pick up and serve out to the report requester.

The solution we (I?) came up with and handed off as the spec for the developer was for the primary RoR app to call an API on the delayed job worker, get a 200 ok back along with a UUID, and then start monitoring the new "binary pdf" table in the database for the new UUID row that they both share and look for a finished flag. Some of these reports can take more than 10 minutes to generate because our clients run the software across an absurd number of units.

Two questions:

a) why is this database method a bad idea
b) what is a better way to do this

Keep in mind the containers are effectively two separate VMs on the open internet.

KoRMaK
Jul 31, 2012



Hadlock posted:

So I had a new problem crop up recently. The product originally started off as RoR on Postgres on a traditional ubuntu LTS machine. Now we're containerizing it.

We have our primary RoR app running in one docker container, and a delayed job worker running in a different container. This is a problem as the system was designed for the delayed job worker to build the massive PDF report and drop it on the file system for the primary app to pick up and serve out to the report requester.

The solution we (I?) came up with and handed off as the spec for the developer was for the primary RoR app to call an API on the delayed job worker, get a 200 ok back along with a UUID, and then start monitoring the new "binary pdf" table in the database for the new UUID row that they both share and look for a finished flag. Some of these reports can take more than 10 minutes to generate because our clients run the software across an absurd number of units.

Two questions:

a) why is this database method a bad idea
b) what is a better way to do this

Keep in mind the containers are effectively two separate VMs on the open internet.

So you have two systems, one talking to another?

Hadlock
Nov 9, 2004

Yeah. Total unrestricted two way communication.

We discussed just feeding back the binary stream to the requesting server in the API response and saving that to the local file system, then serving that to the web client request, but decided it would eat up too much resources. Piping the PDF to a binary data type in the db and then piping it back out seemed more clean? Do you disagree? I'm curious about how we could have done this better. Some kind of secure unix socket over IP?

KoRMaK
Jul 31, 2012



I, unfortunately, agree with you so I can't offer much constructive criticism.


Doin' fine!

Gmaz
Apr 3, 2011

New DLC for Aoe2 is out: Dynasties of India

Hadlock posted:

We have our primary RoR app running in one docker container, and a delayed job worker running in a different container. This is a problem as the system was designed for the delayed job worker to build the massive PDF report and drop it on the file system for the primary app to pick up and serve out to the report requester.
Is there a specific reason why you're storing this report on the file system as opposed to some cloud storage?

EVGA Longoria
Dec 25, 2005

Let's go exploring!

Hadlock posted:

Yeah. Total unrestricted two way communication.

We discussed just feeding back the binary stream to the requesting server in the API response and saving that to the local file system, then serving that to the web client request, but decided it would eat up too much resources. Piping the PDF to a binary data type in the db and then piping it back out seemed more clean? Do you disagree? I'm curious about how we could have done this better. Some kind of secure unix socket over IP?

Please don't store binary data in a SQL database. It will end in tears.

Have the worker POST the file back to the web server and store it on a file system. Paperclip is a gem for taking file uploads and storing them places. Including S3 if there's not a requirement for them to be on site.

EVGA Longoria fucked around with this message at 14:43 on Feb 4, 2016

Pollyanna
Mar 5, 2005

Milk's on them.


I agree. Don't use the state of the database for either communication or concurrency. It always ends badly because nothing's guaranteed to be in sync. Always have your systems talking to each other for purposes like these.

kayakyakr
Feb 16, 2004

Kayak is true

EVGA Longoria posted:

Please don't store binary data in a SQL database. It will end in tears.

Have the worker POST the file back to the web server and store it on a file system. Paperclip is a gem for taking file uploads and storing them places. Including S3 if there's not a requirement for them to be on site.

I like this solution best. Store the file in a 3rd location that is either in the cloud or that is shared storage that both containers can access.

How are you notifying the client that the report is complete?

DONT THREAD ON ME
Oct 1, 2002

by Nyc_Tattoo
Floss Finder
this is from a while back, but don't do a metaprogramming in your application code. it's bad and terrible and no one thinks you're dclever. it's acceptable in true library code (eg gems) but probably not home rolled gems. i only ever meta program in library code that is on github and will have lots of hands touching it. if you don't expect your users to be able to treat that code as a black box, don't meta program.

also DRY is nice in general but making your methods more convoluted just because there's some minor code replication is bad. dogmatic adherence to DRY is bad and something I don't like about the ruby ecosystem.

Slow News Day
Jul 4, 2007

Dogmatic adherence to anything is bad.

Chilled Milk
Jun 22, 2003

No one here is alone,
satellites in every home

MALE SHOEGAZE posted:

this is from a while back, but don't do a metaprogramming in your application code. it's bad and terrible and no one thinks you're dclever. it's acceptable in true library code (eg gems) but probably not home rolled gems. i only ever meta program in library code that is on github and will have lots of hands touching it. if you don't expect your users to be able to treat that code as a black box, don't meta program.

also DRY is nice in general but making your methods more convoluted just because there's some minor code replication is bad. dogmatic adherence to DRY is bad and something I don't like about the ruby ecosystem.

That's what I was getting at. DRY really means don't repeat knowledge, not code. Just because two chunks of code are similar doesn't mean it's repetition. People seem to really lose sight of that, or maybe aren't taught the distinction in the first place.

Pollyanna
Mar 5, 2005

Milk's on them.


I want to talk a little about local development and testing environments.

One of the biggest pains in the rear end I've had with new projects is getting onboarded, i.e. installing all the extraneous crap and setting up the environment variables and all that. Every single time I start a new job or am recruited onto a new project, getting the drat thing running takes a good chunk of time, often more than a day.

This has gotten me to wonder what the best way to automate provisioning and setup for a development environment is. The approach we're taking for the project I'm currently assigned to is to use Docker Compose to manage containers for our Postgres database and for the app itself, the latter of which is run with docker-compose run --service-ports web so we can use pry and other interactive poo poo. This means that developers don't have to sit around and wait for Postgres and stuff to install before they can start hacking away. I'm considering adding a Vagrant machine to make the onboarding process easier (run containers by default, forward ports from internal Docker machine, vagrant up), but I don't know if that's too many layers of abstraction.

What do people here do for managing development environments and making engineer onboarding easier?

manero
Jan 30, 2006

Vagrant by itself into an ubuntu vm has served me well. Provisioned with Ansible or whatever you like.

I have yet to get into Docker, so I can't speak for that.

EVGA Longoria
Dec 25, 2005

Let's go exploring!

I have used Docker.

It is a pain to set up, especially on Mac. Once you get it going, it's fairly simple, if a bit slow, to keep running on your laptop.

The problem is I've yet to find a good service for deploying them to production environments. We tried it for one of our apps to get off of a terrible Capistano service, and it was even more trouble.

It's a cool idea, and in a few years I'm sure it'll have great support. But I don't think it's there yet.

kayakyakr
Feb 16, 2004

Kayak is true

EVGA Longoria posted:

I have used Docker.

It is a pain to set up, especially on Mac. Once you get it going, it's fairly simple, if a bit slow, to keep running on your laptop.

The problem is I've yet to find a good service for deploying them to production environments. We tried it for one of our apps to get off of a terrible Capistano service, and it was even more trouble.

It's a cool idea, and in a few years I'm sure it'll have great support. But I don't think it's there yet.

Looked into a service like tutum for deploy/hosting? https://www.tutum.co/

DONT THREAD ON ME
Oct 1, 2002

by Nyc_Tattoo
Floss Finder
nm

DONT THREAD ON ME fucked around with this message at 19:20 on Feb 6, 2016

Hadlock
Nov 9, 2004

We use vagrant at work to spin up a base CoreOS docker environment on dev laptops.

That said,

If you come up with some magic engineer-onboarding scheme that spins up and runs their entire dev system inside of docker, you should quit first and sell it to someone like JetBrains, or Docker themselves.

Smol
Jun 1, 2011

Stat rosa pristina nomine, nomina nuda tenemus.
In my experience, using Docker Machine to create the virtual machines works way better than Vagrant, since usually Vagrant base images only really support one virtualization provider. So what ends up happening is that a new guy joins the team but he uses Parallels instead of VMware, and since that vagrant image doesn't come with Parallels' kernels modules, stuff like mounting volumes doesn't work.

Docker Machine had worked way better for us, since all their boot2docker images have the necessary kernel modules baked in.

Pollyanna
Mar 5, 2005

Milk's on them.


The Docker experiment has gone well so far. Thanks guys! :)

I've got another question: what's the most acceptable way to model a parent-child relationship within the same kind of resource? For example, parent and child Boards, where each Board must have a parent Board (with the exception of the root), and may have many children Boards. A lot of what I see out there involves something like a parent_id attribute in the model's table, which doesn't really work well in my experience - or at least, I appear to be too dumb to pull it off well cause it's not workinggggg :byodood:

I was thinking that a has_many_through approach might work better, e.g. a Relationship join model with "parent id" and "child id" attributes. Then, a Board has many Boards through Relationship children, and might belong to a Board through a Relationship parent. Does that make sense? Is there some other, canonical way to model hierarchical data in ActiveRecord?

MasterSlowPoke
Oct 9, 2005

Our courage will pull us through
Should be as simple as Board has_many :boards, belongs_to :board.

kayakyakr
Feb 16, 2004

Kayak is true

Pollyanna posted:

The Docker experiment has gone well so far. Thanks guys! :)

I've got another question: what's the most acceptable way to model a parent-child relationship within the same kind of resource? For example, parent and child Boards, where each Board must have a parent Board (with the exception of the root), and may have many children Boards. A lot of what I see out there involves something like a parent_id attribute in the model's table, which doesn't really work well in my experience - or at least, I appear to be too dumb to pull it off well cause it's not workinggggg :byodood:

I was thinking that a has_many_through approach might work better, e.g. a Relationship join model with "parent id" and "child id" attributes. Then, a Board has many Boards through Relationship children, and might belong to a Board through a Relationship parent. Does that make sense? Is there some other, canonical way to model hierarchical data in ActiveRecord?

Does it need to be sorted? If so, use: https://github.com/collectiveidea/awesome_nested_set, otherwise use: https://github.com/amerine/acts_as_tree or something like it.

e: love ruby toolbox: https://www.ruby-toolbox.com/categories/Active_Record_Nesting

kayakyakr fucked around with this message at 02:41 on Feb 21, 2016

Chilled Milk
Jun 22, 2003

No one here is alone,
satellites in every home
I use awesome nested set a lot at work, it works well enough, it could be overkill for a simple parent > child relationship but it will work (and can be sorted and deep nested and all that).


And FYI Ruby Toolbox has been dead/cruising on auto pilot since 2013. Still an ok reference but http://awesome-ruby.com is curated and up to date.

Adbot
ADBOT LOVES YOU

kayakyakr
Feb 16, 2004

Kayak is true

The Milkman posted:

I use awesome nested set a lot at work, it works well enough, it could be overkill for a simple parent > child relationship but it will work (and can be sorted and deep nested and all that).


And FYI Ruby Toolbox has been dead/cruising on auto pilot since 2013. Still an ok reference but http://awesome-ruby.com is curated and up to date.

Cool, thanks for the update. And yes, awesome nested set is total overkill for a simple parent-child, but probably the best option if sorting is necessary.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply