Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Dr. Stab
Sep 12, 2010
👨🏻‍⚕️🩺🔪🙀😱🙀

Hadlock posted:

Has anyone written anything of great interest using Rust yet?

Is Atom/Sublime the new vi/emacs war?

I really don't see this being the case. It's not like there's some fundamental difference between the two. Atom looks and feels a lot like sublime. The real argument is atom/sublime vs vi.

Adbot
ADBOT LOVES YOU

No Gravitas
Jun 12, 2013

by FactsAreUseless

Dr. Stab posted:

I really don't see this being the case. It's not like there's some fundamental difference between the two. Atom looks and feels a lot like sublime. The real argument is atom/sublime vs emacs.

You had a terrible typo here.

TheresaJayne
Jul 1, 2011

No Gravitas posted:

Yeah, and a C program will be the final deliverable. Problem being that C isn't exactly fun to prototype in, when things are very much in flux. It is the prototyping language I'm looking for.

The critical part is kinda goofy. There are components which are engaged in a back-and-forth calling.

A calls B returns to A calls C returns to A calls D returns to A loops and calls B ...

B, C and D are teensy, but need to be separate from A. The idea is a proof-of-concept for a program where A is held constant and people can just write the code for B, C and D separately. In the final product the files with the B, C and D functions would be #included into A. Then A can be written by an expert and any noob can just plug in their B, C and D and still have good stuff happen and fast. Inlining is critical as B, C and D are just usually going to be one or two lines long, a singly-nested loop at worst.

that sounds like you may need to look into plugin functionality, as if you can just drop the B C D into a folder and have the main system A pick it up and know how to call then no need to recompile A just change the B C D code and drop the new plugin in place

Suspicious Dish
Sep 24, 2011

2020 is the year of linux on the desktop, bro
Fun Shoe
atom wins because it is updated more than once every two years or so, and doesn't cost $70 for the privilege of donating to another dead text editor project

ufarn
May 30, 2009
For some reason, Sublime Text 3 is no longer able to run my pre-commit script, when I commit using the Git package. Here is what the start of the script looks like:

code:
#!/usr/bin/env bash

cd "$(git rev-parse --git-dir)" && cd ..
bundle install —quiet
# …
And I get the error

code:
.lint.sh: line 4: bundle: command not found
# …
.lint.sh: line n: xyz: command not found
The script works fine, when I use the CLI, and I’ve made sure to chmod +x just in case. I’ve also reinstalled the package, so I’m not sure what the hell is up. Something with dotfiles maybe? My `subl` command works fine. so I’m not sure what’s busted.

Mogomra
Nov 5, 2005

simply having a wonderful time

Suspicious Dish posted:

atom wins because it is updated more than once every two years or so, and doesn't cost $70 for the privilege of donating to another dead text editor project

Pretty much.

I did have to go back to using Sublime though. Atom (as far as I know) still can't handle really big files, it can open up bigger files than before, but still nothing compared to other editors. Also APM started acting really strange on my computer. Any time I would go to the updates page, or use APM in a console, my computer would hang like a bitch. It would take it a half hour or so to even shutdown, and that would be after I'd manually kill each Atom/APM/Chrome process. Bonkers.

So yeah, never mind. Sublime wins.

Munkeymon
Aug 14, 2003

Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.



Hadlock posted:

Has anyone written anything of great interest using Rust yet?

https://en.wikipedia.org/wiki/Servo_(layout_engine)

yippee cahier
Mar 28, 2005

I'm sorta keeping an eye on https://zinc.rs/. Might really dig into it if I start to feel like I'm spinning my wheels solving the same old bugs at work.

Nippashish
Nov 2, 2005

Let me see you dance!

Mogomra posted:

Atom (as far as I know) still can't handle really big files, it can open up bigger files than before, but still nothing compared to other editors.

Atom will also freeze up all the time if the files you're editing are on a network drive and your connection is slow. Sublime at least only stalls while saving.

JawnV6
Jul 4, 2004

So hot ...

Stealth Like posted:

I am more experience with embedded programming so I'm familiar with C/C++ although like I said all the windows stuff was way over my head with the WinUSB examples. I've been looking at Python and it seems that it may suit my needs. I've seen that there's a library for easy USB integration (PyUSB) and matplotlib for doing graph plotting. Can these be used together with something else to make a workable UI?
I whipped up a cheap DAQ with python reading a serial port and plotting with matplotlib, it was just displaying though and didn't have any interactive UI components. Really didn't take much time at all. If you're coming from embedded, can you just use a serial port instead of a full USB? Any time I've had to use an actual USB device, the vendor provided drivers that made it easier to deal with.

No Gravitas posted:

The interface, yeah. Usually A and the others are in one piece, but this sucks because the user always has to write their own A. The A's internals are nailed down and solid, just a matter of finding a clean line separating A from the rest of the system so that the user can be an idiot and just write the domain-specific parts.
That doesn't hit me as something you can quickly prototype in another language. Assuming you get some awesome interface for A<->B in Python, it's still a mountain of work to replicate that in C. I'd probably start by writing a few different flavors of B and figure out how I want them to plug into the scheduler or what commonalities they share.

Woolwich Bagnet
Apr 27, 2003



JawnV6 posted:

I whipped up a cheap DAQ with python reading a serial port and plotting with matplotlib, it was just displaying though and didn't have any interactive UI components. Really didn't take much time at all. If you're coming from embedded, can you just use a serial port instead of a full USB? Any time I've had to use an actual USB device, the vendor provided drivers that made it easier to deal with.

I'm trying really hard to stick with USB as although this project doesn't require it, my next one will need much higher throughput than serial can provide. I just want to get this part out of the way so I don't need to come back to it in the future. I am using the WinUSB device driver as it stands since it was easy to make a .inf file that allows me to install it. After doing more research it looks like I should be able to use PyUSB, PyQT and PyQTGraph to do everything I need.

kloa
Feb 14, 2007


Is Code Map in VS the best way to visualize a project you are unfamiliar with? Are there better ways?

I joined a startup recently, so I'm trying to get up o speed as quickly as possible.

Flashing Twelve
Mar 20, 2007

Hadlock posted:

Has anyone written anything of great interest using Rust yet?

Is Atom/Sublime the new vi/emacs war?

There's a game engine called Piston that's surprisingly sophisticated for a language that only hit 1.0 a few months ago. Other than that, check out this list of cool stuff. The ecosystem is definitely still in the "replicate the common libraries available in other languages" stage though.

Gravity Pike
Feb 8, 2009

I find this discussion incredibly bland and disinteresting.

kloa posted:

Is Code Map in VS the best way to visualize a project you are unfamiliar with? Are there better ways?

I joined a startup recently, so I'm trying to get up o speed as quickly as possible.

Unless, for some reason, it is Not An Option, the best way to get up to speed on a project is going to be to ask your coworkers. Ask someone if they can draw you a high-level overview of the product. If it's service-oriented, ask someone if they can draw you a lower-level overview of the services that you're working on. This is the kind of stuff that good engineers are really good at. The Code Map can show you the relationships, but someone who has internalized the project can show you the important relationships, point out any common patterns, and best of all, explain why things are the way they are.

madkapitolist
Feb 5, 2006
FFMPEG question!

I am starting with a high res video file and I would like to create 3 variants, low quality, mid quality, and high quality for mobile streaming. I want these mid/low/high variants to be segmented into ts pieces that the m3u8 file will be pointing at. Is there a way to do this in one line in ffmpeg?

I have successfully generated an m3u8 file and ts segments with ffmpeg, do I need to do this 3x and set specs for low/mid/high? If so, how do I get the singular m3u8 file to point to all variants as opposed to one for each variant?

This is the command I used to generate the m3u8 file along with the ts segments.

ffmpeg -i C:\Users\george\Desktop\video\hos.mp4 -strict -2 -acodec aac -vcodec libx264 -crf 25 C:\Users\user\Desktop\video\hos_Phone.m3u8

FAT32 SHAMER
Aug 16, 2012



Hey guys, I have kind of a dilemma. I'm taking a a class called "Information Retrieval and Knowledge Discovery" class that so far has been us sitting in class while the professor reads us slides about how to go about data mining and analyzing, but not teaching us any way to actually do it. He has just assigned a project that involves using any method we would like in order to solve it, and so far I'm extremely too stupid for R, and I don't have enough time to learn how to easy-bake it in RapidMiner.

I have a spreadsheet with approximately 73,000 entries, and each entry has 32 attributes such as RefID, IsBadBuy (boolean, 0/1), make model year etc etc and i'm supposed to form a hypothesis as to how to predict what kind of car is going to be a bad or good buy. What I want to base my hypothesis on is the make, model, transmission, vehicle age, and nationality since those are typically what most people go by when talking about cars and whether or not a car is a good buy.

Here's a little glimpse of what I'm dealing with here:


How would one of you guys tackle this? I seem to remember doing something like this using MySQL, but not at this massive of a volume nor trying to make a learning programme out of it

This may be out of the scope of this thread, so sorry if that is the case.

kloa
Feb 14, 2007


Gravity Pike posted:

Unless, for some reason, it is Not An Option, the best way to get up to speed on a project is going to be to ask your coworkers. Ask someone if they can draw you a high-level overview of the product. If it's service-oriented, ask someone if they can draw you a lower-level overview of the services that you're working on. This is the kind of stuff that good engineers are really good at. The Code Map can show you the relationships, but someone who has internalized the project can show you the important relationships, point out any common patterns, and best of all, explain why things are the way they are.

Yeah, we're meeting up sometime this week to do exactly this. I just had a free weekend and wanted to get ahead on the process of learning it, but it's hard to figure out anything without bouncing ideas off the person that created it :sigh:

Tusen Takk posted:

Hey guys, I have kind of a dilemma. I'm taking a a class called "Information Retrieval and Knowledge Discovery" class that so far has been us sitting in class while the professor reads us slides about how to go about data mining and analyzing, but not teaching us any way to actually do it. He has just assigned a project that involves using any method we would like in order to solve it, and so far I'm extremely too stupid for R, and I don't have enough time to learn how to easy-bake it in RapidMiner.

I have a spreadsheet with approximately 73,000 entries, and each entry has 32 attributes such as RefID, IsBadBuy (boolean, 0/1), make model year etc etc and i'm supposed to form a hypothesis as to how to predict what kind of car is going to be a bad or good buy. What I want to base my hypothesis on is the make, model, transmission, vehicle age, and nationality since those are typically what most people go by when talking about cars and whether or not a car is a good buy.

Here's a little glimpse of what I'm dealing with here:


How would one of you guys tackle this? I seem to remember doing something like this using MySQL, but not at this massive of a volume nor trying to make a learning programme out of it

This may be out of the scope of this thread, so sorry if that is the case.

You could throw it in a database and aggregate on it, but if you want to keep it inside of Excel, go read about Pivot Tables.

ultrafilter
Aug 23, 2007

It's okay if you have any questions.


This is almost perfectly set up for a decision tree, but you will have to use R or RapidMiner to do that.

FAT32 SHAMER
Aug 16, 2012



ultrafilter posted:

This is almost perfectly set up for a decision tree, but you will have to use R or RapidMiner to do that.

Hey this just rang a bell!!! I just realized that only a presentation of what strategy i am using is due tomorrow, not the entire bloody project, so I have time to learn either or, and now I feel significantly less overwhelmed

Literal life-saver itt

Linear Zoetrope
Nov 28, 2011

A hero must cook
Just make sure you use a DT library instead of writing your own. Decision Trees are conceptually simple and if you have any programming experience you could probably write a basic one in a night, but they overfit by design and it takes some finesse to avoid that (usually some form of subsampling, cross validation, pruning, or a mix). You probably could actually fix this by yourself without much trouble, but why reinvent the wheel?

Also, SciKit for Python has a decision tree package if you prefer that.

Linear Zoetrope fucked around with this message at 01:03 on Oct 20, 2015

FAT32 SHAMER
Aug 16, 2012



Jsor posted:

Just make sure you use a DT library instead of writing your own. Decision Trees are conceptually simple and if you have any programming experience you could probably write a basic one in a night, but they overfit by design and it takes some finesse to avoid that (usually some form of cross validation, pruning, or both).

Are there some popular ones that can be imported into RapidMiner or whatever? I'm currently watching this tutorial about it and it's ringing a lot of bells but it's still a pretty overwhelming programme even though it makes my job significantly easier

Linear Zoetrope
Nov 28, 2011

A hero must cook

Tusen Takk posted:

Are there some popular ones that can be imported into RapidMiner or whatever? I'm currently watching this tutorial about it and it's ringing a lot of bells but it's still a pretty overwhelming programme even though it makes my job significantly easier

I'm not familiar with it, but it appears there's a free Python extension if you want to use Sci-Kit.

Linear Zoetrope fucked around with this message at 01:06 on Oct 20, 2015

FAT32 SHAMER
Aug 16, 2012



Jsor posted:

I'm not familiar with it, but it appears there's a free Python extension if you want to use Sci-Kit.

Nice!, cheers man, I'll report back with some results

Linear Zoetrope
Nov 28, 2011

A hero must cook

Tusen Takk posted:

Nice!, cheers man, I'll report back with some results

Google also says there are some R libraries, which I think works natively with Rapid-Miner, but I can't speak to if any are good.

ultrafilter
Aug 23, 2007

It's okay if you have any questions.


I'd consider rpart to be the standard decision tree package, and anything else needs to be compared to it. It predates R, having originated in the S-Plus system.

Linear Zoetrope
Nov 28, 2011

A hero must cook
After searching for libraries that support boosting, it looks like rpart also lets you use adabag which, later in the project, may let you use boosted or bagged decision trees if you need it. (Boosting is a common method to make decision trees better, more common with one-node decision trees called "decision stumps", but can be used with trees too).

FAT32 SHAMER
Aug 16, 2012



So after dicking around in RapidMiner I was able to get it to give me a solid decision tree


0 denotes that the vehicle was a Good Buy, and 1 denotes that it was a Bad Buy.

I'm pretty stoked because my big project went from being 0% done to being 90% done haha

JawnV6
Jul 4, 2004

So hot ...
Perhaps this is a stupid question, but why am I checking the odometer for vehicles in the $13639.500 to $17295 range?

FAT32 SHAMER
Aug 16, 2012



JawnV6 posted:

Perhaps this is a stupid question, but why am I checking the odometer for vehicles in the $13639.500 to $17295 range?

Well, we're trying to predict if a car would be a good buy or a bad buy; if I'm reading the tree correctly, this tells us that a given car is a good buy if it's less than $17k, then from there we can say it's a good buy if it has less than 113k miles on it, then so on from there?

I'm a big dummy and may be reading this wrong though :v:

edit: I ran a performance analysis on my model in RapidMiner and I guess it's pretty good



Now to figure out how to make it better :getin:

FAT32 SHAMER fucked around with this message at 20:51 on Oct 20, 2015

fritz
Jul 26, 2003

Tusen Takk posted:

Well, we're trying to predict if a car would be a good buy or a bad buy; if I'm reading the tree correctly, this tells us that a given car is a good buy if it's less than $17k, then from there we can say it's a good buy if it has less than 113k miles on it, then so on from there?

I'm a big dummy and may be reading this wrong though :v:

edit: I ran a performance analysis on my model in RapidMiner and I guess it's pretty good



Now to figure out how to make it better :getin:

Your model is saying that there are only 4 bad cars out there (instead of 2985).

Xarn
Jun 26, 2015

Tusen Takk posted:

Well, we're trying to predict if a car would be a good buy or a bad buy; if I'm reading the tree correctly, this tells us that a given car is a good buy if it's less than $17k, then from there we can say it's a good buy if it has less than 113k miles on it, then so on from there?

I'm a big dummy and may be reading this wrong though :v:

edit: I ran a performance analysis on my model in RapidMiner and I guess it's pretty good



Now to figure out how to make it better :getin:

Dont forget to either crossvalidate or validate against different datasets than the one you learn from.

FAT32 SHAMER
Aug 16, 2012



fritz posted:

Your model is saying that there are only 4 bad cars out there (instead of 2985).

Hmm, this is how I have it set up to test the accuracy:

With the split data being partitioned at .666/.334

So I'm not sure :shobon:


Xarn posted:

Dont forget to either crossvalidate or validate against different datasets than the one you learn from.

I will definitely do the cross-validate since i don't have any other datasets to compare against other than the master one that the professor has that he's going to compare our model for against his and see how it turns out

fritz
Jul 26, 2003

Tusen Takk posted:

Hmm, this is how I have it set up to test the accuracy:

With the split data being partitioned at .666/.334

So I'm not sure :shobon:

I don't know rapidminer, but from the table you first posted:

code:
21372 2982

1        3
21372 times, you predicted '0' when the answer was '0', 1 time you predicted '1' when the answer was '0', 2982 times you predicted '0' when the answer was '1', and 3 times you predicted '1' when the answer was '1'.

FAT32 SHAMER
Aug 16, 2012



fritz posted:

I don't know rapidminer, but from the table you first posted:

code:
21372 2982

1        3
21372 times, you predicted '0' when the answer was '0', 1 time you predicted '1' when the answer was '0', 2982 times you predicted '0' when the answer was '1', and 3 times you predicted '1' when the answer was '1'.

Right, what I did was split the data so I could compare how two different sets do in the model. I think the wording i'm looking for here is it takes a small sample and sees how it does versus a large sample?

JawnV6
Jul 4, 2004

So hot ...

Tusen Takk posted:

Well, we're trying to predict if a car would be a good buy or a bad buy; if I'm reading the tree correctly, this tells us that a given car is a good buy if it's less than $17k, then from there we can say it's a good buy if it has less than 113k miles on it, then so on from there?

You kinda papered over my point with the "so on from there," the tree goes back to the cost and does a second filter on the same variable. If you turned it into a line of code you'd end up merging all the checks against value into one statement. I'm pretty sure, assuming fritz's reading of your table is right, it's figuring out how to throw out the 2982 actual 1's in small enough batches that the local error isn't too bad.

FAT32 SHAMER
Aug 16, 2012



JawnV6 posted:

You kinda papered over my point with the "so on from there," the tree goes back to the cost and does a second filter on the same variable. If you turned it into a line of code you'd end up merging all the checks against value into one statement. I'm pretty sure, assuming fritz's reading of your table is right, it's figuring out how to throw out the 2982 actual 1's in small enough batches that the local error isn't too bad.

Ahhhhhh yeah, i see what you mean. My professor pointed out that only ~14% of the entries were bad buys, which means that it's really easy to guess that most of the cars are good buys then take that hit if they're wrong.

At least he really liked that my hypothesis was correct based on the data :v:

:doh:

TacoHavoc
Dec 31, 2007
It's taco-y and havoc-y...at the same time!
I need help with SSL and certificates. I have googled the poo poo out of this and I still feel so lost.

So I am working on an embedded product. It has no display or other means of direct user interface. This product is controlled/configured through a phone, tablet, or PC based app that connects directly to the product. It doesn't do the "I talk to the cloud and then the phone talks to the cloud and everything is cool and easy", because some people that use this product don't want it externally accesable. I also don't like saying cloud.

We want to use SSL to encrypt communication between the product and the app. I can't figure out the right way to do this though:
- Self signed certificates appear to be an issue since all modern web browsers/network APIs hate them.
- We can't get a certificate from a global CSA because these products don't have a consistent domain name or IP address.

How do people do this? What am I missing?

TLDR: Certificates are hard because I'm dumb, please help me.

Jabor
Jul 16, 2010

#1 Loser at SpaceChem
If you want meaningful encryption, ultimately you're going to have to come up with some scheme to authenticate that your app is talking to the product it thinks it's talking to. After you've figured that out, everything else will fall into place pretty easily. Certificates are a means to achieve authentication, they're not an end in and of themselves.

Some questions to think about :
- Why do you want to encrypt this data?
- Is it okay if someone who has physical access to the product can intercept and read all the connections being made to it?
- ... all the connections being made to any other product?
- Is this a product that will be set up by sysadmins, or by random consumers?
- How are people connecting to these things to set them up in the first place?
- Do people need to be able to connect through web browsers, or is it just going through your app?

ExcessBLarg!
Sep 1, 2001

TacoHavoc posted:

I need help with SSL and certificates. I have googled the poo poo out of this and I still feel so lost.
Seriously consider hiring a security consultant for this part of the work, at least just to vet your final approach and sign off that there's no vulnerabilities in that. Also Jabor's questions. After that, though:

TacoHavoc posted:

- Self signed certificates appear to be an issue since all modern web browsers/network APIs hate them.
Self-signed CA certificates are fine, you just have to manually add them to the client browsers and applications. The point of purchasing certificates is that the corresponding CA cert is already included in browsers, so you don't have to do any manual setup on the client.

TacoHavoc posted:

- We can't get a certificate from a global CSA because these products don't have a consistent domain name or IP address.
These devices will have to have some stable hostname that points to them in order to use browser-based TLS (SSL) whether you use your own CA or not.

Adbot
ADBOT LOVES YOU

JawnV6
Jul 4, 2004

So hot ...

Tusen Takk posted:

Ahhhhhh yeah, i see what you mean. My professor pointed out that only ~14% of the entries were bad buys, which means that it's really easy to guess that most of the cars are good buys then take that hit if they're wrong.
I've done very, very little ML but I've had that same issue of overfitting to 0. "Gosh, I get 99% if I say there's nothing!" :rolleyes: . We've had to curate our training sets to have a lot of relevant activity to avoid that. It might help to train on a subset of 50/50 data or even 25/75 towards the 14%?

ExcessBLarg! posted:

These devices will have to have some stable hostname that points to them in order to use browser-based TLS (SSL) whether you use your own CA or not.
This is also out of my depth, but I can hit up servers on my local network with just their host name, and to my knowledge I've never added them to my host file. Can you just use a single name like "https://localserver" or does it require something longer? "Fully qualified"?

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply