Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Destroyenator
Dec 27, 2004

Don't ask me lady, I live in beer
They can do entity tracking so when different part of your codebase ask for Car id 7 they get the same in memory object. They can also do change tracking so when you modify that object you can then do a "save" style operation which will generate an appropriate UPDATE rather than a full overwrite.

Sometimes these are good features, sometimes not.

edit: the change tracking can also involve adding or removing from list properties on the object where they make sure the linked item is inserted or deleted and foreign keys all work out etc.

Destroyenator fucked around with this message at 22:43 on Oct 31, 2017

Adbot
ADBOT LOVES YOU

TheBlackVegetable
Oct 29, 2006

Destroyenator posted:

They can do entity tracking so when different part of your codebase ask for Car id 7 they get the same in memory object. They can also do change tracking so when you modify that object you can then do a "save" style operation which will generate an appropriate UPDATE rather than a full overwrite.

Sometimes these are good features, sometimes not.

edit: the change tracking can also involve adding or removing from list properties on the object where they make sure the linked item is inserted or deleted and foreign keys all work out etc.

Are there any that can do bulk load into a user-defined table type then execute a merge on that data? I've always had problems with efficient upserts with ORMs.

B-Nasty
May 25, 2005

TheBlackVegetable posted:

Are there any that can do bulk load into a user-defined table type then execute a merge on that data? I've always had problems with efficient upserts with ORMs.

No, that's pretty much SPROC/table value parameter land.

Once you start to go beyond load an entity (with shallow object graph) and save it, you're rapidly moving out of what ORMs are good at. Pain and unnecessary frustration awaits you if you try to force it.

LongSack
Jan 17, 2003

Speaking of ORM, an update on nHydrate, if anyone cares.

When it works, it works very well. The best part is that you can do everything in the modeler and one of the projects it generates is an installer which will create or update the database for you. Its pretty sweet. When it works.

Unfortunately, it is not at all robust. Something as simple as adding a field to a table can screw things up so badly that its basically unrecoverable. Ive had three projects totaled by (1) adding a field to a table, (2) renaming a table, and (3) deleting a table. Also, if you declare a field as unique, and later change your mind, the installer will never alter the database to make it non-unique, so you have to do it manually.

So I have switched to using EF directly, doing all my database stuff and then generating the model from the database. So far, so good with EF.

LongSack
Jan 17, 2003

Also, can someone help me avoid the unsigned application thing with my installers? I have nearly a dozen tools that I have made available to my coworkers, and every time they run the installer, they have to go through the install anyway thing. Id love to get rid of that warning. This page tells me what I need to do, but doesnt really tell me how to do it.

mystes
May 31, 2006

LongSack posted:

Also, can someone help me avoid the “unsigned application” thing with my installers? I have nearly a dozen tools that I have made available to my coworkers, and every time they run the installer, they have to go through the “install anyway” thing. I’d love to get rid of that warning. This page tells me what I need to do, but doesn’t really tell me how to do it.
You need a code signing certificate. I think the cheapest ones are like $20? You can also probably also use an internal ca if you have that set up.

Edit to expand slightly:
You need a code a code signing certificate. If you don't have an internal certificate authority set up as part of your windows domain, you will have to buy one (like buying an SSL certficate), but an internal certificate authority is probably actually the correct solution. Unfortunately, your company probably either won't have one or won't create a code signing certificate for you.

Actually you might also just be able to generate your own self-signed code signing certificate and get your coworkers to add it to the windows certificate store, but even if windows allows that, I don't know if that would be better than having them click through the warnings.

mystes fucked around with this message at 03:40 on Nov 1, 2017

Portland Sucks
Dec 21, 2004
༼ つ ◕_◕ ༽つ

Joda posted:

Two major advantages off the top of my head is the ability to multi-thread your database queries and doing data pre- and post-processing in a language that isn't SQL.

Also, you can have a single server codebase, in stead of one for database queries and one for all the other stuff.

At what scale does this kind of stuff start to matter as part of the SE process?

I work in industrial manufacturing and the majority of our .NET apps are WinForms desktop applications that just serve as process monitoring applications or HMIs to aid in data collection or interacting with PLCs. None of our individual databases have even close to a gb of data in any one of them even though there are a ton of them. It's mostly just summary data recorded from timeseries data outputed by PLCs.

I can't tell if I have a use case for stuff like ORMs or if our requirements are just simple enough that the reason I don't understand why I'd need it is because we probably don't.

LongSack
Jan 17, 2003

mystes posted:

You need a code signing certificate. I think the cheapest ones are like $20? You can also probably also use an internal ca if you have that set up.

Edit to expand slightly:
You need a code a code signing certificate. If you don't have an internal certificate authority set up as part of your windows domain, you will have to buy one (like buying an SSL certficate), but an internal certificate authority is probably actually the correct solution. Unfortunately, your company probably either won't have one or won't create a code signing certificate for you.

Actually you might also just be able to generate your own self-signed code signing certificate and get your coworkers to add it to the windows certificate store, but even if windows allows that, I don't know if that would be better than having them click through the warnings.

I have an SSL cert from lets encrypt for my website. Can I leverage that or is this something completely different? $20 is no big deal. $20 a year is no big deal. $200 is eh. $200 a year is a non starter.

mystes
May 31, 2006

LongSack posted:

I have an SSL cert from let’s encrypt for my website. Can I leverage that or is this something completely different? $20 is no big deal. $20 a year is no big deal. $200 is eh. $200 a year is a non starter.
No it's similar but not exactly the same. Part of the problem is there's no domain for code signing so it can only identify you by name, meaning that all code signing certs are essentially EV certs and therefore can't be automated with something like let's encrypt.

Actually it looks like I misremembered and the cheapest ones are more like $70, unfortunately.

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

LongSack posted:

I have an SSL cert from lets encrypt for my website. Can I leverage that or is this something completely different? $20 is no big deal. $20 a year is no big deal. $200 is eh. $200 a year is a non starter.

Your company can't shell out $200 a year? That's insane. They probably spend more on straws for the break room.

LongSack
Jan 17, 2003

New Yorp New Yorp posted:

Your company can't shell out $200 a year? That's insane. They probably spend more on straws for the break room.

LOL, no company. Its just me. I am by profession an information security engineer (I manage firewalls and other threat devices for our clients) but I have been programming since the late 1970s and most recently I produce utilities that help me automate things I do as part of my day-to-day work. For example, the most useful of my tools is a program that lets me enter a list of IP addresses. Then I can click one button to produce an excel spreadsheet that contains a list of firewall rules for those addresses. Click another button and you get a text file of commands that can be cut and pasted into the firewall CLI to decommission those addresses. Saves a ton of time. But its not part of my official job duties, and I dont even want to think about the hassle that getting even a single dollar would incur (my company has more than 90,000 employees on six continents, so it is buried in bureaucracy).

putin is a cunt
Apr 5, 2007

BOY DO I SURE ENJOY TRASH. THERE'S NOTHING MORE I LOVE THAN TO SIT DOWN IN FRONT OF THE BIG SCREEN AND EAT A BIIIIG STEAMY BOWL OF SHIT. WARNER BROS CAN COME OVER TO MY HOUSE AND ASSFUCK MY MOM WHILE I WATCH AND I WOULD CERTIFY IT FRESH, NO QUESTION

Portland Sucks posted:

What exactly is the selling point behind using an ORM?

I keep running across stuff like EF as the choice way to interact with data sources, but all of the selling points seem to be stuff like "never use SQL again." If I don't have any issues coming up with SQL queries on the spot are they just not for me?

I've got thousands of DB tables that I could potentially be pulling and aggragating data from. What benefit is there for me to build some secondary abstraction of them to link an ORM to when I could just write a quick SQL query and get the result back as a data structure that is super simple to parse?

I'm trying really hard not to be a close minded goon, but whenever I look at ORM's all I see is enough overhead that I'll probably be finished with my project before I implement the architecture needed to make the ORM function with our databases.

It's an abstraction layer. Sometimes abstraction is useful, sometimes it's overengineering. This will change from project to project.

Writing SQL is easy, and it frustrates me that so many anti-ORM people have this weird belief that ORM users are scared of SQL. What they're seeking is a way to lower cognitive load, which is something every developer should try to do. In some projects adding EF increases the amount of poo poo you have to worry about. In other projects it reduces it. Developers (not you specifically) are weirdly dogmatic about poo poo like this and it drives me up the wall.

I will say that by far my favourite part of EF is type safety. You can't check raw SQL for errors at compile time. Some poo poo exists to simplify the process of checking raw SQL but generally speaking they're not foolproof and have serious limitations. Again, this might matter to you or it might not.

putin is a cunt fucked around with this message at 09:37 on Nov 1, 2017

Pilsner
Nov 23, 2002

Just-In-Timeberlake posted:

Having a really weird issue here, that I'm not even sure how to google for or dive into to see what is going on.

Have a recordset being returned from MSSQL that I'm loading into a DataTable. When I try to populate a string with one of the data fields the DataRow is showing that field is an empty array (so, db NULL I think) so it throws an error. The thing is when I run that stored procedure in SSMS the returned record most certainly has data in that field. If I wrap that output field in the stored procedure with an ISNULL(result, ' ') then the DataRow shows the data = ' ', even though in SSMS that data is not NULL in the result table.

Not really sure where the error is occurring. The ISNULL check makes me think in MSSQL but there's data being returned in the result set, so maybe it's in the DataRow? Any insight or where I can start looking for a solution would be most appreciated.
This is very hard to understand for me. Can you give some code samples? What do you mean "try to populate a string with one of the data fields the DataRow is showing that field is an empty array"??

Portland Sucks posted:

What exactly is the selling point behind using an ORM?

I keep running across stuff like EF as the choice way to interact with data sources, but all of the selling points seem to be stuff like "never use SQL again." If I don't have any issues coming up with SQL queries on the spot are they just not for me?

I've got thousands of DB tables that I could potentially be pulling and aggragating data from. What benefit is there for me to build some secondary abstraction of them to link an ORM to when I could just write a quick SQL query and get the result back as a data structure that is super simple to parse?

I'm trying really hard not to be a close minded goon, but whenever I look at ORM's all I see is enough overhead that I'll probably be finished with my project before I implement the architecture needed to make the ORM function with our databases.
The main selling point is that you write your SQL once, then your entire codebase (for example, .NET code) can re-use said SQL. A lot of the SQL can also be auto-generated by the ORM framework, so you don't have to write out "Select column1, column2, colum3.... from X" for each entity that needs to be select-able. You wrap your SQL in simple object structures (can also be auto-generated), and then other developers need zero SQL skills in order to fetch, insert, etc. data in and out of the database. They just call some CustomerFactory.FetchCustomer("portland@somethingawful.com") to get the Customer with a certain email address. They could also call a seemingly simple method, which behind the scenes has a complex SQL statement, which then only needs to be written once, and maintained in one place.

You say you don't have a problem "coming up with SQL queries on the spot" - and that is terrible code design. Imagine if you need to change a column name, add a column, change a table name, or add a new WHERE clause to an often used query - you will need to manually search through your entire code base and change the SQL everywhere. You're likely to miss some spots this way as well, leaving you open to bugs. Having inline SQL spewed all over the code base is a big no-no - at the very least collect all hand-written SQL in a single Data layer project for a start.

How far you want to go depends a bit on the scale of the project though. You said "my" project, as in a one-man project? But with thousands of DB tables? Did you create all those tables yourself? Are the tables used by anyone else? If the latter, your company should start working on a common data layer that everyone uses, perhaps accessed via a service. That way you won't need to write your own SQL that perhaps some other developer has already written in his/their team's codebase. And when you write the SQL once, or implement an ORM, the next developer who needs to fiddle with the same table, doesn't need to re-write the SQL either.

Pilsner fucked around with this message at 13:37 on Nov 1, 2017

Joda
Apr 24, 2010

When I'm off, I just like to really let go and have fun, y'know?

Fun Shoe

Portland Sucks posted:

At what scale does this kind of stuff start to matter as part of the SE process?

I work in industrial manufacturing and the majority of our .NET apps are WinForms desktop applications that just serve as process monitoring applications or HMIs to aid in data collection or interacting with PLCs. None of our individual databases have even close to a gb of data in any one of them even though there are a ton of them. It's mostly just summary data recorded from timeseries data outputed by PLCs.

I can't tell if I have a use case for stuff like ORMs or if our requirements are just simple enough that the reason I don't understand why I'd need it is because we probably don't.

Like others have said it varies from project to project. I work on a real-time system with fairly large amounts of data, where users expect quick response times and some indication of progress when that is not possible. An example off the top of my head was recently when I had to add 50 odd million rows to a table, but depending on certain parameters, only if 1) the user doing the query had access to certain things, 2) the users being added to the table had access to certain things, and 3) the row did not already exist. Not only is the C# code much easier to read, the combination of faster execution of data processing and multithreading cut the total execution time from 30 odd minutes to 5 and I was able to easily inform the user of progress since selection, filtering and insertion was done in chunks on the different threads.

I have no idea if something like this is relevant for your work, but it does have its advantages, even in run-time.

kitten emergency
Jan 13, 2008

get meow this wack-ass crystal prison

LongSack posted:

LOL, no company. Its just me. I am by profession an information security engineer (I manage firewalls and other threat devices for our clients) but I have been programming since the late 1970s and most recently I produce utilities that help me automate things I do as part of my day-to-day work. For example, the most useful of my tools is a program that lets me enter a list of IP addresses. Then I can click one button to produce an excel spreadsheet that contains a list of firewall rules for those addresses. Click another button and you get a text file of commands that can be cut and pasted into the firewall CLI to decommission those addresses. Saves a ton of time. But its not part of my official job duties, and I dont even want to think about the hassle that getting even a single dollar would incur (my company has more than 90,000 employees on six continents, so it is buried in bureaucracy).

Expense it?

Munkeymon
Aug 14, 2003

Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.



To add the the general "why use an ORM" discussion, the company I'm at now has a lot of stored procedures that are basically a handful of select into #temp gobs, of, crap from heaps of joins; join/edit #temp(s); select from #temps that should probably be moved to EF. For one, you can parallelize the selects, if you want. For another, I suspect that query plans that just encompass one select, even if it is a monster with 20 tables, would "go stale"* less often than query plans that wrap up several of them, which is what happens with a sproc.

* by that I mean become a bad fit for the data later and require a statistics recompute or, my 'favorite', adding a single space to an alter procedure to force a plan update

B-Nasty
May 25, 2005

I once upgraded a 700KLOC, legacy .NET 2.0 Webforms application to 4.6, including bringing in the most recent MVC framework. All the data access code was written using good-old ADO.NET queries and datareaders, and it all still worked perfectly in the new versions of .NET nearly 15 years after it was written. Good luck to whoever inherits your project 10 years from now if you tightly-couple to an (long since deprecated/dead) ORM.

Munkeymon
Aug 14, 2003

Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.



B-Nasty posted:

I once upgraded a 700KLOC, legacy .NET 2.0 Webforms application to 4.6, including bringing in the most recent MVC framework. All the data access code was written using good-old ADO.NET queries and datareaders, and it all still worked perfectly in the new versions of .NET nearly 15 years after it was written. Good luck to whoever inherits your project 10 years from now if you tightly-couple to an (long since deprecated/dead) ORM.

Datareaders are just as dead as LINQ2SQL in that both still work just as well as they did when people stopped working on them, though?

Pilsner
Nov 23, 2002

B-Nasty posted:

I once upgraded a 700KLOC, legacy .NET 2.0 Webforms application to 4.6, including bringing in the most recent MVC framework. All the data access code was written using good-old ADO.NET queries and datareaders, and it all still worked perfectly in the new versions of .NET nearly 15 years after it was written. Good luck to whoever inherits your project 10 years from now if you tightly-couple to an (long since deprecated/dead) ORM.
That doesn't really make sense. If I left a ORM-bound project on the shelf, and re-opened it 10 years later, and of course didn't intentionally update the DLL's/Nuget packages for the ORM framework, there would be just as great a chance of it working as with the homebrew model. Frameworks don't go bad like old food or something. If, to make an extreme example, the keyword "SELECT" in MSSQL was changed to "FETCH", you'd need to update your homebrew framework, just like any ORM tied to MSSQL would require an update (and the homebrew framework would need to be changed a million places compared to one...).

I have to be pretty arrogant about this issue; you're simply wrong if you think having hand-written SQL queries spewed out all over a code base is the best design choice for any even remotely big project.

Also, you can choose an open source ORM framework to protect yourself against future obsoleteness. Then you can either modify the ORM framework yourself; or likely, someone else will.

Just-In-Timeberlake
Aug 18, 2003

Pilsner posted:

This is very hard to understand for me. Can you give some code samples? What do you mean "try to populate a string with one of the data fields the DataRow is showing that field is an empty array"??


Yeah, sorry, wasn't sure how clear that was

code:
Dim row As DataRow = dt1.Rows(0)

Dim customer As String = row.Item("customerName")
When I hover over the "row" variable and look at the value in column "customerName" it shows {}

It's probably not really an empty array but how VS displays NULL because that's the error being thrown (can't convert null to string)

Munkeymon posted:

Your database driver might be setting an option that causes some operator to act differently than SSMS. Try running select @@options through both your driver and SSMS and see if they're different.


Thanks for this, not sure if it helps yet but it's a place to start looking at least. The settings are different, now to see if it makes any difference syncing them up.

chippy
Aug 16, 2006

OK I DON'T GET IT

B-Nasty posted:

I once upgraded a 700KLOC, legacy .NET 2.0 Webforms application to 4.6, including bringing in the most recent MVC framework. All the data access code was written using good-old ADO.NET queries and datareaders, and it all still worked perfectly in the new versions of .NET nearly 15 years after it was written. Good luck to whoever inherits your project 10 years from now if you tightly-couple to an (long since deprecated/dead) ORM.

Yeah I don't really get what point you're making there. Whether you tightly couple your application to SqlConnections, DataSets etc. or a DbContext (for example), it's still tightly coupled, and it's not going to magically stop working just because it stops being supported/developed. EF (for example, I realise you are talking about ORMs generally) is just a wrapper over ADO.NET anyway.

B-Nasty
May 25, 2005

I was referring more to the backwards-compatibility of the ORM library, EF for instance. Will EF Core 18.0 be backwards-compatible with v2 (or whatever it is now), because the basic ADO stuff (System.Data), besides DataTables, is still around without any changes.

Dietrich
Sep 11, 2001

Never update your Entity Framework's major version. Whatever you started with is what you should be using until the day you're prepared to re-write your data access layer.

MisterZimbu
Mar 13, 2006
There are plenty of good reasons to avoid EF. Wanting to remain sane is one of them. "It will be deprecated someday, eventually" is not.

edit: Actually, with my experience of .NET frameworks exploding if you look at them wrong, I retract that last part.

MisterZimbu fucked around with this message at 16:46 on Nov 1, 2017

Portland Sucks
Dec 21, 2004
༼ つ ◕_◕ ༽つ

Pilsner posted:

How far you want to go depends a bit on the scale of the project though. You said "my" project, as in a one-man project? But with thousands of DB tables? Did you create all those tables yourself? Are the tables used by anyone else? If the latter, your company should start working on a common data layer that everyone uses, perhaps accessed via a service. That way you won't need to write your own SQL that perhaps some other developer has already written in his/their team's codebase. And when you write the SQL once, or implement an ORM, the next developer who needs to fiddle with the same table, doesn't need to re-write the SQL either.

I'm in a kind of weird boat at my current company. I'm a new software developer (graduated about 4 months ago) working at an industrial manufacturing company and I'm the only developer here. Everyone else here is either an EE/ME/ChemE of some sort. All of our code base is either python25/26 scripts or VB/C# stuff written in the "most simple way possible." Everyone here is extremely averse to using anything more than the basic constructs of these languages outside of basic DB connectors or vendor supplied and supported libraries for business cases because it has to be supportable by the lowest common denominator.

The manufacturing industry as a whole is currently being spun around by the "Smart Factory" concept, and this is causing a demand for more complex homegrown software products hence my new job. So I'm being asked to both find ways to "modernize" and improve on our systems while being cautioned that I don't want to be the only guy capable of supporting the things I make.

All of our database tables were created more or less adhoc by a few specific engineers who created them for data collection on specific platforms to support software they wrote for specific reasons.

quote:

If the latter, your company should start working on a common data layer that everyone uses, perhaps accessed via a service.


I wish.

Honestly I'm usually not the kind of person who goes around thinking my situation is unique, but I''m starting to think I just live in development bizarro world. There is not a single thing going on here that conforms to any kind of standard and the feedback I get is usually "do whatever you want as long as we don't have to change the way we do things over here."

redleader
Aug 18, 2005

Engage according to operational parameters

Munkeymon posted:

For one, you can parallelize the selects, if you want.

How does this work? Does EF spin up a few threads and query IDs 1-1000 on one thread, 1001-2000 on the next, etc?

Munkeymon
Aug 14, 2003

Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.



redleader posted:

How does this work? Does EF spin up a few threads and query IDs 1-1000 on one thread, 1001-2000 on the next, etc?

In this context, each one is hitting a different (but often partially overlapping) set of tables, so each is a self-contained query that could be run on on its own in parallel pretty easily with TPL. Not a good idea if there's a lot of overlap, but possible.

amotea
Mar 23, 2008
Grimey Drawer

Portland Sucks posted:

The manufacturing industry as a whole is currently being spun around by the "Smart Factory" concept, and this is causing a demand for more complex homegrown software products hence my new job. So I'm being asked to both find ways to "modernize" and improve on our systems while being cautioned that I don't want to be the only guy capable of supporting the things I make.
I'd say this is impossible. Software gets so complex so fast nowadays it's hard to keep up for real software engineers, let alone EE type hobbyists.

A lot of techniques have been created over the last decade to deal with this complexity, but the odds of some tinkering code dudes understanding any of it is small. Just "simple" stuff like DI (which is a no-brainer for large C# projects nowadays) probably goes right over the heads of non-programmers.

Edit: I'm not saying the non-programmers are dumb, it's just that there's a very big difference between hacking on some 1000 line in-house tools or writing the next 1M LOC behemoth.

raminasi
Jan 25, 2005

a last drink with no ice

Portland Sucks posted:

The manufacturing industry as a whole is currently being spun around by the "Smart Factory" concept, and this is causing a demand for more complex homegrown software products hence my new job. So I'm being asked to both find ways to "modernize" and improve on our systems while being cautioned that I don't want to be the only guy capable of supporting the things I make.

I wouldnt go so far as to say that what youre being asked to do is necessarily impossible, but it is very difficult, and critically, the challenges arent technical. Your success will depend on how effectively you can gather requirements from people who have no interest in making time for that, how well you can politick around roadblocks, and how much support you have from people with sticks (because you have only carrots).

Portland Sucks
Dec 21, 2004
༼ つ ◕_◕ ༽つ
I think you're both right. I know the non-technical challenges are very much going to come down to my ability to communicate the importance of sound software engineering principals as we move forward.

I also agree that increasing the complexity of our requirements without increasing the level of competency required to maintain them is probably a fools errand.

More to the point I think the reason why I struggle with seeing how an ORM could fit into our existing process is because it seems like ORMs rely on conforming to some kind of baseline with regards to how your backend is structured and ours is literally chaos (they call it "flexibility").

Destroyenator
Dec 27, 2004

Don't ask me lady, I live in beer

Portland Sucks posted:

More to the point I think the reason why I struggle with seeing how an ORM could fit into our existing process is because it seems like ORMs rely on conforming to some kind of baseline with regards to how your backend is structured and ours is literally chaos (they call it "flexibility").
ORMs are designed for persisting and retrieving Objects in the OO sense (think the classic Order has a Customer and a list of LineItems). If the data you have isn't structured that way or you don't interact with it in code as proper domain modelled Objects then ORMs will always be a bad fit. That doesn't mean the code is necessarily wrong it's just that an ORM isn't the right tool for the abstractions you're using.

Pilsner
Nov 23, 2002

Portland Sucks posted:

I'm in a kind of weird boat at my current company. I'm a new software developer (graduated about 4 months ago) working at an industrial manufacturing company and I'm the only developer here. Everyone else here is either an EE/ME/ChemE of some sort. All of our code base is either python25/26 scripts or VB/C# stuff written in the "most simple way possible." Everyone here is extremely averse to using anything more than the basic constructs of these languages outside of basic DB connectors or vendor supplied and supported libraries for business cases because it has to be supportable by the lowest common denominator.

The manufacturing industry as a whole is currently being spun around by the "Smart Factory" concept, and this is causing a demand for more complex homegrown software products hence my new job. So I'm being asked to both find ways to "modernize" and improve on our systems while being cautioned that I don't want to be the only guy capable of supporting the things I make.

All of our database tables were created more or less adhoc by a few specific engineers who created them for data collection on specific platforms to support software they wrote for specific reasons.
Alright, that's a bit of a special situation for sure, so I don't blame you for not using an ORM. Since you say people use different platforms, for specific reasons, and probably in different languages (and probably also some poo poo in Excel/VBA), it's almost hopeless to create a common data layer, or a common data service.

Is there a trend towards using C# or some other fairly modern language? If so, I would start off simple: Start up a data access layer project that at least attempts to contain all SQL and database calls in one project. Then you can slowly start to create some common helper methods. Or even better, dive in a simple ORM (personally I wouldn't recommend EF, but to each their own), and implement that. The people using the data layer won't know the difference, that's the beauty of it.

I also question why one database needs thousands of tables; perhaps some separation would be in place. But alright, if it is super basic, you probably don't have a test environment, release procedures, failover, all that..... just one big mess and deploy straight to production, am I right? Maybe you should just say gently caress it in such a situation and just get your work done however. I've heard this before; if the "developers" of the company are not really developers, it's kinda hopeless to have ambitions for a decent system architecture. Ah well, you get paid anyway...

Portland Sucks
Dec 21, 2004
༼ つ ◕_◕ ༽つ

Pilsner posted:

Alright, that's a bit of a special situation for sure, so I don't blame you for not using an ORM. Since you say people use different platforms, for specific reasons, and probably in different languages (and probably also some poo poo in Excel/VBA), it's almost hopeless to create a common data layer, or a common data service.

Is there a trend towards using C# or some other fairly modern language? If so, I would start off simple: Start up a data access layer project that at least attempts to contain all SQL and database calls in one project. Then you can slowly start to create some common helper methods. Or even better, dive in a simple ORM (personally I wouldn't recommend EF, but to each their own), and implement that. The people using the data layer won't know the difference, that's the beauty of it.

I also question why one database needs thousands of tables; perhaps some separation would be in place. But alright, if it is super basic, you probably don't have a test environment, release procedures, failover, all that..... just one big mess and deploy straight to production, am I right? Maybe you should just say gently caress it in such a situation and just get your work done however. I've heard this before; if the "developers" of the company are not really developers, it's kinda hopeless to have ambitions for a decent system architecture. Ah well, you get paid anyway...

I'm impressed that you managed to basically ride the entire emotional roller-coaster I've been on to completion fairly accurately.

Why do we have thousands of tables? Because the engineers here don't see database tables as anything other than their own personal stack of yellow sticky notes. So every time someone wants to record data they just make a new table. There is obviously no relationship between any of them, it's common to find columns that hold lists of information, etc...

Thinking of these not as databases, but more like garbage poo poo dumps helps me sleep at night.

I've kinda been working on standing up a development environment. I have a VM that I use as my own personal source control and CI/CD platform. Even though the databases are fractured and pretty dumb, they're small so I'm planning on mirroring all of them onto my VM as well so I can at least have an isolated test environment eventually.

Half of me wants to say gently caress it and just get the work done, but the downside to that is I'm just dooming myself to only working with lovely python scripts and .NET 2.0 code. If I make a plan to slowly improve our architecture my job at least becomes somewhat interesting.

Scikar
Nov 20, 2005

5? Seriously?

LongSack posted:

LOL, no company. Its just me. I am by profession an information security engineer (I manage firewalls and other threat devices for our clients) but I have been programming since the late 1970s and most recently I produce utilities that help me automate things I do as part of my day-to-day work. For example, the most useful of my tools is a program that lets me enter a list of IP addresses. Then I can click one button to produce an excel spreadsheet that contains a list of firewall rules for those addresses. Click another button and you get a text file of commands that can be cut and pasted into the firewall CLI to decommission those addresses. Saves a ton of time. But its not part of my official job duties, and I dont even want to think about the hassle that getting even a single dollar would incur (my company has more than 90,000 employees on six continents, so it is buried in bureaucracy).

There's no cost involved to get a code signing certificate from an internal certificate authority. All you need is for all of your colleagues to have the same root CA installed, which is normally done automatically as part of the domain in an org that size. The only challenge is going to be navigating your company security policy and finding an admin for the CA, but if you're in InfoSec then that should be easy enough?

power crystals
Jun 6, 2007

Who wants a belly rub??

Portland Sucks posted:

The manufacturing industry as a whole is currently being spun around by the "Smart Factory" concept, and this is causing a demand for more complex homegrown software products hence my new job. So I'm being asked to both find ways to "modernize" and improve on our systems while being cautioned that I don't want to be the only guy capable of supporting the things I make.

raminasi posted:

I wouldnt go so far as to say that what youre being asked to do is necessarily impossible, but it is very difficult, and critically, the challenges arent technical. Your success will depend on how effectively you can gather requirements from people who have no interest in making time for that, how well you can politick around roadblocks, and how much support you have from people with sticks (because you have only carrots).

From someone who has effectively the exact same job (.NET guy in the land of industrial controls systems) but has been at it for a number of years, yes, entirely this. Maybe 5% of my actual time is spent architecting stuff, instead it's all about figuring out what people actually want you to do (hint: it is only ever tangentially the same as what they say they want). I've also gotten the exact same speech about "we can't have only you being able to maintain this" speech right after being given a webforms project that emits its entire HTML layout programmatically. With every project I have been ever so slightly nudging us towards a more maintainable architecture including real CI etc and now we're actually in pretty decent shape. Just make sure you can demonstrate why your solution is better than what's already out there for when you get the inevitable "but that's harder, why should we do that?" comment.

If I understand right it sounds like you're building the entire thing in-house, which while messy, does have the advantage of not having to deal with any of the commercial SCADA/HMI packages, which are all completely terrible and I hope you like COM interop, and every database has no foreign keys and often no primary keys either :suicide: On the plus side everybody thinks I am some kind of wizard genius for getting this stuff to work together!

LongSack
Jan 17, 2003

Scikar posted:

There's no cost involved to get a code signing certificate from an internal certificate authority. All you need is for all of your colleagues to have the same root CA installed, which is normally done automatically as part of the domain in an org that size. The only challenge is going to be navigating your company security policy and finding an admin for the CA, but if you're in InfoSec then that should be easy enough?

Thats an interesting possibility - that there may already be an internal CA that I can use. I will def follow up on that. But it raises another question: these programs (a dozen or so now) have all been developed by me, on my own time, and are not a part of my normal job duties, so they are unquestionably mine in terms of copyright. So do I want my certificate(s) ties up with my company? Maybe, maybe not. If I can get a single cert and use it to sign all my apps, then I think Ill go that route. If not, the Ill see if I can ride a company CA.

amotea
Mar 23, 2008
Grimey Drawer
Note that Windows Smartscreen will still complain for a while after you sign your software, it needs to build up reputation by being downloaded and installed without problem by an N amount of Windows users.

The certificates are basically a Microsoft tax. Just like you can pay to be a gold certified partner (all the lovely outsourcing companies are gold deluxe platinum partners, meaning they employ 2+ guys that memorized some multiple choice questions).

B-Nasty
May 25, 2005

amotea posted:

The certificates are basically a Microsoft tax.

To be fair, though, I wish more software used code-signing certificates to sign their installers/binaries. It's not really Microsoft's fault that CAs gouge for CSC, most likely due to them not having the downward price pressure that TLS certs did.

We need a LetsEncrypt for CSCs.

Scikar
Nov 20, 2005

5? Seriously?

LongSack posted:

Thats an interesting possibility - that there may already be an internal CA that I can use. I will def follow up on that. But it raises another question: these programs (a dozen or so now) have all been developed by me, on my own time, and are not a part of my normal job duties, so they are unquestionably mine in terms of copyright. So do I want my certificate(s) ties up with my company? Maybe, maybe not. If I can get a single cert and use it to sign all my apps, then I think Ill go that route. If not, the Ill see if I can ride a company CA.

I'm not a copyright lawyer by any means so take this with a grain of salt. But the CA certificate just means effectively that the CA trusts your installer. It shouldn't have anything to do with the rights to your code, after all you could produce 10 installers for 10 different companies all signed using different certificates obtained from each company's internal CA. The code underneath is still the same.

If you get a cert from a global CA then you should only need the one for all of your code.

Adbot
ADBOT LOVES YOU

mystes
May 31, 2006

B-Nasty posted:

We need a LetsEncrypt for CSCs.
As I stated earlier, all code signing certificates are effectively EV certificates, so this wouldn't be possible. Simply validating that an application really came from a web server at ksjhgsjsghkjgskjhg.com would be pointless (that's what SSL already does)..

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply