Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
ModeSix
Mar 14, 2009

This is both entriguing and baffling at the same time.

We've come full circle from:

HTML - We don't render it's just a page to display poo poo.
PHP - Server Rendering
Jquery/AJAX - A bit of both
React/Angular/etc. - MAKE THE BROWSER RENDER IT AND DO MOST OF THE WORK!
React SSR - poo poo, that thing we did is too slow on mobile devices. Ok let's try PHP again but use javascript instead and tack on some gRPC to it so it's modern.

It's amusing.

Adbot
ADBOT LOVES YOU

prom candy
Dec 16, 2005

Only I may dance

ModeSix posted:

This is both entriguing and baffling at the same time.

We've come full circle from:

HTML - We don't render it's just a page to display poo poo.
PHP - Server Rendering
Jquery/AJAX - A bit of both
React/Angular/etc. - MAKE THE BROWSER RENDER IT AND DO MOST OF THE WORK!
React SSR - poo poo, that thing we did is too slow on mobile devices. Ok let's try PHP again but use javascript instead and tack on some gRPC to it so it's modern.

It's amusing.

I'm seeing this take a lot over the past few weeks and it's really missing the point. The jump from HTML to PHP was "people want to do dynamic poo poo on the web." The jump to using jQuery/AJAX was "people want to make updates to just part of the page rather than having to do a full HTTP Request/Response cycle for everything." The jump to React/Angular was "people want web applications to feel as snappy as the mobile applications that they've now gotten used to and building those interfaces with jQuery is painful" and now the jump to SSR meta-frameworks is "people still want web applications to feel as snappy as mobile applications but also we want all the benefits of SSR and we don't want to duplicate a bunch of code between client and server."

If you can't tell the difference between what something like Next or Remix or SvelteKit offers vs. PHP I imagine we're working on very different types of projects.

Dominoes
Sep 20, 2007

Much of web dev culture has been poisoned. Some cultures are better than others.

On the plus side, some of the tooling is better than ever:
- Modern JS solves many problems of the past
- TS requires a build step and some non-intuitive config, but other than that, is an upgrade over JS.
- Old frameworks like Rails and Django are better than ever
- New micro frameworks are available in Python, Rust etc; of varying quality
- WASM, if used appropriately, can be nice for complex frontends
- Rust in particular is a beautiful language; better suited to other domains, but with WASM, can be applied here too. (carefully)

- NPM/webpack/react/angular/CSS frameworks/dependencies-by-default/complete lack of regard for performance and maintainability can DIAF.

As always, computer hardware has continued to get faster. This means if you ditch the bloat (Bloat isn't there by default! You have to deliberately add it.), computer programs and websites/webapps can be startling responsive and quick-to-download.

Dominoes fucked around with this message at 19:36 on Nov 1, 2022

Head Bee Guy
Jun 12, 2011

Retarded for Busting
Grimey Drawer
I’m teaching myself some webdev while my job is slow, and I’m starting by making a blog/sketchbook. I hope to use these skills for work, so I want to learn best industry practices and follow a logical workflow from design to dev to deployment.

So far, I bought a domain on dreamhost, where it’s hosted, and I was able to successfully upload an html skeleton using sftp that renders on the live sire.

I’d like to use github for version control, and I want the live site to update when I make changes to the production branch of the repository. How do people do this? Most of the documentation I’m finding is about “github pages” but i’d rather not move the hosting.

Also, any suggested resources I read for using a dev server to test out changes before updating the live site? As well as integrating it with the github workflow?

prom candy
Dec 16, 2005

Only I may dance

Head Bee Guy posted:

I’d like to use github for version control, and I want the live site to update when I make changes to the production branch of the repository. How do people do this? Most of the documentation I’m finding is about “github pages” but i’d rather not move the hosting.

One way to do this is with GitHub Actions. GitHub Actions is basically a pipeline that will run arbitrary code when stuff happens with your repos. So you would set up an action that says "when I update the production branch, upload it to my server." Since your server is accessed via FTP, you might be able to use this prebuilt action: https://github.com/marketplace/actions/ftp-deploy

You can also skip the FTP stuff completely and just host your site with a tool like Vercel or Netlify. I think it would be free for a small static site and it would link into your GitHub automatically. Deploying sites via FTP isn't done all that often anymore (although I'm sure it still happens a lot)

Head Bee Guy
Jun 12, 2011

Retarded for Busting
Grimey Drawer

prom candy posted:

One way to do this is with GitHub Actions. GitHub Actions is basically a pipeline that will run arbitrary code when stuff happens with your repos. So you would set up an action that says "when I update the production branch, upload it to my server." Since your server is accessed via FTP, you might be able to use this prebuilt action: https://github.com/marketplace/actions/ftp-deploy

You can also skip the FTP stuff completely and just host your site with a tool like Vercel or Netlify. I think it would be free for a small static site and it would link into your GitHub automatically. Deploying sites via FTP isn't done all that often anymore (although I'm sure it still happens a lot)

Interesting, how do a lot of small/medium sized companies deploy if not through ftp?

prom candy
Dec 16, 2005

Only I may dance

Head Bee Guy posted:

Interesting, how do a lot of small/medium sized companies deploy if not through ftp?

So many ways! Lots of people host their stuff on cloud service providers like Vercel and Netlify. Lots of people containerize their applications and run kubernetes (although this is massively overkill for small and medium size companies). Lots of people containerize their application and host them with some kind of container hosting platform like Railway or Render.com or Fly.io. For a long time in the Rails world we used Capistrano to deploy application which was basically using SCP under the hood. Some people just clone their repos on their servers and run git pull (I don't recommend this).

I personally haven't deployed a site via FTP in... I dunno, a decade? More? It's totally fine, there's nothing wrong with it for a hobby site, I just haven't used that kind of webhosting in a long time, mostly because it's a pain in the rear end to set up NodeJS or Rails hosting on them compared to uploading .html or .php files.

fsif
Jul 18, 2003

prom candy posted:

I'm seeing this take a lot over the past few weeks and it's really missing the point. The jump from HTML to PHP was "people want to do dynamic poo poo on the web." The jump to using jQuery/AJAX was "people want to make updates to just part of the page rather than having to do a full HTTP Request/Response cycle for everything." The jump to React/Angular was "people want web applications to feel as snappy as the mobile applications that they've now gotten used to and building those interfaces with jQuery is painful" and now the jump to SSR meta-frameworks is "people still want web applications to feel as snappy as mobile applications but also we want all the benefits of SSR and we don't want to duplicate a bunch of code between client and server."

If you can't tell the difference between what something like Next or Remix or SvelteKit offers vs. PHP I imagine we're working on very different types of projects.

Just need to say I appreciate your posts being a bastion of sanity. The obtuseness of developers pretending that web dev ten years ago didn't completely suck poo poo compared to today is one of the more baffling parts of the culture.

Last Chance
Dec 31, 2004

prom candy posted:

Some people just clone their repos on their servers and run git pull (I don't recommend this).
Lol we used to do this at a job I worked at.

minato
Jun 7, 2004

cutty cain't hang, say 7-up.
Taco Defender
Pre-containerization, it was not the worst way to do a deploy. Git is really good at moving a versioned set of files between hosts, and making the target directory look exactly how you want it. rsync could only do part of that.

prom candy
Dec 16, 2005

Only I may dance

Last Chance posted:

Lol we used to do this at a job I worked at.

Same and it's not my least favourite way of doing deploys to be honest

fsif posted:

Just need to say I appreciate your posts being a bastion of sanity. The obtuseness of developers pretending that web dev ten years ago didn't completely suck poo poo compared to today is one of the more baffling parts of the culture.

I'm glad I'm not alone in thinking this way. I'm seeing a lot of these kinds of takes on Twitter and it's grating.

Head Bee Guy
Jun 12, 2011

Retarded for Busting
Grimey Drawer

prom candy posted:


Some people just clone their repos on their servers and run git pull (I don't recommend this).



thank you for saving me from doing this.

Data Graham
Dec 28, 2009

📈📊🍪😋



I do it via git and it's pure joy.

I'm sure it will be nicer if I ever get docker working, ever, in any circumstance, and/or have a ci/cd wizard come help me and save me from my own idiocy.

MrMoo
Sep 14, 2000

Dominoes posted:

- TS requires a build step and some non-intuitive config, but other than that, is an upgrade over JS.

I cannot believe how much a pile of poo poo any webdev configuration is. A plethora of options and acronyms.

Even basic things like file extensions are still a joke, is it .ts, .js, or .mjs? I'm currently seeing that a module written in TypeScript with relative imports with an assumed extension fails to import to a TypeScript app using import maps because JS has no idea which extension to use. Common workaround appears to be bundling modules into a single file, or directly importing into the app build process.

barkbell
Apr 14, 2006

woof
its real bad op

prom candy
Dec 16, 2005

Only I may dance

Head Bee Guy posted:

thank you for saving me from doing this.

It's actually probably fine for a solo hobby project. It's a step up from FTP in terms of making sure all the changes are synced and being able to roll back. It just doesn't scale super well, ie if you want to run a bunch of servers behind a load balancer or you have a bunch of devs committing code and you want to gate deploys behind an automated lint/test process.

Vincent Valentine
Feb 28, 2006

Murdertime

It's probably the ideal way for a solo hobby project, honestly. It's good to do that stuff by hand so you can get an idea of how it actually works as you scale it up with automation. And you should definitely implement some form of automation. I've heard really good things about using github actions as a beginner but I can't attest to it personally.

kedo
Nov 27, 2007

Vincent Valentine posted:

It's probably the ideal way for a solo hobby project, honestly. It's good to do that stuff by hand so you can get an idea of how it actually works as you scale it up with automation. And you should definitely implement some form of automation. I've heard really good things about using github actions as a beginner but I can't attest to it personally.

I worked up a full deployment method with Github actions recently and it was a pretty smooth process. Granted I have yet to implement it, but in theory and in tests it worked very nicely and had all the bells and whistles I was looking for.

Summit
Mar 6, 2004

David wanted you to have this.
Hello all. I'm hoping this is the best thread to post this question, as it could easily fit into a few others.

I'm moving to a new company to help remake their backend API server. From my understanding they haven't made a decision on which framework to organize around yet. I'll likely be tasked to weigh in on this decision and so I'm levelling up my understanding of the options. The requirements of this decision are it needs to be Node based, support GraphQL, and there's a strong culture of strictly typing, so the more built in TypeScript is the better.

For the frontend they are firmly in the NextJS camp, and this NextJS frontend application will likely utilize GraphQL to call the backend. This isn't firmly in stone but it's pretty likely at this point, as they prefer the built in documentation of the gql schema to something like a REST API with documentation they'd need to review to use it. One important note is they utilize a monorepo so there's potential that frontend projects could directly import the TS types of the backend project.

So I'm looking through all the options, and what I'm finding is there are a lot! One standout I'm seeing is NestJS. I think the solo dev who's already working on this team likes the framework, as he used it on a prior role. It's purpose built around TypeScript. It's also extremely opinionated, which could be a good thing or bad depending on the team. I've learned a good deal about it as this is the current primary option I've heard about, but I also wanna make sure I know about other alternatives. I'm well aware of simple Express as I've done a lot with that. It's likely the most flexible, unopinionated option one could go with. It can be a simple or complex as one desires really; it's very choose your own adventure. I've also started to learn about things like Koa, fastify, adonisjs - there's many to choose from!

Getting to the point, does anyone have any recent experience developing a backend only server around one of these frameworks? Any pros or cons you could share?

prom candy
Dec 16, 2005

Only I may dance

Summit posted:

Hello all. I'm hoping this is the best thread to post this question, as it could easily fit into a few others.

I'm moving to a new company to help remake their backend API server. From my understanding they haven't made a decision on which framework to organize around yet. I'll likely be tasked to weigh in on this decision and so I'm levelling up my understanding of the options. The requirements of this decision are it needs to be Node based, support GraphQL, and there's a strong culture of strictly typing, so the more built in TypeScript is the better.

For the frontend they are firmly in the NextJS camp, and this NextJS frontend application will likely utilize GraphQL to call the backend. This isn't firmly in stone but it's pretty likely at this point, as they prefer the built in documentation of the gql schema to something like a REST API with documentation they'd need to review to use it. One important note is they utilize a monorepo so there's potential that frontend projects could directly import the TS types of the backend project.

So I'm looking through all the options, and what I'm finding is there are a lot! One standout I'm seeing is NestJS. I think the solo dev who's already working on this team likes the framework, as he used it on a prior role. It's purpose built around TypeScript. It's also extremely opinionated, which could be a good thing or bad depending on the team. I've learned a good deal about it as this is the current primary option I've heard about, but I also wanna make sure I know about other alternatives. I'm well aware of simple Express as I've done a lot with that. It's likely the most flexible, unopinionated option one could go with. It can be a simple or complex as one desires really; it's very choose your own adventure. I've also started to learn about things like Koa, fastify, adonisjs - there's many to choose from!

Getting to the point, does anyone have any recent experience developing a backend only server around one of these frameworks? Any pros or cons you could share?

Is this backend being built just for this frontend or are they expecting multiple clients? If this is just a 1:1 backend-for-frontend doing it with GQL and in a separate codebase is probably the worst way to go. I'd do it with API routes in the NextJS app and use tRPC for end-to-end type safety. Hell if there are multiple clients calling out to the backend but they're all written in TS/JS and they're all in the monorepo I'd still probably look at tRPC instead of GQL. GQL has a lot of tradeoffs which is my hedging way of saying GQL sucks poo poo.

Summit
Mar 6, 2004

David wanted you to have this.

prom candy posted:

Is this backend being built just for this frontend or are they expecting multiple clients? If this is just a 1:1 backend-for-frontend doing it with GQL and in a separate codebase is probably the worst way to go. I'd do it with API routes in the NextJS app and use tRPC for end-to-end type safety. Hell if there are multiple clients calling out to the backend but they're all written in TS/JS and they're all in the monorepo I'd still probably look at tRPC instead of GQL. GQL has a lot of tradeoffs which is my hedging way of saying GQL sucks poo poo.

It's a multiple client situation. There's the current webapp but also a planned mobile app being built in React Native I believe. Both will be in the monorepo.

Also: one thing that comes to mind is versioning. Especially if one consumer is a mobile app, with all the review store issues around that, how does something like trpc handle versioning?

Summit fucked around with this message at 16:39 on Nov 4, 2022

prom candy
Dec 16, 2005

Only I may dance

Summit posted:

It's a multiple client situation. There's the current webapp but also a planned mobile app being built in React Native I believe. Both will be in the monorepo.

Also: one thing that comes to mind is versioning. Especially if one consumer is a mobile app, with all the review store issues around that, how does something like trpc handle versioning?

I don't think tRPC supports versioning necessarily, but neither does GraphQL really. If you want a whole new API you'd set up a new schema at a new endpoint. Most of the time with GraphQL at least you just avoid making breaking changes and you deprecate stuff only after if it's been gone for a long time.

When you build with React Native you can actually push a lot of updates without having to go through app store review. It's only when you change the native bindings that you need to push out a new version. With mobile apps it can also be a good idea to have a killswitch in the app, so your server sends back a minimum app version header with each request and if your app doesn't meet that version you can tell your lazy users they have to upgrade. Ideally you maintain backwards compat for as long as possible but sometimes you just need to stop supporting old versions of the app.

I haven't used it but there's also this project which aims to generate an OpenAPI REST API from a tRPC installation so that you can use it easily with external clients as well https://github.com/jlalmes/trpc-openapi

Edit: I should add, I haven't used tRPC in production myself. I'm stuck maintaining a GraphQL API and a couple of REST APIs built in Ruby on Rails for the time being. But tRPC is 100% what I would reach for if I was starting a greenfield project.

teen phone cutie
Jun 18, 2012

last year i rewrote something awful from scratch because i hate myself

teen phone cutie posted:

we're live with a little less than an MVP:

https://staging.basementcommunity.com/

and here's like the one test thread:

https://staging.basementcommunity.com/threads/1

and again the BE is done with the docs at https://docs.basementcommunity.com/

What works:

1. Create and edit posts
2. Create threads
3. Create forums
4. Login / Register
6. Trigger forgot password email and change password form
7. Favoriting threads
8. Sticky threads appear at top with a sticky icon

What's next

1. Probation / Ban UI
2. Leper's colony type page
3. User profiles and updating account settings
4. Dockerize the front-end
5. Updating posts to show quotes and mentions correctly
6. Private messages
7. View your mentions
8. Reporting post or PMs
9. Bookmarks page
10. Mod/Admin tools
12. Allow markdown in posts
13. UHHHHH a lot more I'm not even close



I've been re-making something awful over in a project.log for fun. I'm writing it with Remix and a python backend (that's mostly finished) and looking for feedback so far if anyone has any to share!!!!!

teen phone cutie fucked around with this message at 01:42 on Nov 6, 2022

Busy Bee
Jul 13, 2004
I wanted to follow up on the below post from a few months ago as I recently finished the MVP of my web app. It took a while but I took the advice I received here and spoke to many people and was able to create the MVP.

I want to take it to the next stage by hiring a web developer to create the web app and wanted to ask for advice here on how to best proceed. I have found a few web developers on Fiverr but wondering if this would be a good way forward.

CarForumPoster posted:

If your goal is to make a fitness app, make a prototype app first. You'll learn so much from the process of getting your app in front of users that you'll laugh at the idea of thinking about the scalability or programming language. Focus solely on the top 5 or 10 functional requirements, then get a prototype that barely works and is clunky in front of people. (In app parlance this is known as a minimum viable product, a concept credited to the excellent book The Lean Startup)

Think of it this way: If you app actually solves a problem, you'll get paying users or you'll run ads. Then you can hire people who can code well to make a good app.

Check this video out:
https://www.youtube.com/watch?v=1hHMwLxN6EM

prom candy
Dec 16, 2005

Only I may dance

Busy Bee posted:

I wanted to follow up on the below post from a few months ago as I recently finished the MVP of my web app. It took a while but I took the advice I received here and spoke to many people and was able to create the MVP.

I want to take it to the next stage by hiring a web developer to create the web app and wanted to ask for advice here on how to best proceed. I have found a few web developers on Fiverr but wondering if this would be a good way forward.

How serious are you about the project? A developer on fiverr is probably not going to be great but they could get you to MVP. A key thing is that you're going to have to be very specific with the dev about what you want done. Alternately if you actually want to build a business here you could try to use your prototype to raise money and hire a CTO, but the investment market isn't great now that interest rates are higher. You could also try to recruit a CTO for no salary and 50% equity but you'd better have a very nice prototype and a very clear go to market plan to get a developer to work for no cash up front.

CarForumPoster
Jun 26, 2013

⚡POWER⚡

Busy Bee posted:

I wanted to follow up on the below post from a few months ago as I recently finished the MVP of my web app. It took a while but I took the advice I received here and spoke to many people and was able to create the MVP.

I want to take it to the next stage by hiring a web developer to create the web app and wanted to ask for advice here on how to best proceed. I have found a few web developers on Fiverr but wondering if this would be a good way forward.

I hired a top rated Pakistani dev on Fiverr once, a few years ago. I'll PM you the one if you want. Theyre currently 4.8 stars with >200 reviews. I paid them fairly but still much cheaper than in the USA and gave them VERY specific requirements. They were remaking something I'd already built a mockup/MVP prototype of, and I ended up needing to redo a lot of their work. Still, it got the project going, which freed me up to work on other stuff. Expect to micromanage them. Expect to be firm and SPECIFIC with your feedback. Expect them to not follow instructions super well. You can say make it exactly like this, with a screenshot and list of requirements, and theyll say they are done and hand over something with clearly different fonts, missing this or that, etc. The value was good but expect junior dev type issues. Definitely agree with poster above that said this could be MVP2.

I'd still suggest seeing how far you can get with low/no code. Are the things youre missing by not having a custom made app things your users want that you simply can't do? Or are they nice to haves that you'd like to add?

If you're past that point and your users need features that you cant get through a no code app or no code + Zapier type solution, MVP2 from a Fiverr person might not be a bad idea. Expect to throw it away. Expect it to be buggy. Make sure you know how to handle your users data security. Frankly you're still gonna need someone who knows a thing or two about coding just to be able to keep the MVP2 running, get it deployed, etc. prom candy identified some possible solutions to this.

kiwid
Sep 30, 2013

kiwid posted:

I work for a grain elevator and we have an inventory tracking application that was written a long time ago that is insecure (sql injection, etc.) and so I've taken it offline. I've been asked to build the successor to this app.

Currently the app has a "site map" where each silo/bin is clickable which allows you to do various tasks with it (like update inventory, etc.).



In its current form this is just styled <button> elements into circles and rectangles and then positioned with tables that have a bunch of rowspan and colspan attributes. I feel like this can be modernized but I've been struggling to figure out an improvement. I've looked at libraries like D3 which are just absurdly complex for what I want to do. I've checked out fabric.js but I'm on the fence with that. Then I decided to see if what I wanted to do was possible with just straight up SVGs but I'm running into issues with that too. Specifically, getting text to be constrained within the shapes.



How would you guys approach this? Should I just stick to what works with <button> and <table>?

For anyone that is curious, I ended up going with a panning and zooming via JavaScript approach with absolute positioned anchors. It's probably an accessibility nightmare but it does work well on both desktop and mobile with touch support and pinch to zoom. As a fallback, they get a table (as it always should have been).

Here is the result:


And the fallback:

spacebard
Jan 1, 2007

Football~

kiwid posted:

via JavaScript approach with absolute positioned anchors. It's probably an accessibility nightmare but it does work well on both desktop and mobile with touch support and pinch to zoom.

Nicely done. Thanks for the update.

I don't think anchors would be much of an accessibility nightmare unlike images embedded in a canvas element. Just need to describe the elements properly, and let the user use keyboard nav / tab equivalent. And it would be possible to add map controls that use the increment the various scroll properties.

Last Chance
Dec 31, 2004

kiwid posted:

For anyone that is curious, I ended up going with a panning and zooming via JavaScript approach with absolute positioned anchors. It's probably an accessibility nightmare but it does work well on both desktop and mobile with touch support and pinch to zoom. As a fallback, they get a table (as it always should have been).

Here is the result:


And the fallback:


this is awesome, great job fulfilling those requirements, but you're right that table view is much nicer and will be for anyone new that needs the tool and doesn't understand why the legacy view looks all "bubbly"

kedo
Nov 27, 2007

I assumed they were "bubbly" because they represent grain silos which are usually big cylinders.

kiwid
Sep 30, 2013

kedo posted:

I assumed they were "bubbly" because they represent grain silos which are usually big cylinders.

Yes exactly, it's supposed to represent a top down view of the site/yard. I would have loved to get high res satellite imagery of the sites and just done like an image area map or something but these facilities are all out in very rural areas that don't get the same google maps resolution that a city would get. Google maps also has very outdated imagery in some areas.

Example:

kiwid fucked around with this message at 15:43 on Nov 15, 2022

LifeLynx
Feb 27, 2001

Dang so this is like looking over his shoulder in real-time
Grimey Drawer

"Here's our new logo, could you add it to the site?"

Roadie
Jun 30, 2013

prom candy posted:

I'm seeing this take a lot over the past few weeks and it's really missing the point. The jump from HTML to PHP was "people want to do dynamic poo poo on the web." The jump to using jQuery/AJAX was "people want to make updates to just part of the page rather than having to do a full HTTP Request/Response cycle for everything." The jump to React/Angular was "people want web applications to feel as snappy as the mobile applications that they've now gotten used to and building those interfaces with jQuery is painful" and now the jump to SSR meta-frameworks is "people still want web applications to feel as snappy as mobile applications but also we want all the benefits of SSR and we don't want to duplicate a bunch of code between client and server."

If you can't tell the difference between what something like Next or Remix or SvelteKit offers vs. PHP I imagine we're working on very different types of projects.

:yeah:

kedo
Nov 27, 2007

LifeLynx posted:

"Here's our new logo, could you add it to the site?"

Stop triggering me.

camoseven
Dec 30, 2005

RODOLPHONE RINGIN'
I have an interview tomorrow for a role I am somewhat underqualified for but REALLY REALLY want. The product is data visualization of real time data streams, and the frontend is written in React. I've been thinking about it and the main issue seems like the bazillions of re-renders that could happen, so I'm thinking a strategy here would be some kind of batching of those rerenders? Like instead of rerendering 10 charts 100 times a second each, you would batch it together and cap it at like 10 renders a second each. Does that seem sound?

Basically I'm asking how I can BS my way through the data viz part so I can focus on my strengths, please help me

Anony Mouse
Jan 30, 2005

A name means nothing on the battlefield. After a week, no one has a name.
Lipstick Apathy

camoseven posted:

I have an interview tomorrow for a role I am somewhat underqualified for but REALLY REALLY want. The product is data visualization of real time data streams, and the frontend is written in React. I've been thinking about it and the main issue seems like the bazillions of re-renders that could happen, so I'm thinking a strategy here would be some kind of batching of those rerenders? Like instead of rerendering 10 charts 100 times a second each, you would batch it together and cap it at like 10 renders a second each. Does that seem sound?

Basically I'm asking how I can BS my way through the data viz part so I can focus on my strengths, please help me

In general, you should lean towards gathering real world data and analyzing that to make decisions. Don't make too many assumptions, and don't overly pre-optimize. If I were in your shoes, my first thought wouldn't be "I bet they're having issues with React re-renders", but "is there even a performance issue in the first place? If so, how does it affect the UI? What tasks are eating up the most time -- blocking CPU tasks, network requests, browser render compositing, or something else?" (I hope you're familiar with both standard and React debug profilers -- if not, become so.) Or an even more straightforward question, "what is the #1 problem you're facing today that I could help solve?"

If you aren't sure, ask questions that will help you better understand the problem. Keep asking questions. Do that until you feel like you have a good understanding of what the core problem to solve is. Don't be afraid to say something like "based on this info I think the root problem is X, and I don't have an immediate solution off the top of my head but I'd start my research there." I'd be 1,000% more impressed with a candidate who asked the right questions to understand the problem, than one who made assumptions and gave a technically competent answer that was way off target. "Solutions" are teachable; intuition, curiosity and communication are far less so.

That said, here are a few simple things I'd keep in mind if I were jumping into a project like that and were concerned about performance bottlenecks:

- Yeah, you could try batching renders or reducing the frequency of updates. How "real-time" does this need to be? Updated once per second? Updated dozens of times per second? Does it matter to the user or not, and why or why not?
- You can leverage React memoization to re-use previous renders for unchanged UI elements; likely the vast majority of it will not change between renders except the visualization itself
- React re-renders are not inherently bad. Even a poorly optimized app probably won't give a poo poo about re-renders. Browser paints/renders/compositing are what's expensive.
- Are there CPU-intensive tasks that could be offloaded to web workers?
- If absolutely necessary, you can step outside of the React render loop to do some things manually, e.g. keeping a reference to a <canvas> element and updating it directly

Anony Mouse fucked around with this message at 22:49 on Nov 16, 2022

camoseven
Dec 30, 2005

RODOLPHONE RINGIN'

Anony Mouse posted:

In general, you should lean towards gathering real world data and analyzing that to make decisions. Don't make too many assumptions, and don't overly pre-optimize. If I were in your shoes, my first thought wouldn't be "I bet they're having issues with React re-renders", but "is there even a performance issue in the first place? If so, how does it affect the UI? What tasks are eating up the most time -- blocking CPU tasks, network requests, browser render compositing, or something else?" (I hope you're familiar with both standard and React debug profilers -- if not, become so.) Or an even more straightforward question, "what is the #1 problem you're facing today that I could help solve?"

If you aren't sure, ask questions that will help you better understand the problem. Keep asking questions. Do that until you feel like you have a good understanding of what the core problem to solve is. Don't be afraid to say something like "based on this info I think the root problem is X, and I don't have an immediate solution off the top of my head but I'd start my research there." I'd be 1,000% more impressed with a candidate who asked the right questions to understand the problem, than one who made assumptions and gave a technically competent answer that was way off target. "Solutions" are teachable; intuition, curiosity and communication are far less so.

That said, here are a few simple things I'd keep in mind if I were jumping into a project like that and were concerned about performance bottlenecks:

- Yeah, you could try batching renders or reducing the frequency of updates. How "real-time" does this need to be? Updated once per second? Updated dozens of times per second? Does it matter to the user or not, and why or why not?
- You can leverage React memoization to re-use previous renders for unchanged UI elements; likely the vast majority of it will not change between renders except the visualization itself
- React re-renders are not inherently bad. Even a poorly optimized app probably won't give a poo poo about re-renders. Browser paints/renders/compositing are what's expensive.
- Are there CPU-intensive tasks that could be offloaded to web workers?
- If absolutely necessary, you can step outside of the React render loop to do some things manually, e.g. keeping a reference to a <canvas> element and updating it directly

Hell yea this was really helpful thanks

MrMoo
Sep 14, 2000

camoseven posted:

I have an interview tomorrow for a role I am somewhat underqualified for but REALLY REALLY want. The product is data visualization of real time data streams, and the frontend is written in React. I've been thinking about it and the main issue seems like the bazillions of re-renders that could happen, so I'm thinking a strategy here would be some kind of batching of those rerenders? Like instead of rerendering 10 charts 100 times a second each, you would batch it together and cap it at like 10 renders a second each. Does that seem sound?

Basically I'm asking how I can BS my way through the data viz part so I can focus on my strengths, please help me

They are all garbage so I wouldn't worry too much, it's good to have solutions as they need to hire to get there. Visualisation is always near real-time, i.e. update the screen every 1-5s, backed by an OLAP.

I need a charting library that can render 10 charts at 300Hz, but I'm lucky enough to get a chart to render under 100ms from any third party so far. WebGL ones for finance don't count as they are all lazy and skip filling under the plot.

Rendering server side and streaming SVG is a wonderful workaround if you have the server capacity for it.

Most interesting vendor so far is probably ag-grid, but slightly arrogant youtube piece.

https://www.youtube.com/watch?v=29ja0liMuv4&t=546s

Basically only render what you can see. Maximum of 4 updates per second per number, and reduce to 1 update per second for numbers one is not directly staring at constantly.

You are not really going to get more than 30,000 updates per second without some serious effort, and that most likely needs to be distributed across a pool of WebWorkers to perform business logic processing if not already done upstream. The renderer thread should be only taking preprocessed data and sticking it in the DOM on a RAF callback.

MrMoo fucked around with this message at 02:58 on Nov 17, 2022

camoseven
Dec 30, 2005

RODOLPHONE RINGIN'

MrMoo posted:

They are all garbage so I wouldn't worry too much, it's good to have solutions as they need to hire to get there. Visualisation is always near real-time, i.e. update the screen every 1-5s, backed by an OLAP.

I need a charting library that can render 10 charts at 300Hz, but I'm lucky enough to get a chart to render under 100ms from any third party so far. WebGL ones for finance don't count as they are all lazy and skip filling under the plot.

Rendering server side and streaming SVG is a wonderful workaround if you have the server capacity for it.

Most interesting vendor so far is probably ag-grid, but slightly arrogant youtube piece.

https://www.youtube.com/watch?v=29ja0liMuv4&t=546s

Basically only render what you can see. Maximum of 4 updates per second per number, and reduce to 1 update per second for numbers one is not directly staring at constantly.

You are not really going to get more than 30,000 updates per second without some serious effort, and that most likely needs to be distributed across a pool of WebWorkers to perform business logic processing if not already done upstream. The renderer thread should be only taking preprocessed data and sticking it in the DOM on a RAF callback.

Ty I am going to read sentences from this post during the interview

Adbot
ADBOT LOVES YOU

MrMoo
Sep 14, 2000

For companies like Tableau, "real-time" can mean as slow as once an hour (https://help.tableau.com/current/online/en-us/schedule_add.htm#create-a-refresh-schedule), and as fast as every 15mins (https://community.tableau.com/s/que...kbook-in-server)

TIBCO is all over the place because they bought so many companies and completely lost their real-time mojo a long time ago. Currently looking like being able to set graph refreshes at 1-120s (https://kb.informationbuilders.com/topic/configuring-automatic-refresh-option-charts-8206).

It's all relative when working with Big Data and not fair comparing with not-Big Data, i.e. SQL stores and similar which offer seconds level of latency.

MrMoo fucked around with this message at 08:20 on Nov 17, 2022

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply