Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Sylink
Apr 17, 2004

Prometheus owns, if anyone has questions we use it all the time.

Adbot
ADBOT LOVES YOU

Sylink
Apr 17, 2004

Mostly node exporter for basic system metrics. Disable all the collectors you dont need, one thing to do in prometheus is to not collect metrics you dont use or want, saves on space and tedium , by default a ton of poo poo is on the node exporter, and it will catch all kinds of useless disk device metrics depending on your setup (looking at u kubernetes)

We also use autodiscovery and we run kubernetes clusters, so there are a ton of useful kubernetes prometheus tie ins, prometheus operators is great.

Anyway, you get all your collectors and exporters up, then the fun begins. Now we build dashboards in grafana and they cover each level of our application, so we a cluster view, a node view, and an application view to drill down levels and into problems.

The dashboards are also committed to code if they are important.

The real skill is creating dashboards that are useful and understanding what each metric really is. A lot of prom stuff is counters which will trick you as they just go up over time, so you have to remember to take irates and so on.

Depending on your app, its very easy to make custom Prometheus exporters for scraping custom metrics. Do this where possible. Every thing you can turn into a simple number metric is less digging through bullshit logs.

Logs loving suck and should be your last restort as much as possible imo. To the point where you go to the log just to confirm what you already know from the metrics in explicit text form.


And alerts are so easy to make and tweak, since they are just PromQL and the same queries you are using in the dashboards but with some condition attached, so you can play around in the Prom dash looking at data, find the right metrics, and copy paste that into an alert pretty much.

Sylink
Apr 17, 2004

No problem, I could talk for days about metrics and data collection.

My only complaint about prometheus is that its documented poorly with regards to actual usage, so you have to dig around. But this site has pretty much every link you'll find on Google https://www.robustperception.io

Sylink
Apr 17, 2004

Does opentracing have anything for PHP that doesn't require editing your code project like the way New Relic works?

I work mainly with performance on PHP apps (they suck) and New Relic traces/transaction data is extremely useful, and easy to use since we dont control the codebases. I.e. the PHP module comes up and does magic so we dont have to deal with editing anyone's code.

I've seen alternatives, but none are as easy and painless.

But New Relic pricing sucks if you have a lot of services you want to use it on.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply