IMHO in a way the constant churn of Grafana made it easier to live with it. When they break your dashboards every major release you just learn to let go. And by break it’s not just making it error and not work, it’s the constant moving of things around, refactoring the UI, replacing one component with another, all accompanied by a number of glitches every time they rewrite things. You just accept and ignore it eventually.
What’s a bigger lock in for me is metrics and promql - you just can’t ever rename a poorly named metric or you face a world of pain. Or when Prometheus releases Native Histograms to replace the old ones, and suddenly everything from rules, alerts, ad-hoc queries and dashboards needs updating.
And PromQL is so opaque, it just never give you an error unless there is a syntax issue. We need tools like https://github.com/cloudflare/pint just to know if my alert description isn’t trying to render a label that’s just not gonna be there, etc
Building an elaborate pile of technical debt is a great way to have an elaborate pile of technical debt, but the lifespan of services being 2-3 years gets painful as you start composing a stack out of enough products that every quarter you need to replace something big.
Mimir is just architected for a totally different order of magnitude of metrics. At that scale, yeah, Kafka is actually necessary. There are no other open-source solutions offering the same scalability, period.
That's besides the point that most customers will never need that level of scale. If you're not running Mimir on a dedicated Kubernetes cluster (or at least a dedicated-to-Grafana / observability cluster) then it's probably over-engineered for your use-case. Just use Prometheus.
Have a look at Victoria Metrics - have run it at a relatively high scale with much more success than any other metric stores. It's one of those things that just work. It's extremely easy to run at in a single-instance mode and handles much more than you would expect. Scaling it is a breeze too.
(I'm not affiliated, but a very happy user across multiple orgs and personal projects)
The project where I looked at Mimir was a 500+ million timeseries project, with the desire to support scaling to the ten-figure level of timeseries (working for a BigCo supporting hundreds of product development teams).
All of these systems that store metrics in object storage - you have to remember that object storage is not file storage. Generally speaking (stuff like S3 One Zone being a relatively recent exception) you cannot append to object files. Metrics queries are resolved by querying historical metrics in object storage plus a stateful service hosting the latest 2 hours of data before it can be compressed and uploaded to object storage as a single block. At a certain scale, you simply need to choose which is more important - being able to answer queries or being able to insert more timeseries. And if you don't prioritize insertion, it just results in the backlog getting bigger and bigger, which especially in the eventual case (Murphy's Law guarantees it) of a sudden flood of metrics to ingest will cause several hour ingestion delays during which you are blind. And if you do prioritize insertion, well the component simply won't respond to queries, which makes you blind anyway. Lose-lose.
Mimir built in Kafka because it's quite literally necessary at scale. You need the stateful query component (with the latest 2 hours) to prioritize queries, then pull from the Kafka topic on a lower priority thread, when there's spare time to do so. Kafka soaks up the sudden ingestion floods so that they don't result in the stateful query component getting DoS'd.
I took a quick look at VictoriaMetrics - no Kafka or Kafka-like component to soak up ingestion floods? DOA.
Again, most companies are not BigCos. If you're a startup/scaleup with one VP supervising several development teams, you likely don't need that scale, probably VictoriaMetrics is just fine, you're not the first person I've heard recommend it. But I would say 80% of companies are small enough to be served with a simple Prometheus or Thanos Query over HA Prometheus setup, 17% of companies will get a lot of value out of Victoria Metrics, the last 3% really need Mimir's scalability.
What's your preferred solution for observability and monitoring of tiny apps?
I'm looking for something with really compact storage, really simple deployment (preferably a single statically linked binary that does everything), and compatible with OpenTelemetry (including metrics and distributed tracing). If/when I outgrow it, I can switch to another OpenTelemetry provider (but realistically this will not happen)
I'm personally not convinced OpenTelemetry is the future. I get the desire to not be vendor-locked to a single provider, but Prometheus and Jaeger are very solid, battle-hardened, popular, well-maintained, easily self-hosted open-source projects. For small deployments you do not need to overthink things here - Grafana, Prometheus, Jaeger (with local disk storage), logging depends on how many machines you're talking about and where they're hosted (e.g. GCP Cloud Logging is fine for GCP-hosted projects, the 50 GB free tier is a lot for a small project) but as a default Loki is also just fine and much better than Elastic/OpenSearch.
OpenTelemetry is, last I looked at it, way too immature, unstable, and resource-hungry to be such a foundational part of infrastructure.
I'm looking at OpenTelemetry because of broad tooling compatibility (both Rust tracing crates, tracing and emit, support it - for logs, tracing and metrics) and it seems like something that will stick around.
Also I'm not sure I will ever need actual performance out of an observability solution; it's a tiny app after all.
What's the most promising alternative to Prometheus/Grafana if you're developing a new solution around OTEL? If you could start today and pick tools, what would you go for?
We also started with the typical kube-prometheus-stack, but we don’t like Prometheus/PromQL. Moreover, it only solves the „metrics“ part - to handle logs and traces, more quite heavy and complex components have to be added to the observability stack.
This didn‘t feel right, so we looked around and found greptimedb https://github.com/GreptimeTeam/greptimedb, which simplifies the whole stack. It‘s designed to handle metrics, logs, and traces. We collect metrics and logs via OpenTelemetry, and visualize them with Grafana. It provides endpoints for Postgres, MySQL, PromQL; we‘re happy to be able to build dashboards using SQL as that’s where we have the most knowledge.
The benchmarks look promising, but our k8s clusters aren’t huge anyway. As a platform engineer, we appreciate the simplicity of our observability stack.
Any other happy greptimedb users around here? Together with OTel, we think we can handle all future obs needs.
Having been using Grafana community/cloud for a number of years, my new gig is currently moving everything to SigNoz. Mostly slick, under active development, communicative team, open source... what's not to love?
I frequently use a docker-compose template with prometheus pushgateway + grafana for deploying on single node servers, as described at the start of the article. It works well and is trivial to setup, but the complexity explodes once your metric volume or cardinality requires more scale like prometheus alternatives a la mimir.
I think this would not need to be an issue as frequently if prometheus had a more efficient publish/scraping mechanism. iirc there was once a protobuf metric format that was dropped, and now there is just the text format. While it wouldn't handle billions of unique labels like mimir, a compact binary metric format could certainly allow for millions at reasonable resolution instead of wasting all that scale potential on repeated name strings. I should be able to push or expose a bulk blob all at once with ordered labels or at least raw int keys.
Not sure what's an alternative for Grafana in the open source world in terms of building dashboards for o11y? I'm not aware of one and Grafana is used very extensively in my company...
I use Signoz for my private purposes, it's not a 100% match, but you can do Prometheus metrics, logs analysis, dashboards, alerts, OTEL spans so depending on your usecase it can be enough
from cursory reading of the article I don’t see that author’s problems are specifically with Grafana in its best use case (metrics), but with other products from Grafana company, for which are a lot of alternatives.
Grafana dashboards itself (paired with VictoriaMetrics and occasionally Clickhouse) is one of the most pleasant web apps IMO. Especially when you don’t try to push the constraints of its display model, which are sometimes annoying but understandable.
I remember that alternative, free/FOSS products existed before Grafana (c2015) but many died, Grafana was everywhere. Now I also cannot find the old-alts. Vague memories of RRD and Nagios...
Munin was what we used for a while, along with a smattering of smokeping.
We're using a combination of Zabbix (alerting) and local Grafana/Prometheus/Loki (observability) at this point, but I've been worried about when Grafana will rug-pull for a while now. Hopefully enough people using their cloud offering sates their appetite and they leave the people running locally alone.
https://github.com/opensearch-project/OpenSearch-Dashboards (Kibana fork) is one. But Grafana is still way better if you just stay away from anything that isn't the core product: data visualization and exploration (explorer and traces).
I mentioned it in another reply, but https://perses.dev/ is probably the most promising alternative.
Besides that, if you're feeling masochistic you could use Prometheus' console templates or VictoriaMetrics' built-in dashboards.
Though these are all obviously nowhere near as feature rich and capable as Grafana and would only be able to display metrics for the single Prom/VM node they're running on. Might enough for some users.
Author here.
I know the old way still works and I respect that. Given the history I ask myself how long will it work, since ist not the default anymore.
> "But I got it all working; now I can finally stop explaining to my boss why we need to re-structure the monitoring stack every year."
Prometheus and Grafana have been progressing in their own ways and each of them is trying to have a fullstack solution and then the OTEL thingy came and ruined the party for everyone
I still haven't got my head around how OTEL fits into a good open-source monitoring stack. Afaik, it is a protocol for metrics, traces, and logs. And we want our open-source monitoring services/dbs to support it, so they become pluggable. But, afaik, there's no one good DB for logs and metrics, so most of us use Prometheus for metrics and OpenSearch for logs.
Does OTEL mean we just need to replace all our collectors (like logstash for logs and all the native metrics collectors and pushgateway crap) and then reconfigure Prometheus and OpenSearch?
logs, spans and metrics are stored as time-stamped stuff. sure simple fixed-width columnar storage is faster, and makes sense to special case for numbers (add downsampling and aggregations, and histogram maintenance and whatnot), but any write-optimized storage engine can handle this, it's not the hard part (basically LevelDB, and if there's need for scaling out it'll look like Cassandra, Aerospike, ScyllaDB, or ClickHouse ... see also https://docs.greptime.com/user-guide/concepts/data-model/ and specialized storage engines https://docs.greptime.com/reference/about-greptimedb-engines... )
I think the answer is it doesn't fit in any definition of a _good_ monitoring stack, but we are stuck with it. It has largely become the blessed protocol, specification, and standard for OSS monitoring, along every axis (logging, tracing, collecting, instrumentation, etc)...its a bit like the efforts that resulted in J2EE and EJBs back in the day, only more diffuse and with more varied implementations.
And we don't really have a simpler alternative in sight...at least in the java days there was the disgust and reaction via struts, spring, EJB3+, and of course other languages and communities.
Not sure how we exactly we got into such an over-engineered mono-culture in terms of operations and monitoring and deployment for 80%+ of the industry (k8s + graf/loki/tempo + endless supporting tools or flavors), but it is really a sad state.
Then you have endless implementations handling bits and pieces of various parts of the spec, and of course you have the tools to actually ingest and analyze and report on them.
I have found Grafana to be a decent product, but Prom needs a better horizontally scalable solution. We use Vector and Clickhouse for logging and works really well.
I second SigNoz. I was paying a fortune for a cloud observability platform that cost more and more every month. Then I switched to self-hosted SigNoz on a cheap Hetzner box and now my observability stack costs $10 a month.
I only know of https://perses.dev/ but haven't had a look at it for ~half a year. It was very barebones back then but I'm hopeful it can replace Grafana for at least basic dashboarding soon.
This is pretty interesting to me, as I do use Grafana in my current role. But none of their other products, and not their helm chart (we're on the Bitnami chart if that's a thing).
So far it's pretty good. We're at least one major version behind, but hey everything still works.
I cannot imagine other products support as many data sources (though I'm starting to think they all suck, I just dump what I can in InfluxDB).
I agree. I think OP has made the mistake of using more than just Grafana for dashboards and perhaps user queries.
I operate a fairly large custom VictoriaMetrics-based Observability platform and have learned early on to only use Grafana as opposed to other Grafana products. Part of the stack used to use Mimir's frontend as caching layer but even that died with Mimir v3.0, now that it can't talk to generic Prometheus APIs anymore (vanilla Prom, VictoriaMetrics, promxy etc.). I went back to Cortex for caching.
Such a custom stack is obviously not for everyone and takes much more time, knowledge and effort to deploy than some helm chart but overall I'd say it did save me some headache. At least when compared to the Google-like deprecation culture Grafana seems to have.
off topic, but prometheus pushgateway is such a bad implementation (once you push the metrics, it always stays there until it's restarted, like counter does not increase, it just pushes a new metric with the new value) that we had to write our own metrics collector endpoint.
That is literally how it is supposed to work. Prometheus grabs metrics --- that is how it works. If you for some reason find yourself unable to host an endpoint with metrics, you can use the fallback pushgateway to push metrics where yes they will stay until restarted. Ask yourself how it could ever work if they are subsequently deleted after read. How would multiple prometheus agents be able to read from the same source?
It sounds like you are using it for the wrong job. It’s supposed to be a solution for jobs / short running processes that don’t expose a /metrics endpoint for Prometheus long enough to be scraped and there you exactly want that kind of behavior.
The pushgateway is itself a horrible hack for the fact that prometheus is designed only for metrics scraping. Unfortunately the whole ecosystem around it is an utter mess.
Remote Write is a viable alternative in Prometheus and its drop-in replacements. I'm not a massive fan of it myself as I feel the pull-based approach is superior overall but still make heavy use of it.
The pushgateway's documentation itself calls out that there are only very limited cirumstances where it makes sense.
I personally only used it in $old_job and only for batch jobs that could not use the node_exporter's textfile collector. I would not use it again and would even advise against it.
FTA > "I know for a fact that that pace is partially driven by career-driven development."
This isn't a Grafana problem, this is an industry wide problem. Resume driven product design, resume driven engineering, resume driven marketing. DO your 2-3 years, pump out something big to inflate your resume. Apply elsewhere to get the pay bump that almost no company is handing out. After the departures there is no one left who knows the system and the next people in want to replace the things they don't understand to pad their resume for the next job.
Wash, rinse, repete.
Loyalty, simply goes unrewarded in a lot of places in our industry (and at a many corporations). And the people who do stay... in many cases they turn into furniture that ends up holding potential good evolution back. They loose out to the technological magpies the bring shiny things to management because it will "move the needle".
Sadly this is just one facet of the problems we are facing, from how we interview to how we run (or rent) our infrastructure things have gotten rather silly...
without any stability, you really can’t blame the player for playing this game.
The days where you could devote your career to a firm and retire with a pension are long gone
The author of this article wants a boring tech stack that just works, and honestly after everything we’ve been through in the last five years, I kinda want a boring job I can keep until I retire, too
what are tested and fairly lightweight alternatives for Loki?
elastic stack is so heavy it's out of question for smaller clusters, loki integration with grafana is nice to have but separate capable dashboard would be also fine
This article comes off sort of low effort and mentions a lot grievances without actual pinpointing precise issues. I think leveraging OTEL as a general processor with a generic output is a good idea, but discounting Grafana for implementing multi tenancy solutions and alloy which is pretty fucking good is kind of pointless.
As someone who runs SaaS products, this post resonates painfully well.
The author is 100% correct: Monitoring should be the most boring tool in the stack. Its one and only job is to be more reliable than the thing it's monitoring.
The moment your monitoring stack requires a complex dependency like Kafka, or changes its entire agent flow every 18 months, it has failed its primary purpose. It has become the problem.
This sounds less like a technical evolution and more like the classic VC-funded push to get everyone onto a high-margin cloud product, even at the cost of the open-source soul.
IMHO in a way the constant churn of Grafana made it easier to live with it. When they break your dashboards every major release you just learn to let go. And by break it’s not just making it error and not work, it’s the constant moving of things around, refactoring the UI, replacing one component with another, all accompanied by a number of glitches every time they rewrite things. You just accept and ignore it eventually.
What’s a bigger lock in for me is metrics and promql - you just can’t ever rename a poorly named metric or you face a world of pain. Or when Prometheus releases Native Histograms to replace the old ones, and suddenly everything from rules, alerts, ad-hoc queries and dashboards needs updating.
And PromQL is so opaque, it just never give you an error unless there is a syntax issue. We need tools like https://github.com/cloudflare/pint just to know if my alert description isn’t trying to render a label that’s just not gonna be there, etc
Building an elaborate pile of technical debt is a great way to have an elaborate pile of technical debt, but the lifespan of services being 2-3 years gets painful as you start composing a stack out of enough products that every quarter you need to replace something big.
Mimir is just architected for a totally different order of magnitude of metrics. At that scale, yeah, Kafka is actually necessary. There are no other open-source solutions offering the same scalability, period.
That's besides the point that most customers will never need that level of scale. If you're not running Mimir on a dedicated Kubernetes cluster (or at least a dedicated-to-Grafana / observability cluster) then it's probably over-engineered for your use-case. Just use Prometheus.
Have a look at Victoria Metrics - have run it at a relatively high scale with much more success than any other metric stores. It's one of those things that just work. It's extremely easy to run at in a single-instance mode and handles much more than you would expect. Scaling it is a breeze too.
(I'm not affiliated, but a very happy user across multiple orgs and personal projects)
The project where I looked at Mimir was a 500+ million timeseries project, with the desire to support scaling to the ten-figure level of timeseries (working for a BigCo supporting hundreds of product development teams).
All of these systems that store metrics in object storage - you have to remember that object storage is not file storage. Generally speaking (stuff like S3 One Zone being a relatively recent exception) you cannot append to object files. Metrics queries are resolved by querying historical metrics in object storage plus a stateful service hosting the latest 2 hours of data before it can be compressed and uploaded to object storage as a single block. At a certain scale, you simply need to choose which is more important - being able to answer queries or being able to insert more timeseries. And if you don't prioritize insertion, it just results in the backlog getting bigger and bigger, which especially in the eventual case (Murphy's Law guarantees it) of a sudden flood of metrics to ingest will cause several hour ingestion delays during which you are blind. And if you do prioritize insertion, well the component simply won't respond to queries, which makes you blind anyway. Lose-lose.
Mimir built in Kafka because it's quite literally necessary at scale. You need the stateful query component (with the latest 2 hours) to prioritize queries, then pull from the Kafka topic on a lower priority thread, when there's spare time to do so. Kafka soaks up the sudden ingestion floods so that they don't result in the stateful query component getting DoS'd.
I took a quick look at VictoriaMetrics - no Kafka or Kafka-like component to soak up ingestion floods? DOA.
Again, most companies are not BigCos. If you're a startup/scaleup with one VP supervising several development teams, you likely don't need that scale, probably VictoriaMetrics is just fine, you're not the first person I've heard recommend it. But I would say 80% of companies are small enough to be served with a simple Prometheus or Thanos Query over HA Prometheus setup, 17% of companies will get a lot of value out of Victoria Metrics, the last 3% really need Mimir's scalability.
What's your preferred solution for observability and monitoring of tiny apps?
I'm looking for something with really compact storage, really simple deployment (preferably a single statically linked binary that does everything), and compatible with OpenTelemetry (including metrics and distributed tracing). If/when I outgrow it, I can switch to another OpenTelemetry provider (but realistically this will not happen)
I'm personally not convinced OpenTelemetry is the future. I get the desire to not be vendor-locked to a single provider, but Prometheus and Jaeger are very solid, battle-hardened, popular, well-maintained, easily self-hosted open-source projects. For small deployments you do not need to overthink things here - Grafana, Prometheus, Jaeger (with local disk storage), logging depends on how many machines you're talking about and where they're hosted (e.g. GCP Cloud Logging is fine for GCP-hosted projects, the 50 GB free tier is a lot for a small project) but as a default Loki is also just fine and much better than Elastic/OpenSearch.
OpenTelemetry is, last I looked at it, way too immature, unstable, and resource-hungry to be such a foundational part of infrastructure.
OpenObserve might be what you're looking for
>compatible with OpenTelemetry
Isn't OpenTelemetry very slow?
What do you suggest instead?
I'm looking at OpenTelemetry because of broad tooling compatibility (both Rust tracing crates, tracing and emit, support it - for logs, tracing and metrics) and it seems like something that will stick around.
Also I'm not sure I will ever need actual performance out of an observability solution; it's a tiny app after all.
What's the most promising alternative to Prometheus/Grafana if you're developing a new solution around OTEL? If you could start today and pick tools, what would you go for?
We also started with the typical kube-prometheus-stack, but we don’t like Prometheus/PromQL. Moreover, it only solves the „metrics“ part - to handle logs and traces, more quite heavy and complex components have to be added to the observability stack.
This didn‘t feel right, so we looked around and found greptimedb https://github.com/GreptimeTeam/greptimedb, which simplifies the whole stack. It‘s designed to handle metrics, logs, and traces. We collect metrics and logs via OpenTelemetry, and visualize them with Grafana. It provides endpoints for Postgres, MySQL, PromQL; we‘re happy to be able to build dashboards using SQL as that’s where we have the most knowledge.
The benchmarks look promising, but our k8s clusters aren’t huge anyway. As a platform engineer, we appreciate the simplicity of our observability stack.
Any other happy greptimedb users around here? Together with OTel, we think we can handle all future obs needs.
Having been using Grafana community/cloud for a number of years, my new gig is currently moving everything to SigNoz. Mostly slick, under active development, communicative team, open source... what's not to love?
I frequently use a docker-compose template with prometheus pushgateway + grafana for deploying on single node servers, as described at the start of the article. It works well and is trivial to setup, but the complexity explodes once your metric volume or cardinality requires more scale like prometheus alternatives a la mimir.
I think this would not need to be an issue as frequently if prometheus had a more efficient publish/scraping mechanism. iirc there was once a protobuf metric format that was dropped, and now there is just the text format. While it wouldn't handle billions of unique labels like mimir, a compact binary metric format could certainly allow for millions at reasonable resolution instead of wasting all that scale potential on repeated name strings. I should be able to push or expose a bulk blob all at once with ordered labels or at least raw int keys.
Not sure what's an alternative for Grafana in the open source world in terms of building dashboards for o11y? I'm not aware of one and Grafana is used very extensively in my company...
I use Signoz for my private purposes, it's not a 100% match, but you can do Prometheus metrics, logs analysis, dashboards, alerts, OTEL spans so depending on your usecase it can be enough
from cursory reading of the article I don’t see that author’s problems are specifically with Grafana in its best use case (metrics), but with other products from Grafana company, for which are a lot of alternatives.
Grafana dashboards itself (paired with VictoriaMetrics and occasionally Clickhouse) is one of the most pleasant web apps IMO. Especially when you don’t try to push the constraints of its display model, which are sometimes annoying but understandable.
I remember that alternative, free/FOSS products existed before Grafana (c2015) but many died, Grafana was everywhere. Now I also cannot find the old-alts. Vague memories of RRD and Nagios...
Munin was what we used for a while, along with a smattering of smokeping.
We're using a combination of Zabbix (alerting) and local Grafana/Prometheus/Loki (observability) at this point, but I've been worried about when Grafana will rug-pull for a while now. Hopefully enough people using their cloud offering sates their appetite and they leave the people running locally alone.
I ran with Centreon for a while because you got Nagios + integrated dashboarding out of the box and a Community option.
I'm out of that game now though so don't have the challenge.
https://www.centreon.com/
https://github.com/opensearch-project/OpenSearch-Dashboards (Kibana fork) is one. But Grafana is still way better if you just stay away from anything that isn't the core product: data visualization and exploration (explorer and traces).
I mentioned it in another reply, but https://perses.dev/ is probably the most promising alternative.
Besides that, if you're feeling masochistic you could use Prometheus' console templates or VictoriaMetrics' built-in dashboards.
Though these are all obviously nowhere near as feature rich and capable as Grafana and would only be able to display metrics for the single Prom/VM node they're running on. Might enough for some users.
We’re using Greylog+Elastic Search which would totally replace a Loki-only stack.
... ugh, they actually made an `o11[a-z]` abbreviation? When I picked this nick, the only term I ever saw in the wild was `i18n`.
K8s (Kubernetes), a11y (accessibility)...
The kicker for me recently was hearing someone say "ally"
a16z, l10n, s11n,
Or without numbers,
authC/authN, authZ...
I work in this area and even I don’t know what AuthC is!
They are absurd abbreviations. The first distinguishing letter comes right after auth, so ... let's hide it?
i d2t s1e t1e p5m...
o11y is not a word. What do you mean?
Observability , in the vein of accessibility which has the silly nickname of a11y
This is all coming from stenography, it's a well established shorthand for long words: first letter, count of middle letters, last letter.
Which is similar to i18n for internationalization and l10n for localization.
> Mimir in version 3.0 needs Apache Kafka to work.
I’d like to adjust this understanding. Kafka is the big new thing, but it’s optional. The previous way using gRPC still works.
I work on Mimir and other things at Grafana Labs.
Author here. I know the old way still works and I respect that. Given the history I ask myself how long will it work, since ist not the default anymore.
> "But I got it all working; now I can finally stop explaining to my boss why we need to re-structure the monitoring stack every year."
Prometheus and Grafana have been progressing in their own ways and each of them is trying to have a fullstack solution and then the OTEL thingy came and ruined the party for everyone
I still haven't got my head around how OTEL fits into a good open-source monitoring stack. Afaik, it is a protocol for metrics, traces, and logs. And we want our open-source monitoring services/dbs to support it, so they become pluggable. But, afaik, there's no one good DB for logs and metrics, so most of us use Prometheus for metrics and OpenSearch for logs.
Does OTEL mean we just need to replace all our collectors (like logstash for logs and all the native metrics collectors and pushgateway crap) and then reconfigure Prometheus and OpenSearch?
logs, spans and metrics are stored as time-stamped stuff. sure simple fixed-width columnar storage is faster, and makes sense to special case for numbers (add downsampling and aggregations, and histogram maintenance and whatnot), but any write-optimized storage engine can handle this, it's not the hard part (basically LevelDB, and if there's need for scaling out it'll look like Cassandra, Aerospike, ScyllaDB, or ClickHouse ... see also https://docs.greptime.com/user-guide/concepts/data-model/ and specialized storage engines https://docs.greptime.com/reference/about-greptimedb-engines... )
I think the answer is it doesn't fit in any definition of a _good_ monitoring stack, but we are stuck with it. It has largely become the blessed protocol, specification, and standard for OSS monitoring, along every axis (logging, tracing, collecting, instrumentation, etc)...its a bit like the efforts that resulted in J2EE and EJBs back in the day, only more diffuse and with more varied implementations.
And we don't really have a simpler alternative in sight...at least in the java days there was the disgust and reaction via struts, spring, EJB3+, and of course other languages and communities.
Not sure how we exactly we got into such an over-engineered mono-culture in terms of operations and monitoring and deployment for 80%+ of the industry (k8s + graf/loki/tempo + endless supporting tools or flavors), but it is really a sad state.
Then you have endless implementations handling bits and pieces of various parts of the spec, and of course you have the tools to actually ingest and analyze and report on them.
I have found Grafana to be a decent product, but Prom needs a better horizontally scalable solution. We use Vector and Clickhouse for logging and works really well.
Signoz is good, and active development https://github.com/SigNoz/signoz
I second SigNoz. I was paying a fortune for a cloud observability platform that cost more and more every month. Then I switched to self-hosted SigNoz on a cheap Hetzner box and now my observability stack costs $10 a month.
But are there good alternatives to grafana in the foss space nowadays?
I only know of https://perses.dev/ but haven't had a look at it for ~half a year. It was very barebones back then but I'm hopeful it can replace Grafana for at least basic dashboarding soon.
This is pretty interesting to me, as I do use Grafana in my current role. But none of their other products, and not their helm chart (we're on the Bitnami chart if that's a thing).
So far it's pretty good. We're at least one major version behind, but hey everything still works.
I cannot imagine other products support as many data sources (though I'm starting to think they all suck, I just dump what I can in InfluxDB).
I agree. I think OP has made the mistake of using more than just Grafana for dashboards and perhaps user queries.
I operate a fairly large custom VictoriaMetrics-based Observability platform and have learned early on to only use Grafana as opposed to other Grafana products. Part of the stack used to use Mimir's frontend as caching layer but even that died with Mimir v3.0, now that it can't talk to generic Prometheus APIs anymore (vanilla Prom, VictoriaMetrics, promxy etc.). I went back to Cortex for caching.
Such a custom stack is obviously not for everyone and takes much more time, knowledge and effort to deploy than some helm chart but overall I'd say it did save me some headache. At least when compared to the Google-like deprecation culture Grafana seems to have.
off topic, but prometheus pushgateway is such a bad implementation (once you push the metrics, it always stays there until it's restarted, like counter does not increase, it just pushes a new metric with the new value) that we had to write our own metrics collector endpoint.
That is literally how it is supposed to work. Prometheus grabs metrics --- that is how it works. If you for some reason find yourself unable to host an endpoint with metrics, you can use the fallback pushgateway to push metrics where yes they will stay until restarted. Ask yourself how it could ever work if they are subsequently deleted after read. How would multiple prometheus agents be able to read from the same source?
It sounds like you are using it for the wrong job. It’s supposed to be a solution for jobs / short running processes that don’t expose a /metrics endpoint for Prometheus long enough to be scraped and there you exactly want that kind of behavior.
The pushgateway is itself a horrible hack for the fact that prometheus is designed only for metrics scraping. Unfortunately the whole ecosystem around it is an utter mess.
Remote Write is a viable alternative in Prometheus and its drop-in replacements. I'm not a massive fan of it myself as I feel the pull-based approach is superior overall but still make heavy use of it.
The pushgateway's documentation itself calls out that there are only very limited cirumstances where it makes sense.
I personally only used it in $old_job and only for batch jobs that could not use the node_exporter's textfile collector. I would not use it again and would even advise against it.
> I want stability for my monitoring; I want it boring, and that’s something Grafana is not offering
I used to be a fan of InfluxDB (back in the days of v1.x) then I went off it for exactly this reason.
Sounds like grafana needed to fork
This reads like a satire. There's so much jargon and so many products involved to just do a little bit of logging. It's ridiculous.
That is to say I agree with the author.
It’s as though they deliberately make it complex and constantly moving to wear you out to give in to the convenient SaaS offer.
I feel this article in my bones. It's rough out there.
FTA > "I know for a fact that that pace is partially driven by career-driven development."
This isn't a Grafana problem, this is an industry wide problem. Resume driven product design, resume driven engineering, resume driven marketing. DO your 2-3 years, pump out something big to inflate your resume. Apply elsewhere to get the pay bump that almost no company is handing out. After the departures there is no one left who knows the system and the next people in want to replace the things they don't understand to pad their resume for the next job.
Wash, rinse, repete.
Loyalty, simply goes unrewarded in a lot of places in our industry (and at a many corporations). And the people who do stay... in many cases they turn into furniture that ends up holding potential good evolution back. They loose out to the technological magpies the bring shiny things to management because it will "move the needle".
Sadly this is just one facet of the problems we are facing, from how we interview to how we run (or rent) our infrastructure things have gotten rather silly...
without any stability, you really can’t blame the player for playing this game.
The days where you could devote your career to a firm and retire with a pension are long gone
The author of this article wants a boring tech stack that just works, and honestly after everything we’ve been through in the last five years, I kinda want a boring job I can keep until I retire, too
what are tested and fairly lightweight alternatives for Loki?
elastic stack is so heavy it's out of question for smaller clusters, loki integration with grafana is nice to have but separate capable dashboard would be also fine
I find the color of the text so light as to be unreadable.
The font doesn't render correctly on my device. It seems like as if some strokes are double, making lines inconsistent.
It’s challenging to read for me too.
the "dark" theme having a *blaring* white banner at the top is an ... interesting design choice
This article comes off sort of low effort and mentions a lot grievances without actual pinpointing precise issues. I think leveraging OTEL as a general processor with a generic output is a good idea, but discounting Grafana for implementing multi tenancy solutions and alloy which is pretty fucking good is kind of pointless.
As someone who runs SaaS products, this post resonates painfully well.
The author is 100% correct: Monitoring should be the most boring tool in the stack. Its one and only job is to be more reliable than the thing it's monitoring.
The moment your monitoring stack requires a complex dependency like Kafka, or changes its entire agent flow every 18 months, it has failed its primary purpose. It has become the problem.
This sounds less like a technical evolution and more like the classic VC-funded push to get everyone onto a high-margin cloud product, even at the cost of the open-source soul.