Monitoring is complicated. If I have all this right:

Grafana is the UI. You only need one instance. It reads data from sources.

Prometheus is a data source for numbers. You feed it with aggregators like CAdvisor (container monitor) and Node Exporter (host monitor). Prometheus does numbers.

Loki is Prometheus, but for log data. It gets fed by Alloy.

Each host has Node Exporter and CAdvisor. Each container has Alloy. You need one Prometheus, one Loki, and one Grafana.

And that's just the stack. Then you get into data collection and transformation. At any point in the chain, you can adjust data. Alloy can parse JSON, add labels, modify timestamps, and more, then send it all to Loki. But Loki can also perform transformations. Oh, and not to be left out, Grafana can transform data, too. Prometheus can visualize and modify data, but it's also used to feed Grafana. But before all that, CAdvisor can, I think, transform data. I never knew monitoring was this complex