We do Grafana + Prometheus for most of our clients but I think that adding Loki into the mix might be necessary. The amount of clients that are missing basic events like "you've run out of disk space...two days ago", is too damn high.

for X in $(seq -f host%02g 1 9); do echo $X; ssh -q $X “grep the shit”; done

:)

But yeah fair, I do actually use a big data stack for log monitoring and searching… it’s just way more usable haha

Just write a bash script to loop over them.

You can run the logs command against a label so it will match all 36 pods

Stern has been around for ever. You could also just use a shared label selector with kubectl logs and then grep from there. You make it sound difficult if not impossible, but it's not. Combine it with egrep and you can pretty much do anything you want right there on the CLI

I don't know how k8s works; but if there is a way to execute just one command in a container and then exit out of it like chroot; wouldn't it be possible to just use xargs with a list of the container names?

Since you are talking about pods, you are obviously emitting all your logs on stdout and stderr, and you have of course also labeled your pods nicely, so grepping all 36 gods is as easy as kubectl logs -l <label-key>=<label-value> | grep <search-term>

Let me introduce you to syslogd.

But well, it's probably overkill, and you almost certainly just need to log on a shared volume.

This is what I was thinking. And you can't really graph out things over time on a graph which is really critical for a lot of workflows.

I get that Splunk and Elastic or unwieldy beasts that take way too much maintenance for what they provide for many orgs but to think grep is replacement is kinda crazy.

That's why tmux has synchronize-panes!

Amazing. Depressing, but amazing.

remember this shit when people talk about how we can't just give people money for doing nothing

we're already just inventing problems for people to fix so we can justify paying them

It's such an insane amount of money

What a nice world you must live in where all your code is perfectly clean, documented and properly tracked.

"Log" is the name of the place you write your tracing information into.

It's a command line tool which filters for all lines containing the query. So something like

cat log.txt | grep Error5

Would output only lines containing Error5

I'm not sure if you're serious or not.

At my job they unilaterally decided that we no longer had access to our application logs in any way other than a single company wide grafana with no access control (which means anyone can see anything and seeing the stats and logs of only your stuff is a PITA).

Half the time the relevant log línes straight up don't show up unless you use a explicit search for their content (good luck finding relevant information for an unknown error) and you're extremely limited in how many log línes you can see at once.

Not to mention that none of our applications were designed with this platform in mind so all the logging is done in a legacy way that conforms to the idea of just grepping a log file and there's no way the sponsors will commit to letting us spend weeks adjusting our legacy applications to actually log in a way that is useful for viewing in grafana and not a complete shitshow.

I've worked with a logstash/elastic/kibana stack for years before this job and I can tell you these solutions aren't meant for seeing lines one by one or context searches (where seeing what happened right before and after matters a lot), they're meant for aggregations and analysis.

It's like moving all your stuff from one house to another in a tiny electric car. Sure technically it can be done but that's not it's purpose at all and good luck moving your fridge.