Ask HN: What is the best enterprise software you use every day? • Hacker News Board

Re: Ask HN: What is the best enterprise software you use every day?

#361

Post by user5994461 » Fri, Jul 17, 2020, 5:41 PM UTC

aprdm wrote: post #351

user5994461 wrote: post #319

Earlier quoted context omitted.

Datadog has incredible integration with clouds (AWS and other), databases (postgresql, mysql, cassandra) and middleware (haproxy, kafka). It can capture all the metrics from all these out of the box with minimal effort, whereas you have to crawl hundreds of broken plugins for prometheus to get one third of that. If you're using clouds (AWS/Azure/Google). Datadog can capture all the AWS metadata automatically and merg…

> Datadog has incredible integration with clouds (AWS and other), databases (postgresql, mysql, cassandra) and middleware (haproxy, kafka). It can capture all the metrics from all these out of the box with minimal effort, whereas you have to crawl hundreds of broken plugins for prometheus to get one third of that. Interesting, I haven't had this experience. I monitor the DBs and middleware you mention and the OSS plu…

The question would be which plugins exactly? because there are tons of plugins spread over GitHub, more or less working, and they're constantly shifting. I've had maybe 15 integrations working perfectly on datadog in 2 weeks of work. The same on prometheus/grafana could have taken 6 months easily (with few exporters to write from scratch).

The cloud does make a difference. Just seeing the daily S3 usage per bucket was life changing. Immediately found that backups were not expiring after a while as they should, costing more and more money. ^^

Do you know how many metrics you are ingesting in prometheus? storage size? and how many tags per host? We were reaching 1 TB of memory usage (mmap) on our server with 1500 hosts. Prometheus was literally grinding to a halt or crashing, was forced to cut down some metrics and stick to the absolute minimum tags.

Try prometheus_tsdb_head_series and prometheus_tsdb_storage_blocks_bytes or du command on the directory.

Re: Ask HN: What is the best enterprise software you use every day?

#362

Post by aprdm » Fri, Jul 17, 2020, 6:15 PM UTC

user5994461 wrote: post #361

aprdm wrote: post #351

Earlier quoted context omitted.

> Datadog has incredible integration with clouds (AWS and other), databases (postgresql, mysql, cassandra) and middleware (haproxy, kafka). It can capture all the metrics from all these out of the box with minimal effort, whereas you have to crawl hundreds of broken plugins for prometheus to get one third of that. Interesting, I haven't had this experience. I monitor the DBs and middleware you mention and the OSS plu…

The question would be which plugins exactly? because there are tons of plugins spread over GitHub, more or less working, and they're constantly shifting. I've had maybe 15 integrations working perfectly on datadog in 2 weeks of work. The same on prometheus/grafana could have taken 6 months easily (with few exporters to write from scratch). The cloud does make a difference. Just seeing the daily S3 usage per bucket wa…

Interesting, I have a very small team that maintains a lot of infrastructure including Prometheus. Setting up the monitoring for 5 datacenters, thousands of hosts and hundreds of services was maybe a ~2 weeks effort. We then organically add and remove things but it is maybe a ~4h/week effort.

It give us a lot of value. We were using other solutions before (sensu/nagios) and it is night and day.

All the plugins that we use are listed in the official Prometheus website: https://prometheus.io/docs/instrumenting/exporters/ we haven't looked outside this page.

--

The cloud does make a difference. Just seeing the daily S3 usage per bucket was life changing. Immediately found that backups were not expiring after a while as they should, costing more and more money. ^^

Yeah this kind of visibility is really lacking on-prems. Storage is something that is hard, at scale, to see what is using what.

--

Do you know how many metrics you are ingesting in prometheus? storage size? and how many tags per host? We were reaching 1 TB of memory usage (mmap) on our server with 1500 hosts. Prometheus was literally grinding to a halt or crashing, was forced to cut down some metrics and stick to the absolute minimum tags.

We have around 500 GB of disk dedicated to prometheus server datacenter with a retention policy of 1 month. The VMs have around 16 GB of RAM.

I would say 95% of the hosts only have node exporter. Then 5% of the hosts will have redis/elasticsearch/postgres/mysql/etc exporters.

Also probably around 5% of the hosts have some sort of custom metrics. We have been leveraging the file exporter for writing custom service metrics.

Custom service metrics goes through a merge request process where we see the best way to structure them to avoid big cardinality. We also use recorded rules to pre-compute things that would be expensive to query (think CPU usage across a datacenter)

1 TB of RAM usage sounds insane. It looks like we can horizontally scale the prometheus servers and use some documented features for that.

For our use case it has been a very smooth ride so far!

Re: Ask HN: What is the best enterprise software you use every day?

#363

Post by jasonv » Fri, Jul 17, 2020, 6:20 PM UTC

hnick wrote: post #162

leokennis wrote: post #10

Two: - UltraEdit. What Photoshop is to images, UltraEdit is to text. The weird thing is, it’s not a super flashy or even immediately intuitive tool. But once you get the hang of it, it never fails to deliver. - Excel. It’s insane the breadth of stuff you can do with it. And as a tool, it’s equally handy and “oh my God this will save me so much work” for a school teacher as it is for a data analyst and stock broker. I…

I got a personal license for UltraEdit years ago too after using it at work (these fancy Electron editors didn't exist back then). It definitely isn't fancy but it works well for larger files and things like column mode and perl regex searches are just so nice in certain cases. I'm finding it hard to put down especially since I do a lot of Perl and I haven't found a big benefit for an IDE there.

+1 UltraEdit

I've kept UltraEdit around for more than 20 years now. Column editing, especially back when it was introduced, saved my bacon more than a few times.

Re: Ask HN: What is the best enterprise software you use every day?

#364

Post by jasonv » Fri, Jul 17, 2020, 6:21 PM UTC

ex3ndr wrote: post #142

Weird but i like Outlook Web, very good replacement for GMail with it's endless "ui improvements".

I don't hate Outlook at all, and the mobile app is 1000% better than the Gmail app.

Re: Ask HN: What is the best enterprise software you use every day?

#365

Post by jasonv » Fri, Jul 17, 2020, 6:22 PM UTC

coblers wrote: post #204

davedx wrote: post #200

JIRA. It's powerful, customizable, flexible and drives workflows and productivity across millions of companies around the world. It's the SAP of issue tracking. Despite all the baggage, they still try continuously to improve the UX of the software so it stays relevant and relatively usable. There's definitely problems with it (its performance can be awful and require dedicated server clusters to keep it up at larger…

It's crazy how different opinions are on JIRA. I think it's the worst tool I'm forced to use - and have been - at all my dev jobs. It's slow, sluggish, the UX is horrible and it gets in my way. A lot of dev teams that are outside of product/project management claws simply opt for using whatever simple kanban board there might be(Trello, for example).

Jira for a team of 20-30 developers, where I'm the CTO and we implement it together... I like it :-)

Re: Ask HN: What is the best enterprise software you use every day?

#366

Post by jasonv » Fri, Jul 17, 2020, 6:23 PM UTC

johnwalkr wrote: post #293

davedx wrote: post #200

JIRA. It's powerful, customizable, flexible and drives workflows and productivity across millions of companies around the world. It's the SAP of issue tracking. Despite all the baggage, they still try continuously to improve the UX of the software so it stays relevant and relatively usable. There's definitely problems with it (its performance can be awful and require dedicated server clusters to keep it up at larger…

Any recommendations from the JIRA haters? I love gitlab but am always wanting a bit more when I use it. Is there an in between?

Jetbrains Space

Clubhouse

Trello

Re: Ask HN: What is the best enterprise software you use every day?

#367

Post by thayne » Fri, Jul 17, 2020, 6:45 PM UTC

wenc wrote: post #94

milani wrote: post #83

Earlier quoted context omitted.

Why is Splunk among the bests for you? What is the cheaper alternative?

For an on-prem general purpose logging server, it's fast and easy to maintain. The query engine supports fairly complex queries. Did I mention it's fast? Plus it's an industry standard so it's an easier sell in the enterprise. It's also actually not that expensive at low data volumes -- prices are comparable to Sumologic. Cheaper alternatives? You can roll your own logging server with fluentd and a database. Some fol…

I love the power of splunks query engine, there really isn't anything comparable that I know of. But setting up forwarders and injesting logs is quite a pain.

Re: Ask HN: What is the best enterprise software you use every day?

#368

Post by ex3ndr » Fri, Jul 17, 2020, 6:50 PM UTC

jasonv wrote: post #364

ex3ndr wrote: post #142

Weird but i like Outlook Web, very good replacement for GMail with it's endless "ui improvements".

I don't hate Outlook at all, and the mobile app is 1000% better than the Gmail app.

iOS app is veeeery good! That's how i eventually moved to paid cloud outlook since it is also so good.

Re: Ask HN: What is the best enterprise software you use every day?

#369

Post by user5994461 » Fri, Jul 17, 2020, 7:36 PM UTC

aprdm wrote: post #362

user5994461 wrote: post #361

Earlier quoted context omitted.

The question would be which plugins exactly? because there are tons of plugins spread over GitHub, more or less working, and they're constantly shifting. I've had maybe 15 integrations working perfectly on datadog in 2 weeks of work. The same on prometheus/grafana could have taken 6 months easily (with few exporters to write from scratch). The cloud does make a difference. Just seeing the daily S3 usage per bucket wa…

Interesting, I have a very small team that maintains a lot of infrastructure including Prometheus. Setting up the monitoring for 5 datacenters, thousands of hosts and hundreds of services was maybe a ~2 weeks effort. We then organically add and remove things but it is maybe a ~4h/week effort. It give us a lot of value. We were using other solutions before (sensu/nagios) and it is night and day. All the plugins that w…

You're lucky to have a team. I was doing that with another person in our spare time and got 0 hour/week to spend. I only give ourselves a couple weeks to setup new software and tune everything perfectly then it has to be done for good. Will check maybe once a quarter for capacity adjustment or software upgrade. (I also supervise a logging system ingesting 1 TB/day with no supervision).

I should probably say that my experience with datadog goes back as far as 5 years ago. Already had monitoring working perfectly back then, when prometheus didn't exist let alone the exporter plugins! So prometheus is really late and sub par to me. ^^

Looks like we got a similar amount of data in prometheus as you (1TB for 60 days) but with 40% of the hosts. Maybe you have many small VM? Got physical hosts with quad CPU (per CPU metrics) and network interfaces and stuff (couldn't tune the node exporter to ignore disabled interfaces and some useless devices). Check how many distinct timeseries you have, prometheus_tsdb_head_series.

Datadog had amazing support for custom metrics (but watch out for extra billing and cardinality!). Applications can just send metrics to localhost:1234 where the agent is listening, and they're enriched automatically with host information and environment. Magic.

This reminds me, prometheus is broken with its idea of pulling metrics, when metrics should be pushed instead. Applications and hosts have to push metrics when they come online, it's not the responsibility of the metrics storage to know about every goddamn thing running in the company and try to talk to them (can't cross firewall anyway). Prometheus worked okay enough for the last company that was on premise with fixed hosts (weeks or months to move anything physical), but it's de facto broken for the previous company that was on AWS with instances created intraday.

Re: Ask HN: What is the best enterprise software you use every day?

#370

Post by man00604 » Fri, Jul 17, 2020, 7:39 PM UTC

adrianbordinc wrote: post #5

Datadog - the amount of observability we have in our infrastructure is insane. I can find out what’s wrong within a few seconds.

Yeah I heard they are pretty good but too bad they don't have mobile access for me. Do you have a need to access tools like Datadog on your mobile phone?