Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Finding Critical Open Source Projects

opensource.googleblog.com

Post reply on HN
Print view View original

61–70 of 102 posts

Re: Finding Critical Open Source Projects

#61

Post by abharya »

throwaway342422 wrote: post #50
hn_throwaway_99 wrote: post #10

As others have mentioned, while this may seem like a good idea, the results are often bizarre, and it's not hard to see why - the metrics and algorithm are here: https://github.com/ossf/criticality_score#criticality-score . That algorithm seems unnecessarily complicated and includes somewhat dubious metrics when, in my mind, the only thing that really "counts" when it comes to "criticality" are "how many other things…

The methodology is pretty silly. It rewards activity and popularity. A lot of critical infrastructure software is not active and not often spoken about! Case in point: The https://www.cip-project.org/faq project highlight the needs of very-long-term support for OS components that run on critical infrastructure such as power stations. The https://www.cip-project.org/faq project is based on Debian. Very little of it is…

We have to start somewhere. It is understandable that this is not complete, so welcome your ideas to discover such projects. Please think of any metrics/ways to find such projects.

Re: Finding Critical Open Source Projects

#62

Post by abharya »

orf wrote: post #34
abharya wrote: post #12

Earlier quoted context omitted.

- Other metrics such as how many contributors and organizations are involved, how many user feature requests and bugs getting reported, those are all important project importance and not just "dependency count". some projects can be standalone, so as per your algo, those should be very low. - Downloads data is not available for most repos, please find a reliable metric to use. - Package repo dependencies works, but i…

You didn’t identify certifi, urllib3, chardet or pytz in your top 10 critical Python dependencies. These are all highly download packages, mostly maintained by one person, which are totally critical to millions of other packages and the Python ecosystem as a whole. A few of your top-10 I can agree with, but when you’re saying a home-automation package (“core”) is more critical than something like pytz then something…

Filed https://github.com/ossf/criticality_score/issues/20, we will fix this, have an idea on the issue.

Re: Finding Critical Open Source Projects

#63

Post by rhencke »

abharya wrote: post #58
arp242 wrote: post #47

Earlier quoted context omitted.

That's probably because SQLite doesn't use git, and this tool seems to require git. Actually, in its current state it seems to require GitHub: https://github.com/ossf/criticality_score/blob/main/critical... This probably omits some other projects as well which don't use git or GitHub.

Yes correct. Right now, we are query-ing projects hosted on Github, but will be expanding to our source control system in the near future.

Please consider allowing scanning tarball/zip distributions of source directly as well. It is a SCM-agnostic method that is also well-supported by GitHub, Gitiles, hgweb, and many old but still-in-use projects that pre-date Git.

Re: Finding Critical Open Source Projects

#64

Post by rhencke »

abharya wrote: post #61
throwaway342422 wrote: post #50

Earlier quoted context omitted.

The methodology is pretty silly. It rewards activity and popularity. A lot of critical infrastructure software is not active and not often spoken about! Case in point: The https://www.cip-project.org/faq project highlight the needs of very-long-term support for OS components that run on critical infrastructure such as power stations. The https://www.cip-project.org/faq project is based on Debian. Very little of it is…

We have to start somewhere. It is understandable that this is not complete, so welcome your ideas to discover such projects. Please think of any metrics/ways to find such projects.

I strongly recommend you use the packages maintained in Linux distributions as a means for discovery. They're well-organized and maintained and easily accessible programmatically - you can even parse the package dependencies programmatically, as well as have full access to the original source code.

Re: Finding Critical Open Source Projects

#65

Post by carapace »

abharya wrote: post #57
eeZah7Ux wrote: post #52

Earlier quoted context omitted.

> of course the random Nebraskan's critical project is going to get a low "criticality score" ...which is the exact opposite of what it should be.

That is not true, check out this as an example - https://groups.google.com/g/wg-securing-critical-projects/c/... . We are just trying to help, so please provide constructive feedback and any ideas on metrics we can use.

I'm not sure if this is relevant, but bash and the readline lib are both maintained by a single unpaid volunteer. (I don't know if he's from Nebraska though.)

Re: Finding Critical Open Source Projects

#66

Post by ludocode »

throwaway342422 wrote: post #50
hn_throwaway_99 wrote: post #10

As others have mentioned, while this may seem like a good idea, the results are often bizarre, and it's not hard to see why - the metrics and algorithm are here: https://github.com/ossf/criticality_score#criticality-score . That algorithm seems unnecessarily complicated and includes somewhat dubious metrics when, in my mind, the only thing that really "counts" when it comes to "criticality" are "how many other things…

The methodology is pretty silly. It rewards activity and popularity. A lot of critical infrastructure software is not active and not often spoken about! Case in point: The https://www.cip-project.org/faq project highlight the needs of very-long-term support for OS components that run on critical infrastructure such as power stations. The https://www.cip-project.org/faq project is based on Debian. Very little of it is…

The activity metric is especially frustrating because it seems to me that the higher quality a project is, the less bugs will be filed, and so the less activity will result.

With one of my open source projects I've got to extreme lengths to test it under a huge number of compiler configurations, so it compiles on anything with no warnings. I get nearly zero bugs filed when people port it to esoteric platforms.

When randomly searching a while back, I found a blog post someone wrote about getting my project running on Arduino [1], something I had never tried. Turns out they just had to set some configuration flags and everything worked fine. They therefore filed zero bugs, and it resulted in zero activity for my project. They didn't even tell me about it!

I was extremely pleased that it worked of course, but disappointed that my project continues to look like it's dead. I imagine Google's criticality score for my project is near zero.

[1]: https://www.thingforward.io/techblog/2017-08-03-compiling-lu...

Re: Finding Critical Open Source Projects

#67

Post by wyldfire »

megous wrote: post #27
Buetol wrote: post #9

Top 10: - Python: salt, core ( https://github.com/home-assistant/core ), pandas, scikit-learn, numpy, airflow, erpnext, matplotlib, pytest & pip - Rust: servo, cargo, rust-clippy, tokio, rust-analyzer, tock, tikv, alacritty, libc & substrate - JS: node, react-native, react, gatsby, three.js, bootstrap, material-ui, odoo, next.js & Rocket.Chat - Java: elasticsearch, flink, spring-boot, hadoop, netty, jenkins, beam, ba…

Which of these match the image in the article, where some random Nebraskan is maintaining some obscure dependency?

I don't think it's explicit in the article, but that's a comic (XKCD). Relevant discussion [1] suggests that there's not a specific project referenced by the comic.

When I first read this comic, ntpd [2] [3] came to mind.

[1] https://www.explainxkcd.com/wiki/index.php/2347:_Dependency#...

[2] https://lwn.net/Articles/701222/

[3] https://lwn.net/Articles/713901/

Re: Finding Critical Open Source Projects

#68

Post by yorwba »

abharya wrote: post #57
eeZah7Ux wrote: post #52

Earlier quoted context omitted.

> of course the random Nebraskan's critical project is going to get a low "criticality score" ...which is the exact opposite of what it should be.

That is not true, check out this as an example - https://groups.google.com/g/wg-securing-critical-projects/c/... . We are just trying to help, so please provide constructive feedback and any ideas on metrics we can use.

Gensim is #119 in the list according to your link, far behind projects with many more active contributors, so hardly a resounding success of your scoring method.

In terms of metrics, you could start by weighing projects with few contributors as more critical, not less. Specifically, gensim does appear to have had quite a few contributors, but the bulk of the code was written by the single maintainer https://github.com/RaRe-Technologies/gensim/graphs/contribut... So maybe you should add a metric "percentage of code in the past year authored by the top contributor".

If you want to go about it in a more data-driven fashion, you could go through the top projects for each language, check whether they actually need your support (e.g. find out what the development goals are, ask whether the current resources are sufficient and what they'd do with the additional resources you can provide) to get a ground-truth labeling of critical projects, then readjust your weights to match the ground truth.

Re: Finding Critical Open Source Projects

#69

Post by commandlinefan »

megous wrote: post #27
Buetol wrote: post #9

Top 10: - Python: salt, core ( https://github.com/home-assistant/core ), pandas, scikit-learn, numpy, airflow, erpnext, matplotlib, pytest & pip - Rust: servo, cargo, rust-clippy, tokio, rust-analyzer, tock, tikv, alacritty, libc & substrate - JS: node, react-native, react, gatsby, three.js, bootstrap, material-ui, odoo, next.js & Rocket.Chat - Java: elasticsearch, flink, spring-boot, hadoop, netty, jenkins, beam, ba…

Which of these match the image in the article, where some random Nebraskan is maintaining some obscure dependency?

That would be "left-pad": https://qz.com/646467/how-one-programmer-broke-the-internet-...

Re: Finding Critical Open Source Projects

#70

Post by sansnom »

abharya wrote: post #57
eeZah7Ux wrote: post #52

Earlier quoted context omitted.

> of course the random Nebraskan's critical project is going to get a low "criticality score" ...which is the exact opposite of what it should be.

That is not true, check out this as an example - https://groups.google.com/g/wg-securing-critical-projects/c/... . We are just trying to help, so please provide constructive feedback and any ideas on metrics we can use.

Nice tool.

I think we could improve it a bit. For example, Spring boot should have a very low score for me. It's backed by a large company Pivotal. They don't need any support I think. Same thing for elasticsearch.

For me:

    - backed by a large company ? 
 
    - number of contributor doing 80% of the work ? or active in the last 12 months ? commits breakdown (99% is done by one guy) ?  

    - issues created/closed ratio  

    - PR created/merged ratio  

    - use critical projects ?  

    - other from your original score
A nice bonus: if we could use the tool to assess critical score for our project (not globally). For local dependency, we could increase the critical value if dependents count is low. Very few person is using it: that's a bad sign. With this, we could find those dependencies.

We could also create a global score (like you did) by using the previous score and scaling it using the dependency usage (dependents_count like you did).

With this calculation, I think it's more likely to find relevant projects.

How to find it's backed by a large company ? Not sure about this, we can check if the project is part of an organization, if contributor have a company or if they have a pro account. For example, if the top 5 contributors are from Google, it's likely it's sponsored by it(could be done during their free time but less likely).

Note: check what happens with a stable project (no new issue and PR).

Post reply on HN