Earlier quoted context omitted.
Well, there is a lot of interest in a secure, scalable, fast, deterministic, simple OS which is also backwards compatible. ;) What do you mean by "safe" though? How is kernel-space safer then user-space?
By being written in systems programming languages safer than C.
What I wish systems researchers would work on
31–40 of 60 posts
Re: What I wish systems researchers would work on
#32And you can be sure that a company like Google will have done configuration using flat files, key-value stores, DSLs, general purpose languages that generate config files, general purpose languages that actually execute stuff, and probably many other ways :-) If there were simple solutions like that, they'd already be in use.
Here's a small list of desirable properties:
- Configurations must be reusable, composable and parametrizable. When running a system with multiple components, you need to be able to take base configurations of those two components and merge them into a useful combination, rather than writing the configurations from scratch. You must also be able to parametrize those base configurations in useful ways -- which external services to connect to, which data center and on how many machines to run on, etc. And note -- this can't be a one time thing. If the base configurations change, there needs to be some way of eventually propagating those changes to all users.
- Configurations must be reproducible. If the configuration is a program, it shouldn't depend on the environment where it's run on (nor should it be able to have side-effects on the environment). Why? Because when somebody else needs to fix your system at 3am, the last thing you want them to do is need to worry about exactly replicating the previous person's setup.
- Tied to the previous point, configurations also need to be static or snap-shottable. If a server crashes and you need to restart a task on another machine, it'd be absolutely inexcusable for it to end up using a different configuration than the original task due to e.g. time dependencies in programmatic configuration.
- It must be possible to update the configuration of running services without restarting them, and it must be possible to roll out config changes gradually. During such a gradual rollout you need to have options for manual and automatic rollback if the new configuration is problematic on the canaries.
- Configurations need to be debuggable. Once your systems grow to having tens of thousands of lines of configuration or hundreds of thousands of programatically generated key-value pairs, the mere act of figuring out "why does this key have this value" can be a challenge.
- It'd be good if configurations can be validated in advance, rather than only when actually starting the program reading the configuration. At a minimum, this would probably mean that people writing configurations that other people use as a base be able to manually add assertions or constraints on whatever is parametrized. But you might also want constraints and validation against the actual runtime environment. "Don't allow stopping this service unless that service is also down", or something.
Some of these desirable properties are of course conflicting. And some are going to be things that aren't universally agreed on. For example there is a real tension between not being able to run arbitrary code from a configuration (important for several of these properties) vs. reuse of data across different services.
My thinking on this is already a few years out of date, it's probable that configuration has again gotten an order of magnitude more complex since the last time I thought about this stuff seriously, and there are already new and annoying properties to think about. The main point is just that this really is a bit more complex than your .emacs file is.
Re: What I wish systems researchers would work on
#33In the OP, the > The "bug" here was not a software bug, or even a bad configuration: it was the unexpected interaction between two very different (and independently-maintained) software systems leading to a new mode of resource exhaustion. can be viewed as pointing to the solution: Broadly in computing, we have 'resource limits' from the largest integer that can be stored in 2's complement in 32 bits to disk partitio…
All of this is done at Google - there's extensive monitoring for all production systems, with alerts firing once parameters move outside of their statistical "normal" range. In practice the range tends to get set more by "How tight can we make it before we get really annoyed by these pagers?" than any rigorous statistical method, but the run-time operation carefully measures average & extreme values and determines wh…
Really, "All"?
First question is, 'parameters'. Usually, and likely at Google, they are monitoring just one parameter at a time. This is a really big bummer. E.g., even if do this for several parameters, are forced into assuming that the 'healthy and well' geometry of the parameters is just a rectangle. Bummer that hurts the combination of false alarm rate and detection rate.
Next, the "normal range" is a bummer because it assumes that what is healthy and well is just a 'range', that is, an interval, and this is not always the case. The result, again, is a poor combination of false alarm rate and detection rate.
Again, yet again, please read again, just for you, as I wrote very clearly, to do well just must be multi-dimensional. I doubt that there is so much as a single large server farm or network in the world that is doing anything at all serious working with multidimensional data for monitoring. Not one.
Next, your remark about false alarm rate points to a problem I point out is solved: The method, with meager assumptions, permits knowing false alarm rate in advance and setting it, actually setting it exactly.
For how many and what variables to monitor, yes, that is a question that can need some overview of the server farm or network and some judgment, but there is some analytical work that should help.
For "rigorous" statistics, the point is not 'rigor' but useful power. Being multidimensional, knowing false alarm rate and being able to adjust it, etc. are powerful.
Re: What I wish systems researchers would work on
#34Earlier quoted context omitted.
By being written in systems programming languages safer than C.
This could catch some programming bugs, sure, but it wouldn't make the OS much safer. You can still have attacks on integrity, privacy and availability.
If you really want to attack a system, nothing beats social engineering anyway.
Re: What I wish systems researchers would work on
#35Anybody near the field has their own list of "taboo" research topics that are very interesting but just don't get funding.
Then you go to the library and realize they have 120 shelf-feet of conference proceedings about Internet QoS that have gone precisely nowhere.
Re: What I wish systems researchers would work on
#36I think people replying here aren't quite appreciating the complexity of configuration properly here. One might think that configuration doesn't get harder with scale, but it does. Suggesting that the solution is simply using a general purpose language for configuration or storing configs in a key-value db or in a version control system is almost adorable. It's basically completely missing where the pain points are.…
Re: What I wish systems researchers would work on
#37I think people replying here aren't quite appreciating the complexity of configuration properly here. One might think that configuration doesn't get harder with scale, but it does. Suggesting that the solution is simply using a general purpose language for configuration or storing configs in a key-value db or in a version control system is almost adorable. It's basically completely missing where the pain points are.…
I would add that the configured program should spit its final configuation back out as it starts up, either as a set of command line parameters into a log file, or if that gets out of hand, as a file in /tmp with all external cofigurable values explicitly written out. This simplifies the 3am fix problem somewhat, since some values will always be missing.
Re: What I wish systems researchers would work on
#38I want to register a complaint. Everyone talks about wanting ground-breaking research and innovation, but in my own personal experience I see little-to-no support for it. It reminds of the scene from the film Election , where a character intones, after you've seen him ski down a double-black diamond slope and crash, "Last winter when I broke my leg, I was so mad at God." We're spending money to squeeze the last drops…
systems are about interactions, and he's identified the challenges in analyzing systems.
this isn't so much about work from the 60s and 70s, but about modern architectures and the new things they enable. we're still working with models form the 60s and 70s and ignoring various aspects of new system architectures and wondering why we still suck.
Re: What I wish systems researchers would work on
#39I want to register a complaint. Everyone talks about wanting ground-breaking research and innovation, but in my own personal experience I see little-to-no support for it. It reminds of the scene from the film Election , where a character intones, after you've seen him ski down a double-black diamond slope and crash, "Last winter when I broke my leg, I was so mad at God." We're spending money to squeeze the last drops…
See https://news.ycombinator.com/item?id=5717481 in this thread.
Re: What I wish systems researchers would work on
#40I want to register a complaint. Everyone talks about wanting ground-breaking research and innovation, but in my own personal experience I see little-to-no support for it. It reminds of the scene from the film Election , where a character intones, after you've seen him ski down a double-black diamond slope and crash, "Last winter when I broke my leg, I was so mad at God." We're spending money to squeeze the last drops…
i originally opened the link expecting to conclude the same thing, however if you read the article again i think the guy is a) highly qualified to complain (after all he chaired HotOS's PC, no small recognition of his stature in the community) and b) he raises good points. systems are about interactions, and he's identified the challenges in analyzing systems. this isn't so much about work from the 60s and 70s, but a…