Live data from Hacker News

“It's The Future”

circleci.com

331–340 of 536 posts

Re: “It's The Future”

#331

Earlier quoted context omitted.

Doesn't statically compiling programs solve the deployment issue better? I mean, as far as I can tell Docker only exists because it's impossible to link to glibc statically, so it's virtually impossible to make Linux binaries that are even vaguely portable. Except now Go and Rust make it very easy to compile static Linux binaries that don't depend on glibc, and even cross-compile them easily. Hell I think it's actual…

Namespaced filesystem and networking, just for one. you seem very eager to dismiss a technology you only barely understand.

Namespaced filesystem shouldn't even be a special requirement - your program should use relative or at least configurable paths. I mean, directories are namespaced filesystems.

What networking problems does Docker solve?

Re: “It's The Future”

#332

Earlier quoted context omitted.

10 million rows in a relational database doesn't need to be bad nor is it big data. Rows is a bad measure of "big" when it comes to data. A measurement of bytes and probably more specifically bytes per field and how many fields the records have, as this gives a better indication into the way this will be written and potentially searched. 10 million rows of 5 integer values is pittance for any relational database wort…

Someone once suggested to me that 'big data' begins when it doesn't fit in RAM in a single rack any more.

Yup, that's essentially what looking that byte-size means. However, just because it doesn't fit in memory might not make it big data if it's just poorly engineered.

But many times this happens because of wasted or bloated indexes that aren't useful. Or it happens when data types are picked incorrectly.

For example, I once worked on a database where the original developer used Decimal(23, 0) as a primary key. This was on MySql and that ended up taking up 11 bytes per row, versus a Long which would have just been 8. In one table, maybe not so bad but when you start putting those primary keys into foreign key relationships... we ended up with a 1 billion row table in MySql that had 4 of these columns in it. That might make it "big data" by that definition but it's also just bad design.

Another example in that same database was using text fields in mysql for storing JSON. Since text fields in mysql are stored as separate files, this meant that every table that had one (and we had several tables that housed multiple) ran in large IO and disk access issues.

"big" data is probably a bad term to use these days because of easy it is to accidentally create a large volume of data but not need a big data solution outside of the fact that it's not the business that needs it, it's the poorly implemented system that does.

But the real reason we talk about fitting in memory comes from the core of the issue: IO. Even a super large memory set could end up being slow if it's postgres and single threaded reader that's scanning a 500 GB index. AWS offers up to 60 GB/s memory bandwidth and we'd need it for this index, since that would still take almost 10 seconds to warm up the indexes in the first place.

Re: “It's The Future”

#333
post #157

Earlier quoted context omitted.

>decoupling that specific service/functionality from your current stack. I do wish people would stop conflating "running in a different service" and "loose coupling". They are completely orthogonal. I've worked on some horrendously tightly coupled microservices.

OSGi makes it easy to end up with a cornucopia of tightly coupled nanoservices all running in the same JVM. Unless you can coax dOSGi into working (which is tons of fun), then you can have services tightly coupled to other services running on entirely different machines causing frequent (and hilarious) cascades of bundle failures whenever the network hiccups. OSGi is a trigger word for me now. I've worked on two larg…

Wholeheartedly agree on OSGi. Disastrous implementations. I worked with servicemix and still have nightmares around class path issues and crazy bundle scoping rules. A plain old maven built jar with shading works much better in practice, but shading itself is shady :)

Re: “It's The Future”

#334
Author here.

To answer some questions: yes this is obviously poking fun at Docker, but I also do really believe in Docker. See the follow-up for more on that: https://circleci.com/blog/it-really-is-the-future/

In a self-indulgent moment I made a "making of" podcast about this blog post, which is kinda interesting (more about business than tech): http://www.heavybit.com/library/podcasts/to-be-continuous/ep...

And if you like this post you'll probably like the rest of the podcast: http://www.heavybit.com/library/podcasts/to-be-continuous/

Re: “It's The Future”

#335
post #304

Earlier quoted context omitted.

10M rows in a relational database is a very low number (depending on the size of the row of course).

I know, it was a sarcastic follow up to the "now they have big data" part of the original comment. "SQL doesn't scale". It needs to be in Mongo or whatever NoSQl database is in right now. I have heard all sorts of nonsense regarding "big data" in the last few years.

ahaha, i didn't read the sarcasm that time, sorry for replying with tmi

Re: “It's The Future”

#336
post #234

Earlier quoted context omitted.

from subprocess import Popen, PIPE p1 = Popen(["foo"], stdout=PIPE) p2 = Popen(["bar"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output, _ = p2.communicate() https://docs.python.org/3.5/library/subprocess.html#replacin...

Thanks. Maybe I'll make a public Gist that's a kind of "foo | bar" cookbook for different languages...

What you actually do in Python is use https://github.com/kennethreitz/envoy

Re: “It's The Future”

#337
Don't listen to hype. Look at the problems you need to solve. If you need a big, complex, distributed system, than maybe microservices are a good idea. If you're building a simple webapp... not so much. Things that work in the large don't necessarily work in the small.

I write stuff in Scheme. I'm a hobbyist, there's no reason for me not to, and I love the language. The apps I write are sometimes single-threaded (or coroutine-based) monoliths. But I only have one machine available for me, and the things I'm writing are fairly simple. It's good ENOUGH. And Worse really is Better[1].

1:and I truly mean that in the Gabriel sense. As in the New Jersey model. Not any other way.

Re: “It's The Future”

#338
post #216
post #205

Earlier quoted context omitted.

bar(foo) When you're not writing shell, just use the tools the language gives you. For the matter, I think a shell script is cleaner than a python script for devops; but I don't think the composability of unix tools is that much of an advantage compared to the amount of python libraries out there.

When I use shell it's often exactly because I want to construct pipelines of processes and FIFOs and do all the other things that shell does very well and has done well for decades. I'm likely to be using Python programs and other programs in those shell scripts. The beauty of shell is that it makes it so easy to compose programs written in different languages.

Shell does things well provided all the intermediate states are naturally expressible as streams of bytes. Otherwise not so much.

I think the advantages of using a single language for everything outweigh the disadvantages - see e.g. http://www.teamten.com/lawrence/writings/java-for-everything... (though actually my single language is Scala)

Re: “It's The Future”

#339
post #247
post #160

Earlier quoted context omitted.

"I am frustrated by the industry as a whole" Unfortunately I have to agree as a developer. My job is to make a fast, reliable, stable product but at the same time I'm questioned the tools I use by people who don't have any knowledge but heard the latest trend. But sometimes it's also very easy to please people. Big data: just insert 10M records in a database and suddenly everyone is happy because they now have big da…

On a positive note, it sounds like proprosals to use newer technology are welcome. I keep seeing the opposite, "No this is too different, could break stuff."

IME that comes with: Sure, the new tool looks cool, but is it battle-tested? How many tools end up being relied on heavily while they're still in beta? And does it solve any of our current problems? or is it just neat?

As a grumpy SA, I see way too many people try to push for new tools because they "seem cool", instead of "Do they solve a problem we have?"

Re: “It's The Future”

#340

The article perfectly summarizes my frustration and sentiment. These days I hear these buzzwords all time. I work as a consultant for an enterprise product and most people whom I meet they somehow catch these buzzwords and blurt it out in front of everyone during meetings and discussions to either showoff that they know technology and things that are in the market these days(also latest iphone, apple news, tesla, spa…

I never thought I would appreciate Java...but the industry has really made me.

Take your .war file, drop it onto JBoss. It deploys across the cluster in a zero downtime manner, isolates configuration, provides consistent log structure, cert management, deployment. You can deploy dozens of small war's to the same server and they can talk to each other. Load balance across the cluster automatically based on actual load. Run scheduled jobs and load balance the scheduled jobs themselves. Allow them to be isolated and unique within the cluster.

I may not like Java as a language, but from an infrastructure standpoint Java was basically Heroku long before Heroku was Heroku. The infrastructure is just...solid. The downside was that the XML config stuff was just messy.

Post reply on HN