Live data from Hacker News

A lot of complex “scalable” systems can be done with a simple, single C++ server

twitter.com

331–340 of 376 posts

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#331
post #317

Earlier quoted context omitted.

There is, though. Go was designed to be easier to reason with than conventional ALGOL-derivatives, while Rust wasn't. To quote Rob Pike: The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use the…

If you can write a ton of code without thinking very much, you're probably writing boilerplate that should have been generated from a human-level description of the problem. Your job is to only spend time writing what needs to be written.

Languages that allow you to write as fast as you think are a blessing.

"Write as fast as you think" is a far better way to program than "Write much slower than you can think."

Eric Raymond has written some pretty substantial things, and he's not as clueless (on programming, at least, the rest of his views are...no) as you're implying.

The idea that intuitive languages are the only ones you should do development in is absurd. A single line of K can do what a hundred lines of C can, and you can write the line of K substantially faster than you could write the C to match. K only has something like 50 primitives. It's simple enough that you can keep it all in your head at once, and that allows you to develop much quicker than almost any ALGOL-derivative. Taking your comment at face value, everything written must be boilerplate. Looking at reality paints a different picture.

Good languages manage complexity in a way that allow you to express complex things in simple terms. That the languages you seem to be familiar with only allow you to describe simple things in simple terms isn't something that's inherent to every programming language. I'd recommend giving APL, J, or K a try.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#332
post #38

Earlier quoted context omitted.

Can't you also defer work to c extensions in threads?

Yes. Numpy releases the GIL for its array operations. Parallelism is possible, but only for computations in native code. FWIW Node.js is essentially the same situation.

Node got background workers recently.

Otherwise yes, Node or Ruby play by the same global lock rules.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#333
post #308
post #227

Earlier quoted context omitted.

If you can set it up so it solves my problems, yes. I could manage my own server and fine tune everything, or I can throw it on snowflake. Snowflake means I've spent almost no time managing anything and it was costing less than an aws box running postgres but absolutely blew it out of the water performance wise. Depends on your workload but it's been perfect for one of my use cases. If they were just using postgres u…

You use Snowflake for OLTP? Can you comment more on how/why?

Not really OLTP, analytical workloads but can't go into much detail I'm afraid. Infrequent, unpredictable and benefitting from rapid scaling (0 to lots of power & ram for short periods) is where cloud type things (can) really shine.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#334
post #332

Earlier quoted context omitted.

Yes. Numpy releases the GIL for its array operations. Parallelism is possible, but only for computations in native code. FWIW Node.js is essentially the same situation.

Node got background workers recently. Otherwise yes, Node or Ruby play by the same global lock rules.

Same in Python, except multiprocessing has actually been around for longer.

Launching multiple processes without mutable shared memory is always an option.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#335

Earlier quoted context omitted.

> Sorry but experienced developer can implement those just as fast as in any scripting language and it will save a ton on maintenance. This is just a no true Scotsman argument. For 10 years I’ve watched python/ruby shops drastically outpace projects in C++/Java shops. What you’re failing to realize is how trivial most apps are and how fast fully functional back ends can be created with frameworks in those languages (…

> What you’re failing to realize is how trivial most apps are and how fast fully functional back ends can be created with frameworks in those languages A vast majority of web dev is incredibly simple. People tend to easily convince themselves that their app is very special and requires a lot of complex solutions to make it work. There’s a lot of incentives that lead people to that conclusion, but most of the time fol…

I run a saas platform and from the outset, I knew that we wouldn't need to do anything special or groundbreaking. We are just sort of line of business. But what I found is that the majority of the complexity started to show up when we are scaling and needed to be able to get results to the user very quickly. That's when you start to deal with queues and pubsub and architecting things to run in parallel. We have a process that only takes about 10 seconds to run completely. Which is acceptable lag time for us. But now, 20 people are simultaneously making that request and the person at the end of the line has to wait 200 seconds, which is not acceptable. This only happens occasionally, so adding servers would be a big waste. That's where the more complicated cloud setups start to help.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#336

Earlier quoted context omitted.

I'd love to keep it in RAM if I could. The problem is, the library I'm familiar with (pandas) typically seems to take more memory than the original csv file once it loads it onto memory. I know this is due to bad data types but in certain cases, I cannot get around those. However, even if I could load it all into memory at once, and assuming it takes 200 gb, I'm still using a master's student access to a cluster. So…

Numpy supports memory mapping `ndarrays` which can back a DataFrame in pandas. This lets you access a dataset far larger than will fit in RAM as if it lived in RAM. Provided it's on fast SSD storage you'll have speedy access to the data and can process huge chunks at once.

Can you provide a link to this please? My current knowledge is that all numpy data lives in memory, and pandas itself has a feature to fragment any data into iterables so I can read upto my memory limit. I cannot use this feature due to the serial nature of some of the operations that I alluded to (I'd have to almost rewrite the entire library for some of these complicated operations like groupby and sorting).

I do have fast SSD storage because it's on the scratch drive of a cluster and from what I've seen it can do ~300-400 MB/s easily. I haven't had a chance to test more than that since I'm mostly memory constrained in much of my testing.

My current attempt is to push this data into a pure database handling system like SQL so that I can query it. But like I said, I work with a less-than-stellar set of tools and I have to literally set up a postgres server from ground up to write to it. Which shouldn't be a big deal except when it's on a non-root user and I have to keep remapping dependencies (took 5-6 hours to set it up on the instance I have access to).

My other option was to write the entire 250 GB to a sqllite database using the sqlalchemy library in Python, but that seems to fail whether I do it with parallel writes, or serial writes. In both cases, it fails after I create ~64-70 tables.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#337
post #208

Earlier quoted context omitted.

Can you expand on this? I have some pretty massive compute loads that need to be scaled onto a cluster with 100+ workers for most computations. This is after I use a library called dask that graphically does its own mapreduce optimisation inside its modules. This is all for a relatively small 250GB raw data file that I keep in a csv (and need to convert to SQL at some point). Are you saying this can be optimised to f…

250GB data is tiny. There are laptops with 128GB RAM, let alone servers. Obligatory read: https://aadrake.com/command-line-tools-can-be-235x-faster-th...

Yes, but I'm heavily constrained because my access to the cluster I'm using is very low level and I get pre-empted quite a bit in my tasks. I'm probably over stretching between the amount of data I need to handle and my severe lack of skills (I'm a quant in training).

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#338

Earlier quoted context omitted.

Can you expand on this? I have some pretty massive compute loads that need to be scaled onto a cluster with 100+ workers for most computations. This is after I use a library called dask that graphically does its own mapreduce optimisation inside its modules. This is all for a relatively small 250GB raw data file that I keep in a csv (and need to convert to SQL at some point). Are you saying this can be optimised to f…

> Are you saying this can be optimised to fit inside a single 10 core server in terms of compute loads? I'm currently employed to write software which does DNA analysis. DNA is known for being Big Data. Some applications are very compute intensive and others are very data-relation intensive. The very compute intensive applications process about 1GB of data in about 30 minutes on a 32 core Xeon 6xxx with 32GB of RAM a…

So my entire dataset is ~24 x 250GB files. That 24 number can be larger if I can find an efficient way of processing each 250GB chunk. Each 250GB chunk is actually stock tick data so it has 500 stocks inside it. A heavily traded stock takes up ~10 GB of memory while a very thinly traded stock can top out at just 700 MB.

While I hope that each 250GB chunk has everything in order and I can separate it cleanly, I don't trust it. So I broke up all the files by stock in an embarrassingly parallelised code I wrote and got 500 separate files. However, the problem is, I'd want to process these 500 files parallely so each stock gets only so much memory (and hence, my original constraint remains). And for each stock, I want the data to be queried on timestamps, so I need a way to quickly say "I have data at 9am on Thursday, I want data from 4pm Wednesday to 8am Thursday to create a model".

I figured the best way to do this efficiently was to have the code create a massive database for all the stocks and query it efficiently in SQL. But I'm stuck there due to a lack of tools.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#339

Earlier quoted context omitted.

Can you expand on this? I have some pretty massive compute loads that need to be scaled onto a cluster with 100+ workers for most computations. This is after I use a library called dask that graphically does its own mapreduce optimisation inside its modules. This is all for a relatively small 250GB raw data file that I keep in a csv (and need to convert to SQL at some point). Are you saying this can be optimised to f…

Not sure which DB you are using, but you can load the csv file into the DB directly on a single thread using something like LOAD DATA INFILE. If you have some good indexes and do some push-down work (give the database aggregation tasks to do instead of your python code), you should probably be more than fine. For a 250Gb file.. should be ok.. maybe add some partitioning too.

I'm open to using any db that I can query over some engine with a python implementation. So any SQL db should be fine. However, I don't know how to convert a csv to an SQL directly. Is the command you mentioned part of some SQL server package? Sounds like it's exactly what I need.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#340
post #331

Earlier quoted context omitted.

If you can write a ton of code without thinking very much, you're probably writing boilerplate that should have been generated from a human-level description of the problem. Your job is to only spend time writing what needs to be written.

Languages that allow you to write as fast as you think are a blessing. "Write as fast as you think" is a far better way to program than "Write much slower than you can think." Eric Raymond has written some pretty substantial things, and he's not as clueless (on programming, at least, the rest of his views are...no) as you're implying. The idea that intuitive languages are the only ones you should do development in is…

Typo: "The idea that unintuitive languages..."
Post reply on HN