Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A lot of complex “scalable” systems can be done with a simple, single C++ server

twitter.com

Post reply on HN
Print view View original

341–350 of 376 posts

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#341

Post by est »

fxtentacle wrote: post #10

Many people don't know about Python's GIL https://wiki.python.org/moin/GlobalInterpreterLock That's the reason why you need to go multi-process if you want to reach a similar level of concurrency in Python as multi-thread in C++. And that surely adds a lot of complexity. As a very practical example of this, TensorFlow has a dedicated page with advice on how to make the Python part that reads the files from disk less…

Many people don't know about every game I played written in C++ can utilize only one CPU core.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#342

Post by westurner »

hnews_account_1 wrote: post #336
giantrobot wrote: post #320

Earlier quoted context omitted.

Numpy supports memory mapping `ndarrays` which can back a DataFrame in pandas. This lets you access a dataset far larger than will fit in RAM as if it lived in RAM. Provided it's on fast SSD storage you'll have speedy access to the data and can process huge chunks at once.

Can you provide a link to this please? My current knowledge is that all numpy data lives in memory, and pandas itself has a feature to fragment any data into iterables so I can read upto my memory limit. I cannot use this feature due to the serial nature of some of the operations that I alluded to (I'd have to almost rewrite the entire library for some of these complicated operations like groupby and sorting). I do h…

Dask groupby example: https://examples.dask.org/dataframes/02-groupby.html

> Generally speaking, Dask.dataframe groupby-aggregations are roughly same performance as Pandas groupby-aggregations, just more scalable.

The dask.distributed scheduler can also run on one high-RAM instance (with threads or processes) https://docs.dask.org/en/latest/setup.html

Pandas docs > Ecosystem > Out-of-core: https://pandas.pydata.org/pandas-docs/stable/ecosystem.html#...

Reading from Parquet into Apache Arrow is much faster than CSV because the data can just be directly loaded into RAM. https://ursalabs.org/blog/2019-10-columnar-perf/

If you have GPU instances, cuDF has a Pandas-like API on top of Apache Arrow. https://github.com/rapidsai/cudf

> Built based on the Apache Arrow columnar memory format, cuDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data.

> cuDF provides a pandas-like API that will be familiar to data engineers & data scientists, so they can use it to easily accelerate their workflows without going into the details of CUDA programming.

Dask-ML makes scalable scikit-learn, XGBoost, TensorFlow really easy. https://dask-ml.readthedocs.io/en/latest/

... re: the OT: While it's possible to write C++ code that's really fast, it's generally inflexible, expensive to develop, and dangerous for devs with experience in their respective domains of experience to write. Much saner to put a Python API on top and optimize that during compilation.

There are a few C++ frameworks in the top quartile of the TechEmpower framework benchmarks. https://www.techempower.com/benchmarks/

Hardware/hosting is relatively cheap. Developers and memory vulnerabilities aren't.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#343

Post by nwallin »

hnews_account_1 wrote: post #339
tkyjonathan wrote: post #202

Earlier quoted context omitted.

Not sure which DB you are using, but you can load the csv file into the DB directly on a single thread using something like LOAD DATA INFILE. If you have some good indexes and do some push-down work (give the database aggregation tasks to do instead of your python code), you should probably be more than fine. For a 250Gb file.. should be ok.. maybe add some partitioning too.

I'm open to using any db that I can query over some engine with a python implementation. So any SQL db should be fine. However, I don't know how to convert a csv to an SQL directly. Is the command you mentioned part of some SQL server package? Sounds like it's exactly what I need.

pandas can read from a CSV file and then write to SQL. Even if you don't go the SQL route, you'd probably gain significant benefits by working with HDF instead of CSV.

https://pandas.pydata.org/pandas-docs/version/0.22/generated...

https://pandas.pydata.org/pandas-docs/version/0.22/io.html#w...

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#344

Post by nwallin »

cogman10 wrote: post #329
CoolGuySteve wrote: post #278

Earlier quoted context omitted.

Except compaction can also take many milliseconds and come from different threads. Writing a trading system in Java is harder than C++ imo because where before you had an allocation problem, now you have a multithreaded randomly stalling allocation problem. Virtu did it but everything I’ve heard about it nullifies the benefits of using java in the first place.

Newer Java GCs are very low latency (microsecond). You trade performance and memory for that low latency though. AFAIK, they are still compacting. Still though, probably makes sense to do it in a lower level language. It's just far easier in C++ to decide that "Hey, you know what, I just want a big memory block that I control". I've even heard of game devs doing things like having per frame allocators. They get super…

The point is that a one millisecond pause is unacceptable. Low latency Java GCs have average latencies of one millisecond, 99th percentile latencies of 10 milliseconds, and 99.9th percentile latencies are neither measured nor optimized for.

I don't consider it realistic to think that garbage collected languages might ever be usable in the context of game engines or HFT.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#345

Post by erik_seaberg »

kick wrote: post #331
erik_seaberg wrote: post #328

Earlier quoted context omitted.

If you can write a ton of code without thinking very much, you're probably writing boilerplate that should have been generated from a human-level description of the problem. Your job is to only spend time writing what needs to be written.

Languages that allow you to write as fast as you think are a blessing. "Write as fast as you think" is a far better way to program than "Write much slower than you can think." Eric Raymond has written some pretty substantial things, and he's not as clueless (on programming, at least, the rest of his views are...no) as you're implying. The idea that intuitive languages are the only ones you should do development in is…

I'm all in favor of concise and expressive languages (even weird ones). I hate being slowed down by the language itself. But writing the first thing that comes into my head leads me to reinventing the wheel a lot, and (at least at work) we have a responsibility to find reusable abstractions and only create new code that's needed.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#346

Post by kick »

erik_seaberg wrote: post #345
kick wrote: post #331

Earlier quoted context omitted.

Languages that allow you to write as fast as you think are a blessing. "Write as fast as you think" is a far better way to program than "Write much slower than you can think." Eric Raymond has written some pretty substantial things, and he's not as clueless (on programming, at least, the rest of his views are...no) as you're implying. The idea that intuitive languages are the only ones you should do development in is…

I'm all in favor of concise and expressive languages (even weird ones). I hate being slowed down by the language itself. But writing the first thing that comes into my head leads me to reinventing the wheel a lot, and (at least at work) we have a responsibility to find reusable abstractions and only create new code that's needed.

I disagree with that. If your code base is small enough, a few redundant lines (idioms) don't matter.

To bring up k again, the language has already done about the maximum amount of abstraction possible. There's really no room for the programmer to make reusable abstractions; any useful ones have already been made. That allows you to do very useful things in very small amounts of code. Picking a random example, here's a complete Sudoku solver in 75 bytes:

http://nsl.com/k/sudoku/aw3.k

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#347

Post by sfifs »

hnews_account_1 wrote: post #336
giantrobot wrote: post #320

Earlier quoted context omitted.

Numpy supports memory mapping `ndarrays` which can back a DataFrame in pandas. This lets you access a dataset far larger than will fit in RAM as if it lived in RAM. Provided it's on fast SSD storage you'll have speedy access to the data and can process huge chunks at once.

Can you provide a link to this please? My current knowledge is that all numpy data lives in memory, and pandas itself has a feature to fragment any data into iterables so I can read upto my memory limit. I cannot use this feature due to the serial nature of some of the operations that I alluded to (I'd have to almost rewrite the entire library for some of these complicated operations like groupby and sorting). I do h…

I'd spin up something like AWS r5.16xlarge node for the processing just for this and shut it down after use - should cost few 10s of dollars per run or so. Of course in some corporate environments, this option may not be available to you.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#348

Post by giantrobot »

hnews_account_1 wrote: post #336
giantrobot wrote: post #320

Earlier quoted context omitted.

Numpy supports memory mapping `ndarrays` which can back a DataFrame in pandas. This lets you access a dataset far larger than will fit in RAM as if it lived in RAM. Provided it's on fast SSD storage you'll have speedy access to the data and can process huge chunks at once.

Can you provide a link to this please? My current knowledge is that all numpy data lives in memory, and pandas itself has a feature to fragment any data into iterables so I can read upto my memory limit. I cannot use this feature due to the serial nature of some of the operations that I alluded to (I'd have to almost rewrite the entire library for some of these complicated operations like groupby and sorting). I do h…

https://docs.scipy.org/doc/numpy/reference/generated/numpy.m...

You can create memory mapped ndarrays, these act like normal numpy arrays but don't need to fit into RAM. Numpy maps the array to a binary file on disk. The array otherwise acts like an ndarray so you can build a DataFrame with it. Whenever you access an array index Numpy in the background (essentially) seeks that many values into the file to grab the value of that index.

Since you're on a fast SSD and Numpy is fairly smart you'll be able to access your arrays close to your drive's speed. It's slower than if the whole database was in RAM but far faster than distributing the data over a network to a bunch of worker nodes. Memory mapped files let you have array-like access to data on disk as if it lived in RAM. When building a pandas DataFrame from a memmapped ndarray I believe you just need to set copy=False in the constructor for it to Just Work.

I don't know what your data looks like but I doubt loading it into SQLite is going to improve your performance.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#349

Post by inetknght »

hnews_account_1 wrote: post #338
inetknght wrote: post #223

Earlier quoted context omitted.

> Are you saying this can be optimised to fit inside a single 10 core server in terms of compute loads? I'm currently employed to write software which does DNA analysis. DNA is known for being Big Data. Some applications are very compute intensive and others are very data-relation intensive. The very compute intensive applications process about 1GB of data in about 30 minutes on a 32 core Xeon 6xxx with 32GB of RAM a…

So my entire dataset is ~24 x 250GB files. That 24 number can be larger if I can find an efficient way of processing each 250GB chunk. Each 250GB chunk is actually stock tick data so it has 500 stocks inside it. A heavily traded stock takes up ~10 GB of memory while a very thinly traded stock can top out at just 700 MB. While I hope that each 250GB chunk has everything in order and I can separate it cleanly, I don't…

Your problem sounds quite similar to a lot of FinTech interview questions for software engineers. They're solved problems but solutions aren't easy.

I have no doubt I could solve your problems. I honestly don't care to do so here though.

I imagine there's a software development/engineering team at your work or school you could ask for guidance though.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#350

Post by pkolaczk »

chusk3 wrote: post #321
pkolaczk wrote: post #190

Earlier quoted context omitted.

Having written lot of Java, Scala and C++ (and recently some Rust), I must say it is much easier to avoid heap allocations in C++ and Rust than in GCed languages, thanks to explicit allocation on the stack and pass by value + move semantics.

A big push in .net core 2.x and 3.x was what we call the Span-ification of the base class library and the runtime. This means there are many new APIs for dealing with slices of memory in a non-allocating manner, and this combined with memory pooling has contributed greatly to an overall performance boost to the runtime by reducing copying and GC time. These same APIs are available to the developer so I'd imagine that…

There exist libraries for native memory pooling in Java as well, and we're using them. I'm not saying low allocation code can't be done in C# or Java. But these languages don't give some nice tools that are present in C++ and Rust - in particular RAII and automatic reference counting.
Post reply on HN