Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A lot of complex “scalable” systems can be done with a simple, single C++ server

twitter.com

Post reply on HN
Print view View original

361–370 of 376 posts

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#361

Post by hnews_account_1 »

giantrobot wrote: post #348
hnews_account_1 wrote: post #336

Earlier quoted context omitted.

Can you provide a link to this please? My current knowledge is that all numpy data lives in memory, and pandas itself has a feature to fragment any data into iterables so I can read upto my memory limit. I cannot use this feature due to the serial nature of some of the operations that I alluded to (I'd have to almost rewrite the entire library for some of these complicated operations like groupby and sorting). I do h…

https://docs.scipy.org/doc/numpy/reference/generated/numpy.m... You can create memory mapped ndarrays, these act like normal numpy arrays but don't need to fit into RAM. Numpy maps the array to a binary file on disk. The array otherwise acts like an ndarray so you can build a DataFrame with it. Whenever you access an array index Numpy in the background (essentially) seeks that many values into the file to grab the va…

Unfortunately my data isn't all numbers. It has text too. The sparse examples in that link only show this for reading in numbers. Do you know off hand if it translates well? There is a dtype parameter, but it'll take me a few days to get back to this code, so I figured I'd check beforehand.

Your second paragraph is essentially what I want. I'm willing to wait a day for code that may run in 1 hour from memory, so time isn't entirely an issue unless it's starting to bleed into weeks. The read_csv function in pandas has a parameter called memory_map, but when I tried using it on a smaller 7GB dataset, it read the whole thing into memory (32GB instance) even when I set it to True.

SQLite is definitely not my best option here. It was the only server-less implementation I could find, so I tried to use it and it didn't work. However, a database like implementation will be helpful because each operation I need to do will require data that satisfies certain timestamp and arithmetic conditions. I figured it'd be best to load the whole thing into a db and query it for every operation to train my model.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#362

Post by hnews_account_1 »

westurner wrote: post #342
hnews_account_1 wrote: post #336

Earlier quoted context omitted.

Can you provide a link to this please? My current knowledge is that all numpy data lives in memory, and pandas itself has a feature to fragment any data into iterables so I can read upto my memory limit. I cannot use this feature due to the serial nature of some of the operations that I alluded to (I'd have to almost rewrite the entire library for some of these complicated operations like groupby and sorting). I do h…

Dask groupby example: https://examples.dask.org/dataframes/02-groupby.html > Generally speaking, Dask.dataframe groupby-aggregations are roughly same performance as Pandas groupby-aggregations, just more scalable. The dask.distributed scheduler can also run on one high-RAM instance (with threads or processes) https://docs.dask.org/en/latest/setup.html Pandas docs > Ecosystem > Out-of-core: https://pandas.pydata.org/p…

Unfortunately, I'm not sure what's wrong with dask, but it doesn't work properly on my cluster. I tested it on an exceedingly simple operation - find all unique values in a very big column (5 billion rows, but I know for a fact that there are only 500-502 unique values in there). With a 100 workers, it still failed. Now this is an embarrassingly parallel operation that can be implemented trivially. So I'm not sure if there's a problem with my cluster or if dask just does not work with slurm clusters very well.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#363

Post by hnews_account_1 »

inetknght wrote: post #349
hnews_account_1 wrote: post #338

Earlier quoted context omitted.

So my entire dataset is ~24 x 250GB files. That 24 number can be larger if I can find an efficient way of processing each 250GB chunk. Each 250GB chunk is actually stock tick data so it has 500 stocks inside it. A heavily traded stock takes up ~10 GB of memory while a very thinly traded stock can top out at just 700 MB. While I hope that each 250GB chunk has everything in order and I can separate it cleanly, I don't…

Your problem sounds quite similar to a lot of FinTech interview questions for software engineers. They're solved problems but solutions aren't easy. I have no doubt I could solve your problems. I honestly don't care to do so here though. I imagine there's a software development/engineering team at your work or school you could ask for guidance though.

> I have no doubt I could solve your problems. I honestly don't care to do so here though.

That's fair. I did not think it'd be such a difficult problem when I first set out to do it myself. But every single turn leading to a dead end kinda bummed me out. I'm going to finally resort to the database method of storing all the data in one query-able file and work off of that.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#364

Post by westurner »

hnews_account_1 wrote: post #362
westurner wrote: post #342

Earlier quoted context omitted.

Dask groupby example: https://examples.dask.org/dataframes/02-groupby.html > Generally speaking, Dask.dataframe groupby-aggregations are roughly same performance as Pandas groupby-aggregations, just more scalable. The dask.distributed scheduler can also run on one high-RAM instance (with threads or processes) https://docs.dask.org/en/latest/setup.html Pandas docs > Ecosystem > Out-of-core: https://pandas.pydata.org/p…

Unfortunately, I'm not sure what's wrong with dask, but it doesn't work properly on my cluster. I tested it on an exceedingly simple operation - find all unique values in a very big column (5 billion rows, but I know for a fact that there are only 500-502 unique values in there). With a 100 workers, it still failed. Now this is an embarrassingly parallel operation that can be implemented trivially. So I'm not sure if…

https://docs.dask.org/en/latest/setup/hpc.html says dask-jobqueue handles "PBS, SLURM, LSF, SGE and other resource managers"

"Dask on HPC, what works and what doesn't" https://github.com/dask/dask-blog/issues/5

Maybe you should spend some time developing a job visualization system for end users from scratch, for end users with lots of C, JS, and HTML experience https://jobqueue.dask.org/en/latest/interactive.html

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#365

Post by hnews_account_1 »

westurner wrote: post #364
hnews_account_1 wrote: post #362

Earlier quoted context omitted.

Unfortunately, I'm not sure what's wrong with dask, but it doesn't work properly on my cluster. I tested it on an exceedingly simple operation - find all unique values in a very big column (5 billion rows, but I know for a fact that there are only 500-502 unique values in there). With a 100 workers, it still failed. Now this is an embarrassingly parallel operation that can be implemented trivially. So I'm not sure if…

https://docs.dask.org/en/latest/setup/hpc.html says dask-jobqueue handles "PBS, SLURM, LSF, SGE and other resource managers" "Dask on HPC, what works and what doesn't" https://github.com/dask/dask-blog/issues/5 Maybe you should spend some time developing a job visualization system for end users from scratch, for end users with lots of C, JS, and HTML experience https://jobqueue.dask.org/en/latest/interactive.html

Yes, I think the library still isn't ready for non HPC experts to use without tinkering. I don't have that level of expertise. I'm a data person who can handle working tools.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#366

Post by Fazel94 »

kitsuac wrote: post #2

I've been programming C++ and assembly for 23 years. Few years ago I became a huge fan of Python. In my opinion Python is amazingly well suited for rapid first revision and can then be swapped out for C++ / asm.

You just have to write a tiny part that uses a lot of CPU in C++/asm or anything else.

Much of code's performance isn't really reflected on to the scalability since mostly a tiny part of code is really ran a lot of time, and the other parts are just glues or management stuff or rarely used(not used in scale) features.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#368

Post by giantrobot »

masklinn wrote: post #353
giantrobot wrote: post #224

Earlier quoted context omitted.

You have 250GB of "raw" data stored in CSV format. The parsed version of this data in memory is likely to be a fraction of the on-disk size. A `long` or `double` only take up eight bytes in memory but 10-20 bytes on disk stored as ASCII in a CSV file. Even if your raw data was 250GB you could store it in memory mapped files. A fast SSD can easily hit a gigabyte per second sequential read speed, far faster than your t…

> A fast SSD can easily hit a gigabyte per second sequential read speed, far faster than your typical network. Nit: 1GB/s is ok, not even solid let alone fast . A fast SSD pretty much saturates 4x3.0 links (which explains why they universally tend to cap out at 3.5GB/s). In fact there are now a few PCIe4 SSDs (e.g. Corsair's MP600) which close in on 5GB/s.

I got the 1GB/s from my laptop I was writing the comment on. The internal NVME drive tops out at 1.5GB/s. I consider that fast but as you point out there's drives that make mine look slow.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#369

Post by bsenftner »

cthalupa wrote: post #359
bsenftner wrote: post #284

Earlier quoted context omitted.

Well, in my case described above, single server solutions include an automated backup sub-system, and my servers expect multiple instances of itself to running on the client network, and these multiple instances synchronize with one another via additional endpoints specific for the purpose. The whole issue of HA and backups is critical and one of the areas my approach shines.

You're still not actually answering the question of how you are HA and backing things up with only a single physical server and nothing else. If you're backing things up to the same server, that's not enough. If the HA instances are running in the same server, that's not enough. If there are other things besides that one physical server and it's power/network, you didn't include them in the cost, so the comparison is…

I do say the deployed system ends up being multiple instances of my single server, which synchronize with one another. Those are separate physical devices each running my one server. Additionally, when a backup runs the data is stored locally as well as on a physical storage device separate from the hardware it is running. Typically clients already have a firewall/router which is used to distribute requests to the various instances. My deployed systems are not one server, they become a server mesh.

Re: A lot of complex “scalable” systems can be done with a simple, single C++ server

#370

Post by bsenftner »

cthalupa wrote: post #360
FpUser wrote: post #267

Earlier quoted context omitted.

I am in the same boat (writing native servers). I will also "disappear" if you start asking me about HA/backups/etc. A particular solutions are very much case specific and can depend on business conducting rules just as much as on pure tech factors. Properly answering your question requires way too much writing and hardly a subject of a single post. I have HA solutions for the products I built but this post is the ex…

I'm not asking about the specifics, and I don't really care about them. But the fact of the matter is quite simply is that any single physical server solution will never be satisfactory for backups or HA purposes. You can't store your backups in the same place as your data and call it good - what do you do when you have multiple disks fail and your RAID can't be rebuilt? This happens. What do you do when operator err…

I can't speak for FpUser, but you misunderstand the idea of creating a single server: one does not just run one of them, they know about and expect multiple copies of themselves to be running at different IP addresses, and they synchronize with one another, as well as maintain individual backups that additional background processes validate between different instances.

Each and every one of your disaster scenarios is handled by the architecture. Each an every one of your disaster scenarios has happened and we've lived through them, as well as after the fact reviewed and optimized how we handled the events. As you are, we're professionals.

Post reply on HN