Live data from Hacker News

Ask HN: What is your ML stack like?

news.ycombinator.com

81–90 of 134 posts

Re: Ask HN: What is your ML stack like?

#81
What has worked fairly well so far:

Models:

- Models are structured as python packages, each model inherits a base class

- base class has define how to train, and how to predict (as well as a few other more specific things)

- ML engineer can override model serialization methods, default is just pickle

Infra:

- Code is checked in to github, Docker container built each merge into master

- Use Sagemaker BYO container to train models, each job gets a job_id that represents the state produced by that job (code + data used)

Inference / deployment:

- Deploy model as http endpoints (SageMaker or internal) using job_id

- Have a service that centralizes all score requests, finds correct current endpoint for a model, emits score events to kinesis, track health of model endpoints

- A/B test either in scoring service or in product depending on requirements

- deploy prediction jobs using a job_id and a data source (usually sql) that can be configured to output data to S3 or our data warehouse

So far this has been pretty solid for us. The tradeoff has been theres a step between notebook and production for ML engineers which can slow them down, but it forces code review and increases the number of tests checked in.

Re: Ask HN: What is your ML stack like?

#82
post #81

What has worked fairly well so far: Models: - Models are structured as python packages, each model inherits a base class - base class has define how to train, and how to predict (as well as a few other more specific things) - ML engineer can override model serialization methods, default is just pickle Infra: - Code is checked in to github, Docker container built each merge into master - Use Sagemaker BYO container to…

> The tradeoff has been theres a step between notebook and production for ML engineers which can slow them down, but it forces code review and increases the number of tests checked in.

This was a game-changer for us. What does your testing story look like?

Re: Ask HN: What is your ML stack like?

#83

We're currently running a single NVIDIA RTX2080 with Tensorflow 2.0 on a Windows 10 station. We'll soon be switching to a standard multi-GPU rig running an air gapped Linux distro. Linux seems overall much better for ML because of better Docker integration and tensor core support on the newer GPUs. Also, we'll probably be switching from Tensorflow to Pytorch for model development. Pytorch requires a little bit more c…

Why airgapped? Is it a business/security requirement? If you have to share the machine, does everybody have to thumb drive over their files to run with the big GPU?

Yes. The air-gapping isn’t ideal, but trying to get our IT org to accommodate a Linux workstation on the network just isn’t worth the hassle.

Re: Ask HN: What is your ML stack like?

#84

Earlier quoted context omitted.

Could you possibly define "pickling" in this context for us ML noobs?

If by ML noob you mean to say that you're like me and have zero formal CS training (as in, I don't know what a data structure is), pickling lets you write your Python workspace to a file just like Matlab's .mat file loading. It's excellent for writing scripts defining different parts of a data pipeline, or just for debugging/trying new things without waiting 20 minutes for something to filter.

> as in, I don't know what a data structure is

Basically everything you work with in programming is a value (the number one-hundred-seventy-five, for example: "175") or the address—location in computer memory, say—of a value. You might record the address of that value above as the count of characters from the beginning of this post, for example, were this post the layout of data in some RAM, just as numbering houses on a street. Add the concept of data "width"—how long the number is, in terms of how many characters represent it (3, in this case) and you've got basically all there is in terms of primitive stuff that computer programs operate on.

Observe that the value stored at a location in memory, say, might itself be an address—location—of some other thing stored in memory.

The width+value concept can get you pretty far, in that you can store a bunch of stuff and find it again, given the address of the beginning and some convention that the first so-many bits of the value describe the width of the rest of the value, or some other means of knowing the size (width) of the value, as long as all its parts are stored right next to each other in memory, and in the correct order. That's called an array, in fact, which is a data structure! One problem with arrays is that if you want to make them longer, you might not have more memory available at the end of them—something else may be using that location already, and if you just overwrite it that'll likely break something. So you'll have to copy your entire array to a larger piece of empty memory to add on to the end of it.

EXAMPLE!

Say we have some RAM large enough to store nine things, and we know that we have something stored at position 4—programmers like to order items starting with zero, but I'll refrain because it's not really important here and makes it more confusing. The RAM contains the stuff we're looking for, plus some other stuff that we don't care about right now:

[429317501]

We look at position 4, and know (by convention, or whatever) that the "3" we find tells us how many more locations to read past that to get our entire value, and extract "175" as our value, by proceeding to do just that. That's an array! One of the most basic data structures. Of course all this is binary under the hood, and those binary numbers can also represent letters or color values of a pixel in an image or whatever, but I'm using simple base-10 numbers to keep things easier to follow.

You can use these basic pieces to build up more complex data structures than that, of course. You can have a series of places in memory, not necessarily next to each other, each containing a value and then the address of another value+address pair. Given the address of the first of these, one could write a program to read each in order, following the addresses to hop from one to the next until it finds one without an address provided. Ta-da, it's (one kind of) a linked list! That's another kind of data structure. Now if you want to add to the end of your value, you just pick an empty spot in memory, add its address to the last piece of the existing list (first "walking" the list to find it by starting at the beginning and reading each piece in turn, if you don't already know where the last part is located), then fill it in with the value you want. This takes up more space than an array, though, and may take a little longer to "read" (get the value[s] from). It's very common for many types of data structure to be able to represent the same thing, just with different trade-offs in terms of space used, or time to locate a given part of the structure, or how much space or time it takes to modify it (recall, having to copy an entire array in order to add on to it) and so on.

EXAMPLE!

[950818972]

If we know (somehow) that our linked list starts at position (address) 5, and know that we can expect a value then an address at that location, next to one another, we see "18", so our value starts with 1 and we should look at address 8 next, where we find "72"—value is 7, and now look at address 2, finding "50". Zero in our little make-believe addressing system here conventionally means there are no further addresses to look up (there is no address 0) so, without knowing how long the list would be when we started, we now know we're done, that there were three values stored in the list, and that they are, in order, "1", "7", and "5".

If you've followed this far, you may be able to see how one could make "trees" (pairs of addresses instead of just one, suggesting a "left" and "right" path) and other things from these fundamental parts, and may be able to think of reasons why one might do this. A linked-list where all the "values" are addresses to parts of some other structure (with the "address" portion of the linked list item used as normal)? That's a sort of index, right? A list where the "value" at each position is the address of the beginning of another list? We call that a multidimensional list (or multidimensional array, if it's laid out as an array) and it's one way a person might represent a grid of, for example, colors in an image (this is basically what a bitmap is). And so on.

Disk storage uses the same fundamental building blocks. FAT, as in FAT32 or FAT16, old DOS and Windows file systems? File Address Table is what FAT stands for. There's a table (kinda like conjoined lists, much as above), starting at a conventional spot (address) on a FAT-formatted disk partition, that describes where all the files are on the rest of the disk, along with some other info about the filesystem. It's basically exactly what it sounds like, and uses precisely the same concepts as above, just applied to locations of files on a disk rather than locations of values in RAM.

One last insight: program code—the instructions for the CPU—is also stored in memory, and that works basically the same way as the stuff above. This unified, undifferentiated storage system is called Von Neumann architecture, and it's what pretty much all computers you're likely to encounter use. Point being, those addresses pointing to things stored in memory? They can also point to places where code is stored, which, once located, one might direct the CPU to execute. A little thought on that, combining it with the above notions, should suggest some cool things this would enable.

And that's about it. That's data structures, and indeed much of programming. It's all values, addresses, widths, and more than a little bit of convention.

Re: Ask HN: What is your ML stack like?

#87
Data ingest: AWS Lambda (JavaScript) using the Serverless Framework to Kinesis Firehose.

Continuous Training: Lambdas triggering SageMaker BYO training jobs.

Continuous Model Deployment: Lambdas polling SageMaker training jobs to update SageMaker BYO endpoints.

Inference endpoint: Lambda proxying to a SageMaker endpoint.

Our BYO endpoints allow us to do batch inference without a bunch of round trips between Lambda and SageMaker.

If endpoint costs get too high, we’ll implement some caching at the lambda layer.

Re: Ask HN: What is your ML stack like?

#88
Almost entirely in Go. I use Gorgonia [0] and Gonum [1]. Granted I wrote Gorgonia. All solutions fit into the company's CI/CD infra with almost no additional overhead.

Sometimes the end result is gRPC services, sometimes its some sort of serialized model (weights). Sometimes the model is specified in protobuf. Very rarely it's a HTTP API. I don't fancy those.

Ironically I haven't done much distributed models. Or if it's distributed, it's not some Kafka-esque monstrosity.

I rarely use Python for anything other than exploratory analyses now.

Being able to type `go build .` and have it run anywhere is pretty awesome

[0] https://gorgonia.org [1] https://gonum.org

Re: Ask HN: What is your ML stack like?

#89
post #85

I'm pretty impressed with the level of automation I'm seeing in general. Looks like many are using docker/k8s or containers in some way or another. Inspiring.

It's pretty horrifying to me. Why are we adding layers and layers of abstraction that does not really make life easier?

Re: Ask HN: What is your ML stack like?

#90
post #69
post #4

crystal / shainet ( https://github.com/NeuraLegion/shainet ) I contract for some clients in fintech and some defense-related stuff.

Is there a specific reason why you chose shainet/crystal?

I love crystal and I have a lot of autonomy so I get to use what I want. Shainet is also really good in general for coming up with new network topologies, which I find is a lot harder to do in a lot of the python-based mega-frameworks, Cafe, etc. In general, fibers are really good for concurrent/parallel data processing, and now that crystal has true parallelism (enabled with a flag), for my purposes there is no reason not to use it.

Occasionally I will use rust or C/C++ for some of these tasks, but I try to keep things in crystal whenever I can.

Post reply on HN