31–36 of 36 posts
Looking at the examples in the landing page.. so I don't need any kind of authentication to do inference? Anyone can run the models I upload?
Or, do your inference using an AVX-512 CPU: https://NN-512.com (open source, free software, no dependencies) With batch size 1, NN-512 is easily 2x faster than TensorFlow and does 27 ResNet50 inferences per second on a c5.xlarge instance. For more unusual networks, like DenseNet or ResNeXt, the performance gap is wider. Even if you allow TensorFlow to use a larger ResNet50 batch size, NN-512 is easily 1.3x faster. If…
Oh that's very interesting, how ready for production is it? It only works for TF right? > If you need a few dozen inferences per second per server, this is the cheapest way. And you're not depending on a proprietary solution whose parent company could go out of business in a year. Definitely the cheapest way. We've been in business for more than a year already actually :)
I'm giving performance comparisons versus TensorFlow, which I consider to be a standard tool.
People who use your proprietary, closed, black-box service are dependent on the well-being of your business. You could vanish tomorrow.
Earlier quoted context omitted.
Well, I guess I know where I am going to host GPT-J-6B then. I don't think it is sustainable.
How are you planning to put a gpt whatever when the service clearly have a model size limit?!
Looks awesome. Do you know if and how you guys support NVIDIA's software. For my project the NVIDIA software I'm using states it needs: CUDA 11.3.0 cuBLAS 11.5.1.101 cuDNN 8.2.0.41 NCCL 2.9.6 TensorRT 7.2.3.4 Triton Inference Server 2.9.0 I'm new to deploying to production inference so I'm not sure if those are easily portable across such platforms or not really.
Looking at the examples in the landing page.. so I don't need any kind of authentication to do inference? Anyone can run the models I upload?
At the moment, no, only the random hash gives some kind of security by obfuscation. More advanced security controls are coming soon.
Looking at the examples in the landing page.. so I don't need any kind of authentication to do inference? Anyone can run the models I upload?
At the moment, no, only the random hash gives some kind of security by obfuscation. More advanced security controls are coming soon.