Loading LLM (Llama-2 70B) 20x faster with Anyscale Endpoints

How long does it take to download Llama2 70B? On the 4x 25 Gbps NICs that aws.p4de's have, it should take ~10s. Yet in production we've observed much higher times, which makes autoscaling less responsive + more expensive. This blog post shows how we've reduced download & init time from 4m25s to 20s, using techniques such as streaming S3->CPU->GPU and multiple TCP streams.