31–40 of 56 posts
Sort uses only a fixed amount of memory, you can sort files larger than memory, but for such situations where you have only a few tens of millions of distinct values you can just use a python dictionary and it works even faster. While sort would shuffle data around a lot, the memory dictionary would just hold a key and a count as it gobbles the logs. It works because it is a special case of sorting where there are re…
Earlier quoted context omitted.
Pretty sure it's unlimited.
Appears that way. Still $920 if you had wanted to extract the 10TB back out though.
func main() {
ch := make(chan string)
for i := 0; i Earlier quoted context omitted.
The results of the Common Crawl project are hosted on AWS Public Data Sets, so it's not in my account. https://aws.amazon.com/datasets/
I see, without CommonCrawl paying for S3 (or AWS maybe eats that cost to help the public); this would be an expensive project.
> AWS is hosting the public data sets at no charge for the community
Sorry, but is this golang concurrent networking pattern correct: func main() { ch := make(chan string) for i := 0; i All goroutines receive on the same channel! Instead a new goroutine should be launched for each net conn. One should be able to spawn 1000s (or 1Ms) of conns and avoid ulimits using buffered chans, waitgroups, timeouts, or counters...
If you set MAX to 1000, you will have 1000 workers — and simultaneous connections.
The flaw is that when the last piece of work gets taken from the channel, the program will end, thus the last pieces of work that at the time are being processed, will get canceled. You could mitigate this by using a second channel, that the workers will access at the end of their work, thus ensuring that it will close only when the last worker finishes its work.
Sorry, but is this golang concurrent networking pattern correct: func main() { ch := make(chan string) for i := 0; i All goroutines receive on the same channel! Instead a new goroutine should be launched for each net conn. One should be able to spawn 1000s (or 1Ms) of conns and avoid ulimits using buffered chans, waitgroups, timeouts, or counters...
However, at a certain point, you may encounter bandwidth issues, timeouts, and the like due to local network congestion; that pattern has its uses there. I've tried writing a downloader that downloads every file it's given at once, and it went about as well as one would expect.
Sorry, but is this golang concurrent networking pattern correct: func main() { ch := make(chan string) for i := 0; i All goroutines receive on the same channel! Instead a new goroutine should be launched for each net conn. One should be able to spawn 1000s (or 1Ms) of conns and avoid ulimits using buffered chans, waitgroups, timeouts, or counters...
This pattern is correct (but has a flaw). It is a simple worker pool. The first available worker will grab the first piece of work from the channel and process it. If you set MAX to 1000, you will have 1000 workers — and simultaneous connections. The flaw is that when the last piece of work gets taken from the channel, the program will end, thus the last pieces of work that at the time are being processed, will get c…
Earlier quoted context omitted.
This pattern is correct (but has a flaw). It is a simple worker pool. The first available worker will grab the first piece of work from the channel and process it. If you set MAX to 1000, you will have 1000 workers — and simultaneous connections. The flaw is that when the last piece of work gets taken from the channel, the program will end, thus the last pieces of work that at the time are being processed, will get c…
The in-article version has a time.Sleep(2 * time.Second) after the scan loop. Not exactly reliable (waitgroups or channel signaling would be better) but better than nothing.
Earlier quoted context omitted.
Appears that way. Still $920 if you had wanted to extract the 10TB back out though.
Yeah... that is one of the really unfortunate lock-ins with AWS. Hopefully they will add data export to their "Snowball" product, but they don't really have a lot of incentive to.
Still, it does appear to reduce the price of getting data out of Amazon compared to using the internet.
Earlier quoted context omitted.
That's awesome. If you are not already doing so, you can download my set from the torrent and include it in your database. https://all-certificates.s3.amazonaws.com/certificates.tar.g... For exporting, pg_dump -F c greatly compresses the data so cost-wise you might be able to put on S3 and publish as a torrent.
Exporting is one possibility, but eventually I'd like to provide a read-only sql access to the database we host. We have a few ideas on how to do this [1], but it's not implemented yet. [1] https://github.com/mozilla/tls-observatory/issues/92