Live data from Hacker News

Stripe-CTF 3 Writeup

muehe.org

41–50 of 74 posts

Re: Stripe-CTF 3 Writeup

#41
post #37

I love his hash table solution to level0! I would never have spent much time optimizing such a trivial challenge as level0, but after seeing the leaderboards with such ridiculous scores, curiosity got the best of me, and I had to figure out how people were achieving such speedups. I used a precomputed bloom filter to get 2641 points on level0, because I couldn't think of another way to avoid doing the expensive proce…

Op here. I have done a lot of work involving hash tables in main-memory database research and therefore really like applying those optimizations. You are right though that it might be overkill :-)

Re: Stripe-CTF 3 Writeup

#42
post #33

I think this particular bit says it all. To get blazingly fast results... > I can say with great confidence that we all cheated. But I disagree with the word used. Pre-computing the hash-table and embedding that into the final binary to eliminate relatively expensive calculations when it matters? That's not cheating, that's effing brilliant . Hard-core software engineering, that's what it is. There's also a saying th…

Op here, just wanted to say thanks. Of course it is not cheating in the sense that I tricked the system to give me a better score than warranted but my solution only works for this ONE dictionary so.. maybe it is not cheating but using tighter assumptions than those which were provided by the original task description :-)

Re: Stripe-CTF 3 Writeup

#43
I really had a blast doing level 1. I wrote my first real C program and it was as blazingly fast as I hoped. After that level 2 was easy but like many others I got stuck in level 3.

Re: Stripe-CTF 3 Writeup

#44
post #38

I read through this and was amazed by the OP's approach to the problems even though I could barely understand 10% of the reading. How do I learn to think and code like this? I honestly have no idea where to begin.

Thank you. What you have to keep in mind is that I do databases research for a living so many of the problems involved in each challenge are very familiar to me. My suggestion to you is to get a stronger systems programming background, I figure that's what helped me the most.

Re: Stripe-CTF 3 Writeup

#45
post #33

I think this particular bit says it all. To get blazingly fast results... > I can say with great confidence that we all cheated. But I disagree with the word used. Pre-computing the hash-table and embedding that into the final binary to eliminate relatively expensive calculations when it matters? That's not cheating, that's effing brilliant . Hard-core software engineering, that's what it is. There's also a saying th…

Op here, just wanted to say thanks. Of course it is not cheating in the sense that I tricked the system to give me a better score than warranted but my solution only works for this ONE dictionary so.. maybe it is not cheating but using tighter assumptions than those which were provided by the original task description :-)

The problem description did say:

    # Our test cases will always use the same dictionary file (with SHA1
    # 6b898d7c48630be05b72b3ae07c5be6617f90d8e).

Re: Stripe-CTF 3 Writeup

#46
Very fair feedback on moving the deadline. We'd debated doing it or not, and ended up moving it in the spirit of distributed systems education. A lot of people in the chat appreciated it, but that's obviously a selected segment.

So we know for next time, how strong are other people's feelings on this?

Re: Stripe-CTF 3 Writeup

#47
post #30

After many hours I failed to complete Level 4. How frustrating! But what a great experience. I think Stripe did a great job at building a programming challenge. By going through the challenge I learned something new in each of the following: Python, Ruby, Scala, Go, Node/Javascript, Git, Hashes, DDOS, Consensus Algorithms, raft, transactional logs, Inverted Indexes, BitCoin, IRC Chat, and more. Thanks Stripe you made…

You're very welcome. Glad you enjoyed :).

Re: Stripe-CTF 3 Writeup

#48
post #46

Very fair feedback on moving the deadline. We'd debated doing it or not, and ended up moving it in the spirit of distributed systems education. A lot of people in the chat appreciated it, but that's obviously a selected segment. So we know for next time, how strong are other people's feelings on this?

I'm glad you did (but I wouldn't have finished if you hadn't).

More people learning & having fun is better. IMO people only complain about the extended deadline because of its effect on the rankings.

The scoring system was an interesting addition over CTF2, but it also created these complaints and silliness like people submitting the same code repeatedly to get the best score possible.

It may have been better to only tell players what broad percentile group they're in. There would still be an incentive to write faster code, without the frustration of being edged out of position #20 because someone else's submission was (arbitrarily) scored a few points higher.

Re: Stripe-CTF 3 Writeup

#49
I went up to level 3 and then work/life took over. Here's my approach to the problems:

(1) Python. Used a set() for the dictionary. Got ~200 points. I was planning to go back and implement this in Go to see the difference in performance but never got around to do it.

(2) Python. I wrote a single-threaded miner which could do around 300K hashes/s. My miner calculated 1M hashes (~3s), then fetched origin, then continued until it got a hit. I stopped after the first Gitcoin.

(3) Node. My solution was very simple and pretty much like the OP's. I kept a map between IP and count and always let through IPs with count 4, then I let them through with 0.3 probability to keep the backends working.

(4) I spend most of my time on this problem, largely trying to keep the memory within the constraints. I started with Node and created a simple inverted index of all words to their positions. Since we needed substring search I had to loop through the keys of the index which was too slow. My next attempt was to index all substrings but that used too much memory. My third, and best, Node attempt was using a prefix tree to index all suffixes, effectively building a suffix tree. This was quite fast, got ~2200 on local tests but was still over the memory limits.

I then gave up on Node and moved to Python. I used Flask for the server and built two solutions. A simple inverted index and a prefix tree (using datrie) with all suffixes. To keep the memory footprint low, I stored file/line locations as bit-shifted longs. Amazingly, the simple index and the prefix tree performed the same (!), so I submitted the inverted index solution and got a passing score.

Thanks for a great week Stripe. I learned tons! Looking forward to the next one.

Re: Stripe-CTF 3 Writeup

#50
Scored in the top 10 on level0 using a Bloom filter in C. Once you had an optimal solution, getting on the leaderboard came down to luck. 90% of the runtime of my program was operating system overhead that also happened on `int main() {}`. Today I realized I could've compiled with -nostdlib or -nostartfiles and halved that overhead - lesson for the future. If your binary needs to run faster than 1.5ms, libc is the next thing to cut :)

I held the top leaderboard spot of level1 for a day or so (my hashrate varied between 1-5GH/s) but was pushed out in the last few hours by pushrax and the Stripe server load. Used an OpenCL + Go solution. Was a close race there by all.

http://bochs.info/~aegis/rounds.tar.gz - tarball of all the Gitcoin round repos if anyone wants to play with the data.

Post reply on HN