Idea for an Amazon rainbow table super computer

[joel]

1. Run a program that will generate a wordlist, put the output in an queue.
   
2. Have a bunch of EC2 instances that take input from the queue and generate MD5, SHA1, etc hashes from the queue. Store hash to password mappings in SimpleDB
   
3. Depending on how fast inserts into SimpleDB are, it might make sense to store the hash/password mappings on disk or in

Comments

[rndmcnlly]

AWS is pretty heavyweight and expensive (compared to free, local resources).

Generating the next word in a sequence, hashing it, and storing it somewhere on a single machine is not much work (1us?). Adding an item to a SQS queue, removing it from the queue with a different program, hashing it and then storing it back to SimpleDB might take up to a whole second with packaging, handshaking, allocation, buffering, whatever. This sounds like a recipe for paying Amazon to do situps.

I think running a few monolithic processes on EC2 instances just once and then saving the resulting data once, un-indexed would be a good first step. Completely seperately you could decide how to provide easy and cheap access to the results.

Calculation?

[anonymous]

Did you ever calculate the cost of generating a rainbow table, say for MD5 of all possible 16-character strings, using EC2?

I would probably have a cluster of machines each using a different character at the start of the string, generating all the hashes from that seed and then posting the results back to a central store, rather than all the overhead of dealing with a queue.

Re: Calculation?

[joel]

No, I still need to do the calculation.

And yes, using a queue is probably a bad way to do this ... it's just so ... shiny.