100M-Row Challenge with PHP

(github.com)

43 points | by brentroose 3 hours ago

4 comments

  • brentroose 3 hours ago
    A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more.

    That's why I built a performance challenge for the PHP community

    The goal of this challenge is to parse 100 million rows of data with PHP, as efficiently as possible. The challenge will run for about two weeks, and at the end there are some prizes for the best entries (amongst the prize is the very sought-after PhpStorm Elephpant, of which we only have a handful left).

    I hope people will have fun with it :)

    • user3939382 5 minutes ago
      exec(‘c program that does the parsing’);

      Where do I get my prize? ;)

    • gib444 19 minutes ago
      > A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds

      That's a huge improvement! How much was low hanging fruit unrelated to the PHP interpreter itself, out of curiosity? (E.g. parallelism, faster SQL queries etc)

  • Retr0id 8 minutes ago
    How large is a sample 100M row file in bytes? (I tried to run the generator locally but my php is not bleeding-edge enough)
  • pxtail 40 minutes ago
    Side note - I wasn't aware that there is active collectors scene for Elephpants, awesome!

    https://elephpant.me/

  • spiderfarmer 51 minutes ago
    Awesome. I’ll be following this. I’ll probably learn a ton.