• Multiprocessing Size and Rank

    I have always thought Python did a great job exposing parallel processing with the multiprocessing package. The Pool class in particular made it relatively simple to jump from the built-in map function, which is a good first step to accelerating loops, to utilizing all cores on a processor without any obscure hoops.


    26 May 2018

  • Generating Different Hash Functions

    Representing genetic sequences using k-mers, or the biological equivalent of n-grams, is a great way to numerically summarize a linear sequence. Depending how unique you need your k-mers to be, you may overallocate your system memory trying to keep track of all 4^k possibilities, where there are 4 possible bases (A, G, C, T) and k-length strings. To circumvent this technological constraint, Bloom filters were designed to probabilisticly track the presence (not count) of items.


    05 Feb 2018

  • Using your own cluster with CyVerse

    The CyVerse SDK currently guides developers through the process of creating apps that run on TACC supercomputers, which requires both a TACC account and an active allocation. For users without an active allocation, they can request to be added to the iPlant-Collabs allocation, which allows developers to prototype CyVerse applications. I stress the word prototype because the iPlant-Collabs allocation is relatively small on purpose to make sure unvetted apps aren’t burning away all the compute time allotted for CyVerse. This methodology is great to generate new apps, the apps needs to be reviewed and published by administrators before they can be run at scale on CyVerse. If a user already has access to a high-performance cluster at their own institution, they can circumvent the constrains of iPlant-Collabs by registering their own executionSystem to the Agave API and running apps on it from the CyVerse Discovery Environment.


    07 Feb 2017

  • Falcon-Zero: Maximum Velocity

    I previously demonstrated how to complete an assembly of the E. coli genome in 54 minutes using Stampede at TACC and CyVerse. While convenient, this is fairly slow for the amount of provided resources compared to the input size. Out of the box, Falcon can run concurrent tasks on multiple compute nodes. However, this is done by submitting each task as a separate batch job to the system scheduler. A system scheduler is not meant to be a high-performance load balancer, but a fair way schedule variable sized workloads to run. Our systems, which are available to local UT-system, national NSF, and global collaborating researchers, are often oversubscribed and subject to fairly long waiting times. This renders any assumption that hundreds or thousands of job submissions can be run to accomplish a given task totally impractical.


    25 Aug 2016

  • Falcon in the NSF Cloud!

    I am proud to announce the release of an optimized version of the Falcon assembler with the help of Cyrus Proctor and Jawon Song at the Texas Advanced Computing Center. The Falcon diploid assembler is an exciting tool for biological research and allows for the assembly of complex genomes in record time. The tool is currently accessible on the CyVerse Discovery Environment and will be deployed as standard module on the Stampede supercomputer in an upcoming maintenance cycle.


    29 Apr 2016