Batch Processing

ThreadPoolExecutor is my new best friend. I've redesigned the data flow (for the third time), and now the system can process over 100 cutouts per minute! The parallel fetching pipeline works like a charm. 

Comments

Post a Comment

Popular posts from this blog

Damn couldn't believe I got in again.

I was glued to reddit and the website the whole time for 4 hours. The results came out later than UTC 18:00 and didn't get the mail until four hours beyond the scheduled time.  None of my friends got in, so was kinda anticipating a rejection too, but to my surprise I got in. I am really excited to go through this journey again. It would be amazing. 
Read more

Proposal Sent! The Waiting Game Begins...

 Drafted, revised, got feedback (mentally, from myself mostly haha), polished, and finally hit submit on the proposal! It feels pretty comprehensive, outlined the pipeline modules, the timeline, potential hurdles... Pouring all that learning from the courses and the tests into it felt good. Now... we wait. Fingers, toes, everything crossed! Whatever happens, learned a ton already.
Read more

Proposal Mode: Activated

 Okay, tests done, now the real writing begins. The proposal. Need to show I understand LSST, DeepLense, the project goals, AND have a solid plan. Drafting sections on objectives, methodology, the timeline... This is where the deep dive into the LSST pipeline docs and the project description really matters. 
Read more