Profiling the Pipeline and Hunting for Bottlenecks

 This week was all about performance analysis. I’ve been using Python’s profiling tools to get a detailed breakdown of where the pipeline is spending its time. The results confirmed my suspicions: a significant amount of time is lost to I/O-bound operations and redundant preprocessing steps that are not being efficiently batched.

Based on this analysis, I've started refactoring the core processing loop. The plan is to implement a producer-consumer pattern, where a pool of worker processes is dedicated to fetching and preparing data, feeding a steady stream of tensors to the GPU for inference. This should decouple data preparation from model execution and allow for much higher throughput.

Comments

Post a Comment

Popular posts from this blog

Damn couldn't believe I got in again.

I was glued to reddit and the website the whole time for 4 hours. The results came out later than UTC 18:00 and didn't get the mail until four hours beyond the scheduled time.  None of my friends got in, so was kinda anticipating a rejection too, but to my surprise I got in. I am really excited to go through this journey again. It would be amazing. 
Read more

Proposal Sent! The Waiting Game Begins...

 Drafted, revised, got feedback (mentally, from myself mostly haha), polished, and finally hit submit on the proposal! It feels pretty comprehensive, outlined the pipeline modules, the timeline, potential hurdles... Pouring all that learning from the courses and the tests into it felt good. Now... we wait. Fingers, toes, everything crossed! Whatever happens, learned a ton already.
Read more

Proposal Mode: Activated

 Okay, tests done, now the real writing begins. The proposal. Need to show I understand LSST, DeepLense, the project goals, AND have a solid plan. Drafting sections on objectives, methodology, the timeline... This is where the deep dive into the LSST pipeline docs and the project description really matters. 
Read more