Batch Processing

ThreadPoolExecutor is my new best friend. I've redesigned the data flow (for the third time), and now the system can process over 100 cutouts per minute! The parallel fetching pipeline works like a charm. 

Comments

Post a Comment

Popular posts from this blog

GSoC 2025: Final Submission and a Look Ahead

And with that, the Google Summer of Code 2025 coding period comes to a close. I’ve just submitted my final term report for the RIPPLe project, marking the culmination of an intense and incredibly rewarding four-month journey. Looking back to May, I remember the steep learning curve of just setting up the LSST Science Pipelines environment. The initial weeks were a deep dive into the complexities of the Butler, its data repositories, and the unique coordinate systems used by the Rubin Observatory. There were moments of frustration, especially when dealing with environment configurations and performance bottlenecks, but each challenge was a significant learning opportunity. From the excitement of the first successful end-to-end pipeline run to the satisfaction of optimising the workflow and seeing the processing time drop from hours to minutes, this project has been a masterclass in building real-world scientific software. The result of this effort is RIPPLe: a robust, configurable pipel...
Read more

RIPPLe: Building a Bridge Between LSST and DeepLense

It's hard to believe seven weeks have flown by. In that time, I've consumed countless cups of green tea, and developed a single obsession: getting petabytes of astronomical data ready for deep learning. When I started this Google Summer of Code project, the mission seemed straightforward enough. I was tasked with building a pipeline to feed data from the Legacy Survey of Space and Time (LSST) into machine learning models for the DeepLense project. The Vera C. Rubin Observatory, which will conduct the LSST, is a firehose of cosmic data, set to produce 20 terabytes every single night. Buried in that data, we expect to find around 100,000 new gravitational lenses—a massive jump from the few hundred we know of today. Each one is a cosmic magnifying glass that can help us understand the mysteries of dark matter. But first, you have to find them. That’s where my project, RIPPLe, comes in. Phase 0: The Foundation  I look back at who I was in February, happily working my way through An...
Read more

Damn couldn't believe I got in again.

I was glued to reddit and the website the whole time for 4 hours. The results came out later than UTC 18:00 and didn't get the mail until four hours beyond the scheduled time.  None of my friends got in, so was kinda anticipating a rejection too, but to my surprise I got in. I am really excited to go through this journey again. It would be amazing. 
Read more