I was causing too much noise. Well, technically it was my laptop fan. But Cadair covered for me. Again.
So, getting to my work on a serious note ( don’t judge a blog by its pictures ), work on Adding Fido entries to the Database continued from the previous fortnight.
Now there is a add_from_fido_search_result() function which enables users to add entries returned from any Fido search result to the database. Currently only JSOC results cannot be added to the database.
Aaaaand as usual, finding a small query for the test functions was frustrating. So I didn’t find one. Used a big one 😀
No, seriously it IS difficult to find a query with a small number of results.
I know I know I should have come up with a better heading. Anyways, let’s begin.
So, here it is. Oooooh, the serious part. The coding period. Again, I dunno what to do.
Unidown had issues ( lots to be fair ) which was not allowing ( honestly, I couldn’t find a better word ) me to work on the first part of my project.
Meanwhile SunPy had started work on their new SunPy Example Gallery which basically contains examples for people to see what SunPy’s different modules can do.
So me and Cadair had a meet and it was decided that I should make examples for the Database Module of SunPy. This had to be short but yet should give a good idea to the people on what the Database Module is capable of. If used on a regular basis, it will allow users to :
manage files easily
use common storage so that a group of people can share the same data
save time and bandwidth as there is no need to download the same data twice
It did take some time to prepare and organize what’ll go where as I had to travel during this time, but I did complete it ( after Sphinx nightmares ). 😀
Aaaand again a meeting with Cadair. This time unidown was in a much (much) better state and we decided that I should start implementing the First Part of my proposal and we’ll tackle any issues as they come. And so work on adding Fido search results to the database began….
This work continues into the Second Half of the Pre Mid Evaluation Coding Period.
Now that the Google Summer of Code 2016 (yes, I wrote that whole thing instead of GSoC ’16) has officially started, I don’t know what to do.
😐 I really don’t. 😐
Well, I had a lot of plans (Really? I wrote those things? Wow…) for this period in my proposal but they just seem trivial now, reading code and all that stuff. Thanks to my work before the proposal submission I have a good idea on exactly what to do and how to do it.
So most of the Community Bonding period went by and I did nothing. Literally, nothing.
So, coming back to topic ( not really 🙂 ), there were approximately 10 days left and I decided to do code up some new feature and after a meeting with Cadair we decided that I should try to implement a VSO ( go here ) caching mechanism which would save bandwidth and hence save time ( a lot of time ).
The problem was that currently the SunPy database module had a cache which did not download files only if any query was exactly identical to a previous query ( for which files were downloaded successfully ). So if there were 2 queries that were slightly different but had a large overlap, all files would be downloaded again for the second query.
So I did manage to accomplish that feature in the end (easy to say). As you may have guessed by now (duh!), an additional advantage was that users can now download files for a large query and then can peacefully 😀 work on any subset of that query. Yes, highly productive work.
After waiting (very) anxiously for the results (and refreshing the page hundreds of times to see, if, by any chance the results were out before 1900 UTC), officially I am in !!!
Now I will be working as a student developer under GSoC for SunPy. I would like to thank Cadair, DavidPSand especially derdon for helping me get started, being (really really) patient throughout my doubts and screw-ups and giving feedback on by proposal.
I plan to start work early, preferably right after my exams end so that I can get sufficient time to complete Part 1 of the project. Looking forward to working with SunPy !!!
P.S. thank you again to derdon, who has been quite elusive recently 😀
I found the “Improvements to the SunPy database” project interesting among the ideas present in SunPy’s wiki page. This project appealed to me the most because it would integrate Fido into SunPy’s database, which in turn would really open up the SunPy’s database module to a world of possibilities.
This would enable researchers, who currently use separate queries for VSO, HEK etc. to use a common Fido client for all queries. So now, just put in whatever query you want, Fido will decide which client will best serve the query and for that client, all database functionalities, like add, download etc. will automatically be taken care of.
The second part of the project will improve the caching mechanism currently in use by the database. Currently, the caching mechanism cannot efficiently handle overlapping queries. This is because if two queries are slightly different, i.e. they have a lot of files in common, the cache will download all files again. The suggested improvement is to use a query result based caching mechanism rather than a query based mechanism that will check if the files exist for each query result. This will be very efficient in handling overlapping queries.
I have been in contact with derdon, who made the database module, for about a month now, and I now have a good understanding of how to implement both parts of this project. Looking forward to working on this!!!
I first contacted SunPy through their IRC channel because I found one of their ideas interesting. Being a newbie to the world of Open Source, I was at first intimidated by the “Intermediate” difficulty tag of the project. So, I started by going through the issues page and went through all the Database issues. I managed to set up the development environment quickly and got started on solving some of the listed issues. Now that I’ve spent time getting acquainted with the database module, it’s become easy for me to quickly solve issues and I’m looking forward to working on this project.