python

All posts tagged python

UPDATE (2018 Jun 6) – I’m trying to learn how to use Sphinx to create documentation from source-code docstrings. The Sphinx manual is impressively opaque. Some googling turned up this document, which seems to provide useful details.

UPDATE (2018 Jun 4) – I’ve found some new resources that are much more up-to-date:

To set up my package so it can be installed via pip, I’m following the somewhat outdated tutorial here – https://python-packaging.readthedocs.io/en/latest/index.html.

I’ll make more notes and update this post as I go along.

As the second entry in my series on learning how to write scientific software, I’m going to describe choosing and configuring my integrated development environment or IDE. This is the program I’ll use to write and edit the source code for my project. It’s more-or-less a fancy text editor.

My text editor of choice is vim or vi improved. It’s highly customizable, powerful, and there’s lots of online help to use it. However, it has a very steep learning curve and the commands, while fast to type out, can be very cryptic. In fact, vim is famously difficult to exit.

In any case, I started using vim back when “Pirates of the Caribbean” movies were still good, so while I’m no vim guru, I feel pretty comfortable at least exiting the program.

Opening a terminal window in Mac, I fire up vim just by typing “vim” or “vi”. That opens an editor window. vim can automagically interpret and color source code, as follows:

A screenshot from vim showing syntax-colored source code.

To turn on syntax coloring, go to your home directory (type “cd” in the terminal window), and edit the vim configuration file .vimrc by typing “vi .vimrc”. That should open an editor window.

Then, in vim, type the letter “i” (that starts “insert” or edit mode, allowing you to enter text into the file) and type “syntax on” <ENTER> “filetype indent plugin on”, giving a file that looks like this:

Press the escape key (which exits insert mode and enters command mode). Save the changes by typing “:wq”. That should save and exit.

Unlike many other languages, Python considers whitespace in its interpretation, and the Python style guide recommends using four spaces for each level of indentation. It would be nice to have the tab key implement that spacing in vim. Ynfortunately, my vim by default inserts eight spaces for each press of the tab key.

But you can modify that behavior by adding file-type plugin files to a special vim configuration folder. I followed the instructions here to create an ftplugin directory inside the .vim directory (by typing “cd && cd .vim && mkdir ftplugin && cd ftplugin” in the terminal window). Then, inside the directory, I created python.vim (“vi python.vim”) and again pressed “i” to enter insert mode.

I typed the following lines into the file:

Then pressed the escape key and typed “:wq”.

Next, I tested the new configuration by typing “vi test.py” (the “.py” is important because that’s how vim knows you are editing a python file and want to use the new python configuration). I pressed tab and got four spaces instead of eight.

I’m sure there are other configuration settings that would be useful, but this’ll do for now.

UPDATE – 2018 May 17: I found this excellent website – http://docs.python-guide.org/en/latest/, which addresses vim set-up, as well as a number of other issues.

Ancient software developers meticulously punched holes in paper cards to write programs.

In the eons before my graduate career, scientists rarely, if ever, publicly distributed their codes, with authors zealously guarding their coding projects.

But just as I was finishing my PhD, it was becoming common for scientists to make the code they developed as part of a published project readily available on the internet. However, the methods to post code online (at least those I knew about) were pretty clunky.

Nowadays, the infrastructure for posting and sharing code online is robust, mature, and relatively easy to use. Consequently, scientists are creating beautiful code repositories, along with accessible documentation.

Open-sourcing code is becoming ever more important: as codes become more complex and capable, readily available codes with good documentation are critical to support reproducibility, a cornerstone of the scientific process. Moreover, federal funding agencies are starting to require investigators to make their code and data products public.

Unfortunately, since I was one of the last generation of grad students before these repositories were common, I never really learned how to distribute and document code properly.

So as part of an ongoing effort to improve my science output (and as an aide to my future students), I’m going to begin a series of semi-regular blog posts describing my process of learning how to write, document, and post scientific code.

A few caveats upfront:

  • I intend to mostly (probably exclusively) write the code in python, which has become (at least in astronomy) the language of choice, so not all of what I write will be generally relevant.
  • I was ushered into the Cult of Mac many years ago, so not all of what I write will be relevant for other OS’s. Here again, though, I’ve found anecdotally that most astronomers use Mac.
  • This blog series is in no way intended to be comprehensive or rigorous. I’m just planning to describe what I learn as I go along, and what time I can devote will almost definitely not suffice to explain all the details, nuances, or technical aspects that intersect the project.

As to the actual science code I intend to write, several years ago my colleagues and I wrote a paper about ellipsoidal variations induced by massive exoplanets orbiting very close to their host stars. The accompanying code, EVIL-MC was written in IDL, an older language still widely used in astronomy but proprietary and requiring the purchase of an expensive site license.

My plan is to convert that IDL code into a Python package over the next several weeks.

EVIL-MC – Ellipsoidal Variations Induced by a Low-Mass Companion


Tidal distortion (exaggerated) of a star (orange-yellow disk) orbited by planet (white/black disk). The plot below shows the brightness variation of the star due to the tidal distortion.

I’m prepping for my classical mechanics course, scheduled to start next week. One of the first things we discuss is uniform circular motion and how it looks projected along the x- and y-axes, so I thought it would be useful to have an animation showing that. I found a few animations online, but none really showed the x and y projections I was looking for.

So I decided to create my own using my go-to language of choice Python. Fortunately, python guru Jake Vanderplas has created a very nice animation module usable in iPython Notebooks.

Based on his example here, I put together the following code to generate the desired animation:

From http://en.wikipedia.org/wiki/Reese%27s_Pieces#/media/File:Reeses-pieces-loose.JPG.

From http://en.wikipedia.org/wiki/Reese%27s_Pieces#/media/File:Reeses-pieces-loose.JPG.

I eat Reese’s pieces almost every day after lunch, and they come in three colors: orange, yellow, and brown.

I’ve wondered for a while whether the three colors occur in equal proportions, so for lunch today, I thought I’d try to infer the occurrence rates using Bayes’ Theorem.

Bayes’ Theorem provides a quantitative way to update your estimate of the probability for some event, given some new information. In math, the theorem looks like

<code>P\left( H | E \right) = \dfrac{ P\left( E | H \right) P\left( H \right)}{P\left( E \right)},</code>

The probability for event H to happen, given that some condition E is met, is the probability that E is met, given that H happened, times the probability for H to happen at all, and divided by the probability for E to be met at all.

The P(H) and P(E) are called the “priors” and often represent your initial estimates for the probability that H and E occur. P\left(E | H \right) is called the “likelihood”, and P(H | E) is the “posterior”, the thing we know AFTER E is satisfied. P(H | E) is usually the thing we’re trying to calculate.

Big bag

Thanks, Winco buy-in-bulk!

So for my case, P(H) will be the frequency with which a certain color occurs, and E will be my experimental data.

For a given frequency f_{\rm orange} of oranges (or browns or yellows), the probability P(f_{\rm orange} | E) that I draw N_{\rm orange} oranges is  ~ f^N (1 –  f)^N(not orange). As I select more and more candies, I can keep re-evaluating P for the whole allowed range of f (0 to 1) and find the value that maximizes P.

Closing my eyes, I pulled ten different candies out of the bag, with following results in sequence: brown, orange, orange, yellow, orange, orange, orange, brown, orange, yellow, orange. These results obviously suggest orange has a higher frequency than yellow or brown.

This ipython notebook implements the calculation I described, and the plots below show how P changes after a certain number of trials n_{\rm trials}:

Applying Bayesian inference to determine the frequency of Reese's pieces colors.

Applying Bayesian inference to determine the frequency of Reese’s pieces colors.

So, for example, before I did any trials n_{\rm trials} = 0, I assumed all colors were equally likely. After the first trial when I chose a brown candy, the probability that brown has a higher frequency than the other colors goes up. After three trials (brown, orange, orange), orange takes the lead, and since I hadn’t seen any yellows, there’s a non-zero probability that yellow’s frequency is actually zero. We can see how the probabilities settle down after ten trials.

Based on this admittedly simple experiment, it seems that oranges have a frequency about twice that of yellows and browns. Although not as much fun, if I’d bothered to check wikipedia, I would have seen that “The goal color distribution is 50% orange, 25% brown, and 25% yellow” — totally consistent with my estimate.

A preliminary investigation showed a project a colleague and I were considering probably isn’t worth doing. But for that investigation, I took a few hours to make a rather complicated plot using pylab, so I thought I’d share how I did that.

First, here’s the plot:

Tidal decay timescales for members of multi-planet systems.

Tidal decay timescales for members of multi-planet systems.

The plot shows the timescales for tidal decay of members of multi-planet systems. Unfortunately, the x-axis labels aren’t legible unless you zoom in, but if you do, you can see the font colors match up with the corresponding line colors.

Below is the ipython notebook I used to generate the plot. The excel spreadsheet with the data is here.

<span class="c">#Show the plot inline</span>
<span class="o">%</span><span class="k">matplotlib</span> <span class="n">inline</span>

<span class="c">#load in the required modules</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="kn">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">pylab</span> <span class="kn">as</span> <span class="nn">pl</span>
<span class="kn">import</span> <span class="nn">itertools</span> <span class="kn">as</span> <span class="nn">it</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>

<span class="c"># using the ExcelFile class</span>
<span class="n">xls</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">ExcelFile</span><span class="p">(</span><span class="s">'exoplanet-archive_2015Mar25.xlsx'</span><span class="p">)</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">xls</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s">'obj of interest'</span><span class="p">,</span> <span class="n">index_col</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="n">pd</span><span class="o">.</span><span class="n">notnull</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'a/(da/dt)Qs=1e6 (Gyrs)'</span><span class="p">])]</span>

<span class="c">#Make a nice, big figure</span>
<span class="n">fig</span> <span class="o">=</span> <span class="n">pl</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">15</span><span class="p">,</span><span class="mi">15</span><span class="p">))</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">add_subplot</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>

<span class="c">#Make a list, indexing the dataframe labels</span>
<span class="n">indices</span> <span class="o">=</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">index</span><span class="p">)))</span>

<span class="c">#Make a list with the indices as the entries</span>
<span class="n">labels</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">index</span><span class="p">))</span>
<span class="c">#For concision, drop "Kepler" wherever it's found</span>
<span class="n">labels</span> <span class="o">=</span> <span class="p">[</span><span class="n">w</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s">'Kepler-'</span><span class="p">,</span> <span class="s">''</span><span class="p">)</span> <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="n">labels</span><span class="p">]</span>

<span class="c">#Make a cycle of line and text colors, blue, green, red, etc.</span>
<span class="n">colors</span> <span class="o">=</span> <span class="n">it</span><span class="o">.</span><span class="n">cycle</span><span class="p">([</span><span class="s">'b'</span><span class="p">,</span> <span class="s">'g'</span><span class="p">,</span> <span class="s">'r'</span><span class="p">,</span> <span class="s">'c'</span><span class="p">,</span> <span class="s">'m'</span><span class="p">,</span> <span class="s">'y'</span><span class="p">,</span> <span class="s">'k'</span><span class="p">])</span>

<span class="c">#Since each member of the multi-system should be plotted with the same x-value,</span>
<span class="c">#  I need to generate a new list of all the same value with as many entries</span>
<span class="c">#  as members. That's what "i" is for.</span>
<span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">unq</span> <span class="ow">in</span> <span class="nb">set</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">index</span><span class="p">):</span>

    <span class="c">#Retrieve the decay timescales calculated in the spreadsheet</span>
    <span class="n">taus</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="n">unq</span><span class="p">,</span> <span class="s">'a/(da/dt)Qs=1e6 (Gyrs)'</span><span class="p">]</span>

    <span class="c">#Generate the list of all the same x-value</span>
    <span class="n">idx</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ones_like</span><span class="p">(</span><span class="n">taus</span><span class="p">)</span><span class="o">*</span><span class="n">i</span>

    <span class="c">#Make the scatter plot points with the current color</span>
    <span class="n">ax</span><span class="o">.</span><span class="n">semilogy</span><span class="p">(</span><span class="n">idx</span><span class="p">,</span> <span class="n">taus</span><span class="p">,</span> <span class="n">marker</span><span class="o">=</span><span class="s">'o'</span><span class="p">,</span> <span class="n">color</span><span class="o">=</span><span class="n">cur_color</span><span class="p">)</span>

    
    <span class="c">#Get the next line color</span>
    <span class="n">cur_color</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span>
    
    <span class="c">#Next x-value</span>
    <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
    
<span class="c">#Give a little space to the left and right of the first and last x-values</span>
<span class="n">pl</span><span class="o">.</span><span class="n">xlim</span><span class="p">([</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">index</span><span class="p">))</span><span class="o">+</span><span class="mi">1</span><span class="p">])</span>

<span class="c">#Switch out the x-values with the system names</span>
<span class="n">pl</span><span class="o">.</span><span class="n">xticks</span><span class="p">(</span><span class="n">indices</span><span class="p">,</span> <span class="n">labels</span><span class="p">,</span> <span class="n">rotation</span><span class="o">=</span><span class="s">'vertical'</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="s">'small'</span><span class="p">,</span> <span class="n">ha</span><span class="o">=</span><span class="s">'center'</span><span class="p">)</span>

<span class="c">#Increase the size of the y-axis label font</span>
<span class="n">pl</span><span class="o">.</span><span class="n">yticks</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="mi">36</span><span class="p">)</span>
<span class="c">#Label the y-axis</span>
<span class="n">pl</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s">'$a/</span><span class="se">\\</span><span class="s">left(</span><span class="se">\\</span><span class="s">frac{da}{dt}</span><span class="se">\\</span><span class="s">right)_{Q_{</span><span class="se">\\</span><span class="s">rm s} = 10^6}$ (Gyrs)'</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">36</span><span class="p">)</span>

<span class="c">#Reset the colors cycle</span>
<span class="n">colors</span> <span class="o">=</span> <span class="n">it</span><span class="o">.</span><span class="n">cycle</span><span class="p">([</span><span class="s">'b'</span><span class="p">,</span> <span class="s">'g'</span><span class="p">,</span> <span class="s">'r'</span><span class="p">,</span> <span class="s">'c'</span><span class="p">,</span> <span class="s">'m'</span><span class="p">,</span> <span class="s">'y'</span><span class="p">,</span> <span class="s">'k'</span><span class="p">])</span>

<span class="c">#Set a new color for each x-axis label</span>
<span class="k">for</span> <span class="n">tick</span> <span class="ow">in</span> <span class="n">ax</span><span class="o">.</span><span class="n">xaxis</span><span class="o">.</span><span class="n">get_ticklabels</span><span class="p">():</span>
    <span class="n">tick</span><span class="o">.</span><span class="n">set_color</span><span class="p">(</span><span class="n">cur_color</span><span class="p">)</span>
    <span class="n">cur_color</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span>

<span class="n">pl</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="s">'Comparing multi-planet system a_dadt.png'</span><span class="p">,</span> <span class="n">bbox_inches</span><span class="o">=</span><span class="s">'tight'</span><span class="p">,</span> <span class="n">orientation</span><span class="o">=</span><span class="s">'landscape'</span><span class="p">,</span> <span class="n">dpi</span><span class="o">=</span><span class="mi">250</span><span class="p">)</span>

 

 

Artist's conception of Pluto's and Charon's surfaces. From http://www.ourpluto.org/home.

Artist’s conception of Pluto’s and Charon’s surfaces. From http://www.ourpluto.org/home.

We talked briefly about several things at Friday’s Journal Club. First, we discussed astrobites.org, a great blog that covers the interesting nitty-gritty of astronomy research. I pointed out that they are requesting submissions from undergrad researchers.

Second, we discussed the New Horizons mission’s request for suggestions for names of features on the surface of Pluto and its moons. After the mission flies by the system, there will be mounds of high resolution images, probably showing a variety of complex surface morphologies. And all that stuff is going to need names.

Third, Jacob presented a recent paper that extends the Titius-Bode relation to extrasolar systems and predict there are about 2 planets in habitable zones per star in our galaxy. A potentially fascinating result, but unfortunately, the T-B relation is probably just an interesting coincidence for our solar system — it has no theoretical basis, and so there’s no reason to believe it can be generalized to other planetary systems. Nevertheless, the article got a lot of press last week.

Finally, we talked about coding in astronomy, and I wanted to post this resource I just heard about, https://python4astronomers.github.io/. Looks to have a lot of helpful tutorials relevant to astronomy.

Friday’s attendees included Jennifer Briggs, Trent Garrett, Nathan Grigsby, Tanier Jaramillo, Emily Jensen, Liz Kandziolka, and Jacob Sabin.