Bokeh slider for “Phenology of two interdependent traits in migratory birds in response to climate change”

(7 January 2017: some updates, see end of the post.)

A while I ago I made a slider using ipywidgets that could be embedded in a html page (handy for blog posts). This week I decided to see where things were at with IPython or Jupyter.

As of July 2015 the ipywidgets package is unsupported. The author recommends using IPython’s built-in interactive tools. However IPython doesn’t have static widgets yet, according to this issue. A StackOverflow answer mentioned Bokeh so I decided to give that a go.

Bokeh slider

Here is a slider that replicates my earlier ipywidgets effort:

https://carlo-hamalainen.net/bokehslider/bokehslider2017

This is pretty nice. It’s an interactive slider, works on desktop and mobile, and doesn’t have any of the notebook stuff around it. Just the graph with the interactive widget. Bokeh also provides tools for zooming and panning around. It’s also worth mentioning that Bokeh provides a GUI library (things like hboxes, vboxes, layouts, etc) and my impression is that you could have multiple plots changing based on one slider, two plots tied together on some parameter, or whatever else you dreamt up.

The slider is implemented in bokehslider.py and is run using bokeh-server –ip 0.0.0.0 –script bokehslider.py. One strange thing that I ran into was that the slider wasn’t interactive unless I opened up port 5006 on my server, even though Nginx is doing the proxy_pass stuff. I suspect that some of the Bokeh-generated Javascript expects to be able to connect to the host on 5006.

Here’s the relevant Nginx config settings:

server {

    # listen, root, other top level config...

    # Reverse proxy for Bokeh server running on port 5006:

    location /bokeh {
        proxy_pass http://104.200.25.78:5006/bokeh;
    }

    location /static {
        proxy_pass http://104.200.25.78:5006;
    }

    location /bokehjs {
        proxy_pass http://104.200.25.78:5006;
    }

    # rest of the config...

}

In terms of coding, the Bokeh model is a bit different to the usual plotting procedure in that you set up
data sources, e.g.

obj.line_source  = ColumnDataSource(data=dict(
                                            x_cV=[],
                                            arrival_date=[],
                                            laying_date=[],
                                            hatching_date=[],))

and then plot commands use that data source. You don’t pass NumPy arrays in directly:

plot = figure(plot_height=400, plot_width=400,
              tools=toolset, x_range=[130, 180], y_range=[110, 180])

plot.line('x_cV', 'x_cV',          source=obj.line_source, line_width=4, color='black')
plot.line('x_cV', 'arrival_date',  source=obj.line_source, line_width=4, color='purple', legend='Arrival time')
plot.line('x_cV', 'laying_date',   source=obj.line_source, line_width=4, color='red',    legend='Laying time')
plot.line('x_cV', 'hatching_date', source=obj.line_source, line_width=4, color='green',  legend='Hatching date')

Then, the input_change method calls my update_data method which actually updates the data sources. It doesn’t have to explicitly make a call to redraw the plot.

def update_data(self):
    u_q = self.u_q_slider.value

    self.line_source.data  = get_line_data_for_bokeh(float(u_q))

Links

https://github.com/carlohamalainen/phenology-two-trait-migratory-bird/tree/bokeh-slider

http://bokeh.pydata.org/en/latest/docs/server_gallery/sliders_server.html

https://www.reddit.com/r/IPython/comments/3bgg7t/ipython_widgets_in_a_static_html_file

https://github.com/ipython/ipywidgets/issues/16

http://stackoverflow.com/questions/22739592/how-to-embed-an-interactive-matplotlib-plot-in-a-webpage

https://jakevdp.github.io/blog/2013/12/05/static-interactive-widgets

Update 7 January 2016

The slider described above doesn’t work with the latest version of Bokeh, so here’s one that does:

bokehslider2017.py

Launching is slightly different (see this script):

bokeh serve bokehslider2017.py --host=carlo-hamalainen.net:443 --prefix=bokehslider 
                               --use-xheaders --port=5100 --allow-websocket-origin=carlo-hamalainen.net

Happily for me this now plays nicely with my https nginx config:

    location /bokehslider {
            proxy_pass http://127.0.0.1:5100;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_http_version 1.1;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host:$server_port;
            proxy_buffering off;
        }

And there is no need to open up port 5006 as I said before.

Classy mtl

This post has a minimal stand-alone example of the classy lenses and prisms from George Wilson’s talk about mtl. The source code for George’s talk is here: https://github.com/gwils/next-level-mtl-with-classy-optics.

Literate Haskell source for this post is here: https://github.com/carlohamalainen/playground/tree/master/haskell/classy-mtl.

First, some imports:

> {-# LANGUAGE OverloadedStrings    #-}
> {-# LANGUAGE TemplateHaskell      #-}
>
> module Classy where
>
> import Control.Lens
> import Control.Monad.Except
> import Control.Monad.Reader
> import Data.Text

Toy program – uses the network and a database

The case study in George’s talk was a program that has to interact with a database and the network. We have a type for the database connection info:

> type DbConnection = Text
> type DbSchema     = Text
>
> data DbConfig = DbConfig
>     { _dbConn :: DbConnection
>     , _schema :: DbSchema
>     }

For the network we have a port and some kind of SSL setting:

> type Port = Integer
> type Ssl  = Text
>
> data NetworkConfig = NetworkConfig
>     { _port     :: Port
>     , _ssl      :: Ssl
>     }

At the top level, our application has a database and a network configuration:

> data AppConfig = AppConfig
>     { _appDbConfig   :: DbConfig
>     , _appNetConfig  :: NetworkConfig
>     }

Types for errors that we see when dealing with the database and the network:

> data DbError = QueryError Text | InvalidConnection
>
> data NetworkError = Timeout Int | ServerOnFire
>
> data AppError = AppDbError  { dbError  :: DbError      }
>               | AppNetError { netError :: NetworkError }

Classy lenses and prisms

Use Template Haskell to make all of the classy lenses and prisms. Documentation for makeClassy and makeClassyPrisms is in Control.Lens.TH.

> makeClassy ''DbConfig
> makeClassy ''NetworkConfig
> makeClassy ''AppConfig
>
> makeClassyPrisms ''DbError
> makeClassyPrisms ''NetworkError
> makeClassyPrisms ''AppError

We get the following typeclasses:

  • HasDbConfig
  • HasNetworkConfig
  • HasAppConfig
  • AsNetworkError
  • AsDbError
  • AsAppError

For example, here is the generated class HasDbConfig:

*Classy> :i HasDbConfig
class HasDbConfig c_a6IY where
  dbConfig :: Functor f => (DbConfig -> f DbConfig) -> c0 -> f c0
  dbConn   :: Functor f => (DbConnection -> f DbConnection) -> c0 -> f c0
  schema   :: Functor f => (DbSchema -> f DbSchema) -> c0 -> f c0
instance HasDbConfig DbConfig -- Defined at Classy.lhs:58:3

If we write HasDbConfig r in the class constraints of a type signature then we can use the lenses dbConfig, dbConn, and schema to get the entire config, connection string, and schema, from something of type r.

In contrast, the constraint AsNetworkError r means that we can use the prisms _NetworkError, _Timeout, and _ServerOnFire on a value of type r to get at the network error details.

*Classy> :i AsNetworkError
class AsNetworkError r_a759 where
  _NetworkError ::
    (Choice p, Control.Applicative.Applicative f) =>
    p NetworkError (f NetworkError) -> p r0 (f r0)

  _Timeout ::
    (Choice p, Control.Applicative.Applicative f) =>
    p Int (f Int) -> p r0 (f r0)

  _ServerOnFire ::
    (Choice p, Control.Applicative.Applicative f) =>
    p () (f ()) -> p r0 (f r0)
    -- Defined at Classy.lhs:63:3

instance AsNetworkError NetworkError -- Defined at Classy.lhs:63:3

Using the class constraints

The first function is loadFromDb which uses a reader environment for database configuration, can throw a database error, and do IO actions.

> loadFromDb :: ( MonadError e m,
>                 MonadReader r m,
>                 AsDbError e,
>                 HasDbConfig r,
>                 MonadIO m) => m Text
> loadFromDb = do
>
>   -- Due to "MonadReader r m" and "HasDbConfig r"
>   -- we can ask for the database config:
>   rdr    let dbconf  = rdr ^. dbConfig :: DbConfig
>
>   -- We can ask for the connection string directly:
>   let connstr  = rdr ^. dbConn :: DbConnection
>
>   -- We have "AsDbError e", so we can throw a DB error:
>   throwError $ (_InvalidConnection #) ()
>   throwError $ (_QueryError #) "Bad SQL!"
>
>   return "foo"

Another function, sendOverNet uses a reader environment with a network config, throws network errors, and does IO actions.

> sendOverNet :: ( MonadError e m,
>                  MonadReader r m,
>                  AsNetworkError e,
>                  AsAppError e,
>                  HasNetworkConfig r,
>                  MonadIO m) => Text -> m ()
> sendOverNet mydata = do
>
>   -- We have "MonadReader r m" and "HasNetworkConfig r"
>   -- so we can ask about the network config:
>   rdr    let netconf = rdr ^. networkConfig  :: NetworkConfig
>       p       = rdr ^. port           :: Port
>       s       = rdr ^. ssl            :: Ssl
>
>   liftIO $ putStrLn $ "Pretending to connect to the network..."
>
>   -- We have "AsNetworkError e" so we can throw a network error:
>   throwError $ (_NetworkError #) (Timeout 100)
>
>   -- We have "AsAppError e" so we can throw an application-level error:
>   throwError $ (_AppNetError #) (Timeout 100)
>
>   return ()

If we load from the database and also send over the network then we get extra class constraints:

> loadAndSend :: ( AsAppError e,
>                  AsNetworkError e,
>                  AsDbError e,
>                  HasNetworkConfig r,
>                  HasDbConfig r,
>                  MonadReader r m,
>                  MonadError e m,
>                  MonadIO m) => m ()
> loadAndSend = do
>   liftIO $ putStrLn "Loading from the database..."
>   t 
>   liftIO $ putStrLn "Sending to the network..."
>   sendOverNet t

Things that won’t compile

We can’t throw the database error InvalidConnection without the right class constraint:

> nope1 :: (MonadError e m, AsNetworkError e) => m ()
> nope1 = throwError $ (_InvalidConnection #) ()

Could not deduce (AsDbError e)
arising from a use of ‘_InvalidConnection’

We can’t throw an application error if we are only allowed to throw network errors, even though this specific application error is a network error:

> nope2 :: (MonadError e m, AsNetworkError e) => m ()
> nope2 = throwError $ (_AppNetError #) (Timeout 100)

Could not deduce (AsAppError e)
arising from a use of ‘_AppNetError’

We can’t get the network config from a value of type r if we only have the constraint about having the database config:

> nope3 :: (MonadReader r m, HasDbConfig r) => m ()
> nope3 = do
>   rdr    let netconf = rdr ^. networkConfig
>
>   return ()

Could not deduce (HasNetworkConfig r)
arising from a use of ‘networkConfig’

What is the #?

The # is an infix alias for review. More details are in Control.Lens.Review.

*Classy> :t review _InvalidConnection ()
review _InvalidConnection () :: AsDbError e => e

*Classy> :t throwError $ review _InvalidConnection ()
throwError $ review _InvalidConnection () :: (AsDbError e, MonadError e m) => m a

What is the monad transformer stack?

We didn’t specify it! The functions loadFromDb and sendOverNet have the general monad m in their type signatures, not a specific transformer stack like ReaderT AppConfig (ExceptT AppError IO) a.

What else?

Ben Kolera did a talk at BFPG about stacking monad transformers. He later modified the code from his talk to use the classy lens/prism approach. You can see the code before and after, and also see a diff. As far as I could see there is one spot in the code where an error is thrown, which motivated me to create the stand-alone example in this post with the body for loadFromDb and sendOverNet sketched out.