Category: Uncategorized

Solar home electricity dataset by Ausgrid

I've started exploring an open dataset that should be very useful for research on residential solar energy (for example studying energy self-consumption with a PV-battery system).

My Python code to manipulate this dataset and some preliminary studies (simple statistics) is freely available on Github: https://github.com/pierre-haessig/ausgrid-solar-data. This data exploration code is extensively using pandas (for slicing, grouping, resampling, etc.).

About the dataset

The “Solar home electricity dataset” is made available by Ausgrid, an Australian electric utility which operates the distribution grid in Sidney and nearby areas.

This dataset contains a pretty rich record:

  • electricity consumption and the PV production
  • of 300 customers (in Sidney and its area),
  • over three years (July 2010 to June 2013),
  • with a 30 minutes timestep.

Here is a small extract of the PV production for 3 randomly chosen customers, over 3 days in July 2011:

https://raw.githubusercontent.com/pierre-haessig/ausgrid-solar-data/master/PV%20production%202011-07%2001-03.png

Many more plots and statistics are given in the Jupyter notebooks available on the Github repository.

Locating postcodes (geocoding)

Also, the exploration of this dataset was an occasion to discover the Google Maps Geocoding API. Indeed, the location of each anonymous customer is given by a postcode only. To enable quantitative study of the spatiotemporal pattern in PV production, I've tried to locate this postcodes. The Python code for this is in the Postcodes location.ipynb Notebook. Map plotting is done with cartopy.

Extracted from this notebook, here is an overview of the locations of the postcodes present in the dataset (in Australia, NSW). Red rectangles are the boundaries of each postcode, as returned by Google maps (small in urban areas along the coast, gigantic otherwise).

2019-11-11 update: fixing broken link to dataset webpage on Ausgrid website.

Installing JModelica on Ubuntu 16.04

This a copy of  the post I've just placed  JModelica forum: http://jmodelica.org/27775

I just wanted to share my experience building JModelica from source (trunk version) on Ubuntu 16.04. After several attempts, I think I've a quite streamlined procedure. Both commands "make install" and "make casadi_interface" are successful. In the end, I was also able to successfully run "make test" (although it took 58 min on the virtual machine).

For most aspects, I've followed the user guide, but using updated versions of the dependencies (see after), in particular using Java 8 instead of 6 (not available anymore).

My main departure from user guide is that I used Ipopt from Ubuntu instead of compiling from source, as recommended. This works well enough (functionally speaking, since I've no ideas about the performance). In the end, I think this approach is simpler/quicker (especially since it also saves from recompiling blas). However, I had to manually patch one Ipopt header for the compilation to work (but then this modification may not be needed when just using the JModelica binary). See below.

It would be also interesting to try with Ipopt compiled from source, but for the moment I've no incentive to replace something that I just got to work!

Dependency versions

Here are the list of versions of the main dependencies (some packages, like g++, come already pre-installed with a fresh Ubuntu install):

g++: 5.4.0
subversion: 1.9.3
gfortran: 5.4.0
cmake: 3.5.1
swig: 3.0.8
ant: 1.9.6
openjdk-8-jdk (8u91) , instead of openjdk-6
python-dev: 2.7.11
python-numpy: 1.11
python-scipy: 0.17
python-matplotlib: 1.5.1
cython: 0.23.4
python-lxml: 3.5.0
python-nose: 1.3.7
python-jpype: 0.5.4.2 (alternative: there is a fork JPype1, on PyP, which seems more up to dateI. not tested)
zlib1g-dev: 1.2.8
libboost-dev: 1.58
jcc: 2.21 (from Ubuntu rather than from PyPI, as suggested in the user guide, since both versions are the same)

extra : python-pip : 8.1.1

For ipython, I'm NOT using 2.4.1 from Ubuntu, but rather 5.1 from PyPI (pip install)

Also, for using Ipopt from Ubuntu, ipopt + some extra headers are needed:

coinor-libipopt1v5: 3.11.9
coinor-libipopt-dev: 3.11.9
libblas-dev: 3.6.0
liblapack-dev: 3.6.0

Patching Ubuntu's Ipopt header

As I said, I had to patch one Ipopt header. The starting point is this compilation error in the "make install" before the patch:

libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../../JMI/src -I../.. -Wall -I/usr/include/coin -DJMI_AD=JMI_AD_NONE -g -I/home/modelica/JModelica_trunk/build/sundials_install/include -fPIC -g -O2 -MT libjmi_solver_la-jmi_opt_coll_ipopt.lo -MD -MP -MF .deps/libjmi_solver_la-jmi_opt_coll_ipopt.Tpo -c ../../../JMI/src/jmi_opt_coll_ipopt.cpp  -fPIC -DPIC -o .libs/libjmi_solver_la-jmi_opt_coll_ipopt.o
In file included from /usr/include/coin/IpJournalist.hpp:15:0,
                 from /usr/include/coin/IpIpoptApplication.hpp:26,
                 from ../../../JMI/src/jmi_opt_coll_ipopt.cpp:21:
/usr/include/coin/IpSmartPtr.hpp:18:4: error: #error "don't have header file for stddef"
 #  error "don't have header file for stddef"
    ^
Makefile:1240: recipe for target 'libjmi_solver_la-jmi_opt_coll_ipopt.lo' failed
make[1]: *** [libjmi_solver_la-jmi_opt_coll_ipopt.lo] Error 1
make[1]: Leaving directory '/home/modelica/JModelica_trunk/build/JMI/src'
Makefile:417: recipe for target 'install-recursive' failed
make: *** [install-recursive] Error 1

So the error comes from line 18 in IpSmartPtr.hpp, in the /usr/include/coin/ directory. I've modified this line by taking inspiration from similar line in IpJournalist.hpp (I don't remember the source of this idea, because I did this back in April. Sorry if I forgot to credit some other source). So I changed line 18 of IpSmartPtr.hpp to remove the error and instead force inclusion of <cstdarg>, as done in IpJournalist.hpp. This is the line after modification (with my initials to remember that this is a dirty patched line, but the rest of the comment really comes from IpJournalist.hpp )

#  include <cstdarg>  // if this header is included by someone who does not define HAVE_CSTDARG or HAVE_STDARG, let's hope that cstdarg is available. PH 2016-09

Now with this modification, I think somebody else should be able to reproduce the build.

Summary of the commands

This list of commands are just adapted from the user guide. Only the include path for Ipopt is specific (using Ubuntu package).

Install directory:

$ mkdir /home/modelica/Programmes/JModelica

in the source tree:

$ mkdir build
$ cd build/
$ ../configure --prefix=/home/modelica/Programmes/JModelica --with-ipopt=/usr
$ make install

then casadi interface

$ make casadi_interface

→ should just work!

PyCOZIR

I've started to play with the pretty nice Cozir CO2 sensor from GSS. This relates to research projects on air quality control.

For testing purpose, the sensor is connected to my computer through a USB-serial converter cable. In order to communicate  with the sensor (e.g. grab the CO2 concentration data), I've written a bit of Python code to wrap the low-level ASCII communication protocol into a higher level, more compact API.

For example, instead of exchanging byte codes, reading the temperature becomes:

>>> from cozir import Cozir
>>> c = Cozir('/dev/ttyUSB0')
>>> c.read_temperature()
20.5

For people that might be interested, all the details and the code are on the Github repository: https://github.com/pierre-haessig/pycozir

Also, the repo contains a basic logging program that can be customized to anyone's needs. Tested for now with Python 2.7, on Linux and Windows.

 

PowerTech 2015 article online

The article I submitted to PowerTech 2015 (Eindhoven, June 2015) has been accepted. I've put online the pdf manuscript. It is an author version, since the copyright will be transfered to IEEE.

Title: "Energy Storage Control with Aging Limitation"

[Update July 7, 2015] presentation slides are now also available.

Abstract

Energy Storage Systems (ESS) are often proposed to mitigate the fluctuations of renewable power sources like wind turbines. In such a context, the main objective for the ESS control (its energy management) is to maximize the performance of the mitigation scheme.

However, most ESS, and electrochemical batteries in particular, can only perform a limited number of charge/discharge cycles over their lifetime. This limitation is rarely taken into account in the optimization of the energy management, because of the lack of an appropriate formalization of cycling aging.

We present a method to explicitly embed a limitation of cycling aging, as a constraint, in the control optimization. We model cycling aging with the usual ``exchanged energy'' counting method. We demonstrate the effectiveness of our aging-constrained energy management using a publicly available wind power time series. Day-ahead forecast error is minimized while keeping storage cycling just under an acceptable target level.

Fake Wind Energy Conference ?

I just received this email on an old academic email address of mine

Dear Mr. Pierre Haessig,

The purpose of this letter is to formally invite you to be honorable Speaker for the upcoming "World Congress and Exhibition on Wind Energy" during June 15-17, 2016 at Berlin, Germany.

The conference highlights the theme “Future Wind Energy Technologies”. The program is a rich mix of formats ranging from keynote sessions featuring many well-known and thought provoking speakers to intense and highly interactive discussion sessions.

Wind Energy-2016 is a remarkable event which explores the technology to showcase mature technologies from lectures, panel discussions, debates by world renowned experts who will discuss the latest advances in wind energy production and planning.

Your contribution and knowledge in the field pulls our attention to keep the proposal of joining our conference and enlighten the audience with your knowledge and experience. It would be our honor to have an eminent delegate like you in our summit.

I would be glad to answer your queries in this regard.

For more details please visit our Website (http://www.windenergy.omicsgroup.com/)

This really looks like an automated email: not any specific words about my "contribution and knowledge in the field". Only my name, and the words "Wind Energy". Not even a specific list of topics or keywords related to Wind Energy (could be "offshore", "grid integration", "wind power forecasting"). Also, the conference website (http://www.windenergy.omicsgroup.com/) is pretty empty, with no visible organizing and scientific committees. Clearly this seems a fake conference.

It's not the first time I receive such an email, so I wonder: what is the business model of this OMICS Group company? What do they want, how do they make money? Is it some kind of hidden travel agency for researchers?

Also, will there be any actual people gathering for this conference in Berlin in June 15-17, 2016?

I really wonder...

 

Visual timer app update

Back in July 2014, I made a small visual timer application for my PhD defense (45 minutes presentation in France, and then unlimited time for questions from the jury). It is a pure SVG graphic created with Inkscape, embedded in a lean html page, with some bits of javascript to control the animation. Also, it is meant to be fully responsive: it scales nicely on laptops, tablets and smartphones.

Visual timerApp update

This week, I've spent some time to update the app to significantly improve the useability. Beyond a strong visual refresh, key improvements are:

  • full offline mode support: no more need of an Internet connection after the first usage. This is achieved using a AppCache manifest file.
  • new pause/resume control button.

How to use the timer

This timer simply works in a (recent) browser: http://éole.net/timer/. On a desktop/laptop computer, using the fullscreen mode gives the best result.

Source code

The app is available under a CC-BY-SA license. Source code is on GitHub: https://github.com/pierre-haessig/visual-timer

Possible future updates

I may take the time to add the necessary manifest file to turn this web app into an actual hosted Firefox OS app.

Original Arduino boards are uncertified ?

I just got a new Arduino Uno board this morning, and got bitten by a strange message in the Arduino IDE (version 1.6.3 from http://arduino.cc/en/Main/Software) when uploading a first test program:

This board comes from an uncertified manufacturer

Since the board comes from a trusted distributor (Lextronic), I was pretty surprised!

The reason behind the madness

Searching for this error message, I was even more surprised to discover that this is a side effect of a split between two Arduino "kingdoms":

  • arduino.cc, where I downloaded the Arduino IDE. They use the Arduino™ logo.
  • arduino.org, which manufactured my "original made in Italy" board. They use the Arduino ® logo. Their board can also be recognized by the Vendor ID 0x2A03.

I made a side-by-side picture comparison of the boards, using photos from the respective websites (the logo difference is particularly small: click to see it fullscreen).

The two official Arduino Uno boards side by side
The two official Arduino Uno boards side by side

How subtle! But behind this surprisingly minor cosmetic difference, I found a good report of the present situation (as of March 2015) in this article: http://hackaday.com/2015/03/12/arduino-v-arduino-part-ii/. It seems things are getting tensed (trademark lawsuits).

In practice

I shouldn't forget to say that, apart from the annoyingly frightening warning message, my "uncertified" board from arduino.org is in perfect condition and works perfectly with the current arduino.cc IDE!

(maybe future releases will behave differently, especially with new generation of boards developed separately)

It's sad that probably many people will get bitten by this quite misleading message in the coming months.

PHP trick for FxOS Manifest file

While exploring the development of a Web App for Firefox OS, I encountered a small trouble with the app manifest file: this text file needs to be served with a specific Content-Type header.

As an illustration, this is the content of the manifest of my app (JSON-formatted description of the App). It's hosted at http://éole.net/immocalc/manifest.webapp.

{
  "name": "ImmoCalc",
  "description": "Calculatrice de prêts immobiliers",
  "launch_path": "/immocalc/index.html",
  "icons": {
    "512": "/immocalc/img/house-icon-512.png",
    "128": "/immocalc/img/house-icon-128.png",
    "60":  "/immocalc/img/house-icon-60.png"
  },
  "developer": {
    "name": "Pierre Haessig",
    "url": "http://pierreh.eu"
  },
  "default_locale": "fr",
  "type": "web",
  "appcache_path": "/immocalc/manifest.appcache",
  "version": "1.0"
}

Normally, this file served as plain text. This can be seen when clicking on the link and opening the Page information dialog in Firefox (Ctrl+I). While plain text may be fine in another context, it is required that the Content-Type header is set to application/x-web-app-manifest+json. More precisely, this is required for the App submission on the Firefox Marketplace. It is recommended on MDN to tune the Apache server configuration with a .htaccess file. However, with my shared hosting plan at OVH, this was not working: the .htaccess file had no effect. [Edit: it seems to work now, so this PHP trick may not be needed anymore !].

To bypass this hosting limitation, I tried with a snippet of PHP code that simply sourcing the original manifest.webapp file, but with the proper Content-Type header.

<?php
// Serving the manifest file with the appropriate header
// https://developer.mozilla.org/en-US/Apps/Build/Manifest#Serving_manifests
header('Content-type: application/x-web-app-manifest+json');
readfile('manifest.webapp');
?>

These two lines were enough to successfully submit my app on the Marketplace. The effect of the header can be seen in the browser: it triggers the Download dialog when opening the url (http://éole.net/immocalc/manifest.webapp.php), instead of displaying the content.

Maybe this trick can help others ?