tag:blogger.com,1999:blog-70253900772222700962024-03-05T23:49:50.469+00:00Code from an English Coffee DrinkerRather than bore my family with code snippets I'll post them here!Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.comBlogger78125tag:blogger.com,1999:blog-7025390077222270096.post-81817005937123603712015-06-17T11:13:00.000+01:002019-01-06T18:11:44.907+00:00SteadyCamOver on one of my other blogs I recently <a href="http://englishcoffeedrinker.blogspot.co.uk/2015/06/hummingbird-hawk-moth.html">wrote a post</a> about a Hummingbird Hawk-Moth which I'd seen in the garden. This post included an animated GIF of the moth in flight. I included an animated GIF rather than a normal video because I'd had problems with the camera and the longest video I manged to shoot before the moth flew away contained just 14 frames.<br />
<br />
<div style="text-align: center;"><iframe allowfullscreen="" frameborder="0" height="225" src="https://www.youtube-nocookie.com/embed/p6CqWj8jGtQ?rel=0" width="400"></iframe><br />
</div><br />
As you can see it finishes almost before you've had chance to realise it's started to play and isn't very helpful at showing why the moth is named after the Hummingbird.<br />
<br />
Now I'm sure there are many ways in which you could turn the video into a looping GIF but I'm going to detail what I did, partly so I don't forget, and partly as I wrote some software to deal with one particular issue.<br />
<br />
My approach to almost any video related task usually starts with <a href="https://www.ffmpeg.org/">FFmpeg</a> and this was no different, with a simple command to scale the video down and produce a GIF as output making sure to keep the right frame rate.<br />
<pre class="brush:bash;gutter:false;">ffmpeg -i 00002.MTS -vf "fps=25,scale=400:225" animated.gif</pre><div class="separator" style="clear: both; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUC1jzJ92p37eavDOg5k8SdWMaJJnE3_O6Wn6Z4ykey6G-VEH7WhJut8uNfFw9cxElEzzCgEPXR6ki8VC9ACyknFKIHBd9Ict6kTMqWPUPJYAyY6dL2Vv48LHfbme1KjC2xBfOGsSJe0w/s1600/step2.gif" /></div><br />
As you can see this works to produce an animated GIF although there are a number of problems with it. Firstly it's very grainy and secondly you can't really see the moth now we've scaled the image down. We'll deal with the second problem first (as the first problem mostly goes away by the end). Again a simple FFmpeg command allows us to crop the video:<br />
<br />
<pre class="brush:bash;gutter:false;">ffmpeg -i 00002.MTS -vf "fps=25,scale=1920:1080,crop=800:450:400:225,scale=400:225" animated.gif</pre><div class="separator" style="clear: both; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIk8ilihRFrOPii2Nu-_m40UPNXJPNexxuJGCe5DMxvhDhdr8OtDtnATeY8z4-aJ-t6MThqMDmHW8JJ0usS14aW6jRHMVevD94AI2nwcIF4-C3NxEVowGjck5mnQTblUx1lwwace4hnGY/s1600/step3.gif" /></div><br />
This gives us a much better view of the moth but that jump as the animation loops around is very very annoying. The problem is that even with just 14 frames there is enough camera movement between the first and last frame for the join to be really obvious. This is something you often see with animated GIFs produced from video and you can clearly see why in this image of the first and last frames superimposed on one another.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPmczvQohl3Bq9ta-8MwlKIq29SLMi9RTgAKQy5159W2RNvI7ZR22RpQ8705KAnAZ7PGqwYC7Q5OO8JNnDGYUKZ2O_z8b23pH1p7WEuwszQt5HV260iDtWKymfqIoN6pyPKEBvOUnI7tk/s1600/first-last-original.jpg" /></div><br />
At this point I tried a number of filters in FFmpeg that are supposed to help remove camera shake etc. but none of them helped as the camera movement is fairly smooth and what I want is to just remove the movement altogether so that the plant stay stills between the frames. While I couldn't find anyway of doing this in FFmpeg I did realise that some of the code I used in <a href="http://englishjavadrinker.blogspot.co.uk/search/label/3DAssembler">3DAssembler</a> might help.<br />
<br />
I've never really described how 3DAssembler works, but essentially it aligns images. In fact it uses <a href="https://en.wikipedia.org/wiki/Speeded_up_robust_features">SURF</a> to detect identical features in a pair of images and then determines the horizontal and vertical shift required to try and overlap as many of the features it found as possible. In 3DAssembler this allows you to automatically align photos to produce a usable stereo pair. Here though we can use the same approach to align the frames of the video.<br />
<br />
The code I wrote (which is currently a mess of hard coded values, so I'll release it once I've had time to clean it up) calculates the horizontal and vertical shifts of each frame against the first frame and then crops each frame appropriately. If we superimpose the first and last of these corrected frames we can see how things have improved.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXB3ToHX-jR17HtcxnZKMTLYwHBtGEQV94vevY9bSi5_-S0WCMezX1U8ylya-qXY5vpXwB6Unn5S1gjTD-Epq2wXnCmBJDo-lSfum0d1bmTo94LcNQk1Yb3-qz7NiGHmc2OaHbBngNkMc/s1600/first-last-steady.jpg" /></div><br />
Producing the final animated GIF is then a multistage process. Firstly we use FFmpeg to turn the video into a series of still images, taking care to deinterlace the original video:<br />
<pre class="brush:bash;gutter:false;">ffmpeg -i 00002.MTS -vf "scale=1920:1080,yadif=1:0,hqdn3d,fps=25" frame%03d.png</pre>My code then aligns, crops, and scales these frames down to the same size we were using before. The set of frames is then reassembled to produce the animated GIF:<br />
<pre class="brush:bash;gutter:false;">convert -delay 4 frame*.png animated.gif</pre><div class="separator" style="clear: both; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhw4Yqvs0saqQ_6iZEiUIcfcO9vj0iVvc30lB1N_3Qs9HLQPNqgXubVCbr7FkQptSlWyCfMxml216MOj3wzec8kfG1g4Yngf7WmiXee7batm-ha0AUGL_w6ZtRq4MJLL6SkL9l4_RHDH6A/s1600/steady001.gif" /></div><br />
While there is still a jump of the background as it loops around it is a lot less obvious than in the original. You'll also notice that the grain present in the original has disappeared. The grain is actually dithering introduced to try and improve the image due to the fact that a GIF image is limited to just 256 colours. I didn't apply a dither filter when assembling the GIF which does mean you can see problems with the colour palette especially in the grass at the bottom left.<br />
<br />
I'm not sure why GIF has such a limited colour pallette but I'm guessing it relates to keeping the filesize down when storage was more expensive and bandwidth was a lot lower. Now that most peoples internet connection can handle full resolution HD video we shuold probably move beyond GIF images. For single images, especially hand drawn or those that use transparency, GIF has been replaced by PNG. The PNG format also <a href="https://en.wikipedia.org/wiki/APNG">supports animation</a>. Unfortunately only Firefox currently has support for showing animated PNG images, in all the other browsers all you see is the first frame.<br />
<br />
Fortunately it is possible to get animated PNGs to play in most modern browsers with a little bit of JavaScript trickery using the <a href="https://github.com/davidmz/apng-canvas">apng-canvas</a> library. Unfortunately the way this library works means that you need to host both the image and the javascript in the same place which makes it difficult to use with blogger, not helped by the fact the you can't currently upload animated PNG files to blogger either. Anyway after a little bit of work hopefully the following should be animated for everyone.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><img class="apng-image" src="https://greenwoodma.github.io/static-blogs/code/apng/example.png"></div><br />
As you can see this is much better than the GIF version as we aren't limited to just 256 colours. This was produced in exactly the same way as the GIF version though, apart from the final command to assemble the file which now looks like:<br />
<pre class="brush:bash;gutter:false;">apngasm animated.png frame*.png 1 25</pre>I'll clean up the code I used and make it available in case anyone else fancies producing weird little animated videos.<br />
<script src="https://greenwoodma.github.io/static-blogs/code/apng/apng-canvas.min.js"></script><br />
<script>
APNG.ifNeeded().then(function() {
var images = document.querySelectorAll(".apng-image");
for (var i = 0; i < images.length; i++) APNG.animateImage(images[i]);
});
</script>Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com3tag:blogger.com,1999:blog-7025390077222270096.post-40208902020542791562014-11-13T10:19:00.000+00:002014-11-18T07:57:40.879+00:00Blogger Stole My Thumbnails<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJHT3667e9aYMmLzUJYjfKZHELnaFDPksvCswVhtv55DDif7Xr2ogLtXyRbKKFel3unPKBrpXtzH0BlanvFGv3CvgFPgWPYPMhZ08wKa3rnaM-xWD1feGGdwn7i0d-o6nD9YzqeXUw_dU/s1600/Fingernails2.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 0em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJHT3667e9aYMmLzUJYjfKZHELnaFDPksvCswVhtv55DDif7Xr2ogLtXyRbKKFel3unPKBrpXtzH0BlanvFGv3CvgFPgWPYPMhZ08wKa3rnaM-xWD1feGGdwn7i0d-o6nD9YzqeXUw_dU/s320/Fingernails2.jpg" /></a></div>I really hoped I was done writing posts about how Blogger was messing with our blogs, but unfortunately here I am writing another post. It seems that something in the way images are added to our posts has changed and this in turn means that Blogger no longer generates and includes a thumbnail for each post in the feeds produced for a blog. Whilst this doesn't effect the look of your blog it will make it look less interesting when viewed in a feed reader or blog list on another blog as only the title of the posts will appear.<br />
<br />
The problem appears to be that when you upload and embed an image, for some reason Blogger is omitting the <code>http:</code> from the beginning of the image URL. This means that all the image URLs now start with just <code>//</code>. This is perfectly valid (it's defined in <a href="http://www.ietf.org/rfc/rfc1808.txt">RFC 1808</a>) and is often referred to as a scheme relative URL. What this means is that the URL is resolved relative to the scheme (http, https, ftp, file, etc.) of the page in which it is embedded.<br />
<br />
I'm guessing this changes means that blogger is looking at switching some blogs to be served over <code>https</code> instead of <code>http</code>. Leaving the scheme off the image URL means that the image will be served using the same scheme as the page regardless of what that is. The problem though is that the code that blogger uses to generate the post feeds seem to ignore images that don't start <code>http:</code> meaning that no thumbnails are generated.<br />
<br />
For now the easy fix is to manually add the <code>http:</code> to the image URLs (or at least of the image you want to use as the thumbnail for the post). Of course it would be better if blogger just fixed their code to spot these URLs properly and include them in the post feed.<br />
<br />
<b><i>Updated 18th November 2014: It looks as if this problem has been resolved as I've just tried posting on one of my other blogs and the <code>http:</code> bit is back.</i></b>Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com1tag:blogger.com,1999:blog-7025390077222270096.post-895187866468431792014-08-23T10:04:00.000+01:002014-08-23T10:04:55.779+01:00In Out, In Out, Shake It All AboutIn the very abstract sense text analysis can be divided into three main tasks; load some text, process it, export the result. Out of the box <a href="http://gate.ac.uk">GATE</a> (both the GUI and the API) provides excellent support for both loading documents and processing them, but until now we haven't provided many options when it comes to exporting processed documents.<br />
<br />
Traditionally GATE has provided two methods of exporting processed documents; a lossless XML format that can be reloaded into GATE but is <a href="http://englishjavadrinker.blogspot.co.uk/2013/08/savings-in-time-and-space.html">rather verbose</a>, or the "save preserving format" option which essentially outputs XML representing the original document (i.e. the annotations in the Original markups set) plus the annotations generated by your application. Neither of these options were particularly useful if you wanted to pass the output on to some other process and, without a standard export API, this left people having to write custom processing resources just to export their results.<br />
<br />
To try and improve the support for exporting documents recent <a href="http://jenkins.gate.ac.uk/job/GATE-Nightly/lastSuccessfulBuild/">nightly builds</a> of GATE now include a common export API in the <a href="jenkins.gate.ac.uk/job/GATE-Nightly/javadoc/gate/DocumentExporter.html"><code>gate.DocumentExporter</code></a> class. Before we go any further it is worth mentioning that this code is in a nightly build so is subject to change before the next release of GATE. Having said that I have now used it to implement exporters for a number of different formats so I don't expect the API to change drastically.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjb0gz0mhxJ6sh96c6T5RuiKV2XRZ4Q2sxPr4bDCx9k4cf-gv7SEEmbF5ztfXDtojywoqp8mguyGs9BPccetRKBzc48GSrJhyphenhyphen-na8EEtoRlt41I98pb1gPjLt8p3SElRZe6zOgEFgZnnwc/s1600/Screenshot+from+2014-08-22+19:32:22.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 0em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjb0gz0mhxJ6sh96c6T5RuiKV2XRZ4Q2sxPr4bDCx9k4cf-gv7SEEmbF5ztfXDtojywoqp8mguyGs9BPccetRKBzc48GSrJhyphenhyphen-na8EEtoRlt41I98pb1gPjLt8p3SElRZe6zOgEFgZnnwc/s320/Screenshot+from+2014-08-22+19:32:22.png" /></a></div>If you are a GATE user, rather than a software developer, than all you need to know is that an exporter is very similar to the existing idea of document formats. This means that they are CREOLE resources and so new exporters are made available by loading a plugin. Once an exporter has been loaded then it will be added to the "Save as..." menu of both documents and corpora and by default exporters for GATE XML and Inline XML (i.e. the old "Save preserving format) are provided even when no plugins have been loaded.<br />
<br />
If you are a developer and wanting to make use of an existing exporter, then hopefully the API should be easy to use. For example, to get hold of the exporter for GATE XML and to write a document to a file the following two lines will suffice:<br />
<pre class="brush:java">DocumentExporter exporter =
DocumentExporter.getInstance("gate.corpora.export.GateXMLExporter");
exporter.export(document, file);</pre>There is also a three argument form of the <code>export</code> method that takes a <code>FeatureMap</code> that can be used to configure an exporter. For example, the annotation types the Inline XML exporter saves is configured this way. The possible configuration options for an exporter should be contained in it's documentation, but possibly the easiest way to see how it can be configured is to try it from the GUI.<br />
<br />
If you are a developer and want to add a new export format to GATE, then this is fairly straightforward; if you already know how to produce other GATE resources then it should be really easy. Essentially you need to extend <code>gate.DocumentExporter</code> to provide an implementation of it's one abstract method. A simple example showing an exporter for GATE XML is given below:<br />
<pre class="brush:java">@CreoleResource(name = "GATE XML Exporter",
tool = true, autoinstances = @AutoInstance, icon = "GATEXML")
public class GateXMLExporter extends DocumentExporter {
public GateXMLExporter() {
super("GATE XML", "xml", "text/xml");
}
public void export(Document doc, OutputStream out, FeatureMap options)
throws IOException {
try {
DocumentStaxUtils.writeDocument(doc, out, "");
} catch(XMLStreamException e) {
throw new IOException(e);
}
}
}</pre>As I said earlier this API is still a work in progress and won't be frozen until the next release of GATE, but the current nightly build now contains export support for Fast Infoset compressed XML (I've <a href="http://englishjavadrinker.blogspot.co.uk/2013/08/savings-in-time-and-space.html">talked about this before</a>), JSON inspired by the format Twitter uses, and HTML5 Microdata (an updated version of the <a href="http://englishjavadrinker.blogspot.co.uk/2014/04/automatically-generating-html5-microdata.html">code I discussed before</a>). A number of other exporters are also under development and will hopefully be made available shortly.<br />
<br />
Hopefully if you use GATE you will find this new support useful and please do let us have any feedback you might have so we can improve the support before the next release when the API will be frozen.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com0tag:blogger.com,1999:blog-7025390077222270096.post-27057522300339978632014-04-08T00:21:00.000+01:002014-04-08T00:21:05.496+01:00Automatically Generating HTML5 MicrodataThe majority of the code I write as part of my day job revolves around trying to extract useful semantic information from text. Typical examples of what is referred to as <i>"semantic annotation"</i> include spotting that a sequence of characters represents the name of a person, organization or location and probably then linking this to an ontology or some other knowledge source. While extracting the information can be a task in itself, usually you want to do something with the information often to enrich or make the original text document more accessible in some way. In the applications <a href="http://gate.ac.uk">we</a> develop this usually entails indexing the document along with the semantic annotations to allow for a richer search experiance and I've <a href="http://englishjavadrinker.blogspot.co.uk/search/label/M%C3%ADmir">blogged about this approach</a> before. Such an approach assumes, however, that the consumer of the semantic annotations will be a human, but what if another computer programme wants to make use of the information we have just extracted. The answer is to use some form of common machine readable encoding.<br />
<br />
While there is already an awful lot of text in the world, more and more is being produced everyday, usually in electronic form, and usually published on the internet. Given that we could never read all this text we rely on search engines, such as <a href="http://www.google.com">Google</a>, to help us pinpoint useful or interesting documents. These search engines rely on two main things to find the documents we are interested in, the text and the links between the documents, but what if we could tell them what some of the text actually means?<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQdSXGdCMZcYawTGRi58ZpW51Cg5D0HeEW-LlCRukzUDczd9pnIYo-zVcwdVqjuyQq9Z_F897E6r7XXlriin6C7KVN9V6LB_Tnl3ly2fF4npPFfZvnclhZBN2eCedx5XWyQOXHYcFA9HM/s1600/microdata.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 0em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQdSXGdCMZcYawTGRi58ZpW51Cg5D0HeEW-LlCRukzUDczd9pnIYo-zVcwdVqjuyQq9Z_F897E6r7XXlriin6C7KVN9V6LB_Tnl3ly2fF4npPFfZvnclhZBN2eCedx5XWyQOXHYcFA9HM/s400/microdata.png" width="300" /></a></div>In the newest version of the <a href="http://en.wikipedia.org/wiki/HTML5">HTML specification</a> (which is a work in progress usually referred to as HTML5) web pages can contain semantic information encoded as <a href=" http://en.wikipedia.org/wiki/Microdata_%28HTML%29">HTML Microdata</a>. I'm not going to go into the details of how this works as there is already a number of great descriptions available, including <a href="http://www.marcofolio.net/webdesign/html5_microdata_what_is_it_and_why_should_you_care_.html">this one</a>.<br />
<br />
HTML5 Microdata is, in essence, a way of embedding semantic information in a web page, but it doesn't tell a human or a machine what any of the information means, especially as different people could embed the same information using different identifiers or in different ways. What is needed is a common vocabulary that can be used to embed information about common concepts, and currently many of the major search engines have settled on using <a href="http://schema.org/">schema.org</a> as the common vocabulary.<br />
<br />
When I first heard about schema.org, back in 2011, I thought it was a great idea, and wrote some code that could be used to embed the output of a GATE application within a HTML page as microdata. Unfortunately the approach I adopted was, to put it bluntly, hacky. So while I had proved it was possible the code was left to rot in a dark corner of my SVN repository.<br />
<br />
I was recently reminded of HTML5 microdata and schema.org in particular when <a href="https://twitter.com/LeonDerczynski/status/448141186965184512">one of my colleges tweeted a link</a> to this <a href="http://searchengineland.com/5-ways-optimize-markup-knowledge-graph-semantic-search-186755">interesting article</a>. In response I was daft enough to admit that I had some code that would allow people to automatically embed the relevant microdata into existing web pages. It wasn't long before I'd had a number of people making it clear that they would be interested in me finishing and releasing the code.<br />
<br />
I'm currently in a hotel in London as I'm due to teach a two day GATE course starting tomorrow (if you want to learn all about GATE then you might be interested in our <a href="https://gate.ac.uk/conferences/fig/fig7.html">week long course to be held in Sheffield in June</a>) and rather than watching TV or having a drink in the bar I thought I'd make a start on tidying up the code I started on almost three years ago.<br />
<br />
Before we go any further I should point out that while the code works the current interface isn't the most user friendly. As such I've not added this to the main GATE distribution as yet. I'm hopping that any of you who give it a try can leave me feedback so I can finish cleaning things up and integrate it properly. Having said that here is what I have so far...<br />
<br />
I find that worked examples usually help convey my ideas better than prose so, lets start with a simple HTML page:<br />
<pre class="brush:html"><html>
<head>
<title>This is a schema.org test document</title>
</head>
<body>
<h1>This is a schema.org test document</h1>
<p>
Mark Greenwood works in Sheffield for the University of Sheffield.
He is currently in a Premier Inn in London, killing time by working on a GATE plugin to allow annotations to be embedded within a HTML document using HTML microdata and the schema.org model.
</p>
</body>
</html></pre>As you can see this contains a number of obvious entities (people, organizations and locations) that could be described using the schema.org vocabulary (<a href="http://schema.org/Person">people</a>, <a href="http://schema.org/Organization">organizations</a> and <a href="http://schema.org/Place">locations</a> are just some of the schema.org concepts) and which would be found by simply running <a href="http://gate.ac.uk/userguide/chap:annie">ANNIE</a> over the document.<br />
<br />
Once we have sensible annotations for such a document, probably from running ANNIE, and a mapping between the annotations and their features and the schema.org vocabulary then it is <i>fairly</i> easy to produce a version of this HTML document with the annotations embedded as microdata. The current version of my code generate the following file:<br />
<pre class="brush:html"><html>
<head>
<title>This is a schema.org test document</title>
</head>
<body>
<h1>This is a schema.org test document</h1>
<p>
<span itemscope="itemscope" itemtype="http://schema.org/Person"><meta content="male" itemprop="gender"/><meta content="Mark Greenwood" itemprop="name"/>Mark Greenwood</span> works in <span itemscope="itemscope" itemtype="http://schema.org/City"><meta content="Sheffield" itemprop="name"/>Sheffield</span> for the <span itemscope="itemscope" itemtype="http://schema.org/Organization"><meta content="University of Sheffield" itemprop="name"/>University of Sheffield</span>.
He is currently in a <span itemscope="itemscope" itemtype="http://schema.org/Organization"><meta content="Premier" itemprop="name"/>Premier</span> Inn in <span itemscope="itemscope" itemtype="http://schema.org/City"><meta content="London" itemprop="name"/>London</span>, killing time by working on a GATE plugin to allow <span itemscope="itemscope" itemtype="http://schema.org/Person"><meta content="female" itemprop="gender"/><meta content="ANNIE" itemprop="name"/>ANNIE</span> annotations to be embedded within a HTML document using HTML microdata and the schema.org model.
</p>
</body>
</html></pre>This works nicely and the embeded data can be extracted by the search engines, as proved using the <a href="http://www.google.com/webmasters/tools/richsnippets">Google rich snippets tool</a>.<br />
<br />
As I said earlier while the code works, the current integration with the rest of GATE definitely needs improving. If you load the plugin (details below) then right clicking on a document will allow you to <i>Export as HTML5 Microdata...</i> but it won't allow you to customize the mapping between annotations and a vocabulary. Currently the ANNIE annotations are mapped to the schema.org vocabulary using a config file in the resources folder. If you want to change the mapping you have to change this file. In the future I plan to add some form of editor (or at least the ability to choose a different file) as well as the ability to export a corpus not just a single file.<br />
<br />
So if you have got all the way to here then you probably want to get your hands on the current plugin, so <a href="http://demos.englishcoffeedrinker.co.uk/gate/Format_HTML5Microdata.zip">here it is</a>. Simply load it into GATE in the usual way and it will add the right-click menu option to documents (you'll need to use a nightly build of GATE, or a recent SVN checkout, as it uses the <a href="http://englishjavadrinker.blogspot.co.uk/2013/08/resource-helpers.html">resource helpers</a> that haven't yet made it into a release version).<br />
<br />
Hopefully you'll find it useful but please do let me know what you think, and if you have any suggestions for improvements, especially around the integration the GATE Developer GUI.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com0tag:blogger.com,1999:blog-7025390077222270096.post-44037835410384047352014-03-12T19:46:00.000+00:002014-03-12T19:46:46.710+00:00At Sixes And Sevens<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPa13zoUJvhExTDpb-1a17Is2EqfsEgtFSwXw6WqIRfn5qGJoDkvjqG0Uj2KdYvWymrSVAk9tYKWOECOc-hJnhBHBfyh3_-Se_vO551L7Ghw6fwtO2rP7okTvUqb2F-Mj7b4XSsscNJBE/s1600/sixessevens.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 0em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPa13zoUJvhExTDpb-1a17Is2EqfsEgtFSwXw6WqIRfn5qGJoDkvjqG0Uj2KdYvWymrSVAk9tYKWOECOc-hJnhBHBfyh3_-Se_vO551L7Ghw6fwtO2rP7okTvUqb2F-Mj7b4XSsscNJBE/s320/sixessevens.jpg" /></a></div>At work we are slowly getting ready for a major new release of <a href="http://gate.ac.uk">GATE</a>. In preparation for the release I've been doing a bit of code cleanup and upgrading some of the libraries that we use. After every change I've been running the test suite and unfortunately some of the tests would intermittently fail. Given that none of the other members of the team had reported failing tests and that they were always running successfully on our <a href="http://jenkins-ci.org/">Jenkins</a> build server I decided the problem must be something related to my computer. My solution then was simply to ignore the failing tests as long as they weren't relevant to the code I was working on, and then have the build server do the final test for me. This worked, but it was exceedingly frustrating that I couldn't track down the problem. Yesterday I couldn't ignore the problem any longer because the same tests suddenly started to randomly fail on the build server as well as my computer and so I set about investigating the problem.<br />
<br />
The <a href="http://sourceforge.net/p/gate/code/HEAD/tree/gate/trunk/src/test/gate/creole/TestPR.java">tests in question</a> are all part of a single standard <a href="http://junit.org/">JUnit</a> test suite that was originally written back in 2001 and which have been running perfectly ever since. Essentially these tests run the main <a href="http://gate.ac.uk/userguide/chap:annie">ANNIE</a> components over four test documents checking the result at each stage. Each component is checked in a different test within the suite. Now if you know anything about <a href="http://en.wikipedia.org/wiki/Unit_testing">unit testing</a> you can probably already hear alarm bells ringing. For those of you that don't know what unit testing is, essentially each test should check a single component of the system (i.e. a unit) and should be independent from every other test. In this instance while each test checked a separate component, each relied on all the previous tests in the suite having run successfully.<br />
<br />
Now while dependencies between tests isn't ideal it still doesn't explain why they should have worked fine for twelve years but were now failing. And why did they start failing on the build server long after they had been failing on my machine. I eventually tracked the change that caused them to fail when run on the build server back to the upgrade from version 4.10 to 4.11 of JUnit but even with the help of a web search I couldn't figure out what the problem was.<br />
<br />
Given that I'd looked at the test results from my machine so many times and not spotted any problems I roped in some colleagues to run the tests for me on their own machines and send me the results to see if I could spot a pattern. The first thing that was obvious was that when using version 4.10 of JUnit the tests only failed for those people running Java 7. GATE only requires Java 6 and those people still with a Java 6 install, which includes the build server (so that we don't accidentally introduce any Java 7 dependencies), were not seeing any failures. If, however, we upgraded JUnit to version 4.11 everyone started to see random failures. The other thing that I eventually spotted was that when the tests failed, the logs seemed to suggest that they had been run in a random order which, given the unfortunate links between the tests, would explain why they then failed. Armed with all this extra information I went back to searching the web and this time I was able to find the problem and an easy solution. <br />
<br />
Given that unit tests are meant to be independent from one another, there isn't actually anything within the test suite that stipulates the order in which they should run, but it seems that it always used to be the case that the tests were run in the order in which they were defined in the source code. The tests are extracted from the suite by looking for all methods that start with the word test, and these are extracted from the class definition using the <a href"http://docs.oracle.com/javase/6/docs/api/java/lang/Class.html#getDeclaredMethods%28%29"><code>Method[] getDeclaredMethods()</code> method from <code>java.lang.Class</code></a>. The documentation for this method includes the following description:<br />
<br />
<blockquote>Returns an array of Method objects reflecting all the methods declared by the class or interface represented by this Class object. This includes public, protected, default (package) access, and private methods, but excludes inherited methods. The elements in the array returned are not sorted and are not in any particular order.</blockquote><br />
This makes it more than clear that we should never have assumed that the tests would be run in the same order they were defined, but it turns out that this was the order in which the methods were returned when using the Sun/Oracle versions of Java up to and including Java 6 (update 30 is the last version I've tested). I've written the following simple piece of code that shows the order of the extracted tests as well as info on the version of Java being used:<br />
<pre class="brush:java">import java.lang.reflect.Method;
public class AtSixesAndSevens {
public static void main(String args[]) {
System.out.println("java version \"" +
System.getProperty("java.version") + "\"");
System.out.println(System.getProperty("java.runtime.name") +
" (build " + System.getProperty("java.runtime.version") + ")");
System.out.println(System.getProperty("java.vm.name") +
" (build " + System.getProperty("java.vm.version") + " " +
System.getProperty("java.vm.info") + ")\n");
for(Method m : AtSixesAndSevens.class.getDeclaredMethods()) {
if(m.getName().startsWith("test"))
System.out.println(m.getName());
}
}
public void testTokenizer() {}
public void testGazetteer() {}
public void testSplitter() {}
public void testTagger() {}
public void testTransducer() {}
public void testCustomConstraintDefs() {}
public void testOrthomatcher() {}
public void testAllPR() {}
}</pre>Running this on the build server gives the following output:<br />
<pre class="brush:html;gutter:false;">java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02 mixed mode)
testTokenizer
testGazetteer
testSplitter
testTagger
testTransducer
testCustomConstraintDefs
testOrthomatcher
testAllPR</pre>While running it on my machine results in a random ordering of the test methods as you can see here:<br />
<pre class="brush:html;gutter:false;">java version "1.7.0_51"
OpenJDK Runtime Environment (build 1.7.0_51-b00)
OpenJDK 64-Bit Server VM (build 24.45-b08 mixed mode)
testTagger
testTransducer
testGazetteer
testSplitter
testTokenizer
testCustomConstraintDefs
testOrthomatcher
testAllPR</pre>Interestingly it would seem that the order only changes when the class is re-compiled, which suggests that the ordering may be related to how the methods are stored in the class file, but understanding the inner workings of the class file format is well beyond me. Even more interestingly it seems that even with Java 6 you can see a random ordering if you aren't using a distribution from Sun/Oracle, as here is the output from running under the Java 6 version of OpenJDK:<br />
<pre class="brush:html;gutter:false;">java version "1.6.0_30"
OpenJDK Runtime Environment (build 1.6.0_30-b30)
OpenJDK 64-Bit Server VM (build 23.25-b01 mixed mode)
testTagger
testTransducer
testCustomConstraintDefs
testOrthomatcher
testAllPR
testTokenizer
testGazetteer
testSplitter</pre>So this explains why switching from Java 6 to Java 7 could easily cause these related tests to fail, but why should upgrading from JUnit version 4.10 to 4.11 while staying with Java 6 cause a problem?<br />
<br />
It turns out that in the new version of JUnit the developers decided to change the default behaviour away from relying on the non-deterministic method ordering provided by Java. Their default approach is now to use a deterministic ordering to guarantee the tests are always run in the same order; as far as I can tell this orders the methods by sorting on the <code>hashCodes</code> of the method names. While this may at least remove the randomness from the test order it doesn't keep them in the same order they are defined in the source file, and so our tests were always failing under JUnit 4.11. Fortunately the developers also allow you to add a class level annotation to force the methods to be ordered alphabetically. I've now renamed the tests so that when sorted alphabetically they are run in the right order (by adding a three digit number after the initial <code>test</code> in their name), and the class definition now looks like:<br />
<pre class="brush:java;">@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class TestPR extends TestCase {
...
}</pre>So I guess there are two morals to this story. Firstly unit tests are called unit tests for a reason and they really should be independent of one another, but more importantly reading the documentation for the language or library you are using and not making assumptions about how they work (especially when the documentation tells you not to rely on something always being true) would make life easier.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com1tag:blogger.com,1999:blog-7025390077222270096.post-30806620573357280182014-01-12T13:43:00.000+00:002014-01-12T13:43:01.149+00:00Improving Your Thumbnails<img style="display:none;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKAn-pBwo8v92NQmEh1G0-xbsATFtZjLz6qN4ImJRoX_jCBz7MzzGiq_ASM1Mj8vc7z3fRnl3ZXMQqfMB-8XmcMVtl7abjv0CTA8-rx6u2OzF9WnVfQ0RPCKNAJY3TOdjIpeROuJVfLEY/s320/thumnail.jpeg" />Now before you all get confused this isn't a post about health or beauty tips for your thumbnails, but rather about a little trick I sometimes use to improve the thumbnails Blogger generates for each post.<br />
<br />
When you publish a post through Blogger not only does the post appear on your blog but Blogger also inserts the content into two news feeds (<a href="http://en.wikipedia.org/wiki/RSS">RSS</a> and <a href="http://en.wikipedia.org/wiki/Atom_%28standard%29">Atom</a> formats) which people can subscribe to in order to know when you have published something new. These feeds are also used to fill in the blog list widget that many people include in their blog template; you can see mine under the heading <i>"I'm Also..."</i> on the right.<br />
<br />
If you have included an image in your post, and the image is hosted by Blogger (i.e. has <code>blogspot</code> in the URL) then as well as putting the post content into the news feed Blogger will generate a small thumbnail to represent the post. It appears to do this by creating a 72 pixel square thumbnail from the first image in the post. Specifically it scales the image so that the short edge is 72 pixels long and then crops the other dimension to retain the middle of the image. You can see this working with the image of a Chinese tin of spam from one of my recent posts.<br />
<br />
<div style="text-align:center;"><img style="padding-right:1em;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEji-5fCSryyIcWx1oSdt4cm3e0NaJX8XWE40olThAaISVYSdCV6yLX9bRJnrDfpaVsPSBDTqJOgcFr-xqAsaXt_Sfw-JwdIEBK4PBDx9cJ2El7QTAcrmFwUogJjGHn9hyphenhyphenK0UQUNriE3Z7w/s300/chinese-spam.jpeg"> <img style="padding-left:1em;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEji-5fCSryyIcWx1oSdt4cm3e0NaJX8XWE40olThAaISVYSdCV6yLX9bRJnrDfpaVsPSBDTqJOgcFr-xqAsaXt_Sfw-JwdIEBK4PBDx9cJ2El7QTAcrmFwUogJjGHn9hyphenhyphenK0UQUNriE3Z7w/s300-c/chinese-spam.jpeg"></div><br />
To make the images easier to see I've used a width of the 300 pixels instead of 72 but the result is the same. On the left you can see the original image sized to 300 pixels wide, whereas on the right we have a 300 pixel thumbnail generated by scaling the height first and then cropping the width. In this example the cropping isn't too bad as it has retained almost all of the important content, but you can easily imagine images where the cropping could result in thumbnails that were far from ideal; chopping off peoples heads is a common example.<br />
<br />
Fortunately it is easy to control the thumbnail that is generated by ensuring that the first image in your post is already square and cropped to your satisfaction. Now of course that would often leave you with an image you don't actually want to use, but that is alright as my trick doesn't actually result in the image appearing in the post anyway.<br />
<br />
To customize the thumbnail all you have to do is upload the square image you want to use (it can be any size although as it will always be displayed as a 72 pixel square there isn't much point making it too big) through the Blogger interface and then switch to the HTML editor view. Now we don't need all the HTML that Blogger generates as all we need is the <code>img</code> tag. So you can remove everything surround the image and move the rest to the very beginning of your post. So for this post that looks something like (I've trimmed the URL to fit the screen):<br />
<pre class="brush:html;gutter:false"><img src="http://3.bp.blogspot.com/.../thumnail.jpeg" /></pre>By placing this at the beginning of the post we ensure that this image is the one Blogger uses when it generates the thumbnail, and we can hide it for all other purposes by adding some CSS to the image as follows:<br />
<pre class="brush:html;gutter:false"><img style="display:none;" src="http://3.bp.blogspot.com/.../thumnail.jpeg" /></pre>In this case the CSS is fairly self-explanatory as it simply turns off the display of the image. And that is it, a very simple trick but one that can make your blog look better in other peoples news feeds.<br />
<br />
As well as using this to customize a thumbnail that Blogger would already generate you can of course use it to generate thumbnails in cases where Blogger otherwise wouldn't. The two main cases where this might be useful are firstly where you host your images somewhere else (maybe <a href="http://www.flickr.com/">flicker</a>) or, and this is where I most often use this trick, if you have embedded a <a href="http://www.youtube.com/">YouTube</a> video instead of an image in your post. In neither case does Blogger generate a thumbnail for you, but you should be able to see how easy it would be to add your own.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com2tag:blogger.com,1999:blog-7025390077222270096.post-11685853389289093872014-01-06T07:32:00.000+00:002014-01-07T09:49:00.059+00:00The Other Kiwi's Grass Is Greener<img style="display:none;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjm6LUpyKmBJ0OFVtCVRSHVeCE6R9keMQtsH_y7F7ALt5GtDI7kx2AH5v6xLsgds3KUYNdxjV816vihORxVTDjXeHwLPxxSXl59rAKvCyQ26g9NPfsU7TEEyO3dVhaGTrL9VsgepesYgkg/s320/kiwi.png" /><noscript><div style="text-align:center; font-weight: bold; font-size:125%;">You will need to enable JavaScript for the examples in this post to function.</div><br />
</noscript><style title="kiwi" type="text/css">
.ground {
fill: #94d31b;
}
.kiwi {
fill: #C19A6B;
}
</style><script type="text/javascript">
//<![CDATA[
// simplify the following code by making sure we can access the URL object
// via window.URL regardless of what it is actually called
window.URL = window.URL || window.webkitURL;
/**
* Returns a DOM element that represents a CSS style sheet. The styles
* included depend upon the value of the names param. If the param is
* undefined then all styles within the document are included. If the
* param has a defined value then it is treated as an array of strings
* and only styles from style sheets whose title is in the array are
* included in the resulting DOM element.
**/
function getStyles(names) {
// create an empty style DOM element
var styles = document.createElement("style");
// set the type attribute to text/css
styles.setAttribute("type","text/css");
// get easy access to the Array slice method
var slice = Array.prototype.slice;
for (var i = 0 ; i < document.styleSheets.length ; ++i) {
// for each style sheet in the document
var sheet = document.styleSheets[i];
if (names == undefined || names.indexOf(sheet.title) != -1) {
// if we are including all styles or this sheet is one we want
slice.call(document.styleSheets[i].cssRules).forEach(
function(rule){
var text = document.createTextNode("\n"+rule.cssText);
styles.appendChild(text);
}
);
}
}
return styles;
}
function saveAsStyledSVG(id, styles, link) {
if (link.href != "#") {
window.URL.revokeObjectURL(link.href);
}
var svg = document.getElementById(id).cloneNode();
svg.insertBefore(getStyles(styles),svg.firstChild);
var content = svg.outerHTML;
if (content == undefined) {
var div = document.createElement("div");
div.appendChild(svg);
content = div.innerHTML;
}
var blob = new Blob([content], {type: "image/svg+xml"});
link.href = window.URL.createObjectURL(blob);
return true;
}
function saveSVG(id) {
var svg = document.getElementById(id).cloneNode();
var content = svg.outerHTML;
if (content == undefined) {
var div = document.createElement("div");
div.appendChild(svg);
content = div.innerHTML;
}
return content;
}
function saveAsSVG(id, link) {
if (link.href != "#") {
window.URL.revokeObjectURL(link.href);
}
var svg = document.getElementById(id).cloneNode();
var content = svg.outerHTML;
if (content == undefined) {
var div = document.createElement("div");
div.appendChild(svg);
content = div.innerHTML;
}
var blob = new Blob([content], {type: "image/svg+xml"});
link.href = window.URL.createObjectURL(blob);
return true;
}
//]]>
</script><svg style="width:250px; height:205px; float:left; margin-right: 1em;" id="svg" version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
viewBox="0 65.326 612 502.174" enable-background="new 0 65.326 612 502.174"
xml:space="preserve"><ellipse class="ground" cx="283.5" cy="487.5" rx="259" ry="80"/><path class="kiwi" d="M210.333,65.331C104.367,66.105-12.349,150.637,1.056,276.449c4.303,40.393,18.533,63.704,52.171,79.03
c36.307,16.544,57.022,54.556,50.406,112.954c-9.935,4.88-17.405,11.031-19.132,20.015c7.531-0.17,14.943-0.312,22.59,4.341
c20.333,12.375,31.296,27.363,42.979,51.72c1.714,3.572,8.192,2.849,8.312-3.078c0.17-8.467-1.856-17.454-5.226-26.933
c-2.955-8.313,3.059-7.985,6.917-6.106c6.399,3.115,16.334,9.43,30.39,13.098c5.392,1.407,5.995-3.877,5.224-6.991
c-1.864-7.522-11.009-10.862-24.519-19.229c-4.82-2.984-0.927-9.736,5.168-8.351l20.234,2.415c3.359,0.763,4.555-6.114,0.882-7.875
c-14.198-6.804-28.897-10.098-53.864-7.799c-11.617-29.265-29.811-61.617-15.674-81.681c12.639-17.938,31.216-20.74,39.147,43.489
c-5.002,3.107-11.215,5.031-11.332,13.024c7.201-2.845,11.207-1.399,14.791,0c17.912,6.998,35.462,21.826,52.982,37.309
c3.739,3.303,8.413-1.718,6.991-6.034c-2.138-6.494-8.053-10.659-14.791-20.016c-3.239-4.495,5.03-7.045,10.886-6.876
c13.849,0.396,22.886,8.268,35.177,11.218c4.483,1.076,9.741-1.964,6.917-6.917c-3.472-6.085-13.015-9.124-19.18-13.413
c-4.357-3.029-3.025-7.132,2.697-6.602c3.905,0.361,8.478,2.271,13.908,1.767c9.946-0.925,7.717-7.169-0.883-9.566
c-19.036-5.304-39.891-6.311-61.665-5.225c-43.837-8.358-31.554-84.887,0-90.363c29.571-5.132,62.966-13.339,99.928-32.156
c32.668-5.429,64.835-12.446,92.939-33.85c48.106-14.469,111.903,16.113,204.241,149.695c3.926,5.681,15.819,9.94,9.524-6.351
c-15.893-41.125-68.176-93.328-92.13-132.085c-24.581-39.774-14.34-61.243-39.957-91.247
c-21.326-24.978-47.502-25.803-77.339-17.365c-23.461,6.634-39.234-7.117-52.98-31.273C318.42,87.525,265.838,64.927,210.333,65.331
z M445.731,203.01c6.12,0,11.112,4.919,11.112,11.038c0,6.119-4.994,11.111-11.112,11.111s-11.038-4.994-11.038-11.111
C434.693,207.929,439.613,203.01,445.731,203.01z"/></svg>I've recently been building a simple proof-of-concept web application which may or may not see the light of day, but essentially it builds an <a href="http://en.wikipedia.org/wiki/SVG">SVG</a> image directly within the browser through user interaction. One useful feature would be to allow the user to download the completed SVG file (as a backup or for use elsewhere). After quite a bit of experimentation with some of the newer HTML features I now have a (mostly) working solution, which I've reduced down to a simple example using the Kiwi to the left (the SVG for this image comes from a useful <a href="http://css-tricks.com/using-svg/">tutorial</a> by Chris Coyier). If you can't see a Kiwi then your browser doesn't supporting displaying SVG images and certainly won't support the techniques in the rest of this post, so if you want to follow along you'll need to find a more modern browser.<br />
<br />
Given that the SVG image is being built through standard DOM manipulation it is easy to get the text of the SVG file directly via JavaScript. For example, if we assume that the SVG element has the id <code>svg</code> then the following will get the full text of the image.<br />
<pre class="brush:javascript">// get a copy of the DOM element with the id "svg"
var svg = document.getElementById("svg").cloneNode();
// get the text of this element, including the element and all children
var content = svg.outerHTML;
if (content == undefined) {
// if there is no content then the outerHTML property isn't supported...
// so create a new DIV element
var div = document.createElement("div");
// append the SVG to the DIV
div.appendChild(svg);
// use the widely supported innerHTML to get at the text of the SVG
content = div.innerHTML;
}</pre>This is a little more convoluted than I would like due to the fact that while all browsers support <code>outerHTML</code> not all of them (specifically Chrome) support it on the SVG element. Either way this code allows us to get the text we need to return as an SVG file to the user. Note that I've cloned the SVG node (line 2) rather than just referencing it directly, because changing a nodes parent in Chrome causes it to disappear from the document, which clearly isn't what we want. The more complex part of the problem then is figuring out how to return the text as an SVG to the user.<br />
<br />
Given that I was looking at this in the context of a large web application, probably the easiest option would have been to send the text back to the server, which could then return it to the browser with a sensible filename and mime type (<code>image/svg+xml</code>), but this would clearly be rather inefficient, especially as the SVG image grows in size. It would also preclude the technique from being used in simple client side applications which didn't use a web server.<br />
<br />
I stumbled upon an almost working example by accident. It turns out that if you simply return the SVG data from within a JavaScript href link the SVG will open within the browser, allowing the user to simply save the page to download the SVG. So we can wrap up the code above in a function:<br />
<pre class="brush:javascript">function saveAsSVG(id) {
var svg = document.getElementById(id).cloneNode();
var content = svg.outerHTML;
if (content == undefined) {
var div = document.createElement("div");
div.appendChild(svg);
content = div.innerHTML;
}
return content;
}</pre>And then call this function from a link<br />
<pre class="brush:html;gutter:false"><a href="javascript:saveAsSVG('svg');">Download</a></pre>And you can see what happens with this <a href="javascript:saveSVG('svg');">live version</a> of the above. If you've just tried that you will have hopefully noticed a few problems:<br />
<ul><li>the image is missing all it's colour information (the kiwi and grass are both black)</li>
<li>when you save the image the browser suggests a HTML filename</li>
<li>the file you save is actually a HTML file not an SVG as the browser adds some extra HTML wrapping</li>
</ul>While these are all clearly problems I was encouraged that I was at least heading in roughly the right direction. The missing colours I had expected (and will come back to later) and so I focused on the final two related problems first. After searching the web I came across the HTML5 <code>download</code> attribute that the <a href="http://www.whatwg.org/specs/web-apps/current-work/multipage/links.html#downloading-resources">specification</a> seemed to suggest was exactly what I needed. Essentially the new <code>download</code> attribute, when applied to a link, signals to the browser that the item the link points to should be downloaded rather than display. It also allows you to specify a filename that the browser should suggest when the user clicks the link. So using this with the above gets us:<br />
<pre class="brush:html;gutter:false"><a href="javascript:saveAsSVG('svg');" download="kiwi.svg">Download</a></pre>which again you can try for your self with this <a href="javascript:saveSVG('svg');" download="kiwi.svg">live version</a> of the above. Now I don't know which browser you are using, but I was developing this in Firefox, and I was amazed to find that this works; clicking the link pops up a "save as" dialog with the correct file name and the saved file is the correct SVG file. Unfortunately trying the same thing in both Opera and Chrome doesn't work. In Opera, the attribute seems to be simply ignored as it doesn't change the behaviour, while in Chrome when you click the link nothing happens, and no errors are reported. As far as Opera is concerned the problem seems to be that I'm running the latest version available for Ubuntu which is v12, while the most recent version for Windows and Mac is 18; so if Opera are abandoning Linux I'm going to abandon testing with Opera.<br />
<br />
Now while I could mandate specific browser versions for the app I was working on I really did want to try and find a more general solution. Also I had noticed that even once you have saved the SVG in Firefox the <a href="http://en.wikipedia.org/wiki/Throbber">throbber</a> never stops spinning, which usually suggests that the browser isn't really happy with what you have just done! Fortunately the search that led me to the <code>download</code> attribute also pointed me at a related new HTML feature; the blob.<br />
<br />
A <a href="https://developer.mozilla.org/en-US/docs/Web/API/Blob">blob</a> <i>"represents a file-like object of immutable, raw data. Blobs represent data that isn't necessarily in a JavaScript-native format"</i>, which sounds like exactly what I need, especially as a blob can be referenced by a URL. I'm not going to detail all the possible features or ways of using blobs, but the following single line of code can be used to create a blob from the SVG content we already know how to extract:<br />
<pre class="brush:javascript">var blob = new Blob([content], {type: "image/svg+xml"});</pre>If you remember from before the variable <code>content</code> holds the text of the SVG file we want to create, so this line creates a binary blob from the SVG data with the right mime type. It is also easy to create and destroy URLs that point to these blobs. For example, the following creates and then immediately destroys a URL pointing to the blob.<br />
<pre class="brush:javascript">var url = window.URL.createObjectURL(blob);
window.URL.revokeObjectURL(url);</pre>You can probably get away without explicitly revoking every URL you create as they should be automatically revoked when the page is unloaded, but it is certainly a good idea to release them if you know that a) the URL is invalid and b) you are creating a lot of these URLs. One thing to note is that Chrome (and other WebKit based browsers) use a vendor prefix for the URL object (i.e. it's called <code>webkitURL</code>) so we need to make sure that regardless of the browser we are using we can access the functions we need. Fortunately this is easy to do by adding the following line (outside of any function):<br />
<pre class="brush:javascript">window.URL = window.URL || window.webkitURL;</pre><br />
So the remaining question is how do we string all these bits together so that clicking a link on a page, builds a blob, generates a URL and then allows the user to download the URL. All the solutions I saw while searching the web did this in two steps. Essentially they had a link or button that when clicked would build the blob, generate the URL, and then add a new link to the document. This new link would use the <code>download</code> attribute and the blob URL to allow the user to download the file. While this works it doesn't seem particularly user friendly. Fortunately it is easy to combine all this functionality into a single link.<br />
<br />
The trick to doing everything within a single link is to use the <code>onclick</code> event handler of the link to build the blob and generate the URL which it then sets as the <code>href</code> attribute of the link. As long as the <code>onclick</code> event handler returns true then the browser will follow the <code>href</code> which now points at the blob URL and when we combine this with the <code>download</code> attribute the result is that the user is prompted to download the blob. So the function I came up with looks like the following:<br />
<pre class="brush:javascript">function saveAsSVG(id, link) {
if (link.href != "#") {
window.URL.revokeObjectURL(link.href);
}
var svg = document.getElementById(id).cloneNode();
var content = svg.outerHTML;
if (content == undefined) {
var div = document.createElement("div");
div.appendChild(svg);
content = div.innerHTML;
}
var blob = new Blob([content], {type: "image/svg+xml"});
link.href = window.URL.createObjectURL(blob);
return true;
}</pre>Essentially this is just all the code we have already seen assembled into a single function, although it assumes that the initial <code>href</code> is <code>#</code> so that it doesn't try and revoke an invalid URL (from testing it seems that trying to revoke something that isn't valid is simply ignored, but this is slightly safer). We can then call this function from a link as follows:<br />
<pre class="brush:html;gutter:false"><a onclick="return saveAsSVG('svg',this);" href="#" download="kiwi.svg">Download</a></pre>And again you can try this for yourself using this <a onclick="return saveAsSVG('svg',this);" href="#" download="kiwi.svg">live version</a> of the above. Now this should work in all browsers that support these new HTML features which, according to the excellent <a href="http://caniuse.com/">Can I Use...</a> website, should be most modern browsers. The only remaining issue is that the kiwi and grass are still both black in the downloadable SVG.<br />
<br />
The reason the colour information is lost is that it was never part of the SVG to start with. In this example, the SVG is styled using CSS within the web page rather than directly within the SVG. When you download the SVG file you don't get the associated CSS styles and hence the elements revert back to being black. If we had used a self contained SVG image then everything would work and we would have no need to go any further. Fortunately, even when we style the SVG using CSS it is fairly easy to merge the styles into the SVG before the user downloads the image. In this example the page contains the following CSS style element.<br />
<pre class="brush:html"><style title="kiwi" type="text/css">
.ground {
fill: #94d31b;
}
.kiwi {
fill: #C19A6B;
}
</style></pre>Fortunately it is easy to add CSS styles to the SVG image before creating the blob object. If we assume that there is a function that returns a DOM <code>style</code> element then we can extend our existing function to embed this into the SVG as follows:<br />
<pre class="brush:javascript">function saveAsStyledSVG(id, styles, link) {
if (link.href != "#") {
window.URL.revokeObjectURL(link.href);
}
var svg = document.getElementById(id).cloneNode();
svg.insertBefore(getStyles(styles),svg.firstChild);
var content = svg.outerHTML;
if (content == undefined) {
var div = document.createElement("div");
div.appendChild(svg);
content = div.innerHTML;
}
var blob = new Blob([content], {type: "image/svg+xml"});
link.href = window.URL.createObjectURL(blob);
return true;
}</pre>Here you can see that on line 8 we call a function <code>getStyles</code> to get the DOM element and then append this as the first child of the root element of the SVG document, but the rest of the function is identical (other than the name as we can't overload functions in JavaScript). Now all we need to do is to define the <code>getStyles</code> function, which we do as follows:<br />
<pre class="brush:javascript">function getStyles(names) {
// create an empty style DOM element
var styles = document.createElement("style");
// set the type attribute to text/css
styles.setAttribute("type","text/css");
// get easy access to the Array slice method
var slice = Array.prototype.slice;
for (var i = 0 ; i < document.styleSheets.length ; ++i) {
// for each style sheet in the document
var sheet = document.styleSheets[i];
if (names == undefined || names.indexOf(sheet.title) != -1) {
// if we are including all styles or this sheet is one we want
slice.call(document.styleSheets[i].cssRules).forEach(
// slice the style sheet into separate rules
function(rule){
// create a new text node with the text of each CSS rule
var text = document.createTextNode("\n"+rule.cssText);
// add the rule to the style sheet we are building
styles.appendChild(text);
}
);
}
}
// return the completed style sheet
return styles;
}</pre>Hopefully the comments make it clear how this works, but essentially is finds all the rules within a named set of style sheets and adds them to a newly created <code>style</code> DOM element. It's worth noting that it is much safer to explicitly specify the style sheets you want to use because a) it will be quicker, b) the resultant file will be smaller but most importantly c) you can't access the content of style sheets loaded from another domain than the current page as this goes against the browser security model.<br />
<br />
So given these new functions we can now use a link defined as follows to allow the kiwi to be downloaded in all it's glorious colours:<br />
<pre class="brush:html;gutter:false"><a onclick="return saveAsStyledSVG('svg',['kiwi'],this);" href="#" download="kiwi.svg">Download</a></pre>And again here is a <a onclick="return saveAsStyledSVG('svg',['kiwi'],this);" href="#" download="kiwi.svg">live version</a> of the above for you to try, which should allow you to download the full colour SVG.<br />
<br />
Hopefully you have found this a useful and interesting exploration of some of the newer features of HTML, but it is worth noting that the techniques discussed could be used for downloading a whole range of different file types without requiring server interaction.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com0tag:blogger.com,1999:blog-7025390077222270096.post-4117924968750458622013-11-30T12:02:00.000+00:002013-11-30T12:02:09.389+00:00Why Aren't They Spamming The Chinese?<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEji-5fCSryyIcWx1oSdt4cm3e0NaJX8XWE40olThAaISVYSdCV6yLX9bRJnrDfpaVsPSBDTqJOgcFr-xqAsaXt_Sfw-JwdIEBK4PBDx9cJ2El7QTAcrmFwUogJjGHn9hyphenhyphenK0UQUNriE3Z7w/s1600/chinese-spam.jpeg" imageanchor="1" style="clear: left; float: right; margin-bottom: 0em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEji-5fCSryyIcWx1oSdt4cm3e0NaJX8XWE40olThAaISVYSdCV6yLX9bRJnrDfpaVsPSBDTqJOgcFr-xqAsaXt_Sfw-JwdIEBK4PBDx9cJ2El7QTAcrmFwUogJjGHn9hyphenhyphenK0UQUNriE3Z7w/s320/chinese-spam.jpeg" width="300" /></a></div>Whilst trying to drink my first cup of coffee this morning, I was rudely interrupted by click-jacking malware affecting my wife’s computer. All she was trying to do was look at some Google search results, but clicking on them would take her to a suspicious looking shopping search site. From a little bit of Googling it looked as if it might be a real nasty trojan which would have taken ages to clean up. Fortunately it turned out that all the pages she was having the problem with had been infected with the same bit of malicious JavaScript. I'm not sure how (probably through a malicious banner ad or something) but a reference to the following JavaScript had been inserted at the very end (after the <code></html></code>) of each affected page:<br />
<pre class="brush:javascript">if (navigator.language)
var language = navigator.language;
else
var language = navigator.browserLanguage;
if(language.indexOf('zh') == -1) {
var regexp = /\.(aol|google|youdao|yahoo|bing|118114|biso|gougou|ifeng|ivc|sooule|niuhu|biso|ec21)(\.[a-z0-9\-]+){1,2}\//ig;
var where = document.referrer;
if (regexp.test(where)) {
window.location.href="http://www.bbc.co.uk/news";
}
}</pre>To make the script easier to read I've reformatted it, and replaced the redirect with a safe URL (who doesn't trust the BBC?) rather than giving the spammers free advertising, but I haven't changed any of the functional aspects of the script.<br />
<br />
Essentially all it does is check the URL that you were on when you clicked the link leading you to the current page, and if that looks like a search results page from one of 14 different companies, then it redirects you. The regular expression it uses to check the referring page is simple yet effective and will catch any of the sub-domains of these search services as well. What I find weird is why the script checks the language of the browser.<br />
<br />
The first four lines of the script get the language the browser is using. There are two ways of doing this depending on which browser you are using hence the <code>if</code> statement. On my machine this gets me <code>en-US</code> (which means I need to figure out why it has switched from <code>en-UK</code> which is what I thought I'd set it to). Line 6 then checks to make sure the language doesn't include the string <code>zh</code>, which according to <a href="http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes">Wikipedia</a> is Chinese. I'm assuming that the spammers behind the script are Chinese and don't want to be inconvenienced by their own script, but it seems odd, especially when you consider that at least one of the search engines covered by the regular expression (118114 on many different top-level domains) seems to be a Chinese site.<br />
<br />
Looking at this script there is of course another way to defeat it, other than disabling JavaScript. One of the privacy or security options in most browsers concerns the referer (yes I know it is spelt wrong, but that is the correct spelling in the HTTP spec) header. Essentially this header tells a web server the page you were on when you clicked the link leading to the page you are requesting. Some sites will use this to provide functionality so disabling it can cause problems but it does mitigate against scripts like this one. Because it can cause problems it's often an advanced setting, for example here are <a href="http://kb.mozillazine.org/Network.http.sendRefererHeader">the details for Firefox</a>.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com0tag:blogger.com,1999:blog-7025390077222270096.post-2198108055846054202013-11-09T15:33:00.000+00:002013-11-09T15:33:02.768+00:00Serializing To A Human Interface DeviceIf you've read <a href="http://englishjavadrinker.blogspot.co.uk/2013/10/the-caffeine-button.html">my previous post</a> you'll know that I've been looking at a cheap and simple way of adding serial communication to a breadboard Arduino clone (such as <a href="http://englishjavadrinker.blogspot.co.uk/2012/12/double-speed-for-just-35p.html">this one</a>). To summarise the situation so far; adding true <a href="http://en.wikipedia.org/wiki/Rs-232">RS-232 serial communication</a> is both expensive and difficult as the required part is only available as a surface mount component but I discovered V-USB which allows me to emulate low speed USB devices. The end result was that I managed to use V-USB to emulate a USB keyboard. Being able to pass data from the Arduino to the PC by simply emulating key presses is useful but a) it is rather slow and b) different keyboard mappings lead to different characters being and typed and more importantly c) it doesn't allow me to send data to the Arduino. So on we go...<br />
<br />
Let's start with what I haven't managed to achieve; a <a href="http://en.wikipedia.org/wiki/USB_communications_device_class">USB CDC ACM device</a> for RS-232 communication. Unfortunately CDC ACM devices require bulk endpoints (these allow for large sporadic transfers using all remaining available bandwidth, but with no guarantees on bandwidth or latency) and these are only officially supported for high speed USB devices. V-USB only allows me to emulate low speed USB devices, and while most operating systems used to allow low speed devices to create bulk endpoints, even though this is contrary to the spec, modern versions of Linux (and possibly Windows) do not. I did manage to get a device configured correctly but as soon as I plugged it in the bulk endpoints were detected and converted to interrupt endpoints which stopped the device from working. However, all is not lost as I do have a solution which I think is just as good; serializing data to and from a generic <a href="http://en.wikipedia.org/wiki/USB_human_interface_device_class">USB Human Interface Device</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://geektechntings.wordpress.com/2011/10/25/principles-of-computer-graphics-mock-brief-results/" imageanchor="1" style="clear: left; float: left; margin-top: 0.5em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwOka82WPlZnJx0d18rhuVkc-jBAkzdDqmssE5f5WhP7O0doZz7zl1PhxLjtyrijvESBf_3OQ9IggPZ_d9TTPWtUUoz7aoltDjQT2KeLlW840yldTgDw4kt4aNM7iTXBlmn8fdjl4aBC8/s320/usb-arm-sockets.jpg" /></a></div>The USB specification defines the USB Human Interface Device (HID) class to support, as the name suggests, devices with some form of human interface. This doesn't mean sticking a USB cable into your arm, but rather defines common devices such as keyboards, mice and game controllers as well as devices like exercise machines, audio controllers and medical instruments. While such devices may communicate data in a variety of forms it all passes to and from the device using the same protocol. This means that when you plug any such device into practically any computer with a USB port it will be recognised and basic drivers will be loaded.<br />
<br />
Writing code to communicate with a USB HID device isn't that much more complex than interfacing with a classic serial port and given the standard driver support we can rely on the operating system taking care of most of the communication for us.<br />
<br />
For what follows I'm assuming the same basic USB circuit that I <a href="http://englishjavadrinker.blogspot.co.uk/2013/10/the-caffeine-button.html">described in the previous post</a> as we know it works and it is cheap to build.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh45UrmMqUb1Hz0l3mESW8gy5QlobAXast6hT3oR3dmtc4AV_abbRMCfqf_w63xJHqzCKvsgwFpKZD265O5tDVg_hprIVlhGwg5d-UjdICtRTMJs2Fji7gQqXNeKu77a77RLmaYxThxrnY/s1600/breadboard.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh45UrmMqUb1Hz0l3mESW8gy5QlobAXast6hT3oR3dmtc4AV_abbRMCfqf_w63xJHqzCKvsgwFpKZD265O5tDVg_hprIVlhGwg5d-UjdICtRTMJs2Fji7gQqXNeKu77a77RLmaYxThxrnY/s320/breadboard.png" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgg6fdG-J_GrtnulkcVLt6JKGrAHZjQ0PSV-Vt4ghXow7oMkiD7OBSqsUVP_jITgk03HU0BnTYuP5BiGTXz-CDnkx6zPfoFaQXgDrcs1eZ7O6Pkt9zoFeYkLwFr7VytZZeSiP2nAZqBnjE/s1600/schematic.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgg6fdG-J_GrtnulkcVLt6JKGrAHZjQ0PSV-Vt4ghXow7oMkiD7OBSqsUVP_jITgk03HU0BnTYuP5BiGTXz-CDnkx6zPfoFaQXgDrcs1eZ7O6Pkt9zoFeYkLwFr7VytZZeSiP2nAZqBnjE/s320/schematic.png" /></a></div><br />
Now we have the circuit let's move on to the software we need to write. Unlike with the USBKeyboard library, that powered <a href="http://englishjavadrinker.blogspot.co.uk/2013/10/the-caffeine-button.html">The Caffeine Button</a>, we will need both firmware for the Arduino and host software that will run on the PC and interface with the basic HID drivers the operating system provides. Given that you can't test the host software until we have a working device we'll start by looking at the firmware.<br />
<br />
The first thing you have to do when constructing a HID is to define its descriptor. The descriptor is how the device presents itself to the operating system and defines the type of device as well as the size and type of any communication messages. Now you will probably never need to edit this but I thought it was worth showing you the full descriptor we are using:<br />
<pre class="brush:c">PROGMEM char usbHidReportDescriptor[USB_CFG_HID_REPORT_DESCRIPTOR_LENGTH] = {
0x06, 0x00, 0xff, // USAGE_PAGE (Generic Desktop)
0x09, 0x01, // USAGE (Vendor Usage 1)
0xa1, 0x01, // COLLECTION (Application)
0x15, 0x00, // LOGICAL_MINIMUM (0)
0x26, 0xff, 0x00, // LOGICAL_MAXIMUM (255)
0x75, 0x08, // REPORT_SIZE (8)
0x95, OUT_BUFFER_SIZE, // REPORT_COUNT (currently 8)
0x09, 0x00, // USAGE (Undefined)
0x82, 0x02, 0x01, // INPUT (Data,Var,Abs,Buf)
0x95, IN_BUFFER_SIZE, // REPORT_COUNT (currently 32)
0x09, 0x00, // USAGE (Undefined)
0xb2, 0x02, 0x01, // FEATURE (Data,Var,Abs,Buf)
0xc0 // END_COLLECTION
};</pre>In this descriptor we define two reports of different sizes which we will use for transferring data to and from the device. The first thing to point out is that the specification defines input and output with respect to the host PC and not the device. So an input message is actually used for writing out from the device rather than for receiving data. Given this, we can see that the descriptor defines two message types. Firstly (lines 7 to 10) we define an 8 byte (<code>OUT_BUFFER_SIZE</code> is defined as 8) input report (the size is defined in bits so we have 8 bits times the count to give 8 bytes) which means we can write 8 byte blocks of data back to the PC we are connected to. The second message type is defined as a <code>FEATURE</code> message of 32 bytes (because <code>IN_BUFFER_SIZE</code> is defined as 32 and the <code>REPORT_SIZE</code> hasn't been redefined so it still 8 bits) which we will use for passing data from the PC to the USB device. As I said you will probably never need to edit this structure especially as you can tweak the message sizes, if necessary, by adjusting the two constants instead. If you do decided to change the descriptor it is worth noting that some operating systems are more forgiving than others. For example, with Linux if you have a defined a message of 8 bytes but only have two to send then you can do that and everything will work. Under Windows, however, if you only send two bytes the device will simply stop functioning altogether so you will need to pad the message to be exactly 8 bytes. This also means that it is easy to check that your descriptor matches what you are actually doing by quickly testing under Windows (I've been doing this with a copy of Windows XP running under <a href="https://www.virtualbox.org/">VirtualBox</a>).<br />
<br />
Now that we know the size of the messages we will send and receive we still need to decide upon their format, i.e. the protocol we will use for our data that we are sending on top of the USB protocol. If we were only interested in sending textual data then we could send null terminated data (i.e. put a zero value byte into the array after the last byte of data), but if we want to send arbitrary bytes then using 0 as an end of data marker seems an odd choice. For this reason I've opted to set the first byte of each message to the length of the valid data in the array. This is both simple to use and results in firmware code that is slightly simpler (and hence smaller) than checking for the null terminator. This does of course mean that in an 8 byte message we can only fit 7 bytes of actual data plus the length marker (this is no different than with null terminated data of course). If you know the messages you want to send will always be of a fixed length then tweaking the buffer sizes to suit might make for a more efficient transfer of data. In general, as you will see shortly, as a user of the library a lot of these details are dealt with for you.<br />
<br />
From the very beginning my aim was to find a drop-in replacement for the standard <a href="http://arduino.cc/en/reference/serial">Arduino Serial</a> object and so I've made the USBSerial library implement the same <a href="http://arduino.cc/en/Reference/Stream">Stream interface</a>. This means you can use any of the methods defined in the Stream interface for reading and writing data and the details about buffer sizes etc. are hidden within the library.<br />
<br />
To show how easy the library is to use, here is a simple example where the sketch simply echos back any bytes that it is sent.<br />
<pre class="brush:c">#include <USBSerial.h>
char buffer[IN_BUFFER_SIZE];
void setup() {
USBSerial.begin();
}
void loop() {
USBSerial.update();
if(USBSerial.available() > 0) {
int size = USBSerial.readBytes(buffer, IN_BUFFER_SIZE);
if (size > 0) {
USBSerial.write((const uint8_t*)buffer, size);
}
}
}</pre>Note that I've used the same <code>IN_BUFFER_SIZE</code> constant in this example as within the library itself, as there is no reason to define a buffer that is bigger than we can ever expect to fill. The only line that you wouldn't find in a similar example using the standard Serial object is line 10 where we make sure that the USB connection is up to date (you need to do this approximately once every 50ms to keep the connection alive). Before we move on to looking at the host software there are a few things you need to know before trying to use the library.<br />
<br />
Unfortunately the V-USB part of the library needs customizing for each project, so you can't simply drop the library into the Arduino sketchbook folder, as the USB manufacturer and product identifiers have to be unique for different devices. These identifiers are set in the <code>usbconfig.h</code> file. V-USB doesn't actually provide a copy of <code>usbconfig.h</code> what is provided is a file called <code>usbconfig-prototype.h</code> which you can copy and rename as a starting point. I've already done a lot of the configuration for you by editing <code>usbconfig-prototype.h</code> leaving just four lines you need to edit for yourself. Firstly you need to set the vendor name property by editing lines 244 and 245:<br />
<pre class="brush:c; first-line: 244;">#define USB_CFG_VENDOR_NAME 'o', 'b', 'd', 'e', 'v', '.', 'a', 't'
#define USB_CFG_VENDOR_NAME_LEN 8</pre>and then the device name by editing lines 254 and 256:<br />
<pre class="brush:c first-line: 254;">#define USB_CFG_DEVICE_NAME 'T', 'e', 'm', 'p', 'l', 'a', 't', 'e'
#define USB_CFG_DEVICE_NAME_LEN 8</pre>These values have to be changed and can't be set to any random value because as part of the V-USB license agreement you need to conform to the following rules (taken verbatim from the file <code>USB-IDs-for-free.txt</code>):<br />
<br />
<blockquote>(2) The textual manufacturer identification MUST contain either an Internet domain name (e.g. "mycompany.com") registered and owned by you, or an e-mail address under your control (e.g. myname@gmx.net"). You can embed the domain name or e-mail address in any string you like, e.g. "Objective Development http://www.obdev.at/vusb/".<br />
<br />
(3) You are responsible for retaining ownership of the domain or e-mail address for as long as any of your products are in use.<br />
<br />
(4) You may choose any string for the textual product identification, as long as this string is unique within the scope of your textual manufacturer identification.</blockquote>Once properly configured you should be able to compile (<a href="http://englishjavadrinker.blogspot.co.uk/2013/07/arduino-without-ide.html">I recommend using arduino-mk</a> instead of the Arduino IDE) and use the library without issue, and without understanding how it actually works internally (if you are interested in the details then both my code and the V-USB library contain vast amounts of code comments which should help you get a better understanding) so let's move on to looking at the host software.<br />
<br />
As I've already mentioned connecting the device to a PC usually causes generic HID drivers to be loaded by the operating system. This means that you should be able to use any programming language you like to write the host software as long as it can talk to the generic USB drivers. I've included host software written in Java using <a href="http://code.google.com/p/javahidapi/">javahidapi</a> but, for instance, you could also use <a href="http://sourceforge.net/apps/trac/pyusb/">PyUSB</a> if you prefer to program using Python. The important thing to remember is the protocol for passing data that we defined earlier: data to the USB device is sent as 32 byte feature requests with the first byte being the length of the valid data in the rest of the array, while data from the USB device is in 8 byte chunks again with the first byte being the length of the valid data.<br />
<br />
As with the firmware code we have already discussed, I've written a simple Java library to hide most of the details behind standard interfaces, which allow you to read and write data using the standard Java <a href="http://docs.oracle.com/javase/7/docs/api/java/io/InputStream.html">InputStream</a> and <a href="http://docs.oracle.com/javase/7/docs/api/java/io/OutputStream.html">OutputStream</a> interfaces. Full details of the available methods can be found in the <a href="http://greenwoodma.servehttp.com/jenkins/view/All/job/USBSerial%20For%20Arduino/javadoc/">Javadoc</a> but a simple echo example shows most of the important details.<br />
<pre class="brush:java">import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.PrintStream;
import englishcoffeedrinker.arduino.USBSerial;
public class SimpleEcho {
public static void main(String[] args) throws Exception {
// get an instance of the USBSerial class for the specified device
USBSerial serial =
USBSerial.getInstance("englishcoffeedrinker.co.uk", "EchoChamber");
// open the underlying USB connection to the device
serial.connect();
// create an output stream to write characters to the device
PrintStream out = new PrintStream(serial.getOutputStream());
// send a simple message
out.println("hello world!");
// ensure the message has been sent and not buffered internally somewhere
out.flush();
// create a reader for getting characters back from the device
BufferedReader in =
new BufferedReader(new InputStreamReader(serial.getInputStream()));
String line;
while((line = in.readLine()) == null) {
// keep checking the device until a line of text is returned
}
// display the message sent from the device
System.out.println(line);
// we have finished so disconnect our connection to the device
serial.disconnect();
}
}</pre>Essentially, lines 10 to 14 get an instance of the USBSerial class for a specific device, in this case found via the manufacturer and product identifiers although other methods are available, and then opens the connection. Lines 16 to 23 then use the OutputStream to write data to the device while lines 26 to 35 read it back, with line 38 cleaning up by closing the connection. For anyone who is happy programming in Java this should look no different than reading or writing to and from any other type of stream, which means it should be easy to integrate within any project where you want to communicate with an Arduino.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgp5oPtV0LH-95g8U3o6AieXdMjLXSqXTk1Kty1W5P-YqiyUlXvOJYO7jBcwNCipzHyt0lYtLUJrf5c_im4y4d52JaS2QYZ84-NVIFAoPKGAFN-6HhyphenhyphensmkRE-PZHKhErLjdrcvJp2iwpK4/s1600/USBSerialMonitor.png" imageanchor="1" style="clear: left; float: left; margin-top: 0.5em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgp5oPtV0LH-95g8U3o6AieXdMjLXSqXTk1Kty1W5P-YqiyUlXvOJYO7jBcwNCipzHyt0lYtLUJrf5c_im4y4d52JaS2QYZ84-NVIFAoPKGAFN-6HhyphenhyphensmkRE-PZHKhErLjdrcvJp2iwpK4/s1600/USBSerialMonitor.png" /></a></div>To make life a little easier I've also included a slightly more useful demo application that effectively reproduces the Serial Monitor from the Arduino IDE. You can see it running here connected to a device running the simple echo sketch from earlier in this post, but it should work with any device that uses the USBSerial library.<br />
<br />
I've included another example with the USBSerial library that shows you can use this for more than just echo commands. This is the CmdMsg example, which is an example <a href="http://englishjavadrinker.blogspot.co.uk/2012/06/message-passing.html">I've talked about before on this blog</a>, but this version uses the USBSerial library, and hence can be controlled through this new USB Serial Monitor, rather than using the standard Serial library.<br />
<br />
If you've read all the way to here then I'm guessing you might want to know where you can get the library from, well it is available under the GPL licence (a restriction imposed because I'm using the free USB vendor and product IDs from Objective Development) from my <a href="http://greenwoodma.servehttp.com/svn/repos/open-source/list/blog-code/arduino/libraries/USBSerial/">SVN repository</a>. Do let me know if you find it useful or if you have any suggestions for improvement.<br />
<br />
When I set out to try and add serial support to a breadboarded Arduino (specifically <a href="http://englishjavadrinker.blogspot.co.uk/2012/12/double-speed-for-just-35p.html">this circuit</a>) I did have a device I wanted to build in mind, so I'm sure at some point I'll blog again about using this library in a real device rather than just the simple examples included with the library that do nothing more than prove everything works.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com1tag:blogger.com,1999:blog-7025390077222270096.post-44143756456328857592013-10-26T18:04:00.000+01:002013-10-27T10:13:31.305+00:00The Caffeine Button<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7IAZu2fFe6m4rHkzFDF19z2zLrzhRMk2jmoYGMCNpOoak_KrzEy7dURjcHcLBqf5lDUmK3bTDPjZsRKt1VtYyLEZkzK6ckZrXokQwwgBd2Bqt3X3m7w84TYtzxKtCCGsf5Rvfp16p8Hw/s1600/button_push.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 0em; margin-left: 0.5em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7IAZu2fFe6m4rHkzFDF19z2zLrzhRMk2jmoYGMCNpOoak_KrzEy7dURjcHcLBqf5lDUmK3bTDPjZsRKt1VtYyLEZkzK6ckZrXokQwwgBd2Bqt3X3m7w84TYtzxKtCCGsf5Rvfp16p8Hw/s200/button_push.png" width="200" /></a></div>In a couple of previous posts (<a href="http://englishjavadrinker.blogspot.co.uk/2012/12/honey-i-shrunk-arduino.html">here</a> and <a href="http://englishjavadrinker.blogspot.co.uk/2012/12/double-speed-for-just-35p.html">here</a>) I've shown how easy and cheap it is to go from a prototyped setup using an <a href="http://www.arduino.cc/">Arduino</a> to a standalone circuit built from just a handful of components. While those posts were more of an academic exercise to prove it was possible I've also now built, and <a href="http://caffeine-train.blogspot.co.uk/2013/07/the-flickering-fires-of-hell.html">blogged about</a>, such a circuit that I'm actually using in anger. The problem is that an actual Arduino isn't just an <a href="http://www.atmel.com/devices/atmega328p.aspx">Atmel ATMega328P-PU</a> it also has accompanying electronics which enable you to talk to a computer via the USB connection which is great both for debugging and for interfacing external hardware to a PC.<br />
<br />
When you connect an Arduino to a PC what actually happens is that the supporting circuitry creates a <a href="http://en.wikipedia.org/wiki/USB_communications_device_class">USB CDC ACM device</a> which emulates a good old fashioned <a href="http://en.wikipedia.org/wiki/Rs-232">RS-232 serial port</a>. If I wanted to add similar functionality to my standalone circuits then the most common way of doing so would be to use an <a href="http://uk.farnell.com/ftdi/ft232rl/ic-usb-to-uart-smd-28ssop/dp/1146032RL">FT232RL</a>, but the chip alone would almost double the cost of the circuit plus it is only available as a surface mount part making it difficult to experiment with on a breadboard and I'm not sure my soldering skills are good enough to deal with surface mount parts either.<br />
<br />
After pondering this for a bit and doing a little research I came across a potential solution in the form of <a href="http://www.obdev.at/products/vusb/index.html">V-USB</a>. V-USB is a software only implementation of low speed USB for Atmel microcontrollers, such as the ATMega328P-PU. Unfortunately the distribution doesn't directly support the Arduino (it supports the ATMega328P-PU but not through the Arduino IDE etc.), however, I did find a <a href="http://code.google.com/p/vusb-for-arduino/">previous attempt</a> to add Arduino support although this project seems to have been abandoned as it hasn't seen any updates in over three years. It did, however, give me a good point to start from.<br />
<br />
So far I haven't managed to emulate a serial port, but I have managed to make the Arduino behave like a USB keyboard which means that I can use it for debugging by having it pretend to press lots of keys in sequence, which is better than nothing. Before we get to looking at how to make use of V-USB we need to wire a USB plug up to the Arduino.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh45UrmMqUb1Hz0l3mESW8gy5QlobAXast6hT3oR3dmtc4AV_abbRMCfqf_w63xJHqzCKvsgwFpKZD265O5tDVg_hprIVlhGwg5d-UjdICtRTMJs2Fji7gQqXNeKu77a77RLmaYxThxrnY/s1600/breadboard.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh45UrmMqUb1Hz0l3mESW8gy5QlobAXast6hT3oR3dmtc4AV_abbRMCfqf_w63xJHqzCKvsgwFpKZD265O5tDVg_hprIVlhGwg5d-UjdICtRTMJs2Fji7gQqXNeKu77a77RLmaYxThxrnY/s320/breadboard.png" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgg6fdG-J_GrtnulkcVLt6JKGrAHZjQ0PSV-Vt4ghXow7oMkiD7OBSqsUVP_jITgk03HU0BnTYuP5BiGTXz-CDnkx6zPfoFaQXgDrcs1eZ7O6Pkt9zoFeYkLwFr7VytZZeSiP2nAZqBnjE/s1600/schematic.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgg6fdG-J_GrtnulkcVLt6JKGrAHZjQ0PSV-Vt4ghXow7oMkiD7OBSqsUVP_jITgk03HU0BnTYuP5BiGTXz-CDnkx6zPfoFaQXgDrcs1eZ7O6Pkt9zoFeYkLwFr7VytZZeSiP2nAZqBnjE/s320/schematic.png" /></a></div><br />
<table style="margin-left:1em; clear:both; float: right; text-align:center; border: 1px solid gray;"><tr><th colspan="3">USB Connection Parts List</th></tr>
<tr><th>Part</th><th>Unit Cost</th><th>Quantity</th></tr>
<tr><td><a href="http://uk.farnell.com/jsp/search/productdetail.jsp?SKU=1308876"> USB Socket, Type B</a></td><td>£0.50</td><td>1</td></tr>
<tr><td><a href="http://uk.farnell.com/jsp/search/productdetail.jsp?SKU=1700904">3.6V, 0.5W Zener Diode</a></td><td>£0.071</td><td>2</td></tr>
<tr><td><a href="http://uk.farnell.com/jsp/search/productdetail.jsp?SKU=1700221">68Ω Resistor</a></td><td>£0.008</td><td>2</td></tr>
<tr><td><a href="http://uk.farnell.com/jsp/search/productdetail.jsp?SKU=1700241">2.2kΩ Resistor</a></td><td>£0.008</td><td>1</td></tr>
</table>As you can see it isn't a particularly complex circuit to build. Essentially we have the two data lines D- and D+ linked to pins 2 and 4 of the Arduino and as the USB spec states that these run on 3.3V we restrict the voltage using two 3.6V zener diodes to step down from the 5V output of the Arduino. The Arduino itself is powered from the other two USB pins which provide 5V/GND (this also means that you could use USB to power a standalone ATMega328P-PU without needing a voltage regulator and smoothing capacitors). The final connection is between pin 5 and D- via a 2.2kΩ pull-up resistor which allows the connection to be connected/disconnected from within software (in theory you can link this to V+ instead if you don't need this flexibility but I found that this meant that the hardware wasn't always recognized correctly when it was connected to a PC). For the USB plug itself, I opted to use a Type B plug (the same as the Arduino) so that I could get away with a single cable snaking across my desk, but you should be able to use any USB plug with the same circuit. One thing to be careful with when constructing the circuit is that in most cases the metal case of the USB plug is connected to the ground pin, so be careful that you don't end up with any of the other connections catching the case otherwise you might end up with them pulled to ground which will cause weird things to happen; depending which pin is involved either your PC won't recognize you have anything plugged in or it might end up thinking you have a high speed or high power device connected and things won't work properly.<br />
<br />
Now we have the hardware what we need is some working software we can upload to the Arduino. As I said above I'm using a <a href="http://code.google.com/p/vusb-for-arduino/">previous attempt</a> to get this working as my starting point. This code hasn't been updated for over three years and I'm guessing that a number of things have changed within the Arduino libraries since then as the code didn't just work. After a bit of trial and error I discovered that the problems were mostly related to timer code that had been added to get V-USB to work with the Arduino and which was now doing the exact opposite. Removing this code got me to a point where when I plugged in the cable the Arduino was now being recognized as a USB v1.01 HID Keyboard, which I know is my device because of the vendor and product strings being shown in this screenshot.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiIBaYbq2Xqsur8xfWZHNivBYFBYvuAQu5nvCC4YBioWE8KKCTCBNmlZWdtODz5oklZTbeVRvjcy79EpRVEfqk-gwr0wyfHFIiVw2ltTv2xyoj6GFFmJmdXtKtxYM8hZiWwMriaDJAbT8/s1600/caffeinebutton-kernel.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="51" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiIBaYbq2Xqsur8xfWZHNivBYFBYvuAQu5nvCC4YBioWE8KKCTCBNmlZWdtODz5oklZTbeVRvjcy79EpRVEfqk-gwr0wyfHFIiVw2ltTv2xyoj6GFFmJmdXtKtxYM8hZiWwMriaDJAbT8/s640/caffeinebutton-kernel.png" width="640" /> </a></div><br />
Having got the code running I then took the plunge to update the version of V-USB being used to the latest version (currently 20121206) which, after a few little tweaks, was a success. Unfortunately because of the way the library has to be configured a single copy can't be shared between multiple projects, but in the rest of this post I'll explain how to customize the library and show you a fully worked example: TheCaffeineButton.<br />
<br />
Firstly I've had problems compiling the library through the Arduino IDE, so I'd recommend compiling any sketches that use the library via the <a href="http://ed.am/dev/make/arduino-mk"><code>arduino-mk</code></a> project that I discussed in a <a href="http://englishjavadrinker.blogspot.co.uk/2013/07/arduino-without-ide.html">previous post</a>. This allows you to have libraries local to a sketch by putting them in a <code>libs</code> subfolder. A basic sketch to use the libary then looks like the following:<br />
<pre class="brush:c">// pull in the USB Keyboard library
#include <USBKeyboard.h>
void setup() {
// TODO: setup like stuff in here...
}
void loop() {
// poll the USB connection
USBKeyboard.update();
// TODO: whatever you want in here...
}</pre>You simply include the library (line 2) and then poll the connection every time through the main loop (line 6). As I mentioned though, the library needs customizing for each project and so if you try and compile the sketch at this point you'll end up with an error:<br />
<pre class="brush:bash; gutter:false">libs/USBKeyboard/usbdrv.h:12:23: fatal error: usbconfig.h: No such file or
directory</pre>V-USB doesn't provide a copy of <code>usbconfig.h</code> as it is project specific, what they do provide is a file called <code>usbconfig-prototype.h</code> which you can copy and rename as a starting point. I've already done a lot of the configuration for you by editing <code>usbconfig-prototype.h</code> leaving just four lines you need to edit for yourself. Firstly you need to set the vendor name property by editing lines 244 and 245:<br />
<pre class="brush:c; first-line: 244;">#define USB_CFG_VENDOR_NAME 'o', 'b', 'd', 'e', 'v', '.', 'a', 't'
#define USB_CFG_VENDOR_NAME_LEN 8</pre>and then the device name by editing lines 254 and 256:<br />
<pre class="brush:c first-line: 254;">#define USB_CFG_DEVICE_NAME 'T', 'e', 'm', 'p', 'l', 'a', 't', 'e'
#define USB_CFG_DEVICE_NAME_LEN 8</pre>These values have to be changed and can't be set to any random value because as part of the V-USB license agreement you need to conform to the following rules (taken verbatim from the file <code>USB-IDs-for-free.txt</code>):<br />
<br />
<blockquote>(2) The textual manufacturer identification MUST contain either an Internet domain name (e.g. "mycompany.com") registered and owned by you, or an e-mail address under your control (e.g. myname@gmx.net"). You can embed the domain name or e-mail address in any string you like, e.g. "Objective Development http://www.obdev.at/vusb/".<br />
<br />
(3) You are responsible for retaining ownership of the domain or e-mail address for as long as any of your products are in use.<br />
<br />
(4) You may choose any string for the textual product identification, as long as this string is unique within the scope of your textual manufacturer identification.</blockquote>If you glance back at the screenshot I showed earlier of my example device being recognized then you can see that I followed these rules by setting the vendor name to <a href="http://englishcoffeedrinker.co.uk">englishcoffeedrinker.co.uk</a> and the device name to <code>TheCaffeineButton</code>.<br />
<br />
So having created a valid <code>usbconfig.h</code> file you can now compile the basic sketch shown above. Of course this does nothing other than allow the Arduino to be recognized as a USB keyboard. If you want to actually do something interesting then you need a little more code. As an example, I'll finally introduce you to The Caffeine Button!<br />
<br />
Caffeine (usually in the form of coffee) is my one true addiction and so I thought I'd build a device with a single button that when pressed types out the chemical formula for caffeine: C<sub>8</sub>H<sub>10</sub>N<sub>4</sub>O<sub>2</sub>. The full sketch is fairly simple and assumes the basic circuit above with a push button between pin 12 and ground (it uses the internal pull-up resistor to keep the part list down to the single button):<br />
<pre class="brush:c">// pull in the USB Keyboard library
#include <USBKeyboard.h>
// using Bounce makes working with buttons nice and easy
// http://www.arduino.cc/playground/Code/Bounce
#include <Bounce.h>
// we have the button on pin 12
#define BUTTON_PIN 12
// create a Bounce instance to manage the button
Bounce button = Bounce(BUTTON_PIN, 5);
void setup() {
// initialise the pin the button is connected to
pinMode(BUTTON_PIN, INPUT);
// enable the internal pull-up resistor
digitalWrite(BUTTON_PIN, HIGH);
}
void loop() {
// poll the USB connection
USBKeyboard.update();
// check the status of the button
button.update();
if (button.fallingEdge()) {
// if the button has just been pressed then...
// ...print out the formula for caffeine
USBKeyboard.print("C8H10N4O2");
}
}</pre>There are actually four different methods you can use to emulate pressing keys:<br />
<pre class="brush:c;gutter:false">void write(byte keycode);
void write(byte keycode, byte modifiers);
void print(const char* text);
void println(const char* text);</pre>The first two method send a USB key usage code (with or without a modifier, such as the shift key) and then release the key, while the last two methods translate alphanumeric characters (and space) into a sequence of keystrokes (followed by the enter key in the <code>println</code> case) to make the library a little easier to use. Unfortunately there is no guarantee that my simple sketch will always result in C8H10N402 being displayed when you press the button.<br />
<br />
When you press a key on a keyboard a keycode is sent to the computer which determines which letter has been pressed. This allows the same physical keyboard to be used for different languages simply by changing the printed labels on each key. Unfortunately this makes it impossible to translate a string into a sequence of keycodes which will always display the same on every computer. For example, if you send keycode 28 on a computer with an English keyboard mapping you'll get a 'y', but if the keyboard mapping is German you'll get a 'z'. I've defined a bunch of constants in <code>USBKeyboard.h</code> (i.e. <code>KEY_A</code>) which work for English, and I've used the same mapping in the <code>print</code> and <code>println</code> methods. If you can set the keyboard mapping for the device to English then these methods will work properly for you, if not you might need to tweak the mappings to get what you need. You can find the full list of mappings in <a href="http://www.usb.org/developers/devclass_docs/Hut1_11.pdf#page=53">chapter 10 of the Universal Serial Bus HID Usage Tables</a> document should you need more details.<br />
<br />
<iframe width="420" height="236" style="float:left; margin-right:1em; margin-bottom: 0.5em; margin-top: 0.5em;" src="//www.youtube-nocookie.com/embed/_pijVkGqzXc?rel=0" frameborder="0" allowfullscreen></iframe>So here we have the final working item. As you can see I built the main USB circuit onto a prototyping shield so I can experiment with lots of different circuits without having to keep recreating the basics every time, and in this case have simply jammed the button between pins 12 and ground.<br />
<br />
If you've read all the way to here then I'm guessing you might want to know where you can get the library from, well it is available under the GPL licence (a restriction imposed because I'm using the free USB vendor and product IDs from Objective Development) from my <a href="http://greenwoodma.servehttp.com/svn/repos/open-source/list/blog-code/arduino/libraries/USBKeyboard/">SVN repository</a>.<br />
<br />
Whilst I haven't yet been able to emulate a serial port I haven't given up and when/if I'm successful I'm sure there will be a post about it and another Arduino library for you to play with.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com2tag:blogger.com,1999:blog-7025390077222270096.post-48011330574707935082013-10-20T10:14:00.000+01:002013-10-20T10:14:26.102+01:00#newsHACK<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTuQo_dis6aPzvYlfSmqWkj6dauFhJHaDht5mwlMtB7uwlC9MLTDkP3CZFEoKlVgsh2kQtgWLIAqKcz_roZa_4WeIYt2ShQf3SLVe6zedqLV3lR2TXHbtC0mlPxwWH8bhDbWWYBE4Pa9o/s1600/newshack.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTuQo_dis6aPzvYlfSmqWkj6dauFhJHaDht5mwlMtB7uwlC9MLTDkP3CZFEoKlVgsh2kQtgWLIAqKcz_roZa_4WeIYt2ShQf3SLVe6zedqLV3lR2TXHbtC0mlPxwWH8bhDbWWYBE4Pa9o/s1600/newshack.jpg" /></a></div>Last week I spent two interesting days at the BBC's <a href="http://newshack.co.uk/">#newsHACK</a> event at <a href="http://shoreditchtownhall.com/">Shoreditch Town Hall</a> in London. The BBC have recently started to open up APIs to a lot of useful indexing and annotation work that they have been doing and this event was aimed at getting news organizations and other interested parties using some of the APIs to produce interesting tech demos.<br />
<br />
Our team consisted of Ian Roberts, Dominic Rout and myself from the University of Sheffield and Helen Lippell from the Press Association. We were kind of the odd ones out as our expertise centres around processing large amounts of text to expose interesting information and to make that searchable or useable in some way; which is exactly what some of the BBC APIs already did. None of us claim to be great user interface peopel so there was no point us trying to generate a really fancy interface over the BBC APIs. In the end we decided that we would play to our strengths and so our demo (which you can <a href="https://annomarket-demo.services.gate.ac.uk/newshack/">go and play with</a>) tried to go one step further than the existing BBC APIs by using the power of Mímir to allow for complex searches over text, annotations and Linked Open Data to allow journalists a deeper view into the news archive when writing a story. One of our <a href="https://annomarket-demo.services.gate.ac.uk/newshack/index.jsp?query=%28%28{Money+value+%3E+5000000}+AND+bonus%29+IN+{Sentence}%29">examples</a> (that seemed to go down well) was; imagine you are writing a story about a CEO who has just been given a £5m bonus and you want to find other people who have been awarded more in the past.<br />
<br />
Our demo didn't win any of the categories but we certainly didn't embarrass ourselves and there were some really good ideas presented. It's worth looking at the <a href="http://newshack.co.uk/hacks/">list of hacks</a> as some of them have really cool demos you can play with.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com0tag:blogger.com,1999:blog-7025390077222270096.post-65504315386607686392013-08-04T17:31:00.000+01:002013-08-04T17:31:54.651+01:00Savings In Time And Space<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjn-CecQyvM8eDnyXVdi5Vr3snfQV56a-YrECGzA88nsIORxac5BVuQYu_vJw_G5CtcrxRU5836fxaTnHo3jNsvcfJ6a1HNqceuKecDpDL1lXW_ftOAqy1Oa2of6kvN3Ai-U6qzYGeoiSY/s1600/XMLCompression.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjn-CecQyvM8eDnyXVdi5Vr3snfQV56a-YrECGzA88nsIORxac5BVuQYu_vJw_G5CtcrxRU5836fxaTnHo3jNsvcfJ6a1HNqceuKecDpDL1lXW_ftOAqy1Oa2of6kvN3Ai-U6qzYGeoiSY/s1600/XMLCompression.png" /></a></div>In the <a href="http://englishjavadrinker.blogspot.co.uk/2013/08/resource-helpers.html">previous post</a> I talked about a new feature that fixes my main annoyance with the <a href="http://gate.ac.uk">GATE</a> Developer interface. In this post I'm going to address a problem that can affect any use of GATE; the size of GATE XML files.<br />
<br />
I'll explain what I mean with this very simple XML document:<br />
<pre class="brush: xml"><doc>Mark is running a test.</doc></pre>This document is just 35 bytes in size, and when loaded into GATE contains just one <code>doc</code> annotation in the Original markups set. If we now save this document as GATE XML we end up with this 831 byte file:<br />
<pre class="brush: xml"><?xml version='1.0' encoding='UTF-8'?>
<GateDocument version="2">
<!-- The document's features-->
<GateDocumentFeatures>
<Feature>
<Name className="java.lang.String">gate.SourceURL</Name>
<Value className="java.lang.String">file:/home/mark/Desktop/simple-test.xml</Value>
</Feature>
<Feature>
<Name className="java.lang.String">MimeType</Name>
<Value className="java.lang.String">text/xml</Value>
</Feature>
</GateDocumentFeatures>
<!-- The document content area with serialized nodes -->
<TextWithNodes><Node id="0"/>Mark is running a test.<Node id="23"/></TextWithNodes>
<!-- The default annotation set -->
<AnnotationSet>
</AnnotationSet>
<!-- Named annotation set -->
<AnnotationSet Name="Original markups">
<Annotation Id="0" Type="doc" StartNode="0" EndNode="23">
</Annotation>
</AnnotationSet>
</GateDocument></pre>Clearly we could save a few bytes by removing the comments and line separators but there is nothing much we can do to get the size down anywhere near the size of the original document. Now of course comparing the two documents isn't really fair as they are encoding very different things.<br />
<ul><li>The original document uses inline tags to annotate the document, whereas GATE uses a standoff markup for added flexibility (e.g. inline XML can't handle crossing annotations).</li>
<li>GATE XML explicitly encodes information only implicit in the original document; in this case it's MIME type and filename.</li>
</ul>Now clearly this is a slightly contrived example where I've tried to make GATE XML look as bad as possible, but this can quickly become a real problem. While it is true that "disk space is cheap" it only holds when you want reasonable sized disks. If you are looking at processing terabytes of text and wanting to store the output then you really don't want to find that your processed documents are 24 times the size of the original! This is a problem we often run into with even reasonably sized corpora, as it becomes difficult to move the processed document around just due to their size. The solution we usually adopt is to simply zip up a folder of GATE XML documents. While this helps with moving them around you still need to unpack them if you want to reload them into GATE. One solution would be to look at creating a new GATE document format that would require less diskspace but be as flexible. Given the amount of effort that would require I'm ruling it out without even thinking about how I would do it. A better solution would be file level compression that would leave the documents in a format that GATE could still load easily.<br />
<br />
My first thought was to simply alter GATE to wrap the input/output streams used for writing GATE XML so that they did <a href="http://en.wikipedia.org/wiki/Gzip">gzip compression</a>. This would probably work and would give a reasonable space saving (using gzip to compress the above example brings the file size down to 403 bytes), but it would increase the time taken to load/save documents in GATE due to the compression overhead. Now given that we often want to process thousands or millions of documents time is important so I'd prefer not to introduce code that would slow down large scale processing.<br />
<br />
Fortunately I'm not the only person who has ever wanted to compress an XML document and there are a number of libraries out there that claim to be able to efficiently compress XML documents. I settled on looking at <a href="http://en.wikipedia.org/wiki/Fast_Infoset">Fast Infoset</a> because a) it follows a <a href="http://www.itu.int/rec/T-REC-X.891-200505-I/en">recognised standard</a> and b) there is an easy to use <a href="http://fi.java.net/">Java library</a> available.<br />
<br />
Fast Infoset is a binary encoding (I was going to say lossless but strictly it isn't as the formatting information of the the original XML file is lost) of XML. I won't go into the technical details other than to say that it supports both <a href="https://en.wikipedia.org/wiki/Simple_API_for_XML">SAX</a> and <a href="http://en.wikipedia.org/wiki/StAX">StAX</a> access and claims to be more efficient than equivalent reading of text based XML documents (see the <a href="http://www.itu.int/rec/T-REC-X.891-200505-I/en">spec</a> or the <a href="http://en.wikipedia.org/wiki/Fast_Infoset">Wikipedia page</a> for more details). What this means in GATE is that we can continue to use the existing methods for reading/writing GATE XML documents by just swapping out one SAX/StAX parser/writer for another. So let's have a look at a more "real world" example; the GATE <a href="http://gate.ac.uk">home page</a>.<br />
<br />
I saved a copy of the GATE home page to my local disk (to remove network effects) and then ran a few experiments, which are summarised in the following table.<br />
<br />
<table style="margin-left: auto; margin-right: auto; text-align: left;"><tbody>
<tr align="center"><th>File</th><th>File Size</th><th colspan="2">Time (ms)</th></tr>
<tr align="center"><th>Format</th><th>(bytes)</th><th>Save</th><th>Load</th></tr>
<tr align="center"><td>HTML</td><td>17,853</td><td>N/A</td><td>45</td></tr>
<tr align="center"><td>XML</td><td>70,044</td><td>284</td><td>140</td></tr>
<tr align="center"><td>FI</td><td>19,774</td><td>190</td><td>73</td></tr>
</tbody></table><br />
It should be clear from this table that the original HTML file is clearly the smallest representation and the quickest to load, which given the previous discussion shouldn't be surprising. We can also see that the XML encoding sees a huge increase in file size as well as loading time; the XML file is almost 4 times the size of the HTML file. The result of using Fast Infoset, however, is really promising. The file size grew by just 1,921 bytes! Loading time is slower (it takes almost twice as long) but is around twice as fast as loading from XML. The time to save the XML and Fast Infoset versions show little difference which is kind of to be expected (the code is mostly the same so the difference in speed will be down to the number of bytes written, i.e. the speed of my hard drive). It's also worth noting that the speed results were gathered through the GATE GUI and as such are probably not the most rigorous set of data ever collected, although they seemed to be fairly stable when repeated.<br />
<br />
From these numbers we would rightly assume that if we just want to store a HTML document efficiently then we should simply store it as HTML; converting it to a GATE document doesn't actually give us any real benefit, and no matter how we store the document on disk we pay a penalty in space and time. Of course storing HTML documents isn't really what GATE is designed for, so let's look at a slightly different scenario; load the same HTML document, run <a href="http://gate.ac.uk/userguide/chap:annie">ANNIE</a> over it to produce <i>lots</i> of annotations and then save the processed document. The performance numbers in this scenario are then as follows:<br />
<br />
<table style="margin-left: auto; margin-right: auto; text-align: left;"><tbody>
<tr align="center"><th>File</th><th>File Size</th><th colspan="2">Time (ms)</th></tr>
<tr align="center"><th>Format</th><th>(bytes)</th><th>Save</th><th>Load</th></tr>
<tr align="center"><td>XML</td><td>591,763</td><td>191</td><td>162</td></tr>
<tr align="center"><td>FI</td><td>112,919</td><td>166</td><td>87</td></tr>
</tbody></table><br />
In this scenario (where we can't use HTML to save the document) we can see that using Fast Infoset is a lot better than storing the documents as raw XML. Not only can we re-load the document in half the time it would take to load the XML we also make a space saving of 81%! In case you still need convincing that you really do save a lot of disk space using Fast Infoset I'll give a final large scale, real world(ish) example.<br />
<br />
As part of the <a href="http://www.khresmoi.eu/">Khresmoi</a> project I'd previously processed 10,284 medical Wikipedia articles. Due to disk space requirements I hadn't kept the processed documents and as the Khresmoi application can take a while to run (it's a lot more complex than ANNIE) I've simply run ANNIE over these documents to generate some annotations (it will certainly produce lots of tokens and sentences even if it doesn't find too many people or organizations in the medical documents). The results are as follows (I haven't shown save time as the GUI doesn't give it when saving a corpus, and the load time represents populating the corpus into a datastore):<br />
<br />
<table style="margin-left: auto; margin-right: auto; text-align: left;"><tbody>
<tr align="center"><th>File</th><th>File Size</th><th>Time (s)</th></tr>
<tr align="center"><th>Format</th><th>(GB)</th><th>Load</th></tr>
<tr align="center"><td>XML</td><td>13.7</td><td>1,145.387</td></tr>
<tr align="center"><td>FI</td><td>2.6</td><td>914.741</td></tr>
</tbody></table><br />
Now if a space saving of 81% (or 11GB) and a time saving of 4 minutes doesn't convince you then I don't know what will!<br />
<br />
So assuming you are convinced that storing GATE documents using Fast Infoset instead of standard XML is the way to go, then you'll want to have a look at the new Format_FastInfoset plugin (you'll need either a recent <a href="http://jenkins.gate.ac.uk/job/GATE-Nightly/lastSuccessfulBuild/">nightly build</a> or a recent SVN checkout). Documents with a <code>.finf</code> extension will automatically load correctly once the plugin is loaded, if you want to use a different extension then you will have to specify the <code>application/fastinfoset</code> MIME type. You can save documents/corpora using the new "Save as Fast Infoset XML..." menu option (courtesy of the new <a href="http://englishjavadrinker.blogspot.co.uk/2013/08/resource-helpers.html">resource helper</a> feature), which will also appear when you load the plugin.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com0tag:blogger.com,1999:blog-7025390077222270096.post-10493024604106148862013-08-04T09:06:00.000+01:002013-08-04T09:06:53.526+01:00Resource Helpers<div class="separator" style="clear: both; text-align: center;"><a href="http://marozidawn.deviantart.com/art/Help-Across-the-Street-251445262" imageanchor="1" style="clear: left; float: left; margin-bottom: 0em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEXqiaWC-XQu7yFsOidBxpA05Sd6N92vLkXLF0EPlT9gqaT02lxzr8rCRfpwZfZ5ZGVRifmnjPuMdjhtfgXL3cAHpGuuhx-Th8UvFyjE1asSgla3qAKVxG4AaUc3qeq-MY7al78QIlVrE/s320/help_across_the_street_by_marozidawn-d45pcem.jpg" /></a></div>For a long time there has been one issue with the <a href="http://gate.ac.uk/">GATE</a> Developer interface that has really annoyed me; the inability to add new items to the right click menu of a resource (that you didn't develop) without creating a new visual resource and opening that viewer. This is the reason that you can't run an application in GATE without having the application editor open (the editor adds the run option to the menu). I know from a few conversations that this has annoyed other people in the past as well.<br />
<br />
Last year I added support for opening <a href="http://www.mediawiki.org/wiki/MediaWiki">MediaWiki</a> documents to GATE. The MediaWiki markup is used on a lot of wikis, including <a href="http://en.wikipedia.org/wiki/Main_Page">Wikipedia</a>, so having support for it in GATE is really useful as we often end up processing some or all of Wikipedia. There are two main ways you might encounter MediaWiki markup. Firstly you might just have a piece of text with the markup embedded in it (e.g. the text copied out of the page editor on Wikipedia), but you are more likely to have an XML file containing the content of multiple articles in a single file. I added <a href="http://gate.ac.uk/userguide/sec:creole:mediawiki">support for both options</a> to GATE, but the support for multiple articles in a single XML file doesn't work very well; you can get the article content, but you lose the title and any other associated metadata. What I really wanted was to add a new option to all corpus resources that would allow you to properly populate the corpus from a MediaWiki XML dump file, but of course I couldn't (at least not without creating a pointless visual resource, and then the option would only be visible if the corpus viewer was open).<br />
<br />
A few days ago I was thinking about this problem again when I had a burst of inspiration and realised that there was a fairly easy way to add support, for what I'm calling <i>Resource Helpers</i>, to GATE. Essentially I'm re-using the idea of a <a href="http://gate.ac.uk/userguide/sec:creole-model:tools">GATE resource being a tool</a> (usually used to add items to the Tools menu) to allow any resource to provide items for the right-click menu of any other resource. This did require some minor changes to the core GATE code, so if you want to try this yourself you will need to use a <a href=" http://jenkins.gate.ac.uk/job/GATE-Nightly/lastSuccessfulBuild/">nightly build</a> (or a recent SVN checkout). There will eventually be some details in the userguide, but in essence all you need to know can be gleaned from the following simple example:<br />
<br />
<pre class="brush: java">package gate.test;
import gate.Document;
import gate.Resource;
import gate.creole.metadata.AutoInstance;
import gate.creole.metadata.CreoleResource;
import gate.gui.MainFrame;
import gate.gui.NameBearerHandle;
import gate.gui.ResourceHelper;
import java.awt.event.ActionEvent;
import java.util.ArrayList;
import java.util.List;
import javax.swing.AbstractAction;
import javax.swing.Action;
import javax.swing.JOptionPane;
@CreoleResource(name = "Resource Helper Test Tool",
tool = true,
autoinstances = @AutoInstance)
public class HelperTest extends ResourceHelper {
@Override
protected List<Action> buildActions(final NameBearerHandle handle) {
// a list to hold any actions we want to add
List<Action> actions = new ArrayList<Action>();
// if the resource isn't a document then we are finished
if(!(handle.getTarget() instanceof Document)) return actions;
// create a new action that will add a menu item labelled "Helper Test"
// which will show a simple message dialog box
actions.add(new AbstractAction("Helper Test") {
@Override
public void actionPerformed(ActionEvent arg0) {
JOptionPane.showMessageDialog(MainFrame.getInstance(),
"Testing the resource helper for " +
((Document)handle.getTarget()).getName());
}
});
// return the list of actions
return actions;
}
}</pre><br />
If you already understand how tool support in GATE works, then this should be easy to follow, but basically we are creating a new GATE resource that is marked as a tool, and a single instance of which is automatically created. The resource extends <code>ResourceHelper</code> which forces us to implement the single <code>buildActions</code> method. The example implementation of <code>buildActions</code> shown here, simply checks to see if we are being asked to help a Document instance, and if so adds a single new menu item, labelled "Helper Test", which when clicked pops up a dialog box showing the name of the document.<br />
<br />
The one thing to note is that the <code>buildActions</code> method is only ever called once per resource as the return value is cached for performance reasons. If you want to have a truly dynamic menu then you will need to also look at overriding the the <code>getActions</code> method of <code>ResourceHelper</code>, but that is beyond the scope of this post.<br />
<br />
Having added this new functionality, as you will have already guessed, I've now added a new "Populate from MediaWiki XML Dump" option to every corpus instance, you just need to load the MediaWiki document format plugin for it to appear.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com0tag:blogger.com,1999:blog-7025390077222270096.post-85620880961721343122013-07-19T07:00:00.000+01:002013-07-20T17:55:16.235+01:00Arduino Without The IDE<div class="separator" style="clear: left; float: left; margin-bottom: 0em; margin-right: 1em; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9zm9t42Te_dU-9T5HTFYGv1iXT6l0TyQ1_CCq5WP7uB4euQ5uidQVx6KUmoFlcxcUrfdVApXvOMnzw1Tjl1UU8pDGWVjHk7Ez8ypl5rC2PsEuIU0ebJWW6JN9EjVJa8r1Urnt0NeN-sQ/s400/arduino-cli.png" width="300" /></div>Some of you may remember that I've <a href="http://englishjavadrinker.blogspot.co.uk/2012/12/honey-i-shrunk-arduino.html">previously blogged about shrinking down an Arduino</a> by buying the <a href="http://uk.farnell.com/jsp/search/productdetail.jsp?SKU=1715487">ATmega328P-PU</a> and associated components. This is a great way of using the Arduino in a more permanent setting as it is much cheaper than using a full Arduino. I did, however, have a problem in that the Arduino IDE seemed to use the wrong upload speed when trying to burn the bootloader or upload a sketch. This meant I had to mess around copying the command out of the IDE and into a terminal and then tweaking it so that it used the right upload speed. Honestly while the IDE is easy to use (especially for novices) it isn't really a very good IDE and suffers from a number of problems which really annoy me. So I set out to look for an alternative way of compiling and uploading code to the Arduino.<br />
<br />
Given that I already knew that the Arduino IDE is just a front-end to a number of other tools which are responsible for the compiling and uploading (which is why I could copy the relevant commands into a terminal and tweak them) I knew that it was likely someone else had already figured out how to use the Arduino without the IDE. In fact a couple of people have published details on how to do this, but the most comprehensive approach I found was by Tim Marston.<br />
<br />
Tim's <a href="http://ed.am/dev/make/arduino-mk"><code>arduino-mk</code></a> project supports compiling Arduino code, uploading it to an Arduino, and has a serial monitor as well. As the project name suggests this is all implemented as a classic <a href="http://en.wikipedia.org/wiki/Make_%28software%29">Makefile</a> meaning that it works from the command line, leaving me free to choose whichever code editor I want to use. The really impressive part though was that it worked first time; not only to upload code to my Arduino, but also to upload code to my hand-built replacement. There were, however, a few missing features.<br />
<br />
Firstly the build process didn't pull in any libraries in the users sketchbook folder (which the IDE does), meaning some code I had didn't initially compile. Secondly the project didn't have any support for burning the bootloaders to new chips, something the IDE should be able to do, although in my experience it doesn't work properly (the same upload speed problem). Fortunately <code>arduino-mk</code> is an open-source project and Tim is more than happy to accept patches for bug fixes or new features. Whilst I don't think I've ever written a Makefile from scratch before, I know enough about how they work to add to an existing file and so with a little effort I managed to figure out adding support for both libraries in the sketchbook folder and the burning of bootloaders. Both features have now been incorporated into the latest release which means everyone else now has access to them as well.<br />
<br />
If you are happy working from the command line and aren't really a fan of the Arduino IDE then I would definitely recommend trying out <a href="http://ed.am/dev/make/arduino-mk"><code>arduino-mk</code></a>.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com0tag:blogger.com,1999:blog-7025390077222270096.post-54025759961703750132013-07-04T06:53:00.002+01:002013-07-04T06:53:37.989+01:00Silence, Most Definitely, Isn't GoldenI travel quite a bit for work, which means I end up spending quite a lot of time in hotel rooms, often working late into the night. These long evenings are made slightly better with a little music, and I prefer not to spend my time wearing ear phones. Fortunately my laptop has a decent sounding set of speakers.... when they work at all.<br />
<br />
When I first got the laptop the speakers were working, but the first thing I did was to do a fresh install of Ubuntu (version 12.04 at the time). It took me a few weeks to notice, but the fresh install had left me with a laptop with no sound.<br />
<br />
Periodically I tried to fix the problem always without any success. The final straw came when I had a long morning in <a href="http://englishcoffeedrinker.blogspot.co.uk/search/label/Hannover">Hannover</a>, with only a little work to do before the meeting was to start in the afternoon. It was too cold to spend too long outside so I set about finding a fix.<br />
<br />
Three hours and a lot of swearing and rebooting and I finally had sound working. Annoyingly I didn't write down exactly what I did to fix the problem, and so when last week I upgraded Ubuntu to the newest version (13.04) I found that I had no sound and couldn't remember how to fix the problem. It's taken me almost two hours to find the solution again so this time I'm writing this post to both help anyone else having the same problem and to stop me having to figure it all out for a third time next time I upgrade Ubuntu.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBbDACgzEDoh5ZVY7PMjqc7q_6mSMvzhrFZu9nU7uQyo3XMnKteWO8VyE72JzujaOtel8u47CMN_Rf6U1xbWMatw8aVmeH1Q88Gmi612iSLqXsAbfGcLDTuXqgcoAxh-_h7IJ2twkUPpI/s724/alsamixer.png" imageanchor="1" style="clear: left; float: left; margin-top: 0.5em; margin-right: 1em;"><img border="0" height="204" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBbDACgzEDoh5ZVY7PMjqc7q_6mSMvzhrFZu9nU7uQyo3XMnKteWO8VyE72JzujaOtel8u47CMN_Rf6U1xbWMatw8aVmeH1Q88Gmi612iSLqXsAbfGcLDTuXqgcoAxh-_h7IJ2twkUPpI/s320/alsamixer.png" width="320" /></a></div>I followed almost every suggestion I could find on the web to diagnose sound problems, yet in most cases my laptop was already configured as suggested. I was almost ready to give up when I came across a page which suggested disabling the <b>Auto-Mute Mode</b> in the <code>alsamixer</code> control panel; you can see this in the screenshot. This looked promising but there was one problem -- When I ran <code>alsamixer</code> I didn't have a Auto-Mute Mode option.<br />
<br />
Another forty-five minutes of fruitless web searching followed before I eventually discovered a way of changing the audio driver to show all the available options. Unfortunately this uses the daily build of alsa which I expect is why it got disabled after the upgrade to the newest version of Ubuntu. Anyway, the trick is to run the following commands to add the relevant repository and then to install the daily build of alsa.<br />
<pre class="brush:bash">sudo apt-add-repository ppa:ubuntu-audio-dev/alsa-daily
sudo apt-get update
sudo apt-get install oem-audio-hda-daily-dkms</pre>Once completed you can then run <code>alsamixer</code> and you should find that the Auto-Mute Mode has now helpfully appeared. Simply disable this option and you should find that you're sound starts working. As far as I understand this option is related to sensing if you have plugged headphones in and if the internal speakers should be disabled or not. This means, although I haven't tested this, that if you disable the Auto-Mute then plugging headphones in my not stop sound coming out of the speakers, but that is an easy problem to deal with.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com1tag:blogger.com,1999:blog-7025390077222270096.post-51648899740389425052013-05-01T11:20:00.000+01:002013-05-01T11:20:26.097+01:00(File) Size Does Matter!<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAR7yhh2wAh9qNEov3C6Gjtc52ioPuwVwJu1wP1FgYffpOghI4MJiRdlyxbvZfjJmhXc3SJ8gFRNwtY9tBGVl25JPJyuXWL5YNwY3kxOyRnv173s9hoOLcf4Fg31wsgbS_sleTnzugW1o/s1600/size-does-matter.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 0em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAR7yhh2wAh9qNEov3C6Gjtc52ioPuwVwJu1wP1FgYffpOghI4MJiRdlyxbvZfjJmhXc3SJ8gFRNwtY9tBGVl25JPJyuXWL5YNwY3kxOyRnv173s9hoOLcf4Fg31wsgbS_sleTnzugW1o/s320/size-does-matter.jpg" /></a></div>I like <a href="http://en.wikipedia.org/wiki/Portable_Document_Format">PDF</a> files. Everyone (or as near as makes no difference) can open them, and you can be fairly certain that the file will look the same on every computer. Sometimes though I struggle to produce PDF files that are of a high enough quality, but small enough to e-mail or host on a website.<br />
<br />
The most recent problem I've faced involved producing a PDF from an <a href="http://en.wikipedia.org/wiki/Scalable_Vector_Graphics">SVG</a> file (authored using <a href="http://inkscape.org/">Inkscape</a>) which included linked digital photos. The SVG file itself is just 77kB in size, and the five photos it links to total just over 7MB. Even if the images were simply included as is, I wouldn't expect the resulting PDF to be more than 8MB in size, although I would hope that some reasonable down sampling would be applied. Printing to a PDF from within Inkscape, however, results in a file of 22MB in size! That's over three times the size of the included photos, which is just ridiculous. Exporting to a PDF instead of printing, results in a file of the same size so isn't much help.<br />
<br />
Now I don't know enough about the inner workings of a PDF file, but a 22MB file (which is incidentally a single A4 page) just isn't right, and certainly isn't sensible for trying to send around via e-mail etc. I've tried searching the web for suggestions, but none of the ideas I've found have turned out to be much use; either they don't alter the size of the file, or they reduce the quality so far that the text becomes unreadable.<br />
<br />
The first alternative I tried was to produce a PNG image from the SVG instead of a PDF. While an image file wasn't really what I wanted I thought it might be a good place to start. Using Inkscape to produce a 300 DPI image resulted in a file of just 1.7MB in size; a file, I should add, with a quality that is perfectly adequate. The problem with an image file is that it is easy to alter and can be a pain to print at the right size, so I'd still like to use a PDF file. Fortunately, on linux at least, it is easy to create a PDF from a PNG, using the simple command<br />
<pre class="brush:bash">convert input.png output.pdf</pre>Given that Inkscape can also be controlled from the command line I can easily write a reusable script that converts an SVG direct to a PDF of a reasonable size:<br />
<pre class="brush:bash">#!/bin/sh
inkscape -f "$1.svg" -e out.png -C -d 300
convert out.png "$1.pdf"
rm out.png
</pre>I can then use this by passing in the name of the SVG file (minus the <code>.svg</code> extension) and out will pop a PDF file. Now I know this approach isn't perfect but given my requirements (a PDF of a reasonable size that is really only needed for printing) this works well. I'm intending to see if I can find a better solution (one that will keep the text as text) so if you have any suggestions please feel free to leave a comment.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com3tag:blogger.com,1999:blog-7025390077222270096.post-82228246152006975762013-03-03T12:48:00.000+00:002013-03-03T12:48:09.395+00:00CORPSE: COld stoRage backuP SoftwarE<a href="http://artistic-feet.deviantart.com/art/C-S-I-Toes-165192054" imageanchor="1" style="clear: left; float: left; margin-bottom: 0em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiJdMt1NbRw5eTePzB3qewTCJRMuKFpis8mIMu5uGjEgkzlfmqo5k_GtZTTt35hhg_0LR6cUTKQWqylrpQxos1Nq_Mu1TPQDYja6rWkMWW8Z3qlvM4WMKhakX6vfrlu1x5JrCnqRnlbVY/s320/C_S_I__Toes_by_Artistic_Feet.jpg" /></a>I've recently started using <a href="http://aws.amazon.com/glacier/">Amazon Glacier</a> to store the <i>backup of last resort</i> of all my digital photos (I've <a href="http://englishcoffeedrinker.blogspot.com/2013/03/cold-storage.html">discussed the motivation behind this on my other blog</a>). While I think the idea of cold storage, and Glacier in particular, is great there is just one problem; Amazon don't provide a friendly user interface for Glacier. Glacier is part of the Amazon Web Service (AWS) framework, and as such is designed as a service around which other people can develop applications.<br />
<br />
A quick web search will bring up links to quite a few different tools for using Glacier. I'm currently uploading my data using <a href="http://simpleglacieruploader.brianmcmichael.com/">SAGU</a> (the Simple Amazon Glacier Uploader). While SAGU does make uploading data to Glacier easy it has a number of shortcomings which the developer is working to address. I've proposed a couple of patches, but I'm also looking at implementing more useful functionality (for example, range based downloads are a must have feature if you want to keep the costs under control). While I'm hopping that some of the ideas I'm working on might be incorporated into SAGU, I'm currently developing them in a separate project which I'm calling CORPSE (which of course stands for COld stoRage backuP SoftwarE).<br />
<br />
Even if CORPSE doesn't result in a full application, I'm using it to a) experiment with a number of Java libraries I've not used before (including <a href="http://aws.amazon.com/sdkforjava/">AWS</a> itself, <a href="http://wiki.fasterxml.com/JacksonHome">Jackson</a>, <a href="http://flywaydb.org/">Flyway</a>) and b) using <a href="http://en.wikipedia.org/wiki/Git_%28software%29">git</a> for version control. In the past I've always used a centralised version control system (be that <a href="http://en.wikipedia.org/wiki/Concurrent_Versions_System">CVS</a>, <a href="http://en.wikipedia.org/wiki/PVCS">PVCS</a> or <a href="http://en.wikipedia.org/wiki/Apache_Subversion">SVN</a>), but the decentralised nature of git is intriguing and with a free account on <a href="https://github.com/">GitHub</a> I don't have anything to lose. I'm sure I'll mention CORPSE again on this blog in the future but if you want to take a look at what is there now or to follow development then you can find it in <a href="https://github.com/greenwoodma/CORPSE">this GitHub project</a>.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com0tag:blogger.com,1999:blog-7025390077222270096.post-53785371586709803432013-01-19T12:49:00.000+00:002013-01-19T12:49:27.658+00:00To Track, Or Not To Track, That Is The Question<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBqbBXfmM_7FDFxdJMIxEhh0bgIldeiu9hyphenhyphenC6iVz-qLXv_JqMi4liFs8t1jDirtADsMj2AJ_S5uFvCDT5CQbOy9uPUTWsH3LDI0URAxlLqw253dczimWbhL0TWn9WGp4qe5ElXMTrHAAo/s1600/totrackornot.jpg" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:0em"><img border="0" width="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBqbBXfmM_7FDFxdJMIxEhh0bgIldeiu9hyphenhyphenC6iVz-qLXv_JqMi4liFs8t1jDirtADsMj2AJ_S5uFvCDT5CQbOy9uPUTWsH3LDI0URAxlLqw253dczimWbhL0TWn9WGp4qe5ElXMTrHAAo/s320/totrackornot.jpg" /></a></div>If you read any of my other blogs, then you may well be aware that I've recently opened a <a href="http://www.shapeways.com/shops/penistone-railway-works">shopfront on Shapeways</a> to make a few 3D models available for sale. While I really like this feature of Shapeways I wanted to customize the experience a little and so built a <a href="http://www.penistone-railway-works.co.uk/">seperate, more customizable, shopfront</a>. One of the options when setting up the Shapeways store was to provide a <a href="http://www.google.com/analytics/">Google Analytics</a> tracking code. I've never used any tracking code before on any of the websites I've built as I've never really seen a strong reason to. Given I was being prompted to though, I thought I'd investigate, especially as it would allow me to see how often people looked at my models, separate to how often people buy them.<br />
<br />
Registering for a Google Analytics account is free and easy (as with most Google services) and within a few minutes I had the tracking information to register with Shapeways. Having got this far I decided to also add the tracking information to the separate shop front I'd built, and of course this is where life got interesting.<br />
<br />
The first problem I faced was that I couldn't get Google to recognise that I'd added the tracking code to my website. No matter what I tried it simply kept telling me that I hadn't yet added the relevant JavaScript to my pages, yet I knew it was there. After a lot of head scratching and web searching I eventually found <a href="http://qyerty.blogspot.co.uk/2012/07/solved-google-analytics-tracking-not.html">the problem and an easy solution</a>. The domain I purchased doesn't actually include the initial www of most web addresses, but the way I have the DNS records and forwarding configured if you type the bare domain name you will be forwarded to the www version automatically. So when completing the Google Analytics form I included the www, which turns out was my mistake. I'm still not sure why this should make a difference (I'm guessing that Google aren't following the forwarding information when checking the page) but anyway removing the www from the default URL field fixed the problem.<br />
<br />
The second problem was that I then wanted to turn the tracking off. Not so that I could <a href="http://www.ico.gov.uk/for_organisations/privacy_and_electronic_communications/the_guide/cookies.aspx">comply with the EU directive on cookies or anything</a> (I'm taking the applied consent rule, on the basis that if you visit an online shop you have to assume, at the bare minimum, that your page views are being tracked), but so that I wouldn't end up tracking my own page views.<br />
<br />
I tend to have two copies of every website I develop running at the same time; one is the publicly visible (hopefully stable) version, while the second is my internal development version. Once I'm happy with updates to the development version I then push them to the live version. So when it comes to tracking page views not only do I not want to track my page views on the live site, but I definitely don't want to track page views on the development version.<br />
<br />
My solution to this is to use server side code to only embed the tracking related JavaScript into the page, if it should be used. For this I wrote a utility function to determine if I should be tracking the page view or not:<br />
<pre class="brush:java">public static boolean trackPageViews(HttpServletRequest request)
throws Exception {
//get the address the request is coming from
InetAddress client = InetAddress.getByName(request.getRemoteHost());
//the client is the same machine (an address such as 127.0.0.1)
if (client.isLoopbackAddress()) return false;
//the client is within the same site (i.e. not using a world visible IP)
if (client.isSiteLocalAddress()) return false;
//the client is at the same world visible IP address as the server
if (client.toString().equals(getServerIP(false))) return false;
//track all other page views
return true;
}</pre>Essentially this code checks the request and says not to track page views if it is coming from a loop back address (i.e. 127.0.0.1 always refers to the local machine, no matter what the machine is), a site local address (i.e. within the same local network and hence using a non-public IP address, often something in the 192.168.*.* range), or if the request is from the same public IP as the server is running on. The first two checks (lines 8 and 11) use standard Java libraries to do the checking, and catch all page views of the development versions of my websites (which are always only accessible from within my local network). The third check is, however, slightly more complex.<br />
<br />
The problem is that you're machine itself probably doesn't have a world visible IP address; general a home network will be connected to a broadband router which will have a world visible address. So what we need to do is find the external IP address of the router. There are a number of websites available that will show you the world visible IP from which you are connecting and so we could use one of those to figure out the external address of the server, or we could write our own.<br />
<pre class="brush:java">package utils;
import java.io.IOException;
import java.net.InetAddress;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class IPAddressServlet extends HttpServlet
{
@Override
public void doGet(HttpServletRequest request,
HttpServletResponse response) throws IOException {
InetAddress client = InetAddress.getByName(request.getRemoteHost());
response.getWriter().println(clientAddress.toString());
}
}</pre>Assuming you access this servlet via a public IP address then it will simply echo the address back to you. We can now fill out the missing <code>getServerIP</code> method from our <code>trackPageViews</code> method.<br />
<pre class="brush:java">private static String myip = null;
private static String getServerIP(boolean refresh)
throws IOException {
if (myip == null || refresh) {
URL url = new URL("http", HOST, "/myip");
BufferedReader in = new BufferedReader(
new InputStreamReader(url.openStream()));
myip = in.readLine();
in.close();
}
return myip;
}</pre>I'm assuming this will work in all cases where you connect to the server using it's standard web address, as that will always be converted via a DNS server to a world visible IP address, although if you have a strange network setup where you might have multiple world visible IPs through which you could connect then it might not work correctly. The important point is it works for me, so I can now happily browse the public version and development version of the site from anywhere within my home network safe in the knowledge that my own page views aren't being tracked, so any analytics Google collects for me are form <i>real</i> visitors; albeit only those with JavaScript enabled, but that's an issue for another day.<br />
<br />
Before I forget, thanks to <a href="http://galenote.blogspot.co.uk/2011/10/walk-in-sun.html">GB for the photo</a> I used to brighten up this post!Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com1tag:blogger.com,1999:blog-7025390077222270096.post-90893511912116385452013-01-07T18:59:00.000+00:002013-01-07T18:59:47.191+00:00Generating Sitemaps<img style="background: white; float:left; margin-right:1em; margin-bottom: 0.5em; width:200px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIA4cNR594B3iaVtayJNPSzf6bIF7ekCfTEPeLYHbs4ZU_m5GZKXMXimwBEEPa3g-fpG78eJPqRBO7jiwwVos4YouW0AcU3fZZvE7wAjLkNmpgmstToHUG5QePsbsZ4vNcVGIuKr_-Mmg/s320/Sitemaps.png" />A long time ago (okay 2009) on an entirely different blog, I introduced <a href="http://englishcoffeedrinker.blogspot.co.uk/search/label/SitemapGenerator">v0.1 of the SitemapGenerator</a>. This was a command line tool that I was running once an hour to monitor my blog, and if anything had changed it would re-generate a <a href="http://www.sitemaps.org/">sitemap</a> and notify Google of the changes. I wanted this predominantly so that a Google custom search engine would stay up-to-date, something I no longer need to worry about given that I use <a href="http://www.postvorta.co.uk">Postvorta</a> to index all my blogs. I haven't used the code in a number of years but recently I again need to generate a sitemap. This time though I wanted to generate the file from within the web application I was building rather than via a separate recurring script.<br />
<br />
I did a quick search for a better version than my original, and while I found a few Java libraries none really met my requirements; I wanted something lightweight that would allow me to take advantage of <a href="https://support.google.com/webmasters/bin/answer.py?hl=en&answer=183668#2">the extensions to the protocol Google has introduced</a> for documenting images, videos and news articles. So I went back and re-wrote the guts of my original application. The new and improved version now supports a simple API that I can call from within my web application (although I've maintained the command line tool in case anybody was still using it).<br />
<br />
If you are interested then you can <a href="http://greenwoodma.servehttp.com/svn/repos/open-source/list/SitemapGenerator/trunk/">grab the new version direct from SVN</a>.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com2tag:blogger.com,1999:blog-7025390077222270096.post-67585085636012398192012-12-24T18:48:00.000+00:002012-12-24T18:48:12.537+00:00Back To Front But Functional<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjl0NmgZyEDVg5cV7KqVGHmtx_r8rD_cbM0Endgh6e3yth3OD3CDytJ1sZC9buNVChyphenhyphenAJz-Yq71ny9Jai6JfA2kr77SSTn3aycnQkLqZvonn68_xoblhP7hCk6Ph6YFcG3SHukUrYcMkAs/s1600/programmer-shield.jpg" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:0em"><img border="0" height="240" width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjl0NmgZyEDVg5cV7KqVGHmtx_r8rD_cbM0Endgh6e3yth3OD3CDytJ1sZC9buNVChyphenhyphenAJz-Yq71ny9Jai6JfA2kr77SSTn3aycnQkLqZvonn68_xoblhP7hCk6Ph6YFcG3SHukUrYcMkAs/s320/programmer-shield.jpg" /></a></div>While the circuit required for programming an ATmega328P-PU chip (i.e. a standalone Arduino) is fairly simply (regardless of whether you are running at <a href="http://englishjavadrinker.blogspot.co.uk/2012/12/honey-i-shrunk-arduino.html">8MHz</a> or <a href="http://englishjavadrinker.blogspot.co.uk/2012/12/double-speed-for-just-35p.html">16MHz</a>) it would still be a pain to have to set everything up on a breadboard each time I wanted to programme a chip. The solution was to build a shield for the Arduino which I could simply slot on whenever I wanted to programme a new ATmega328P-PU.<br />
<br />
Now I could have gone all fancy and designed a custom PCB for this project, but for such a simple project (where I wouldn't ever need more than one shield) this seemed overkill. Instead I built the shield out of a <a href="http://www.oomlout.co.uk/budget-proto-shield-x3-p-261.html">budget prototyping shield</a> from <a href="http://www.oomlout.co.uk/">oomlout</a>. Obviously I don't want to solder an ATmega328P-PU to the shield as that would defeat the purpose, so I've added a <a href="http://uk.farnell.com/jsp/search/productdetail.jsp?SKU=1103850">28pin DIP socket</a> in its place. It took me a while to figure out the best placement of the components and to cut appropriate sized pieces of wire (lots of the connections are actually made simply by soldering the leads together on the underside of the board), although actually soldering everything in place was fairly quick (the advantage of lots of planning time). Most of the components were held in place by bending their leads slightly. The DIP socket, however, kept falling out. Unfortunately one of those occasions was just as I was about to solder it into place, and I managed to put it in back to front :(<br />
<br />
Fortunately, the only problem with having the DIP socket back to front is that the notched end of the chip doesn't match up with the notch on the socket, which is why I've added a white paint mark to the socket to remind me to put the chip in the wrong way around. Anyway I now have a functioning shield which I can use to programme an ATmega328P-PU to run at either 8MHz or 16MHz and to upload sketches to it. I'd call that a success.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com3tag:blogger.com,1999:blog-7025390077222270096.post-38464586328077932892012-12-22T16:13:00.000+00:002012-12-22T16:13:34.348+00:00Double The Speed For Just 35pIn my <a href="http://englishjavadrinker.blogspot.co.uk/2012/12/honey-i-shrunk-arduino.html">previous post</a> I explained how I'd eventually managed to figure out the procedure for configuring and programming an ATmega329P-PU micro-controller. The configuration in that post was deliberately designed to be as minimal as possible, which was why I was limited to using the internal 8MHz <a href="http://en.wikipedia.org/wiki/RC_oscillator">RC oscillator</a>. While 8MHz is possibly fast enough for a lot of projects, there are occasions where speed is important; <a href="http://www.glacialwanderer.com/hobbyrobotics/?p=16">a lightening trigger for a camera</a> for example. In these cases running the chip at 16MHz would be preferable. Fortunately adding an external 16MHz crystal to double the speed is both easy and cheap!<br />
<br />
Before we get to the hardware though, we need add a new entry to the <code>boards.txt</code> file defining the new setup (see <a href="http://englishjavadrinker.blogspot.co.uk/2012/12/honey-i-shrunk-arduino.html">the previous post</a> for details on this and how to upload code etc.).<br />
<pre class="brush:bash">atmega328p16mhz.name=ATmega328P (16 MHz external crystal)
atmega328p16mhz.upload.protocol=stk500v1
atmega328p16mhz.upload.maximum_size=30720
atmega328p16mhz.upload.speed=9600
atmega328p16mhz.bootloader.low_fuses=0xD7
atmega328p16mhz.bootloader.high_fuses=0xDF
atmega328p16mhz.bootloader.extended_fuses=0x07
atmega328p16mhz.bootloader.unlock_bits=0x3F
atmega328p16mhz.bootloader.lock_bits=0x0F
atmega328p16mhz.build.mcu=atmega328p
atmega328p16mhz.build.f_cpu=16000000L
atmega328p16mhz.build.core=arduino:arduino
atmega328p16mhz.build.variant=standard</pre>Most of this is identical to the definition for the 8MHz version. The two main changes are to the low fuse (to specify that we are using an external full-swing crystal) and to the processing speed (changed from 8000000 to 16000000).<br />
<br />
To make this work we need to add three small components to the previous circuits: a <a href="https://uk.farnell.com/jsp/search/productdetail.jsp?SKU=1611761">16MHz crystal</a> and two <a href="http://uk.farnell.com/jsp/search/productdetail.jsp?SKU=9411674">22pF capacitors</a>. Fortunately these are very cheap components and add just 36p to the cost of the circuit (making a total cost of £5.54 or £3.87 if you ordered parts for ten circuits). You can see how these are wired up in these diagrams (for programming on the left and in use on the right).<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbDBGHR72FMmrkAg7Me2f0KkhFbA7JT4ww4wtNtRMkOMMngJyxiOo86fKeeRNef9SRjDTmRa3Kzo3XI2HxQJjB58vIbwtkCFRaGbjbacCUFJ1UixHefrATae8a3JORThAEk0pVEzmlpzQ/s1600/Programming-WithCrystal_bb.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="320" width="253" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbDBGHR72FMmrkAg7Me2f0KkhFbA7JT4ww4wtNtRMkOMMngJyxiOo86fKeeRNef9SRjDTmRa3Kzo3XI2HxQJjB58vIbwtkCFRaGbjbacCUFJ1UixHefrATae8a3JORThAEk0pVEzmlpzQ/s320/Programming-WithCrystal_bb.png" style="vertical-align:middle;"/></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEH1njx-eyGy_bvvD2Q6svwL-uCHbpJm1wIbsEB0ZAi_Xtkawjbun-DQfpY5FVIAfowQLUsgVjldeLjHR_Zys62cKSCLgUHsj0Zo3QcJ95W0SlvJ8Z1QCe9zArKjQCKyTzI_7_XyfZXnU/s1600/InUse-WithCrystal_bb.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="173" width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEH1njx-eyGy_bvvD2Q6svwL-uCHbpJm1wIbsEB0ZAi_Xtkawjbun-DQfpY5FVIAfowQLUsgVjldeLjHR_Zys62cKSCLgUHsj0Zo3QcJ95W0SlvJ8Z1QCe9zArKjQCKyTzI_7_XyfZXnU/s320/InUse-WithCrystal_bb.png" style="vertical-align:middle;"/></a></div><br />
Configuring the fuses and uploading sketches to this new circuit is <a href="http://englishjavadrinker.blogspot.co.uk/2012/12/honey-i-shrunk-arduino.html">exactly the same procedure as for the 8MHz version</a>, other than ensuring you select the 16MHz version from the boards menu.<br />
<br />
There is, however, one important thing to note. Once you have configured the chip to run at 16MHz, you can't then use it without a 16MHz crystal present. You can't even re-programme the fuses to set it back to run at 8MHz. So if you want to go back to using the internal RC oscillator re-programme the fuses with the crystal present and then simplify the circuit.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com1tag:blogger.com,1999:blog-7025390077222270096.post-59714836436666156332012-12-17T18:52:00.000+00:002012-12-22T15:27:02.040+00:00Honey, I Shrunk The ArduinoAs you might be able to tell from <a href="http://englishjavadrinker.blogspot.co.uk/search/label/Arduino">recent posts</a>, I've been doing quite a bit of work with an Arduino. I've now got at least one project that I'd like to make a little more permanent, rather than it just being a bunch of components on a breadboard.<br />
<br />
The easiest option would probably be to solder the components onto a prototyping shield (for example, <a href="http://www.oomlout.co.uk/budget-proto-shield-x3-p-261.html">these budget models from oomlout</a>). There are two problems with this approach though. Firstly not every component is ideally suited for the form factor of an Arduino shield (for example a 16x2 character LCD takes up most of the prototyping space), and secondly to be any use a shield still needs to be connected to an Arduino. While Arduino's aren't extortionately expensive at just over £20 (depending where you buy them from), that's quite a cost to add to a project.<br />
<br />
There are cheaper versions of the Arduino available, such as the <a href="http://arduino.cc/en/Main/ArduinoBoardMini">Arduino Mini</a>, which would bring the cost down to around £10 and which might be ideal for some projects. If you want to move your project onto a custom PCB though, you probably don't want to have to solder on a second PCB. The solution is to build the Arduino components directly into your circuit rather than relying on a pre-built solution. This should be cheaper than buying an Arduino and allows you to lay out the components in the most efficient way on a custom PCB.<br />
<br />
The problem of course is figuring out how to construct an Arduino from the component parts. I spent quite a while trawling the internet trying to find the answers, but couldn't find a single page that collected everything you needed to know in one place, so I'm sure you can guess what the rest of this post is going to cover :)<br />
<br />
The first decision you have to make is which Arduino compatible microcontroller you are going to base your circuit around. This will depend on the number of input/output pins you need as well as the size of the compiled sketch you intend to use. Given that I'm prototyping my ideas using an Arduino UNO, I decided to stick with the ATmega328P-PU chip the UNO is based around. This is by far the most expensive component you will need to buy, but you can pick them up for around £3. You will find that a number of places sell them with the Arduino bootloader programmed in, for which they charge a premium. Unless you are simply wanting to replace the chip in an existing Arduino UNO you don't actually need to use the bootloader, so save yourself a few pounds and buy the plain un-programmed chips.<br />
<br />
Most of the tutorials on using a ATmega328P-PU expect you to run the chip at 16MHz (the same speed as the UNO) but this requires you to provide an external clock signal using a 16MHz crystal. As components mean costs, I decided I'd be a cheapskate and run the chip just using the internal 8MHz clock. We don't need any other components in order to programme the chip, but we do need to tell the Arduino software how to work with the chip.<br />
<br />
Extra boards/chips are made available through the Arduino software by adding them to the <code>boards.txt</code> file in the <code>hardware/arduino/</code> folder of the Ardunio installation (for full details of the format of this file try <a href="http://code.google.com/p/arduino/wiki/Platforms">this Arduino wiki page</a>). After a fair amount of messing around (starting from <a href="http://arduino.cc/en/Tutorial/ArduinoToBreadboard">this example</a>) I eventually figured out the following definition:<br />
<pre class="brush: bash">atmega328p8mhz.name=ATmega328P (8 MHz internal clock)
atmega328p8mhz.upload.protocol=stk500v1
atmega328p8mhz.upload.maximum_size=30720
atmega328p8mhz.upload.speed=9600
atmega328p8mhz.bootloader.low_fuses=0xE2
atmega328p8mhz.bootloader.high_fuses=0xDF
atmega328p8mhz.bootloader.extended_fuses=0x07
atmega328p8mhz.bootloader.unlock_bits=0x3F
atmega328p8mhz.bootloader.lock_bits=0x0F
atmega328p8mhz.build.mcu=atmega328p
atmega328p8mhz.build.f_cpu=8000000L
atmega328p8mhz.build.core=arduino:arduino
atmega328p8mhz.build.variant=standard</pre>I'm sure that to most people (me included) that isn't self explanatory, so let's step through it to see what it means.<br />
<br />
Firstly, line 1 simply assigns a human readable name to the board definition (this is what will apeartr on the <i>Boards</i> menu). Lines 3-5 state that we will be uploading to the chip using the right protocol to use the Arduino as the ISP (In System Programmer), the maximum sketch size, and the speed at which we will upload to the chip. Unfortunately, at least in version 1.0 of the Arduino software the upload speed seems to be ignored which will come back to bite us later.<br />
<br />
Lines 7 to 11 specify the fuse settings. These are essentially used to configure the chip. I used <a href="http://www.engbedded.com/fusecalc/">Engbedded's fuse calculator</a> to figure out the correct settings. Note that I'm not using the default fuse settings that the calculator suggests as I've turned off the <i>Divide clock by 8 internally</i> option. I found that otherwise the chip seemed to be running at 1MHz rather than 8MHz (i.e. the blink example sketch while setup to blink at 1 second intervals appeared to be actually blinking at 8 second intervals). I also specified the smallest bootloader space option, given that we aren't going to use one. Lines 13 and 14 specify the chip type and speed and finally lines 15 and 16 specify which internal libraries to compile the sketch against (in this example the standard variant which is the UNO).<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXETeXPKsmrtNFeIBkeGYOhf1m5g463tC8Gw1isA7bGuvZlihX4rvWZwxwG9pPWiip2GZp925SP3NpzWLQVl56pV_EdYe-C0rIxL1x97USi_Q0eidFBLR6IhrRpMIur6XymRDgWx9Tp7o/s1600/Programming_bb.png" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:1em"><img border="0" height="320" width="253" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXETeXPKsmrtNFeIBkeGYOhf1m5g463tC8Gw1isA7bGuvZlihX4rvWZwxwG9pPWiip2GZp925SP3NpzWLQVl56pV_EdYe-C0rIxL1x97USi_Q0eidFBLR6IhrRpMIur6XymRDgWx9Tp7o/s320/Programming_bb.png" /></a></div>So now that the Arduino software knows how to programme the board, we need to turn our existing Arduino UNO into an ISP which we can use to programme the chip. Now there is an example sketch called ArduinoISP bundled with the Arduino software, but try as I might I couldn't get it to work. Instead I'm using a slightly improved version released by <a href="http://www.adafruit.com/">Adafruit</a>. Simply <a href="https://github.com/adafruit/ArduinoISP">download the new version</a> into your sketchbook folder and then upload it to your Arduino UNO the same way you would any normal sketch. Now wire the ATmega382P-PU you want to programme to the Arduino as shown to the left (this is also documented in the comments at the top of the ArduinoISP sketch, which also talks about adding LEDs for feedback; feel free to add these if you wish). Now in the IDE, from the Tools menu set the Programmer to <i>ArduinoISP</i> and the board to <i>ATmega328P (8 MHz internal clock)</i>.<br />
<br />
Now before we can actually programme the chip we need to set the fuses to the correct values as discussed above. To do this simply choose <i>Burn Bootloader</i> from the Tools menu (note that this is slightly misleading in that we aren't burning a bootloader as we didn't specify one in <code>boards.txt</code>, but this menu item sets the fuses as well as burning the bootloader if specified). Hopefully this will "just work", but for some reason my version of the IDE uses the wrong upload speed and as such this fails. Fortunately it's fairly easy to do manually. The easiest way is to get the IDE to generate the command line for you. To do this open the Preferences dialog from the File menu, and turn on the verbose output during upload. Now try to burn the bootloader again. It will still fail, but the first message will be the command line it is trying to use. On my machine this is:<br />
<br />
<code>/usr/share/arduino/hardware/tools/avrdude -C/usr/share/arduino/hardware/tools/avrdude.conf -v -v -v -v -patmega328p -cstk500v1 -P/dev/ttyACM0 -b19200 -e -Ulock:w:0x3F:m -Uefuse:w:0xFF:m -Uhfuse:w:0xD9:m -Ulfuse:w:0x62:m</code><br />
<br />
This command is setting the baud rate to 19200 which is an awful lot faster than both the upload speed we specified in <code>boards.txt</code> and the baud rate specified in the ArduinoISP sketch, both of which use 9600. So simply change the <code>-b19200</code> to <code>-b9600</code> and execute the command manually (from a terminal or command prompt etc.). Hopefully that should work correctly. Once you have the fuses set, assuming you don't want to change the way the chip is configured, you won't need to do this step in the future.<br />
<br />
Now you are ready to actually upload a sketch to the new chip. I usually test that everything is working using the simple Blink example that comes with the Arduino IDE. As you have already connected the Aruino to pin 13 of the new chip (this doubles as the SCK pin) I usually change the sketch to use pin 8 which is the bottom right pin if you have yours in the same orientation as the above diagram, so that you can see the sketch working without having to change any of the wiring used for programming. Now to upload to the new chip you need to use the "Upload Using Programmer" option from the File menu (making sure that the programmer and board are still set correctly) and not the Upload button on the toolbar. If setting the fuses failed the first time and you had to manually change the baud rate then you will probably run into the same problem here as well, but the solution is the same (run the command from the IDE then copy it out, change the baud rate, and then run it manually). Once it has uploaded successfully you should find your LED is blinking on and off at one second intervals.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1Q59hTySc51Iwq7VnyHunjF3gSAZOsHKacEMYqfDVhs8V_o3q24JOvTamU28Xdc5yZR8KiBpbeEEB263ZjLeGPWn2iOZJrE-myuzEp6ijaTTLl2ZOe7jQ0tS1KATZHV3ObOd1URFZySE/s1600/InUse_bb.png" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:1em"><img border="0" height="190" width="292" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1Q59hTySc51Iwq7VnyHunjF3gSAZOsHKacEMYqfDVhs8V_o3q24JOvTamU28Xdc5yZR8KiBpbeEEB263ZjLeGPWn2iOZJrE-myuzEp6ijaTTLl2ZOe7jQ0tS1KATZHV3ObOd1URFZySE/s320/InUse_bb.png" /></a></div>Now, as the main aim was to remove the need to dedicate a full Arduino UNO to each project we need to power the chip in some other way than by being connected to the UNO. Fortunately this is also fairly simple as you can see from the setup to the left. This uses just four components; a voltage regulator, two capacitors, and a 9V battery connector.<br />
<br />
I've actually found that the supply from a fresh 9V battery is stable enough that you can do without the two capacitors, but as they are stipulated by the datasheet for the voltage regulator I've included them.<br />
<br />
<table style="margin-right:1em; clear:both; float: left; text-align:center; border: 1px solid gray;"><tr><th colspan="2">Minimal Arduino Part List</th></tr>
<tr><th>Part</th><th>Unit Cost</th></tr>
<tr><td><a href="https://uk.farnell.com/jsp/search/productdetail.jsp?SKU=1715487">ATMEL ATmega328P-PU</a></td><td>£3.02</td></tr>
<tr><td><a href="https://uk.farnell.com/jsp/search/productdetail.jsp?SKU=1467758">5V Voltage Regulator</a></td><td>£0.40</td></tr>
<tr><td><a href="https://uk.farnell.com/jsp/search/productdetail.jsp?SKU=1281807">100µF Capacitor</a></td><td>£0.151</td></tr>
<tr><td><a href="https://uk.farnell.com/jsp/search/productdetail.jsp?SKU=1650675">9V Battery Connector</a></td><td>£0.60</td></tr>
</table>These extra components are, as you can see from this table, much cheaper than the main ATmega328P-PU, and adding the two capacitors adds just 30p to the cost. In total the components add up to just £4.32 + VAT for a grand total of £5.19 -- or around half the cost of the Arduino Mini, or a quarter of the cost of an Arduino UNO. The circuit could be made even cheaper if you were willing to buy the components in bulk; for example if you buy 10 ATmega328P-PU then they cost just £1.63 each, so you could make 10 of these circuits for just £3.52 each.<br />
<br />
The one thing I haven't tried yet is using any analog inputs with this setup. The datasheet for the ATmega328P-PU says that the AREF pin needs to be connected to power via a low-pass filter to remove noise if the analog to digital converter is being used. I've seen a number of websites that at least suggest this isn't necessary, especially if powering via a battery, rather than an alternating current based power source. I'll figure this out in due course but even if I do need to add extra components they will be priced in the pence so won't add much to the overall cost.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com0tag:blogger.com,1999:blog-7025390077222270096.post-30898784321757549032012-12-08T12:29:00.000+00:002012-12-08T12:29:55.255+00:00Levelling Out The Wear<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEia2MVI0Sa9BxoTMxUfbt87hhFptTn2ZgRUcJ4u97XVxGHwqYlcLJyieCihyLAdNQKzt3plXbhpNx_g49qXHU-buS0M0_kItS1EqinMzI-0P2JdWds7J_gk-Zy0D5cnAiZ2UUbzfDhdbcw/s1600/worn-tires.jpg" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-top:0.5em"><img border="0" width="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEia2MVI0Sa9BxoTMxUfbt87hhFptTn2ZgRUcJ4u97XVxGHwqYlcLJyieCihyLAdNQKzt3plXbhpNx_g49qXHU-buS0M0_kItS1EqinMzI-0P2JdWds7J_gk-Zy0D5cnAiZ2UUbzfDhdbcw/s320/worn-tires.jpg" /></a></div>I've been <a href="http://englishjavadrinker.blogspot.co.uk/search/label/Arduino">playing around with an Arduino</a> since May, but up until now I haven't had any need to use the 1024 bytes of EEPROM embedded in the <a href="http://www.atmel.com/devices/atmega328.aspx">ATMEGA328 microcontroller</a> at the heart of the Arduino UNO. <a href="http://en.wikipedia.org/wiki/Eeprom">EEPROM</a>, or to give it its full name Electrically Erasable Programmable Read-Only Memory, is really useful for storing configuration data as any values saved to EEPROM are retained even when the power is removed.<br />
<br />
The main problem with EEPROM is that it can quickly wear out. The datasheet for the ATMEGA328 gives an expected life of just 100,000 writes for each 32 byte page. If you are just using the EEPROM to save configuration data, which won't change very often, then this won't be a problem, but if you intend to frequently write to the EEPROM this could quickly become a real issue. To mitigate this problem what we need is a <a href="http://en.wikipedia.org/wiki/Wear_leveling">wear levelling</a> algorithm.<br />
<br />
In the sketch I'm working on I have just four bytes that I need to store as configuration data. The easiest option would be to simply store them at offsets 0, 1, 2 and 3, but of course this would mean that every time they were changed the same four bytes (and hence the same 32 byte page) would be re-written, while the rest of the EEPROM would remain unused. Clearly this won't result in even wear across the EEPROM.<br />
<br />
My solution, which probably isn't optimal, is to simply shift the data along by one byte each time it needs to be written. This completely ignores the idea of 32 byte pages, but if data is being written often enough then hopefully the pages should get fairly evenly used.<br />
<br />
Now the main problem of shifting the data along by one byte each time it changes is knowing where to read it back from when the Arduino starts up. The trick here is to employ some form of multi-byte marker that denotes the location of the data block. In this example I'm going to use the four characters "TS01" to stand for Test Sketch version 1. This allows me to increment the version number if I change the format of the stored data in the future, and tells me which sketch the data refers to. This second point is important if you are going to use the EEPROM across different sketches (less important if you dedicate an Arduino to a single project). So enough talking lets start putting this into practice.<br />
<br />
Firstly, while the core Arduino software does include an <a href="http://arduino.cc/en/Reference/EEPROM">EEPROM library</a> it's fairly rudimentary. Fortunately another Arduino user has contributed the <a href="http://playground.arduino.cc/Code/EEPROMex">EEPROMex library</a> which offers a whole bunch of useful methods (note that I did have a few problems getting this library to work; you have to make sure that the folder and filenames all share the same capitalization -- EEPROMex, and the header file doesn't need to include the original EEPROM library). This extended library is capable of writing arbitrary structures to EEPROM, so instead of having separate variables for the configuration data it's easier to store them in a custom structure giving us the following code:<br />
<pre class="brush:c">#include <EEPROMex.h>
#define CONFIG_VERSION "TS01"
struct ConfigStructure {
byte a, b, c, d;
char version[5];
} config = {
76, 74, true, 15,
CONFIG_VERSION
}, stored;
int offset = 0;</pre>As you can see this snippet imports the library, defines a constant to hold the multi-byte marker and then creates the custom structure. We also create two instances of the structure <code>config</code> and <code>stored</code> giving the first one some sensible default values for each variable, so that if we don't find any stored settings the sketch can still operate. You'll also notice that I've put the multi-byte marker at the end. This is so that it only gets written if we have successfully written the rest of the data -- I'll come back to this point later.<br />
<br />
Now before we get into actually reading and writing the settings I'm going to introduce a utility function for comparing the current and stored settings.<br />
<pre class="brush:c">boolean compareSettings() {
if (config.a != stored.a) return false;
if (config.b != stored.b) return false;
if (config.c != stored.c) return false;
if (config.d != stored.d) return false;
return true;
}</pre>Ideally I would have liked to actually pass the two instances to the function so that it wasn't a hard coded comparison of the <code>config</code> and <code>stored</code> instances, but there is an issue with passing user defined structures to functions in the version of the Arduino IDE I'm using (it has been fixed in 1.0.2 but I'm still on 1.0.0). Anyway this function simply checks that the two instances hold identical values for each variable.<br />
<br />
So now we can turn our attention to reading stored settings out of the EEPROM.<br />
<pre class="brush:c">void setup() {
//search through the EEPROM for a valid config structure
for ( ; offset < EEPROMSizeUno-sizeof(stored) ; ++offset) {
//read a struct sized block from the EEPROM
EEPROM.readBlock(offset, stored);
if (strcmp(stored.version, CONFIG_VERSION) == 0) {
//if the block has the right version tag then we can
//assume that the rest of the struct is valid
//overwrite the default settings with those we have
//just loaded from the EEPROM
config = stored;
//we don't need to look any further so break out of
//the loop to speed things up a bit
break;
}
}
if (!compareSettings()) {
//if they haven't changed the settings from the default then
//there is no reason to waste an EEPROM write in storing them
//so assume the stored version is the default
stored = config;
//as there wasn't anything in the EEPROM set the offset to -1
offset = -1;
}
}</pre>Hopefully the comments make this code easy to understand, but essentially all it does is read a block of data from the EEPROM, see if it is a valid settings object by checking if the <code>version</code> variable is the same as the <code>CONFIG_VERSION</code> constant. If it is then we ensure that the current <code>config</code> structure is the same as the <code>stored</code> structure, otherwise we move on 1 byte and try again (this all happens in lines 3 to 20, ). Once we have searched through the entire EEPROM, either we will have found some settings and <code>config</code> and <code>stored</code> will be equal, or we haven't found any and they will differ. Note that we control the loop using the <code>EEPROMSizeUno</code> constant defined by the EEPROMex library, if you have a different Arduino then you will need to change this. If, when we have searched through the EEPROM, <code>config</code> and <code>stored</code> aren't the same then lines 22 to 30 make them so (so we don't store the default values to EEPROM for no reason) and reset the offset so that when it is incremented to save the settings it moves to the beginning of the address range.<br />
<br />
Now in the main part of your sketch when you decided the settings need to change you can simply update the <code>config</code> instance and call a <code>saveSettings</code> function to actually write them to the EEPROM.<br />
<pre class="brush:c">void saveSettings() {
if (!compareSettings()) {
//we need to store the new settings in EEPROM
//move on one position from the previous place we stored things
++offset;
//if writing at offset would mean going outside the EEPROM limit then
//reset the pointer to the beginning of the EEPROM
if (offset > EEPROMSizeUno-sizeof(config)) offset = 0;
//do the actual EEPROM writing
EEPROM.updateBlock(offset, config);
//stored should now be the same as config
stored = config;
}
}</pre>Essentially if the <code>config</code> and <code>stored</code> instances differ then we need to write <code>config</code> to the EEPROM. We increment the offset (the wear levelling part), check we haven't gone outside the valid range, and then write the block (we actually use update so that we don't write bytes that won't actually change) and now we know that <code>config</code> and <code>stored</code> are the same again.<br />
<br />
I mentioned earlier that the reason for putting the multi-byte marker at the end of the config structure was so that I knew once it had been written then the rest of the data must have been as well. The problem here is that if writing fails before the start of the marker we will have no way of knowing, other than we will see random configuration data. The easiest option would be to add a different marker at the start. If both markers are present and in the right place then the data between them must be valid. The problem here is that a four character marker takes up more space than the config data I'm saving so having two of them seems like a waste.<br />
<br />
The approach I'm taking is to compute a hash of the four config values and store this as the first piece of data. This means that when I read the structure back I can check for the multi-byte end marker and that the data matches the hash. The problem is that the hash function is specific to the number and type of config variables you want to store and so isn't a generic solution but something you would have to adapt to your needs. As an example though here is how I'm currently hashing the four bytes of configuration data.<br />
<pre class="brush:c">const int prime = 31;
int result = 1;
result = prime * result + config.a;
result = prime * result + config.b;
result = prime * result + config.c;
result = prime * result + config.d;</pre>This code (which is based on what Eclipse generates for a four byte java structure) generates an integer (two bytes on the Arduino) based upon the four config values. Changing any of the config values changes the result of the hash function. There is also a second reason for using a hash function: data stored in EEPROM can degrade over time and the hash function allows us to check that the data hasn't changed since we stored it.<br />
<br />
I know this hasn’t led to a single example sketch you can use, but hopefully the snippets I've presented will come in handy.Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com2tag:blogger.com,1999:blog-7025390077222270096.post-62453055542444996652012-10-22T17:30:00.000+01:002012-10-22T17:35:32.098+01:00Write Once, Debug Everywhere<img style="float:left; padding-right:5px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUKUWtgE3WW0uWXFWciRbWIy6AIf5BgrwDJehGqGr5lUcSno4AfdsFg1bulKTDyg-kVV8ZfMAwcpjsJngyJzvqAewmVdauPpaO3-Fcj7sE_UfBsEA4msQsmo3Pgem7-DyfeUGiC_lN5oI/s320/computer-bug.jpg" />One of the reasons that I quite enjoy developing software in Java is the promise of WORA or <i>write once, run anywhere</i>. In principle this means that I can develop software on one machine and it should work exactly the same on any other computer, regardless of operating system, as long as a copy of Java is installed. I've been writing Java code for over a decade and in all that time I don't think I've ever personally seen a case where the principal falls down -- and certainly not if the code is restricted to just using the core Java libraries. So this morning, with a deadline fast approaching, I hit the opposite of WORA, WODE; <i>write once, debug everywhere</i>!<br />
<br />
I'd written and debugged some code on my PC, and it was working flawlessly. I'd integrated the code into a tomcat hosted web app, and again testing on my own machine showed no problems at all. I uploaded the web app to one of the servers at work (more processing power and a better internet connection for demo purposes) and it instantly crashed! A quick look at the log showed the problem to be a <a href="http://docs.oracle.com/javase/6/docs/api/java/util/ConcurrentModificationException.html">ConcurrentModificationException</a>. I've seen this exception before when I've tried to modify a list (or some other <a href="http://docs.oracle.com/javase/6/docs/api/java/util/Collection.html">Collection</a> instance) while looping through the list, but in this case a) I couldn't see anything wrong with my code and b) I had a machine where the code was working fine!<br />
<br />
After a little debugging it became clear that the exception was being thrown by the following line of code:<br />
<pre class="brush:java">items.removeAll(items.subList(0, items.size()/2));</pre>Essentially this should halve the original list, retaining the second half (the items in the list were sorted based on an assigned score and I wanted to throw away the lower scoring items). It's worth noting that the <code>subList</code> call returns a view backed by the original list. In other words changes to one list should be reflected in the other. As the code worked fine on my machine I was more than a little confused.<br />
<br />
Given that uploading the full web app to the server at work was a relatively time consuming process I wrote a small test application instead to try and figure out what was happening, including the expected output in the comments.<br />
<pre class="brush:java">// create a list of four strings
List<String> items = new ArrayList<String>(
Arrays.asList(new String[]{"a","b","c","d"}));
System.out.println(items); // [a, b, c, d]
// get a view over the first half of the list
List<String> someItems = items.subList(0, items.size()/2);
System.out.println(items); // [a, b, c, d]
System.out.println(someItems); // [a, b]
// change the first item in the sub-list to "e"
System.out.println(someItems.set(0, "e")); // a
System.out.println(items); // [e, b, c, d]
System.out.println(someItems); // [e, b]
// change the first item in the original list to "f"
System.out.println(items.set(0,"f")); // e
System.out.println(items); // [f, b, c, d]
System.out.println(someItems); // [f, b]
// remove all items in the sub-list from the original list
System.out.println(items.removeAll(someItems)); // true
System.out.println(items); // [c, d]
System.out.println(someItems); // will probably throw an exception</pre>I then ran this on my local machine, where my original code had been working and got the following result:<br />
<pre class="brush:bash">mark@mag:~$ java -version
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.4) (6b24-1.11.4-1ubuntu0.12.04.1)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
mark@mag:~$ java CMETest
[a, b, c, d]
[a, b, c, d]
[a, b]
a
[e, b, c, d]
[e, b]
e
[f, b, c, d]
[f, b]
true
[c, d]
Exception in thread "main" java.util.ConcurrentModificationException
at java.util.ArrayList$SubList.checkForComodification(ArrayList.java:1091)
at java.util.ArrayList$SubList.listIterator(ArrayList.java:972)
at java.util.AbstractList.listIterator(AbstractList.java:300)
at java.util.ArrayList$SubList.iterator(ArrayList.java:968)
at java.util.AbstractCollection.toString(AbstractCollection.java:431)
at java.lang.String.valueOf(String.java:2838)
at java.io.PrintStream.println(PrintStream.java:788)
at CMETest.main(CMETest.java:24)</pre>So far so good. The output matches what I expected to happen, including the fact that the last line throws an exception. It makes sense to throw an exception at this point as I'm essentially trying to access a list consisting of items which I've deleted. So having checked that my test case behaved as I expected I uploaded this to the server at work and ran it.<br />
<pre class="brush:bash">mark@derwent:~$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
mark@derwent:~$ java CMETest
[a, b, c, d]
[a, b, c, d]
[a, b]
a
[e, b, c, d]
[e, b]
e
[f, b, c, d]
[f, b]
Exception in thread "main" java.util.ConcurrentModificationException
at java.util.SubList.checkForComodification(AbstractList.java:752)
at java.util.SubList.listIterator(AbstractList.java:682)
at java.util.AbstractList.listIterator(AbstractList.java:284)
at java.util.SubList.iterator(AbstractList.java:678)
at java.util.AbstractCollection.contains(AbstractCollection.java:82)
at java.util.AbstractCollection.removeAll(AbstractCollection.java:336)
at CMETest.main(CMETest.java:22)</pre>Fortunately for my sanity this shorter example failed to behave in the same way on the server as it did on my machine. You can see from the exception that it is failing on line 22 which is the call to <code>removeAll</code>. While that at least proves that the <code>removeAll</code> call is at fault it still leaves the question as to why it works on one machine and fails on another?<br />
<br />
Even when the Java language wasn't available under an open source license the source code to the core APIs has always been available. Back in the early days of Java I even used this fact on a number of occasions to submit bug reports to Sun (who originally developed Java before they were bought by Oracle) along with suggested fixes. Unfortunately I haven't been able to track down the source bundles for the exact two versions of Java involved but I did manage to track down the OpenJDK source for version 1.6.0 build 26. This doesn't match either version exactly as I'm running b12 of OpenJDK whereas the server is running buld 26, but the Oracle version rather than OpenJDK. Having looked at the stack trace and this version of the source code I think I can now see why it fails.<br />
<br />
The stack trace shows that when I call <code>removeAll</code> on list (which is actually an instance of <code>ArrayList</code>), the implementation in <code>java.util.AbstractCollection</code> is the one that is actually used. This method is a very simple iterator based implementation:<br />
<pre class="brush:java">public boolean removeAll(Collection<?> c) {
boolean modified = false;
Iterator<?> e = iterator();
while (e.hasNext()) {
if (c.contains(e.next())) {
e.remove();
modified = true;
}
}
return modified;
}</pre>Now the <a href="http://docs.oracle.com/javase/6/docs/api/java/util/List.html#subList(int, int)">documentation for the <code>subList</code> method</a> states that<br />
<blockquote>The semantics of the list returned by this method become undefined if the backing list (i.e., this list) is <b>structurally modified</b> in any way other than via the returned list. (Structural modifications are those that change the size of this list, or otherwise perturb it in such a fashion that iterations in progress may yield incorrect results.)</blockquote>Clearly the call to <code>e.remove()</code> on line 6 <i>structurally modifies</i> the backing list, which will render the sublist, <code>c</code> undefined, which explains why an exception is thrown (probably on the call to <code>contains</code> on line 5). So now I know why it fails on the server, but why does it work on my machine.<br />
<br />
I'm guessing that in the version of Java running on my machine, when I call <code>removeAll</code>, rather than using the default implementation in <code>AbstractCollection</code>, it instead uses the <code>ArrayList</code> specific version.<br />
<pre class="brush:java">public boolean removeAll(Collection<?> c) {
return batchRemove(c, false);
}
private boolean batchRemove(Collection<?> c, boolean complement) {
final Object[] elementData = this.elementData;
int r = 0, w = 0;
boolean modified = false;
try {
for (; r < size; r++)
if (c.contains(elementData[r]) == complement)
elementData[w++] = elementData[r];
} finally {
// Preserve behavioral compatibility with AbstractCollection,
// even if c.contains() throws.
if (r != size) {
System.arraycopy(elementData, r,
elementData, w,
size - r);
w += size - r;
}
if (w != size) {
for (int i = w; i < size; i++)
elementData[i] = null;
modCount += size - w;
size = w;
modified = true;
}
}
return modified;
}</pre>As you can see this implementation is slightly more complex, but I'm guessing that it's a lot more efficient. Essentially this implementation modifies the underlying array directly rather than via the public API -- it works by shifting each entry towards the beginning of the array overwriting any entries that need to be removed. Similarly the implementation of <code>contains</code> called from line 11, access the array directly meaning that the entire <code>removeAll</code> call never results in a check for a concurrent modification and hence never throws an exception.<br />
<br />
So now I understand why the same code behaves differently on two different machines. Well I <i>almost</i> understand. The missing piece of the puzzle is why the rather naive version of <code>removeAll</code> is used on the server? Without access to the source code of the exact versions of Java involved I can't be certain, but I'm assuming that the <code>ArrayList</code> specific implementation is an OpenJDK efficiency improvement that isn't in the Oracle branded version.<br />
<br />
Fortunately my original code is easy to fix, I can achieve the same effect (deleting the items in the first half of the list) using the following line of code which does appear to <i>run anywhere</i>.<br />
<pre class="brush:java">items.subList(0, items.size()/2).clear();</pre>Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com0tag:blogger.com,1999:blog-7025390077222270096.post-36601231133544608102012-09-19T19:17:00.002+01:002022-10-19T08:47:19.686+01:00Formatted Comments<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDvZwSajmvk5JDNU6nJjUQGPGGNbMnCpVXBw0KS_RY8RlXjpKZjTOc2exg324hTtK9gS-mD3Aa0naOoHV9Tpa3f4LPSawLBOsrvzAIFLpE7Y26vu99sacZo68Vru-aot8ULHp3IIWEeiE/s1600/blogger-comment-editor.png" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:0em"><img border="0" height="200" width="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDvZwSajmvk5JDNU6nJjUQGPGGNbMnCpVXBw0KS_RY8RlXjpKZjTOc2exg324hTtK9gS-mD3Aa0naOoHV9Tpa3f4LPSawLBOsrvzAIFLpE7Y26vu99sacZo68Vru-aot8ULHp3IIWEeiE/s320/blogger-comment-editor.png" /></a></div>One thing that has always confused me when using Blogger is why the comment form lacks any real features. When we use Blogger to write a post the dashboard gives us lots of options for formatting text and adding links etc. but when, as a reader of a blog, you want to add a comment you are presented with a simple text box. If I want to add a link or make some text standout I can, but I have to manually enter the required HTML. This isn't too much of a problem for me, but I'm sure there are many people that read blogs and leave comments who aren't comfortable writing HTML. I'm guessing that this explains why I see so many comments that contain URLs which aren't actually links.<br />
<br />
So to make my life a little easier, and to enable other people to easily add formatting and links to their comments, I've developed a script that provides buttons for bold, italic and for adding links. Unfortunately I couldn't find a way of adding something to a blogs template to make this work, rather I've developed a <a href="http://wiki.greasespot.net/Greasemonkey">Greasemonkey</a> script that, once installed in your browser, will add this functionality automatically to every Blogger hosted blog you visit.<br />
<br />
If you are using Firefox then you will need to have the <a href="https://addons.mozilla.org/en-US/firefox/addon/greasemonkey/">Greasemonkey extension</a> installed. If you use Chrome then (due to an annoying change Google have recently made) you will need to install <a href="https://chrome.google.com/webstore/detail/dhdgffkkebhmkfjojejmpbldmpobfkfo">Tampermonkey</a>, and for Safari you will need to install <a href="http://wiki.greasespot.net/Cross-browser_userscripting#NinjaKit">NinjaKit</a>. The script also works in <a href="http://www.chromium.org/Home">Chromium</a> (the open source version of Chrome without the Google branding) without requiring you to install an extension first.<br />
<br />
Once you have your browser ready you can simply <a href="http://apps.englishcoffeedrinker.co.uk/greasemonkey/blogger_comment_editor.user.js">click this link</a> to install my script. If you are using Tampermonkey then, when asked, click OK to install into Tampermonkey rather than Cancel to install natively in Chrome.<br />
<br />
Once installed whenever you go to add a comment on a Blogger hosted blog you should find links for bold, italic and to add a link just above the comment box. Hopefully there shouldn't be any bugs as a) it's a fairly simple script and b) thanks to <a href="http://adriansimages.blogspot.co.uk/">Adrian</a> it's actually undergone some user testing, but if you have any problems or any suggestions for improvements then leave me a comment and I'll see what I can do.<br />
<hr><i>Updated 14th July 2013: Added the link to NinjaKit for getting it to work in Safari.</i>
<br/>
<i>Updated to v0.6 18th October 2022: Now supports the newer blogger comment form.</i> Markhttp://www.blogger.com/profile/02418527698793489162noreply@blogger.com14