Wi-Fi and Bluetooth USB dongles for Raspberry Pi 1 model B

I wanted to expand my Raspberry Pi with Wi-Fi and Bluetooth capabilities. For this task I bought two cheap USB dongles on the AliExpress website. A 802.11n Wi-Fi dongle with Realtek's RTL8188EUS chipset and a USB Bluetooth adapter compatible with Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode) adapter.

Both of these devices can be used with the latest version of Raspbian. Check out list of Raspberry compilant hardware:

• http://elinux.org/RPi_USB_Wi-Fi_Adapters
• http://elinux.org/RPi_USB_Bluetooth_adapters

First of all I had to upgrade Raspbian to latest version.

$ rpi-update

# Check out Raspbian's version
$ uname -a
Linux raspberrypi 4.1.16+ #833 Wed Jan 27 14:29:11 GMT 2016 armv61 GNU/Linux

$ sudo apt-get update & sudo apt-get upgrade

After upgrading I checked connected hardware:

$ lsusb
Bus 001 Device 004 ID 0a12:0001 Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)
Bus 001 Device 005 ID 0bda:0179 Realtek Semiconductor Corp.

Then I configured configured connection to my Wi-Fi router guided by Oliver's article.

#/etc/wpa.config
network={
    ssid=""
    proto=RSM
    key_mgmt=WPA-PSK
    pairwise=CCMP TKIP
    group=CCMP TKIP
    psk=""
}

#/etc/network/interfaces
allow-hotplug wlan0
auto wlan0

iface wlan0 inet dhcp
wpa-conf /etc/wpa.config

Wi-Fi started working on reboot of Raspbian.

Installation of Bluetooth keyboard was quite easy as well. I followed steps described in Tyler's article. First I've installed bluetooth support into my Raspbian and then connected wireless keyboard via bluez tools.

# To scan nearby Bluetooth hardware ready to be paired
$ hcitool scan
Scanning ...
  7C:1E:52:AA:0B:E3 Microsoft Sculpt Wireless Keyboard
# To connect keyboard and to 
$ sudo bluez-simple-agent hci0 7C:1E:52:AA:0B:E3
$ sudo bluez-test-device trusted 7C:1E:52:AA:0B:E3 yes 
$ sudo bluez-test-input connect 7C:1E:52:AA:0B:E3

Gradle multi-project using cross dependent modules

I have a framework which consists of several modules. On top of this framework I'd like to build various projects where each project can use one or more modules of this framework. In real world these projects can represent framework's implementations for customers. Framework's modules can depend on each other. For this task I decided to use Gradle build system.

Part of this article is an example project on GitHub.

Setting up the build

I've created three modules that are shared by two root projects. Project1 and project2. Second project doesn't use one of the modules as it shown by following diagram. Both root projects has very similar build scripts so I will focus on project1 for the rest of this article.

:project1
+--- :module1
|    \--- :core-module
\--- :module2
     \--- :core-module

:project2
+--- :module1
     \--- :core-module

Root project's build script contains buildscript block which introduces plugin dependency and other common configuration used by Gradle when compiling/building project. The build script applies spring-boot-multi-project on the root project. This plugin is wrapper of the Gradle Spring Boot plugin and enhances it's functionality. The plugin also adds new task that generates serialized version of dependency graph. We will get to the dependency graph later.

There are three modules (sub-projects). A core module which is Spring Boot project and two modules (module1 and module2) which depends on core module and extends its functionality. These modules forms the framework. The framework is just a collection of libraries so it can't work as a standalone application.

Core module depends on Spring Boot starter project so to be compiled it needs to have compile dependency configuration set to spring-boot-starter. To be able to test this module it also need to have declared testCompile dependency on spring-boot-starter-test.

Note that module's build scripts does not contain repository configuration or does not apply any plugin. Repository configuration is included in spring-boot-multi-project plugin which is applied in root project's build script. This plugin adds Maven Central Repository and JitPack repository to every module including root project.

Module1 depends on core module. And because core module contains Spring Boot project it's not necessary to declare Spring Boot compile dependency in module1. Compile dependencies are transitive so all dependencies from core module are going to be placed on classpath of module1 too.

This rule doesn't apply to testCompile dependencies. When a module (subproject in Gradle's terminology) is tested it's tested separately. This means module is first compiled with all it's dependencies (not root project's dependencies) and it's own testCompile dependencies (not a root project's or subproject's dependencies) and then tests are executed. This means three things.

1. You need to add necessary testCompile dependencies to each subproject's build script.
2. If a module depends on a resource which is not added in compile dependencies you need to add this resource as a testCompile dependency to build script of the module. For example the resource can be a configuration file located in root project. To add directory as a testCompile dependency you can use file command testCompile files("path/to/resource").
3. If you want to use some shared class in your tests you will need to place it to main source set or a new source set and declare a dependency to it. In this example a source set called testFixtures is created in core module. Support for testFixtures source set is added in spring-boot-multi-project plugin.

Of course Gradle's multi-project build has to contain settings.gradle file which tells Gradle where all modules (in Gradle's terminology sub projects) are located.

Project dependencies in application

Sometimes it is useful to know module dependencies in the application. For example if you want to execute data model updating scripts you will need to know which script should run first (core project's) and which last (root project's which depends on everything else).

To do this you can use project dependencies file generated by discoverProjectDependencies task. With this graph you can easily sort resources on classpath in certain order as it is shown in ProjectDependencyManager service.

Running the project

To start projects or to build it you can experiment with tasks provided by Gradle in project1 or project2 directories. For example:

gradle discoverProjectDependencies - to serialize project dependencies into a file
gradle bootRun - to run the application
gradle build - to build jar file
gradle test - to run all tests

If you'd like to open project in IntelliJ's IDEA use the import Gradle project functionality instead of simple open project. Import project will open all framework modules as a project modules so you'll have all required source codes available via project panel. You can also use IDEA's Gradle plugin to easily modify build scripts.

When running project in IDEA please make sure that your run configuration does have root project selected in use classpath of module option.

Be aware of if you don't add any source code into Gradle project then Gradle won't generate empty build directories for this project. Unfortunately it adds these non-existing build directories to a classpath when bootRun or a simmilar task is executed. Invalid paths breaks Java's class loader and it causes problems with loading libraries which usually ends up with java.io.FileNotFoundException: class path resource [] cannot be resolved to URL because it does not exist error message. This can happen in early stages of a project when you create an empty module with some static content but without any Java code.

Generating Java classes with the Java 8 date and time API types by the JAXB

The JAXB's XJC is very popular tool for generating Java classes from a specification files (XSD or WSDL). Java 8 introduced new Date and Time API. Unfortunately this API does not come with (un)marshaller so XJC still generates code using XMLGregorianCalendar type by default.

Because I want to work with the LocalDateTime I had to find and link (un)marshaller for the new API to my project. I decided to use Mikhail Sokolov's adapters.

Then I had to tell XJC that it should generate code with the LocalDateTime for the specification's date-time types. This can be achieved by a binding file. So create a new binding.xjb file and put mapping of xsd:dateTime to LocalDateTime type by using LocalDateTimeXmlAdapter adapter/(un)marshaller.

This binding file uses nonstandard adapter parameter so you need to add -extension switch to be able to execute XJC. Whole command should look like this: xjc -d <target_directory> -b binding.xjb -extension <specification_file.xsd>

The generated code is pretty ok except for one small thing. For initialisation of generic types the generated code uses verbose form instead of diamond operator. IntelliJ IDEA's static code analysis complains about it. To fix this I decided to post-process the code by regular expression and replace all verbose initialisations by the shorter diamond.

Whole process of the generating and the post-processing can be put into a shell script generate.sh for easy use.

Development of the Apache Nutch 1.10 plugin in IntelliJ IDEA using the Maven project

Apache Nutch is quite old project using Apache Ant to build itself. I wanted to develop a plugin for filtering content of parsed pages and because I am IntelliJ IDEA user I wanted to do it in this IDE. This how-to helps you to setup IDEA project so you can develop and debug the Nutch (parse) plugin. I must mention great article by Emir Dizdarevic the Precise data extraction with Apache Nutch which helped me a lot.

Part of the article is a template project: https://github.com/vkuzel/Nutch-Plugin-Development-Template

Overview

When started from IDEA the project is first built by Maven and then deployed to Nutch binary installation. Nutch task (in this case parse) is then started and Nutch itself runs the plugin. IDEA attaches a debugger to Nutch process and you can debug the code.

Maven project

Apache Nutch is present in the Maven repository but unfortunately contains some weird dependencies. To compile plugin you need to add dependency to Nutch org.apache.nutch.nutch version 1.10 and to Hadoop org.apache.hadoop.hadoop-core version 1.2.0 (Hadoop is a core library of Nutch). To compile the project you need to exclude org.apache.cxf dependency from Nutch library because Maven cannot resolve it. Btw. I figured out that different versions of Nutch need to exclude different libraries.

To allow Nutch to run the plugin it has to be built and deployed to Nutch installation directory during every debug session. This is managed by external shell script deploy_plugin_to_nutch_for_debug.sh which is executed by Maven on a install phase of a build process. For this task I incorporated the exec-maven-plugin plugin into the pom.xml file.

Nutch installation

Since this article covers development of Nutch plugin there's no need to have complete Nutch source codes. To run plugin you need properly configured Nutch binary installation that can be downloaded from it's official site. In the template project there's empty directory nutch-1.10 where Nutch should be copied. Only necessary change to Nutch cofiguration (apart from default installation process) is to add the plugin to the plugin.includes directive so Nutch can recognize it.

There's also archive test_data.zip with the pre-downloaded page that can be used to test the parser plugin. This archive is extracted to the nutch-1.10/test_data directory so the plugin is always working with same data.

Project (debug) configuration

Because project uses an external application (Nutch) an custom application run/debug configuration is needed in IDEA. Add new debug configuration with following parameters:

• Main class: org.apache.nutch.parse.ParseSegment. Nutch is started directly by calling its class not by usual shell script located in the bin directory of its installation.
• Program arguments: test_data/crawl/segments/20151010172800. This is path to test data extracted from the test_data.zip archive. Test data contains one html page just to test one pass through the plugin.
• Working directory: nutch-1.10. Besides of this working directory it is also necessary to add Nutch's conf and lib directories to a classpath. Do it by adding them in the Project Settings -> Modules -> Dependencies menu.
• Before launch run Maven Goal: clean install. Before every execution module has to be built and deployed to Nutch installation. Because of it it's necessary to execute an install goal of Maven's build process.

Now by running the debug configuration you should be able to debug the plugin.

Creating offline fallback page for the AngularJs web application using ApplicationCache

I needed to create a page that is going to be displayed to user who opened my application in the past but currently is without internet connection or he is working with the application but his connection is suddenly lost.

Client is opening the application but does not have connection available

For this purpose I decided to use ApplicationCache. I wanted that user will be redirected to offline.html page if index.html page is not loaded because connection is lost. This means I couldn’t put manifest directive to index.html page because page with manifest is automatically cached but I needed only offline.html page to be cached. So I placed manifest to offline.html instead and invoked it by placing object element to index.html.

Then defined failover page in ApplicationCache manifest file.

From now if user tries to open the application without internet connection or without server working a offline.html page will be displayed to him instead.

Client is working with the application and connection is suddenly lost

At this state only requests to server are done via XHR so we just need to intercept responses and if error occurs because connection is lost to redirect client to offline page. Interceptor can be added by $httpProvider.

How to connect Apache Solr 5.2.1 with Apache Nutch 1.10 and build concordancer on top of this stack

Apache Solr is search server based on Apache Lucene search library that allows you to index and search text content. This is great base for concordancer service. Solr itself cannot gather data from the internet. Apache Nutch was created to handle this job. This article describes basic steps of connecting Nutch to Solr and configuring concordancer service.

This text covers configuration for Solr 5.2.1 and Nutch 1.10 on Linux/Unix/OSX operating system. It's possible that configuration is going to change in future releases.

At first download binary versions of Solr and Nutch from project's websites. Then unpack both projects into one directory, let's say concordancer.

Go to Solr direcotry (solr-5.2.1) and start it by calling

bin/solr start

Then create search core named "concordancer".

bin/solr create -c concordancer

Core is the database of indexed data and configuration of how to perform search on this data. For example you can have several cores one for searching intranet sites another for searching internet websites.

Newly created core concordancer uses default configuration that needs to be changed. Without these changes Nuch can't interoperate with Solr. Open the configuration file server/solr/concordancer/conf/solrconfig.xml.

Almost at the end of file you can see directive <updateRequestProcessorChain name="add-unknown-fields-to-the-schema”>. This tells to Solr that when indexing data Solr should define index structure (de facto database structure for indexed data) dynamically. This means if new data structure is indexed (like website with all its metadata) new fields will be added to index structure described in generated managed-schema file. We don't need this feature so remove <updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> and it's content. Also remove <schemaFactory class="ManagedIndexSchemaFactory"> and it's content. Try to find <initParams path="/update/**”> it should contain add-unknown-fields-to-the-schema parameter so remove this directive too.

Search for all occurences of string _text_ in config file and change them to text. Newly created schema file below does not contain field named _text_ so that’s why we have to change it's name.

Now remove generated file managed-schema from configuration directory and replace is by schema.xml file prepared by Nutch developers. This means copying apache-nutch-1.10/conf/schema.xml to solr-5.2.1/server/solr/concordancer/conf directory.

Open the copied configuration schema.xml and find directive <field name="content" type="text" stored="true" indexed="true”/>. Change parameter stored from value false to true. This means that you don’t want to just index page's content but also store it's text for concordancer purposes.

Find directive <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt”/> and delete it. Then close the file and restart Solr by calling:

bin/solr restart

Let’s turn our attention to crawling websites by Nutch so go to directory apache-nutch-1.10 and try to run program without parameters:

bin/nutch

You should see output like this:

If there's some problem please go to Nutch tutorial for more details.

Open Nutch configuration file conf/nutch-site.xml and add configuration of Nutch User-agent http header:

Then create directory that is going to contain files with URLs to be crawled. In this directory create file seed.txt with one URL per line. You should end up with structure like this:

apache-nutch-1.10/urls/seed.txt

Then execute crawling and indexing of pages by calling crawl command.

bin/crawl -i -D solr.server.url=http://localhost:8983/solr/concordancer urls/ crawl/ 1

Troubleshooting: If some error appear while crawling check out the log files solr-5.2.1/server/logs/solr.log and apache-nutch-1.10/logs/hadoop.log Actually Solr's schema.xml file contains some problems like missing field types, etc. Solving of these problems is up to you.

Open the Solr's query page and try to search newly indexed data.

Let's configure Solr's concordancer ability. Open the configuration file server/solr/concordancer/conf/solrconfig.xml and locate configuration directives of highlighter (tag <searchComponent class="solr.HighlightComponent" name="highlight”>). Change the default boundary scanner from simple boundary scanner to break iterator and configure break iterator’s bs.type to SENTENCE. Boundary scanner finds boundaries of sentence so from now concordancer can display whole sentences.

Finally we need to modify request handler to display found sentences. Go to required request handler and add highlighter configuration. I choose to modify /query handler so its configuration looks like following:

Open the Solr admin http://localhost:8983/solr/#/concordancer/query and fire some queries on /query request handler. You should see result similar to this:

Apache 2.4.7 v Xubuntu 14.04 nenačítá některé konfigurační soubory

Po přechodu z Xubuntu 12.04 na verzi 14.04 s instalací nového web-serveru Apache2 mi přestaly fungovat VirtualHosty. Konfiguraci VirtualHostů mám uloženou v adresářích jednotlivých projektů a nalinkovanou do složky s konfigurací webů sites-enabled.

/etc/apache2/sites-enabled/projekt1 -> /home/vkuzel/projects/projekt_dir/projekt1

Problém je ale v tom, že poslední verze Apache2 odmítají načítat konfigurační soubory, které nemají příponu conf. Prý je to z důvodu, že u těchto souborů nelze použít nástroje jako a2enconf, a2ensite atd. Řešením je tedy přidat příponu souboru .conf ke konfiguracím.

/etc/apache2/sites-enabled/projekt1.conf -> /home/vkuzel/projects/projekt_dir/projekt1.conf