Index and Search structured XML documents using Apache Solr

Learn how to index and search structured XML documents using Apache Solr Search Platform.

Apache Solr is highly scalable search engine with lots of goodies in-built. In this guide we will learn how to get our structured data in XML can be indexed and searched effectively.

We will learn following concepts:

  1. Starting up Apache Solr
  2. Importing structured XML document for indexing in Apache Solr

Tools & Library used in this project:

  1. Apache Solr 5.3.0
  2. Java 8
  3. Mac OSX

Downloading & Starting Apache Solr

Download Apache Solr Binary Distribution

We can download Apache Solr latest version from their official website. When we click on major or mirror download distribution link, we got a page like it:

apache solr download page package to choose

Tip: Apache Solr downloadable package size is around 130 MB. Make sure you have this much bandwidth left on your internet connection.

Unpack Apache Solr Binary Download Zip

When we unpack Apache Solr Binary Download Zip we see following files and folder inside main folder:

apache solr 5.3 binary folder structure Apache Solr 5.3 installation folder structure

Starting, Stopping and Restarting Apache Solr

Starting Apache Solr Server

$ cd /Volumes/Drive2/App/solr-5.3.0/

Start Solr Server

$ bin/solr start

Apache Solr has been started at https://localhost:8983/solr.

Stopping Apache Solr

$ cd /Volumes/Drive2/App/solr-5.3.0/

Stop Solr

$ bin/solr stop -p 8983

Restarting Apache Solr

$ cd /Volumes/Drive2/App/solr-5.3.0/

Stop Solr

$ bin/solr restart -p

Note: Replace Solr folder path with your installation path

Let’s create a core (or Collection) “xmlhub”

$ bin/solr create -c xmlhub

Setup new core instance directory: /Volumes/Drive2/App/solr-5.3.0/server/solr/xmlhub

Creating new core ‘xmlhub’ using command: https://localhost:8983/solr/admin/cores?action=CREATE&name=xmlhub&instanceDir=xmlhub

## Indexing XML files

### Sample XML File

We will be indexing xml files kept in a folder (In our application its at _<solr\_installtion\_root\_dir>/example-data_). An example of a XML file content:

**File: example1.xml**
<?xml version="1.0" encoding="UTF-8"?>
<ele xmlns:dc="">
  Atrr1 Value 1
  Attr2 Value 1
<meta property="meta1">
  Meta 1 Val 1
<meta property="meta2">
  Meta 2 Val 1
<meta name="name1">
  Name 1 value 1
<meta name="name2">
  Name 2 value 1

Uploading XML structured data for Indexing using Data Import Handler

Step 1 : Configure solrconfig.xml

We will find solrconfig.xml file in location <solr_installtion_root_dir>/solr/<collection/node_name>/conf.

Location of solrconfig.xml in solr 5.3 installation Location of solrconfig.xml in solr 5.3 installation

File: solrconfig.xml

<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.\*.jar" />
<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
    <lst name="defaults">
      <str name="config">xmlhubconfig.xml</str>

We can place this code in solrconfig.xml

Step 2: Create Data Import configuration

We may can provide data import configuration in solrconfig.xml file, but we choose to do that in external file xmlhubconfig.xml.

File: xmlhubconfig.xml

  <dataSource type="FileDataSource"/>
    <!-- this outer processor generates a list of files satisfying the conditions specified in the attributes -->
    <entity name="f" processor="FileListEntityProcessor" fileName=".\*.xml$" recursive="true" rootEntity="false" dataSource="null" baseDir="/Volumes/Drive2/App/solr-5.3.0/example-data">

      <!-- this processor extracts content using Xpath from each file found -->

      <entity name="nested" processor="XPathEntityProcessor" forEach="/ele | /metadata" url="${f.fileAbsolutePath}" >
              <field column="attr1\_s" xpath="/ele/attr1"/>
              <field column="attr2\_s" xpath="/ele/attr2"/>
              <field column="meta1\_s" xpath="/ele/meta\[@property='meta1'\]"/>
              <field column="meta2\_s" xpath="/ele/meta\[@property='meta2'\]"/>
              <field column="name1\_s" xpath="/ele/meta\[@name='name1'\]"/>
              <field column="name2\_s" xpath="/ele/meta\[@name='name2'\]"/>

This configuration is specific to XML file structure. Pay attention on how we had used XPATH. You should also replace baseDir with your path.

Step 3: Configure to generate unique id automatically

In solrconfig.xml we will be using updateRequestProcessorChain to setup UUIDUpdateProcessorFactory to generate unique UUID for id column.

File: solrconfig.xml

      <processor class="solr.UUIDUpdateProcessorFactory">
        <str name="fieldName">id</str>
      <processor class="solr.LogUpdateProcessorFactory" />
      <processor class="solr.RunUpdateProcessorFactory" />

Index File

We should restart Apache Solr.

Go to  https://localhost:8983/solr/#/xmlhub/dataimport//dataimport:

Execute dataimport handler to index xml Execute dataimport handler to index xml

It will index the xml files and create documents. You can browse the document at  https://localhost:8983/solr/xmlhub/browse.

Browse indexed documents in built-in solr collection browser Browse indexed documents in built-in solr collection browser

Using the built-in collection browser we can search indexed documents. Learn more about Solr Query Syntax at official documentation. Apache Solr also provides API to access search interfaces with all the available features.


  1. Learn about the Apache Solr Query Syntax
  2. Apache Solr Data Import Helper Documentation
  3. Apache Solr Site
Mrityunjay Kumar
Mrityunjay Kumar
Principal Architect

My interests include software architecture, cloud computing, data & Artificial Intelligence.

comments powered by Disqus