Showing posts with label setup. Show all posts
Showing posts with label setup. Show all posts
, , , , , , , ,

MongoDB - Replica Sets with Spring Data MongoDB (Part 2)

Review

In the previous section, we have introduced and discussed Replica Sets. In this section, we will install MongoDB and configure Replica Sets.


Installation

Our first step is to download and install MongoDB in each server. Since our server is running CentOS, we can download the CentOS MongoDB package by following the instructions from this link. However, it is easier to simply install the prebuilt binaries as stated in the MongoDB downloads section, so we will follow this advice instead.

1. Open a browser and visit the MongoDB download section at http://www.mongodb.org/downloads

2. Under the Production Release, choose the file that matches your operating system. In our case, it's Linux 32-bit.

3. Once downloaded, transfer the compressed file to all servers. (In each server, we have created a directory named /home/mongo. This is where we will extract the contents of the compressed file.)

4. Extract the contents by running the following command (you might need to modify the directory). Remember to do these on all servers.
tar -C /home/mongo/ -zxvf/home/mongo/mongodb-linux-i686-2.0.2.tgz 


5. Now we need to create the database directory for our MongoDB servers. By default, /data/db is used by MongoDB, so we'll create those directories by running the following command (Remember to do these on all servers):
mkdir -p/data/db 


6. Next, we will run MongoDB servers in a Replica Set using the following command (Again, do these step on all servers):
/home/mongo/mongodb-linux-i686-2.0.2/bin/./mongod --replSet cluster1


Server 1 should output the following log:
[Server #1] /home/mongo/mongodb-linux-i686-2.0.2/bin/./mongod --replSet cluster1
Sat Feb 18 11:14:36
Sat Feb 18 11:14:36 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Sat Feb 18 11:14:36
Sat Feb 18 11:14:36 [initandlisten] MongoDB starting : pid=5921 port=27017 dbpath=/data/db/ 32-bit host=28125_2_85413_357231
Sat Feb 18 11:14:36 [initandlisten]
Sat Feb 18 11:14:36 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Sat Feb 18 11:14:36 [initandlisten] ** see http://blog.mongodb.org/post/137788967/32-bit-limitations
Sat Feb 18 11:14:36 [initandlisten] ** with --journal, the limit is lower
Sat Feb 18 11:14:36 [initandlisten]
Sat Feb 18 11:14:36 [initandlisten] db version v2.0.2, pdfile version 4.5
Sat Feb 18 11:14:36 [initandlisten] git version: 514b122d308928517f5841888ceaa4246a7f18e3
Sat Feb 18 11:14:36 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_41
Sat Feb 18 11:14:36 [initandlisten] options: { replSet: "cluster1" }
Sat Feb 18 11:14:36 [initandlisten] waiting for connections on port 27017
Sat Feb 18 11:14:36 [websvr] admin web console waiting for connections on port 28017
Sat Feb 18 11:14:36 [initandlisten] connection accepted from 127.0.0.1:39621 #1
Sat Feb 18 11:14:36 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Sat Feb 18 11:14:36 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Sat Feb 18 11:14:46 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Sat Feb 18 11:14:56 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)

Server 2 should output the following log:
[Server #2] /home/mongo/mongodb-linux-i686-2.0.2/bin/./mongod --replSet cluster1
Sat Feb 18 11:15:34
Sat Feb 18 11:15:34 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Sat Feb 18 11:15:34
Sat Feb 18 11:15:34 [initandlisten] MongoDB starting : pid=13534 port=27017 dbpath=/data/db/ 32-bit host=28125_2_82937_349828
Sat Feb 18 11:15:34 [initandlisten]
Sat Feb 18 11:15:34 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Sat Feb 18 11:15:34 [initandlisten] ** see http://blog.mongodb.org/post/137788967/32-bit-limitations
Sat Feb 18 11:15:34 [initandlisten] ** with --journal, the limit is lower
Sat Feb 18 11:15:34 [initandlisten]
Sat Feb 18 11:15:34 [initandlisten] db version v2.0.2, pdfile version 4.5
Sat Feb 18 11:15:34 [initandlisten] git version: 514b122d308928517f5841888ceaa4246a7f18e3
Sat Feb 18 11:15:34 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_41
Sat Feb 18 11:15:34 [initandlisten] options: { replSet: "cluster1" }
Sat Feb 18 11:15:34 [initandlisten] waiting for connections on port 27017
Sat Feb 18 11:15:34 [websvr] admin web console waiting for connections on port 28017
Sat Feb 18 11:15:34 [initandlisten] connection accepted from 127.0.0.1:46620 #1
Sat Feb 18 11:15:34 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Sat Feb 18 11:15:34 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Sat Feb 18 11:15:44 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)

Server 3 should output the following log:
[Server #3] /home/mongo/mongodb-linux-i686-2.0.2/bin/./mongod --replSet cluster1
Sat Feb 18 11:15:35
Sat Feb 18 11:15:35 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Sat Feb 18 11:15:35
Sat Feb 18 11:15:35 [initandlisten] MongoDB starting : pid=8902 port=27017 dbpath=/data/db/ 32-bit host=28125_2_85413_357219
Sat Feb 18 11:15:35 [initandlisten]
Sat Feb 18 11:15:35 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Sat Feb 18 11:15:35 [initandlisten] ** see http://blog.mongodb.org/post/137788967/32-bit-limitations
Sat Feb 18 11:15:35 [initandlisten] ** with --journal, the limit is lower
Sat Feb 18 11:15:35 [initandlisten]
Sat Feb 18 11:15:35 [initandlisten] db version v2.0.2, pdfile version 4.5
Sat Feb 18 11:15:35 [initandlisten] git version: 514b122d308928517f5841888ceaa4246a7f18e3
Sat Feb 18 11:15:35 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_41
Sat Feb 18 11:15:35 [initandlisten] options: { replSet: "cluster1" }
Sat Feb 18 11:15:35 [initandlisten] waiting for connections on port 27017
Sat Feb 18 11:15:35 [websvr] admin web console waiting for connections on port 28017
Sat Feb 18 11:15:35 [initandlisten] connection accepted from 127.0.0.1:52051 #1
Sat Feb 18 11:15:35 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Sat Feb 18 11:15:35 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Sat Feb 18 11:15:45 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Sat Feb 18 11:15:55 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)

Server 4 should output the following log:
[Server #4] /home/mongo/mongodb-linux-i686-2.0.2/bin/./mongod --replSet cluster1
Sat Feb 18 11:15:37
Sat Feb 18 11:15:37 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Sat Feb 18 11:15:37
Sat Feb 18 11:15:37 [initandlisten] MongoDB starting : pid=11685 port=27017 dbpath=/data/db/ 32-bit host=28125_2_84690_354582
Sat Feb 18 11:15:37 [initandlisten]
Sat Feb 18 11:15:37 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Sat Feb 18 11:15:37 [initandlisten] ** see http://blog.mongodb.org/post/137788967/32-bit-limitations
Sat Feb 18 11:15:37 [initandlisten] ** with --journal, the limit is lower
Sat Feb 18 11:15:37 [initandlisten]
Sat Feb 18 11:15:37 [initandlisten] db version v2.0.2, pdfile version 4.5
Sat Feb 18 11:15:37 [initandlisten] git version: 514b122d308928517f5841888ceaa4246a7f18e3
Sat Feb 18 11:15:37 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_41
Sat Feb 18 11:15:37 [initandlisten] options: { replSet: "cluster1" }
Sat Feb 18 11:15:37 [initandlisten] waiting for connections on port 27017
Sat Feb 18 11:15:37 [websvr] admin web console waiting for connections on port 28017
Sat Feb 18 11:15:37 [initandlisten] connection accepted from 127.0.0.1:51100 #1
Sat Feb 18 11:15:37 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Sat Feb 18 11:15:37 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Sat Feb 18 11:15:47 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Sat Feb 18 11:15:57 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)

7. Next step is to configure our MongoDB servers to act as a cluster. Follow the steps below:

  1. Login to Server 1
  2. Run a MongoDB client using the following command:
    /home/mongo/mongodb-linux-i686-2.0.2/bin/./mongo

  3. Initiate the MongoDB cluster
    rs.initiate({_id: 'cluster1', members: [
    {_id: 0, host: '123.456.78.90:27017'},
    {_id: 1, host: '123.456.78.91:27017'},
    {_id: 2, host: '123.456.78.92:27017'},
    {_id: 3, host: '123.456.78.93:27017', arbiterOnly: true}]
    })


    If you have accidentally initiated an incorrect configuration, you can reconfigure the configuration with:
    rs.reconfig({_id: 'cluster1', members: [
    {_id: 0, host: '123.456.78.90:27017'},
    {_id: 1, host: '123.456.78.91:27017'},
    {_id: 2, host: '123.456.78.92:27017'},
    {_id: 3, host: '123.456.78.93:27017', arbiterOnly: true}]
    }, true)


    Notice, the last host is an arbiter only server. This will participate in electing a primary but receive no data.

    You should see the following output from MongoDB client:
    > rs.initiate({_id: 'cluster1', members: [
    bye
    [Server #1] /home/mongo/mongodb-linux-i686-2.0.2/bin/./mongo
    MongoDB shell version: 2.0.2
    connecting to: test
    > rs.initiate({_id: 'cluster1', members: [
    ... {_id: 0, host: '123.456.78.90:27017'},
    ... {_id: 1, host: '123.456.78.91:27017'},
    ... {_id: 2, host: '123.456.78.92:27017'},
    ... {_id: 3, host: '123.456.78.93:27017', arbiterOnly: true}]
    ... })
    {
    "info" : "Config now saved locally. Should come online in about a minute.",
    "ok" : 1
    }

    Examine the output from Server 1
    Sat Feb 18 11:16:22 [conn2] replSet replSetInitiate admin command received from client
    Sat Feb 18 11:16:22 [conn2] replSet replSetInitiate config object parses ok, 4 members specified
    Sat Feb 18 11:16:22 [conn2] replSet replSetInitiate all members seem up
    Sat Feb 18 11:16:22 [conn2] ******
    Sat Feb 18 11:16:22 [conn2] creating replication oplog of size: 47MB...
    Sat Feb 18 11:16:22 [FileAllocator] allocating new datafile /data/db/local.ns, filling with zeroes...
    Sat Feb 18 11:16:22 [FileAllocator] creating directory /data/db/_tmp
    Sat Feb 18 11:16:22 [FileAllocator] done allocating datafile /data/db/local.ns, size: 16MB, took 0.053 secs
    Sat Feb 18 11:16:22 [FileAllocator] allocating new datafile /data/db/local.0, filling with zeroes...
    Sat Feb 18 11:16:22 [FileAllocator] done allocating datafile /data/db/local.0, size: 16MB, took 0.106 secs
    Sat Feb 18 11:16:22 [FileAllocator] allocating new datafile /data/db/local.1, filling with zeroes...
    Sat Feb 18 11:16:23 [FileAllocator] done allocating datafile /data/db/local.1, size: 32MB, took 1.124 secs
    Sat Feb 18 11:16:23 [FileAllocator] allocating new datafile /data/db/local.2, filling with zeroes...
    Sat Feb 18 11:16:26 [FileAllocator] done allocating datafile /data/db/local.2, size: 64MB, took 3.228 secs
    Sat Feb 18 11:16:29 [conn2] ******
    Sat Feb 18 11:16:29 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
    Sat Feb 18 11:16:29 [conn2] replSet info saving a newer config version to local.system.replset
    Sat Feb 18 11:16:29 [conn2] replSet saveConfigLocally done
    Sat Feb 18 11:16:29 [conn2] replSet replSetInitiate config now saved locally. Should come online in about a minute.
    Sat Feb 18 11:16:29 [conn2] command admin.$cmd command: { replSetInitiate: { _id: "cluster1", members: [ { _id: 0.0, host: "123.456.78.90:27017" }, { _id: 1.0, host: "123.456.78.91:27017" }, { _id: 2.0, host: "123.456.78.92:27017" }, { _id: 3.0, host: "123.456.78.93:27017", arbiterOnly: true } ] } } ntoreturn:1 reslen:112 7192ms
    Sat Feb 18 11:16:39 [rsStart] replSet STARTUP2
    Sat Feb 18 11:16:39 [rsHealthPoll] replSet member 123.456.78.91:27017 is up
    Sat Feb 18 11:16:39 [rsHealthPoll] replSet member 123.456.78.92:27017 is up
    Sat Feb 18 11:16:39 [rsHealthPoll] replSet member 123.456.78.93:27017 is up

Next

In the next section, we will configure our Spring application to support Replica Sets. Click here to proceed.
Continue reading MongoDB - Replica Sets with Spring Data MongoDB (Part 2)
, , , , , , ,

MongoDB - Replica Sets with Spring Data MongoDB (Part 1)

Introduction

In this tutorial, we will study how to setup MongoDB Replica Sets using a Spring Data MongoDB-based application for testing our replica sets. We will not create a MongoDB application here but instead we will reuse an existing one from the Spring MVC 3.1 - Implement CRUD with Spring Data MongoDB guide. For this study, we will use production-ready, cloud servers to demonstrate in real-time the results.


What is a Replica Set?

Replica sets are a form of asynchronous master/slave replication, adding automatic failover and automatic recovery of member nodes.

  • A replica set consists of two or more nodes that are copies of each other. (i.e.: replicas)
  • The replica set automatically elects a primary (master). No one member is intrinsically primary; that is, this is a share-nothing design.
  • Drivers (and mongos) can automatically detect when a replica set primary changes and will begin sending writes to the new primary.

Replica sets have several common uses:
  • Data Redundancy
  • Automated Failover / High Availability
  • Distributing read load
  • Simplify maintenance (compared to "normal" master-slave)
  • Disaster recovery

Source: MongoDB Replica Sets

Servers

We have four cloud servers (for security purposes, I have modified their actual IP addresses). All servers are running with the following configuration:
OS: CentOS release 5.2 (Final)
RAM: 512MB
HD: 30GB

Here are our servers:
Server #IP addressMongoDB portComments
Server 1123.456.78.9027017primary server
Server 2123.456.78.9127017slave server
Server 3123.456.78.9227017slave server
Server 4123.456.78.9327017arbiter server (see note below)

Although Server 1 is the primary server, this is not permanent. Whenever the primary server is down, another server is elected as a primary. Server 4 is an arbiter server which is useful for certain cases in electing a primary server.

Note: Our Spring Data MongoDB-based application is hosted in Server 1.

What is an Arbiter?
Arbiters are nodes in a replica set that only participate in elections: they don't have a copy of the data and will never become the primary node (or even a readable secondary). They are mainly useful for breaking ties during elections (e.g. if a set only has two members).

When to add an arbiter
  • Two members with data : add an arbiter to have three voters. 2 out of 3 votes for a member establishes it as primary.
  • Three members with data : no need to add an arbiter. In fact having 4 voters is worse as 3 of 4 needed to elect a primary instead of 2 of 3. In theory one might add two arbiters thus making number of votes five, and 3 of 5 would be ok; however this is uncommon and generally not recommended.
  • Four members with data : add one arbiter.

Source: MongoDB - Adding an Arbiter

Do I need an Arbiter?
You need an arbiter if you have an even number of votes. As an extension to this, at most you should only ever have 1 arbiter. If you aren't sure how many votes you have, it's probably the same as the number of servers in the set you have (including slaves, hidden, arbiters).

Source: Does My MongoDB Replica Set Need An Arbiter?

Next

In the next section, we will install MongoDB on our servers and configure Replica Sets. Click here to proceed.
Continue reading MongoDB - Replica Sets with Spring Data MongoDB (Part 1)
, , , , , , , ,

MongoDB - Replica Sets with Spring Data MongoDB (Part 4)

Review

In the previous section, we have introduced MongoDB Replica Sets, installed MongoDB, configured Replica Sets, and updated our Spring application. In this section, we will test our cluster to verify if we have achieved our goal.


Testing

Start Spring App

Kindly start our Spring application from Server 1. You may need to deploy the application first (it's up to you whether to deploy it on the cloud or on a localhost machine).

Notice the log output from Server 1. It acknowledges two connections from our Spring application, and it starts allocating space for the spring_mongodb_tutorial database:
Sat Feb 18 11:18:50 [conn14] getmore local.oplog.rs query: { ts: { $gte: new Date(5710309296642719745) } } cursorid:7697632718916849147 reslen:20 4588ms
Sat Feb 18 11:18:51 [initandlisten] connection accepted from 123.456.78.90:58023 #27
Sat Feb 18 11:18:53 [initandlisten] connection accepted from 123.456.78.90:58027 #28
Sat Feb 18 11:18:53 [conn28] CMD: drop spring_mongodb_tutorial.role
Sat Feb 18 11:18:53 [conn28] CMD: drop spring_mongodb_tutorial.user
Sat Feb 18 11:18:53 [FileAllocator] allocating new datafile /data/db/spring_mongodb_tutorial.ns, filling with zeroes...
Sat Feb 18 11:18:54 [FileAllocator] done allocating datafile /data/db/spring_mongodb_tutorial.ns, size: 16MB, took 0.081 secs
Sat Feb 18 11:18:54 [FileAllocator] allocating new datafile /data/db/spring_mongodb_tutorial.0, filling with zeroes...
Sat Feb 18 11:18:56 [FileAllocator] done allocating datafile /data/db/spring_mongodb_tutorial.0, size: 16MB, took 1.534 secs
Sat Feb 18 11:18:56 [FileAllocator] allocating new datafile /data/db/spring_mongodb_tutorial.1, filling with zeroes...
Sat Feb 18 11:18:57 [FileAllocator] done allocating datafile /data/db/spring_mongodb_tutorial.1, size: 32MB, took 1.27 secs
Sat Feb 18 11:18:57 [conn28] build index spring_mongodb_tutorial.user { _id: 1 }
Sat Feb 18 11:18:57 [conn28] build index done 0 records 0.074 secs
Sat Feb 18 11:18:57 [conn28] insert spring_mongodb_tutorial.user 4131ms
Sat Feb 18 11:18:57 [conn28] build index spring_mongodb_tutorial.role { _id: 1 }
Sat Feb 18 11:18:57 [conn28] build index done 0 records 0 secs
Sat Feb 18 11:18:57 [conn14] getmore local.oplog.rs query: { ts: { $gte: new Date(5710309296642719745) } } cursorid:7697632718916849147 nreturned:4 reslen:1048 6948ms
Sat Feb 18 11:18:57 [conn13] getmore local.oplog.rs query: { ts: { $gte: new Date(5710309296642719745) } } cursorid:844650061822854635 nreturned:4 reslen:1048 5872ms

Watch how Server 2 synchronizes with Server 1
Sat Feb 18 11:18:51 [initandlisten] connection accepted from 123.456.78.90:43302 #20
Sat Feb 18 11:18:57 [FileAllocator] allocating new datafile /data/db/spring_mongodb_tutorial.ns, filling with zeroes...
Sat Feb 18 11:18:57 [FileAllocator] done allocating datafile /data/db/spring_mongodb_tutorial.ns, size: 16MB, took 0.042 secs
Sat Feb 18 11:18:57 [FileAllocator] allocating new datafile /data/db/spring_mongodb_tutorial.0, filling with zeroes...
Sat Feb 18 11:18:57 [FileAllocator] done allocating datafile /data/db/spring_mongodb_tutorial.0, size: 16MB, took 0.041 secs
Sat Feb 18 11:18:57 [rsSync] build index spring_mongodb_tutorial.user { _id: 1 }
Sat Feb 18 11:18:57 [rsSync] build index done 0 records 0 secs
Sat Feb 18 11:18:57 [rsSync] build index spring_mongodb_tutorial.role { _id: 1 }
Sat Feb 18 11:18:57 [rsSync] build index done 0 records 0 secs
Sat Feb 18 11:18:57 [FileAllocator] allocating new datafile /data/db/spring_mongodb_tutorial.1, filling with zeroes...
Sat Feb 18 11:18:58 [FileAllocator] done allocating datafile /data/db/spring_mongodb_tutorial.1, size: 32MB, took 0.078 secs

Also, observe how Server 3 synchronizes with Server 1
Sat Feb 18 11:18:51 [initandlisten] connection accepted from 123.456.78.90:52987 #20
Sat Feb 18 11:18:57 [FileAllocator] allocating new datafile /data/db/spring_mongodb_tutorial.ns, filling with zeroes...
Sat Feb 18 11:18:57 [FileAllocator] done allocating datafile /data/db/spring_mongodb_tutorial.ns, size: 16MB, took 0.043 secs
Sat Feb 18 11:18:57 [FileAllocator] allocating new datafile /data/db/spring_mongodb_tutorial.0, filling with zeroes...
Sat Feb 18 11:18:57 [FileAllocator] done allocating datafile /data/db/spring_mongodb_tutorial.0, size: 16MB, took 0.044 secs
Sat Feb 18 11:18:57 [rsSync] build index spring_mongodb_tutorial.user { _id: 1 }
Sat Feb 18 11:18:57 [rsSync] build index done 0 records 0 secs
Sat Feb 18 11:18:57 [rsSync] build index spring_mongodb_tutorial.role { _id: 1 }
Sat Feb 18 11:18:57 [rsSync] build index done 0 records 0 secs
Sat Feb 18 11:18:57 [FileAllocator] allocating new datafile /data/db/spring_mongodb_tutorial.1, filling with zeroes...
Sat Feb 18 11:18:58 [FileAllocator] done allocating datafile /data/db/spring_mongodb_tutorial.1, size: 32MB, took 0.091 secs

Server 4 does not synchronize because it's an arbiter-only server! Take note of that.

Start Killing

I don't mean killing people but rather MongoDB servers. Let's add a new record first. I have chosen mary, Mary, Jane, zzzzzzz, Regular as the properties of the new record. Feel free to vary.


Now, let's kill Server 1
Sat Feb 18 11:21:02 got kill or ctrl c or hup signal 2 (Interrupt), will terminate after current cmd ends
Sat Feb 18 11:21:02 [conn14] getmore local.oplog.rs query: { ts: { $gte: new Date(5710309296642719745) } } cursorid:7697632718916849147 exception: interrupted at shutdown code:11600 reslen:20 2565ms
Sat Feb 18 11:21:02 [conn13] getmore local.oplog.rs query: { ts: { $gte: new Date(5710309296642719745) } } cursorid:844650061822854635 exception: interrupted at shutdown code:11600 reslen:20 2565ms
Sat Feb 18 11:21:02 [interruptThread] now exiting
Sat Feb 18 11:21:02 Sat Feb 18 11:21:02 [conn14] got request after shutdown()
Sat Feb 18 11:21:02 [conn13] got request after shutdown()
dbexit:
Sat Feb 18 11:21:02 [interruptThread] shutdown: going to close listening sockets...
Sat Feb 18 11:21:02 [interruptThread] closing listening socket: 5
Sat Feb 18 11:21:02 [interruptThread] closing listening socket: 6
Sat Feb 18 11:21:02 [interruptThread] closing listening socket: 8
Sat Feb 18 11:21:02 [interruptThread] removing socket file: /tmp/mongodb-27017.sock
Sat Feb 18 11:21:02 [interruptThread] shutdown: going to flush diaglog...
Sat Feb 18 11:21:02 [interruptThread] shutdown: going to close sockets...
Sat Feb 18 11:21:02 [interruptThread] shutdown: waiting for fs preallocator...
Sat Feb 18 11:21:02 [interruptThread] shutdown: closing all files...
Sat Feb 18 11:21:02 [interruptThread] closeAllFiles() finished
Sat Feb 18 11:21:02 [interruptThread] shutdown: removing fs lock...
Sat Feb 18 11:21:02 dbexit: really exiting now
Sat Feb 18 11:21:02 [conn1] end connection 127.0.0.1:39621
Logstream::get called in uninitialized state
Sat Feb 18 11:21:02 [conn38] end connection 173.204.91.83:37424

Reload our Spring application. Notice, we still have the same data. We lost nothing. The replicas are working!

Kill More

Let's kill more servers. Add new record again. I have chosen anna, Anna, Williams, zzzzzzz, Admin as the properties of the new record.


Kill Server #2. This means only Server 3 and Server 4 are running, one slave and one arbiter remains respectively. Refresh our Spring application. However the application fails to load the data. Why?

The log explains why:
Sat Feb 18 11:23:46 [rsSync] replSet syncThread: 10278 dbclient error communicating with server: 173.204.91.84:27017
Sat Feb 18 11:23:46 [conn42] end connection 123.456.78.92:53291
Sat Feb 18 11:23:47 [rsHealthPoll] couldn't connect to 123.456.78.90:27017: couldn't connect to server 173.204.91.90:27017
Sat Feb 18 11:23:48 [rsHealthPoll] DBClientCursor::init call() failed
Sat Feb 18 11:23:48 [rsHealthPoll] replSet info 123.456.78.92:27017 is down (or slow to respond): DBClientBase::findN: transport error: 123.456.78.92:27017 query: { replSetHeartbeat: "cluster1", v: 1, pv: 1, checkEmpty: false, from: "123.456.78.93:27017" }
Sat Feb 18 11:23:48 [rsHealthPoll] replSet member 123.456.78.92:27017 is now in state DOWN
Sat Feb 18 11:23:48 [rsMgr] replSet can't see a majority, will not try to elect self

It's because our servers can't elect a primary server. In general, you need two non-arbiter servers in order to have a primary server! The arbiter is designed for breaking the tie-break between equal votes. Now let's restore the other two dead servers. Run mongod again for in Server 1 and Server 2.

Refresh our Spring application again. Notice we have all four records, including Mary and Anna. Remember when we added Anna, the Server #1 was dead!

Let's kill Server 2 and see if Anna is still displayed. Yes, it's diplayed!

Let's kill Server 3 and see if Anna is still displayed. And run Server 3 again. Yes, it's still diplayed!

Conclusion

That's it! We've successfully implemented a MongoDB Replica Set. We have shown step-by-step how to configure our servers and our Spring application. We've also made some quick tests to verify the replication feature of our servers.
Continue reading MongoDB - Replica Sets with Spring Data MongoDB (Part 4)
, , , , , , , ,

MongoDB - Replica Sets with Spring Data MongoDB (Part 3)

Review

In the previous section, we have installed and configured our servers for MongoDB replication. In this section, we will update our Spring application to support MongoDB Replica Sets.


Spring App Configuration

We have setup a MongoDB Replica Set using four servers, and we've examined the output from the logs and verified that all instances are running. Our next step is to configure our Spring MongoDB-base application.

Since we're reusing our application from the tutorial Spring MVC 3.1 - Implement CRUD with Spring Data MongoDB, adding a MongoDB Replica Set support in Spring is trivial. All we need to do is modify the spring.properties file with the following contents (this file is located under the WEB-INF directory):



Then we have to modify as well the spring-data.xml file with the following updates:



That's all we need to for our Spring application!

Next

In the next section, we will run and test our servers to verify and test our MongoDB cluster for replication. Click here to proceed.
Continue reading MongoDB - Replica Sets with Spring Data MongoDB (Part 3)
, , , , , , , ,

LDAP - Apache Directory Studio: A Basic Tutorial

In this tutorial we will setup a basic LDAP structure containing users and roles. We will be using the excellent Apache Directory Studio IDE. This tutorial will be the basis for our other Spring LDAP integration tutorials.

What is Apache Directory Studio?
The Eclipse based LDAP browser and directory client

Apache Directory Studio is a complete directory tooling platform intended to be used with any LDAP server however it is particularly designed for use with ApacheDS. It is an Eclipse RCP application, composed of several Eclipse (OSGi) plugins, that can be easily upgraded with additional ones. These plugins can even run within Eclipse itself.

Source: http://directory.apache.org/studio/
What is LDAP?
The Lightweight Directory Access Protocol (LDAP) is an application protocol for reading and editing directories over an IP network. A directory is an organized set of records. For example, the telephone directory is an alphabetical list of persons and organizations, with each record having an address and phone number. A directory information tree often follows political, geographic, or organizational boundaries. LDAP directories often use Domain Name System (DNS) names for the highest levels. Deeper inside the directory might appear entries for people, departments, teams, printers, and documents.

Source: http://en.wikipedia.org/wiki/LDAP
If this is your first time to LDAP, you might be wondering how is this different from an RDBMS. I suggest my readers to visit the following article Should I Use a Directory, a Database, or Both?

A Brief Background
We have a small startup company named Mojo Enterprises. We have four people, and two of them are admins. Our task is to create a hierarchical structure of our organization using LDAP because we anticipate the eventual growth of the company. We may have hundreds of people from different departments in five years time. Each has their own information and structure. These information and structure will be shared among different applications of the company. LDAP is a good protocol to meet all these requirements.

Layout the Structure
Let's define the important elements of the company.

Company Name: Mojo Enterprises
Members:
  • Hugo Williams
  • John Keats
  • John Milton
  • Robert Browning
Admins:
  • Hugo Williams
  • John Keats
LDAP is a hierarchical tree structure, so our design will be influenced by that.

We need to assign the topmost parent of our structure. Logically, the name of the company fits that requirement. We'll pick the name mojo as the topmost parent.

Under the mojo we will assign our members. There are many ways to organize our members. We can organize them by gender, by job function, and etc. For this tutorial we'll organize them based on identity and roles. We'll put all the identities of each person in a separate element, while the roles will be placed on another element.

Under the roles we have two sub-divisions. Remember we have the regular users and the admins. What we'll do is divide the roles into elements.

Here's how our structure would like:
mojo
|
|--roles
| |
| |--admin
| |--regular
|
|--identities
|
|--guy1
|--guy2
|--guy3
|--guy4
We'll make the names a little bit formal, and make it conform with the naming convention of LDAP. For the topmost parent, we'll retain mojo. For roles, we'll use groups instead. For identities, we'll use users

When we're done with this tutorial, we should have the following structure:
It's really simple to do. All you need is an Apache Directory Studio, this tutorial, and patience.

Install the Apache Directory Studio
Before we can do any LDAP-related work, we need to have an IDE first. Although it's not required, we'll require it for this tutorial. And why not? It's free anyway.

To install the Apache Directory Studio, visit the following link http://directory.apache.org/studio/downloads.html Follow the instructions on that link
for the installation steps.

Create a New Server
Once you have installed the studio, we need to create a server. Here are the steps:

1. Open Apache Directory Studio.

3. Go to File, and click New. A popup window will open.

4. Expand the Apache DS folder, and select the Apache DS Server

5. Click Next.

6. Type-in any name for the server. For example, apache-ds-server

7. Click Finish.

8. A new server has been added on your Servers panel.
If you can't see the Servers panel, go the menu bar then click Window > Show View > Apache DS > Servers

9. Select your server, then click the Run button.
Your server should now be running.

Create a Connection
To browse the contents of the server, we need to create a connection. Here are the steps:

1. Right-click on your server.

2. Select LDAP Browser

3. Select Create a Connection. An alert message will popup indicating the a new connection has been created.

4. Go to the Connections panel.
If you can't see the Connections panel, go the menu bar then click Window > Show View > LDAP Browser > Connections

5. Double-click the name of the new connection you've created earlier.

6. The LDAP Browser panel should refresh and show the contents of the server.
Notice the topmost entry of this server is DIT, followed by the Root DSE.

7. Expand the Root DSE folder. There are two sub-entries: ou=schema and ou=system.
These are partitions in the server. Do not modify them unless you know what you're doing.

Create a New Partition
To add our company to the root tree, we need to create a new partition. Everything related to the company will be attached to this new partition. Here are the steps:

1. Go to the Servers panel, and right-click your server.

2. Select Open Configuration. A server.xml editor will appear on the main panel.

3. At the bottom part of server.xml, there are five tabs. Click the Partitions tab.
The Partitions tab should appear.

4. Click on Add and enter the following details for this new partition.

ID: mojo
Cache Size: 100
Suffix: o=mojo

5. Save your changes (CTRL + S). A new partition has been added.

6. Restart your Apache DS server.

7. Refresh the LDAP Browser panel.

8. Click on the Root DSE folder. A new editor will open containing the details of this folder.
Notice the namingContexts attribute. The namingContexts o=mojo shows in the lists. However it doesn't show under the Root DSE tree because we need to add the organization manually in the tree.

Add the Parent Organization
Our company is an organization. In LDAP, to represent a company we use the organization object which is represented by the alias o. So if our company's name is mojo, the Distinguished Name (dn) of the company is o=mojo. It's a naming convention. (The Distinguished Name is like the primary key or primary identity).

To create the organization object, follow the steps below:
1. Right-click on the Root DSE folder. Select New. Select New Context Entry

2. Select Create entry from scratch. Click Next. The Object classes window will appear.

3. Find the organization object. Select it then click Add

4. Click Next. Now you need to enter a Distinguished Name (dn). Click on the pulldown menu. Select o=mojo.

5. Click Next. The Attributes window will appear. Examine the values.

6. Click Finish. Notice the new partition now appears under the Root DSE.

Add the Organizational Units
Earlier we mentioned we'll structure our company based on users (contains personal information of the user) and groups (contains the authorization level of each person).
Both of these represent an organizational unit.

In LDAP, to represent an organizational unit we use the organizationalUnit object which is represented by the alias ou. So if we have a unit name users, the Distinguished Name (dn) is ou=users,o=mojo. Why is there an o=mojo? It's a naming convention. The same convention applies to groups. The Distinguished Name (dn) is ou=users,o=mojo. This can be likened the way we name URLS. For example, users.mojo.com or groups.mojo.com.

We'll add first the users unit. Here are the steps:
1. Go to the LDAP Browser panel. Expand the Root DSE folder.

2. Right-click the o=mojo entry. Select New. Select New Entry.
The Entry Creation Method window will appear.

3. Select Create entry from scratch. Click Next. The Object Classes window will appear.

4. Find the organizationalUnit object. Select it then click Add.

5. Click Next. Now you need to enter a Distinguished Name (dn).
The Parent field should read o=mojo.

On the RDN field enter ou. On the value field enter users. The DN Preview should read ou=users,o=mojo

6. Click Next. The Attributes window will appear. Examine the values.

7. Click Finish. We've just created the ou=users organizational unit.

Add the Second Organizational Units
We've just added the ou=users organizational unit. We need to add the organizational unit for groups as well. We'll follow the same steps.

1. Go to the LDAP Browser panel. Expand the Root DSE folder.

2. Right-click the o=mojo entry. Select New. Select New Entry.
The Entry Creation Method window will appear.

3. Select Create entry from scratch. Click Next. The Object Classes window will appear.

4. Find the organizationalUnit object. Select it then click Add.

5. Click Next. Now you need to enter a Distinguished Name (dn).
The Parent field should read o=mojo.

On the RDN field enter ou. On the value field enter groups. The DN Preview should read ou=groups,o=mojo

6. Click Next. The Attributes window will appear. Examine the values.

7. Click Finish. We've just created the ou=groups organizational unit.

Add the Staff
Now we need to add our four people:
  • Hugo Williams
  • John Keats
  • John Milton
  • Robert Browning
Admins:
  • Hugo Williams
  • John Keats
We'll place their personal information under the ou=users; whereas we'll place their authorization levels under the ou=groups.

Let's start with the ou=users. We'll be adding four persons. We'll represent each person using the inetOrgPerson object.

What's an inetOrgPerson object?
The inetOrgPerson object class is a general purpose object class that
holds attributes about people. The attributes it holds were chosen
to accommodate information requirements found in typical Internet and
Intranet directory service deployments.
Source: http://www.faqs.org/rfcs/rfc2798.html
An inetOrgPerson can contain a user id (uid) and password (userPassword) which will be useful later for authenticating users from using LDAP.

Here are the steps we need to do:
1. Go to the LDAP Browser panel. Expand the Root DSE folder.

2. Expand the o=mojo entry.

3. Right-click the ou=users entry. Select New. Select New Entry.
The Entry Creation Method window will appear.

4. Select Create entry from scratch. Click Next. The Object Classes window will appear.

5. Find inetOrgPerson object. Select it then click Add.

6. Click Next. Now you need to enter a Distinguished Name (dn).
The Parent field should read ou=users,o=mojo.

On the RDN field enter cn. On the value field enter Hugo Williams.
The DN Preview should read cn=Hugo Williams,ou=users,o=mojo (cn represent Common Name).

7. Click Next. The Attributes window will appear. Examine the values.

8. Under the sn attribute, enter Williams (sn stands for Surname)

9. We need to add a username for this user. Right-click on the same window. Select New Attribute. The Attribute Type window will appear.

10. On the Attribute type field, enter uid. This will serve as the username of the person.

11. Click Next, then click Finish.

12. You're back on the Attributes window. On the uid attribute value, enter hwilliams


13. We need to add a password for this user. Right-click on the same window. Select New Attribute. The Attribute Type window will appear.

14. On the Attribute type field, enter userPassword. This will serve as the password of the person.

15. Click Next, then click Finish.

16. You will be asked to enter a password. Enter pass as the new password. Make sure that the Select Hash Method is set to Plaintext

17. Click OK.

A new entry has been added under the ou=users. The new entry is cn=Hugo Williams.

Now we need to add the remaining three users. In order to do that, just repeat the same steps earlier. Here are the details of the three remaining users.
Name: John Keats
uid: jkeats
userPassword: pass

Name: John Milton
uid: jmilton
userPassword: pass

Name: Robert Browning
uid: rbrowning
userPassword: pass
Add the Authorization Levels
We have added the personal information, as well as the usernames and passwords, for each person under the ou=users. Now, we will be adding the authorization level for each of these persons.

We'll add them under ou=groups. We'll use the groupOfUniqueNames object to represent each role.

Let's add the User role first.
1. Go to the LDAP Browser panel. Expand the Root DSE folder.

2. Expand the o=mojo entry.

3. Right-click the ou=groups entry. Select New. Select New Entry.
The Entry Creation Method window will appear.

4. Select Create entry from scratch. Click Next. The Object Classes window will appear.

5. Find the groupOfUniqueNames object. Select it then click Add.

6. Click Next. Now you need to enter a Distinguished Name (dn).
The Parent field should read ou=groups,o=mojo.

On the RDN field enter cn. On the value field enter User
The DN Preview should read cn=User,ou=groups,o=mojo

8. Click Next. The Attributes window will appear. Examine the values.

Notice there's a uniqueMember attribute. We'll be placing the Distinguished Name (dn) of our users in this entry. One uniqueMember attribute will represents one user. This means we need to add three more uniqueMember attributes for a total of four uniqueMember attributes.

9. Right-click on the same window. Select New Attribute. The Attribute Type window will appear.

10. On the Attribute type field, enter uniqueMember.

11. Click Next, then click Finish.

12. We're back on the Attributes window. We need to add two more uniqueMembers (for a total of four uniqueMembers). Repeat the same steps for adding an attribute.

13. Now we need to fill-in the values for these attributes. In each entry add the dn of each user. Here are the Distinguished Name for each user.
cn=Hugo Williams,ou=users,o=mojo
cn=John Keats,ou=users,o=mojo
cn=John Milton,ou=users,o=mojo
cn=Robert Browning,ou=users,o=mojo

14. Click Finish when you're done.

A new entry has been added under the ou=groups. The new entry is cn=User

Now we need another entry for the Admin role. We'll repeat the same steps.
1. Go to the LDAP Browser panel. Expand the Root DSE folder.

2. Expand the o=mojo entry.

3. Right-click the ou=groups entry. Select New. Select New Entry.
The Entry Creation Method window will appear.

4. Select Create entry from scratch. Click Next. The Object Classes window will appear.

5. Find the groupOfUniqueNames object. Select it then click Add.

6. Click Next. Now you need to enter a Distinguished Name (dn).
The Parent field should read ou=groups,o=mojo.

On the RDN field enter cn. On the value field enter Admin
The DN Preview should read cn=Admin,ou=groups,o=mojo

7. Click Next. The Attributes window will appear. Examine the values.

Notice there's a uniqueMember attribute. We'll be placing the Distinguished Name (dn) of our users in this entry. One uniqueMember attribute will represent one user. This means we need to add one more uniqueMember attributes for a total of two uniqueMember attributes.

8. Right-click on the same window. Select New Attribute. The Attribute Type window will appear.

9. On the Attribute type field, enter uniqueMember.

10. Click Next, then click Finish.

11. We're back on the Attributes window. We need to add one more uniqueMembers (for a total of two uniqueMembers). Repeat the same steps for adding an attribute.

12. Now we need to fill-in the values for these attributes. In each entry add the dn of each user. Here are the Distinguished Name for each user.
cn=Hugo Williams,ou=users,o=mojo
cn=John Keats,ou=users,o=mojo

13. Click Finish when you're done.

A new entry has been added under the ou=groups. The new entry is cn=Admin

Here's the final structure:

Exporting the Data
If you need to backup your data or replicate the information in your LDAP, you can export the data. When the data is exported, it's saved in LDIF format. This format is actually a human-readable file. Here's the LDIF for this tutorial:
version: 1

dn: o=mojo
objectClass: organization
objectClass: extensibleObject
objectClass: top
o: mojo

dn: ou=users,o=mojo
objectClass: extensibleObject
objectClass: organizationalUnit
objectClass: top
ou: users

dn: ou=groups,o=mojo
objectClass: extensibleObject
objectClass: organizationalUnit
objectClass: top
ou: groups

dn: cn=User,ou=groups,o=mojo
objectClass: groupOfUniqueNames
objectClass: top
cn: User
uniqueMember: cn=John Milton,ou=users,o=mojo
uniqueMember: cn=Robert Browning,ou=users,o=mojo
uniqueMember: cn=Hugo Williams,ou=users,o=mojo
uniqueMember: cn=John Keats,ou=users,o=mojo

dn: cn=Admin,ou=groups,o=mojo
objectClass: groupOfUniqueNames
objectClass: top
cn: Admin
uniqueMember: cn=Hugo Williams,ou=users,o=mojo
uniqueMember: cn=John Keats,ou=users,o=mojo

dn: cn=Robert Browning,ou=users,o=mojo
objectClass: organizationalPerson
objectClass: person
objectClass: inetOrgPerson
objectClass: top
cn: Robert Browning
sn: Browning
uid: rbrowning
userPassword:: cGFzcw==

dn: cn=John Keats,ou=users,o=mojo
objectClass: organizationalPerson
objectClass: person
objectClass: inetOrgPerson
objectClass: top
cn: John Keats
sn: Keats
uid: jkeats
userPassword:: cGFzcw==

dn: cn=Hugo Williams,ou=users,o=mojo
objectClass: organizationalPerson
objectClass: person
objectClass: inetOrgPerson
objectClass: top
cn: Hugo Williams
sn: Williams
uid: hwilliams
userPassword:: cGFzcw==

dn: cn=John Milton,ou=users,o=mojo
objectClass: organizationalPerson
objectClass: person
objectClass: inetOrgPerson
objectClass: top
cn: John Milton
sn: Milton
uid: jmilton
userPassword:: cGFzcw==
That's it. We've managed to setup our basic LDAP structure using Apache Directory Studio. We've also covered some of the popular LDAP objects.
Continue reading LDAP - Apache Directory Studio: A Basic Tutorial