Develop an online document library

Assignment Help Computer Engineering
Reference no: EM132191487

Systems Engineering - Assignment Organization

Task description

You are required to develop an online **document library**.

Users are presented with an input form, where they can submit *documents* (e.g., books, poems, recipes) along with *metadata* (e.g., author, mime type, ISBN).

For the sake of simplicity, they can view *all* stored documents on a single page.

Hence, create an application with the following architecture.

Don't worry, in this repository you can find some Makefiles, Dockerfiles, configuration files and source code to get you started.

Nginx

Nginx is a web server that delivers static content in our architecture.

Static content comprises the landing page (index.html), JavaScript, css and font files located in ``nginx/www``.

1. complete the ``nginx/Dockerfile``

a) upgrade the system
#) install nginx
#) copy ``nginx/nginx.conf`` from host to container's ``/etc/nginx/nginx.conf``
#) use port 80 in the container
#) run nginx on container startup

#. in docker-compose

a) build the image
#) assign nginx to the ``se_backend`` network
#) mount the host directory ``nginx/www`` to ``/var/www/nginx`` in the container

#. verify your setup (it should display the landing page)

HBase
~~~~~

We use HBase, the open source implementation of Bigtable, as database.
``hbase/hbase_init.txt`` creates the ``se2`` namespace and a ``library`` table with two column families: ``document`` and ``metadata``.

1. build the image for the container description located in ``hbase/``
#. in docker-compose

a) add hbase to the ``se_backend`` network

The Dockerfile exposes different ports for different APIs.
We recommend the JSON REST API, but choose whatever API suits you best.

1. `HBase REST documentation <http://hbase.apache.org/book.html#_rest>`_
#. the client port for REST is 8080
#. employ curl to explore the API

a) ``curl -vi -X PUT -H "Content-Type: application/json" -d '<json row description>' "localhost:8080/se2:library/fakerow"``
#) yes, it's really *fakerow*

#. ``gserve/src/gserve/HbaseJSON.go`` contains helpers to convert data from frontend JSON via Go types to base64-encoded HBase JSON and back
#. you might want to use the (Un)marshal functions from the `encoding/JSON package <https://golang.org/pkg/encoding/json/>`_

ZooKeeper
~~~~~~~~~

Deviating from the architecture image, you don't need to create an extra ZooKeeper container.
The HBase image above already contains a ZooKeeper installation.

1. add an alias to the hbase section in docker-compose such that other containers can connect to it by referring to the name ``zookeeper``

.. note::

1. you are allowed to use the `go-zookeeper <https://github.com/samuel/go-zookeeper>`_ library

grproxy
~~~~~~~

This is the first service/server you have to write by yourself.
Implement a reverse proxy that forwards every request to nginx, except those with a "library" prefix in the path (e.g., ``http://host/library``).
Discover running gserve instances with the help of ZooKeeper and forward ``library`` requests in circular order among those instances (Round Robin).

1. complete ``grproxy/Dockerfile``
#. in docker-compose

a) build grproxy
#) add grproxy to both networks: ``se_frontend`` and ``se_backend``

.. note::

1. you are allowed to use `httputil.ReverseProxy <https://golang.org/pkg/net/http/httputil/>`_
2. you don't need to handle the case where an instance registered to ZooKeeper doesn't reply

gserve
~~~~~~

Gserve is the second service you need to implement, and it serves basically two purposes.
Firstly, it receives ``POST`` requests from the client (via grproxy) and adds or alters rows in HBase.
And secondly, it replies to ``GET`` requests with an HTML page displaying the contents of the whole document library.
It only receives requests from grproxy after it subscribed to ZooKeeper, and automatically unsubscribes from ZooKeeper if it shuts down or crashes.

1. gserve shall return all versions of HBase cells (see output sample above)
#. the returned HTML page must contain the string *"proudly served by gserve1"* (or gserve2, ...) without HTML tags in between
#. complete ``gserve/Dockerfile``
#. in docker-compose

a) build gserve
#) start two instances *gserve1* and *gserve2*
#) add both instances to ``se_backend``
#) make sure, that both instances start after hbase and grproxy
#) provide the names of the instances (gserve1, gserve2) via environmental variables

Hints
-----

* Start small, don't try to solve every problem at once.
* Test your components against single Docker containers (e.g., gserve with HBase container), and integrate them into docker-compose later on.
* The developer tools of your browser may help you to capture and analyse requests and responses.

Frequently Asked Questions
--------------------------

1. How do I use the JSON/Base64-encoding/(Un)Marshaling code?

.. code:: go

package main

import "encoding/json"

func main() {
// unencoded JSON bytes from landing page
// note: quotation marks need to be escaped with backslashes within Go strings: " -> \"
unencodedJSON := []byte("{\"Row\":[{\"key\":\"My first document\",\"Cell\":[{\"column\":\"document:Chapter 1\",\"$\":\"value:Once upon a time...\"},{\"column\":\"metadata:Author\",\"$\":\"value:The incredible me!\"}]}]}")
// convert JSON to Go objects
var unencodedRows RowsType
json.Unmarshal(unencodedJSON, &unencodedRows)
// encode fields in Go objects
encodedRows := unencodedRows.encode()
// convert encoded Go objects to JSON
encodedJSON, _ := json.Marshal(encodedRows)

println("unencoded:", string(unencodedJSON))
println("encoded:", string(encodedJSON))
}

/*
output:

unencoded: {"Row":[{"key":"My first document","Cell":[{"column":"document:Chapter 1","tiny_mce_markerquot;:"value:Once upon a time..."},{"column":"metadata:Author","tiny_mce_markerquot;:"value:The incredible me!"}]}]}
encoded: {"Row":[{"key":"TXkgZmlyc3QgZG9jdW1lbnQ=","Cell":[{"column":"ZG9jdW1lbnQ6Q2hhcHRlciAx","tiny_mce_markerquot;:"dmFsdWU6T25jZSB1cG9uIGEgdGltZS4uLg=="},{"column":"bWV0YWRhdGE6QXV0aG9y","tiny_mce_markerquot;:"dmFsdWU6VGhlIGluY3JlZGlibGUgbWUh"}]}]}
*/

#. Do I need a library to connect with HBase?

No, we recommend the REST interface. You might also consider using Thrift, but we haven't tested it.

#. Could you provide an example for an HBase scanner?

Yes, for the command line:

.. code:: bash

#!/usr/bin/bash

echo "get scanner"

scanner=`curl -si -X PUT \
-H "Accept: text/plain" \
-H "Content-Type: text/xml" \
-d '<Scanner batch="10"/>' \
"http://127.0.0.1:8080/se2:library/scanner/" | grep Location | sed "s/Location: //" | sed "s/\r//"`

echo $scanner

curl -si -H "Accept: application/json" "${scanner}"

echo "delete scanner"

curl -si -X DELETE -H "Accept: text/plain" "${scanner}"

#. What is meant by "build gserve"?

Build the docker image with docker compose, **not** the gserve binary.

Optional

You had a lot of fun and want more?
No problem!
Select a topic you're interested in, and enhance any of the components.
For instance, query single documents or rows, replace nginx with a web server written by yourself, improve the error handling in Grproxy, write test cases or in the worst case just beautify the HTML/CSS.
But keep in mind: your application *shall still conform to the task description*.

Attachment:- document library.rar

Reference no: EM132191487

When could be a transition be appropriate

Some animation schemes include slide transitions and some do not. When would a transition be appropriate? When would it not be appropriate? If a slide transition is applied

Collection of the volatile data for investigation

Discuss with others what kinds of the data are considered volatile and the methods through which the investigators should collect and preserve the volatile data. Recognize t

What will offer as a network suggestion

With true VPN access to basically an Intranet Environment at each office; there really should not be any need for "Routing" per se unless I am missing something. I only have

Write down a program for the following decision table

The National Earthquake Information Center has asked you to write down a program for the following decision table to characterize an earthquake based on its Richter scale nu

Write a function that converts an input of some integer

Write a function that converts an input of some integer number of inches into outputs of integer numbers of yards, feet, and inches. Do not write a main function. Your funct

Explain principles of digital analysis

Explain Principles of digital analysis discussion. When creating forensic images, verifying an image's integrity (i.e., whether or not it is an exact bit-for-bit copy of the

Write a multi-layer neural network in the language

Principles of Neural Systems and Implementation Project Guidelines - Each project requires the development of a program. This means that some programming skills are essential

Examine your thoughts on the effects of indexes

Discuss the thoughts on the effects of indexes, data types, filegroups, and transaction logs on space considerations. Which of those database constructs do you feel are most

Reviews

Write a Review

 
Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd