# Java Scala Mongo Installation

```python
Getting Scala to work on EC2

Ref: https://www.digitalocean.com/community/tutorials/how-to-install-java-with-apt-get-on-ubuntu-16-04

# Java installation - installing Oracle JDK 8
 apt-get -y update
 add-apt-repository ppa:webupd8team/java
 apt-get -y update
 apt-get install -y oracle-java8-installer

# if there are multiple JDK installations on one server, then we can switch between versions using:
# update-alternatives --config java

# Setting JAVA_HOME
# Put the location of java from the config command above at the end of the file
# JAVA_HOME="/usr/lib/jvm/java-8-oracle"
# Then do source /etc/environment

# Scala installation
# First see which version of Scala we should install - We are using 2.11.8
# Got to Scala website and find the link to set.
# Make a directory like /tf_files/scala_bkend and get this tgz file
# wget https://github.com/sbt/sbt/releases/download/v1.0.2/sbt-1.0.2.tgz
# gunzip sbt-1.0.2.tgz
# tar xvf sbt-1.0.2.tar

 #Ref: https://gist.github.com/alexislucena/5947c925afc3a0f26b04c3785d482e9f
 # Scala Install
 mkdir -p /tf_files/DOWNLOAD/Scala_install
 cd /tf_files/DOWNLOAD/Scala_install
 apt-get remove scala-library scala
 wget www.scala-lang.org/files/archive/scala-2.11.8.deb
 dpkg -i scala-2.11.8.deb

 # SBT install
 # Ref: http://www.scala-sbt.org/0.13/docs/Installing-sbt-on-Linux.html
 echo "deb https://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list
 apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
 apt-get update
 apt-get install sbt


 # Install MongoDB
 # Ref: https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04
 # Ref: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
 apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6
 echo "deb [ arch=amd64,arm64 ] http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 multiverse" | tee /etc/apt/sources.list.d/mongodb-org-3.4.list
 apt-get -y update
 apt-get install -y mongodb-org

 # Install MongoDB in ubuntu18.04
 ubuntu@ip-10-215-55-167:~$ cat mongo_install.sh 
#!/bin/bash
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org
echo "mongodb-org hold" | sudo dpkg --set-selections
echo "mongodb-org-server hold" | sudo dpkg --set-selections
echo "mongodb-org-shell hold" | sudo dpkg --set-selections
echo "mongodb-org-mongos hold" | sudo dpkg --set-selections
echo "mongodb-org-tools hold" | sudo dpkg --set-selections

sudo service mongod start

./mongo_install.sh



# then follow to initiate systemd as given in the documentation - edit /etc/systemd/system/mongodb.service
# finally start the service - DID NOT WORK IN DOCKER
# systemctl start mongodb

# to start the mongo service manually:
# /usr/bin/mongod --quiet --config /etc/mongod.conf &

# Get AgShift mongoldb zip file from GitHub and then sep to the EC2 docker
# Start mongodb
# 1. To dump the entire database at some location, use the following command.
#   mongodump -d databasename -o /path/of/output/
# Then restore the database using the following command
# mongorestore --host localhost --port 27017 --db agshift /path/to/Downloaded unzipped file/agshift

# Ref image: https://s3-us-west-2.amazonaws.com/agskore/Users/IOS/devrelease/A23571BC-A7D4-4739-8778-C57FB36AE54A/2017-09-14/JPEGImages/pic_1505417603796.jpg
# Curl command: curl -X POST -d '{"to":"Support"}' -H "Content-Type: application/json" http://localhost:809
0/api/misc/contact-us -v
```

## MongoDB connect

```
 mongo <DB Name> --host 127.0.0.1 --port 38270 -u <user id> -p <passwd>

 # to start the mongo service manually:
 /usr/bin/mongod --quiet --config /etc/mongod.conf &

 # to restore a DB
  mongorestore --host localhost --port 27017 --db <DB name> </path/to/Downloaded_unzipped_file>/<DB name>
```

## MongoDB upgrade (community edition 4.0.7) and start service

Ref: <https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/>

```
# Hope that you don't fear losing the DB directory
sudo apt list --installed | grep mongodb
sudo apt-get purge mongodb-org*

sudo rm -r /var/log/mongodb
sudo rm -r /var/lib/mongodb

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org

# starting service might give an error
sudo systemctl start mongodb
Failed to start mongodb.service: Unknown unit: mongodb.service

# to rectify the above error and successfully restarting mongod service
sudo systemctl enable mongod
sudo service mongod start

# edit /etc/mongod.conf and change the port
port 39817

then start mongo shell as:
mongo --port=39817
```

## MongoDB commands

```
 > db.<collection_name>.remove({})
 WriteResult({ "nRemoved" : 112 })


Data types reference:

Data Type    Number
Double            1
String            2
Object            3
Array             4
Binary Data       5
Undefined         6
Object Id         7
Boolean           8
Date              9
Null              10
Regular Expression    11
DBPointer       12
JavaScript        13
Symbol            14
JavaScript with scope    15
Integer 32 bit   16  
Integer 64 bit   18
Decimal          19
Timestamp        17
Min Key          -1
Max Key          127

Ref: https://examples.javacodegeeks.com/software-development/mongodb/mongodb-foreach-example/

 # to change the data type
 db._company.find({"subCompanyId":{$type:1}}).forEach(function(x) { \
     x.subCompanyId = new NumberInt(x.subCompanyId); \
     db._company.save(x); \
 });


 # to insert a new field "subCOmpanyId" in the collection for all the documents
 db._user.find().forEach( function (elem) {   \
    elem.subCompanyId = elem.company_code;   \
    db._user.save(elem); \
 } )

 # you may need to change the data type for "subCompanyId" after doing the above
 db._company.find({"subCompanyId":{$type:1}}).forEach(function(x) { \
    x.subCompanyId = new NumberInt(x.subCompanyId); \
    db._company.save(x); \
 });


 # to update nested array
 # Ref: https://docs.mongodb.com/manual/reference/operator/update/positional-all/
 db.students3.update({"_id":2}, {$set:{"grades.$[].questions.$[score]":100 }}, \
 { arrayFilters: [  { "score": { $gt: 10 } } ], multi: true})


 # update nested arrays - requires MongoDb 3.6
 Ref: https://docs.mongodb.com/manual/reference/operator/update/positional-all/#position-nested-arrays
 Ref: 
 db.acl_test.insert([
   { "companyCode" : 1,
      "ACL" : [
        { functionality: "new", roles: [ "birddog", "reviewer", "marketing"] },
        { functionality: "assign", roles: [ "birddog", "reviewer", "sales" ] },
        { functionality: "delete", roles: [ "birddog", "reviewer" ] },
        { functionality: "edit", roles: [ "reviewer" ] }
      ]
   },
   { "companyCode" : 2,
      "ACL" : [
        { functionality: "new", roles: [ "birddog", "reviewer", "marketing"] },
        { functionality: "assign", roles: [ "birddog", "reviewer", "sales" ] },
        { functionality: "delete", roles: [ "birddog", "reviewer" ] },
        { functionality: "edit", roles: [ "reviewer" ] }
      ]
   },
   { "companyCode" : 3,
      "ACL" : [
        { functionality: "new", roles: [ "birddog:desc1:new", "reviewer:desc1:new", "marketing:desc9:new"] },
        { functionality: "assign", roles: [ "birddog:desc2:assign", "reviewer:desc7:assign", "sales:desc2:assign" ] },
        { functionality: "delete", roles: [ "birddog:desc3:delete", "reviewer:desc6:delete" ] },
        { functionality: "edit", roles: [ "reviewer:desc4:edit" ] }
      ]
   }

])

db.acl_test.update(    {"companyCode":3},    \
{ $set: { "ACL.$[].roles.$[roleName]": "hawk:desc42:delete" } }, \
{ arrayFilters: [  { "roleName": { $in: ["birddog:desc3:delete"] } } ]} )


# for findAndModify
Ref: https://docs.mongodb.com/manual/reference/method/db.collection.findAndModify/#db.collection.findAndModify

# for $in[] operator
Ref: https://docs.mongodb.com/manual/reference/operator/query/in/


# multiple queries
db._input_data.find({inputDataId :{$gt :"2882",$lt :"2962"}, "annotatedFoodAwsUrl" : /INPUTEDITEDImages/}).

# querying with $gt for strings - javascript syntax
db._input_data.find( "this.inputDataId > 2888" ).count()

# querying a list using regex
db._input_data.find({"foodAwsUrl": { $regex: /pic_004_1548458142_good_top_0/}}, {"inputDataId":1}).pretty()

#removing using a query
 db._input_data.remove({"inputDataId":"3395"})

# removing a range of records/documents
 db._input_data.remove({"inputDataId":{$gte:4703}})

 # query using a "dateTime" string and getting out multiple projections (inputDataId, licenseKey, annotatedCoordinates)
 db._input_data.find({"dateTime" : "2019-03-26"}, {"inputDataId":1, "licenseKey":1, "annotatedCoordinates.top":1}).pretty()

 # multiple queries, multiple projections - for Olam (extracting the cashew_DT info)
 db._input_data.find({"subCompanyId":"6", "dataCollectionMode" : 3}, {"dateTime":1, "inputDataId" : 1, "commodity" : 1, "licenseKey":1, "foodAwsUrl" :1}).pretty()

 # querying a regex in an array object in a document
 db._input_data.find({"foodAwsUrl":{$elemMatch: {$regex: "pic_080_1560963470_generic_bot_1"}}}, {"inputDataId":1}).pretty()
```

## MongoDB security

<https://medium.com/@raj_adroit/mongodb-enable-authentication-enable-access-control-e8a75a26d332>

**Getting MongoDB secured in Kubernetes**

<http://pauldone.blogspot.com/2017/06/deploying-mongodb-on-kubernetes-gke25.html>

## MongoDB Kubernetes ec2\_kops\_shadow

```
 mongo -u <DB name> -p <passwd> -authenticationDatabase <authentication DB name>

root@mongod-0:/# mongo -u <user Id> -p <passwd> -authenticationDatabase <authentication DB name>
 use <DB name>
 show collections


 Dev machine
 To get the DB dump:
 ubuntu@ip-172-31-0-119:~/MongoDB_BACKUP$ sudo mongodump --host 127.0.0.1 --port 38270 --db <DB name> --username <user ID> --password <passwd> -o <output>

 scp the dump from dev to ec2_kops_shadow


 use kubectl command to copy from ec2_kops_shadow host to kubernetes mongod-o
 kubectl cp /tmp/<output_name>.tar.gz mongod-0:/tmp/<output_name>.tar.gz


 db.dropDatabase() => not able to drop

 So delete collections one by one
 use <DB name>
 db.<collection_name>.drop()

 Exhaustive list:
 db._access_token.drop();
db._acl.drop();
db._commodity.drop();
db._commodityDetails.drop();
db._company.drop();
db._company_client.drop();
db._counters.drop();
db._device_mapping.drop();
db._growing_region.drop();
db._input_data.drop();
db._iot_enclosure_config.drop();
db._item_master.drop();
db._market.drop();
db._market_group.drop();
db._menu.drop();
db._qt_counters.drop();
db._roi.drop();
db._temporary_user.drop();
db._user.drop();
db._variety.drop();
db._varietyDetails.drop();
db._version.drop();

# This programatic way will be even better, but could not make it work yet
var collectionNames = db.getCollectionNames();
for(var i = 0, len = collectionNames.length; i < len ; i++){
    var collectionName = collectionNames[i];
    if(collectionName.indexOf('cache_') == 0){
        db[collectionName].drop()
    }
}

# THIS PROGRAMMATIC WAY WORKS!!
db.getCollectionNames().forEach(function(collname) { print(collname);db[collname].drop();} );


 quit()


This appended the current database
mongorestore -d <DB name> -u <user ID> -p <passwd>  /tmp/<path>/<DB name>
```

## Replicate MongoDB from Production DB to Staging DB

```
# Login to ec2_kops_staging machine
ssh ec2_kops_staging


# copy DB from prod to staging and to staging mongo pod
 cd ~/StagingMongoBackup/
 scp -P 39989 -i ~/.ssh/ubuntu1403_keypair.pem ubuntu@54.245.190.37:/volxvdf/HydraMogoDBbackup/mongoDB_DUMP_agshift_12_21_2018_03_00_01.tar.gz .
 tar xvzf mongoDB_DUMP_agshift_12_21_2018_03_00_01.tar.gz
 kubectl cp ~/StagingMongoBackup/tmp/mongoDB_DUMP_agshift_12_21_2018_03_00_01 mongod-0:/tmp/mongoDB_DUMP_agshift_12_21_2018_03_00_01


# Delete all existing collections in staging DB
 cd ~/Staging/Production_Access
 ./mongo_pod_access_new.sh mongod-0
 mongo -u agshift -p yrhy4Nqfuc9XEcdBqKyNCpeL -authenticationDatabase agshift
 use agshift
 db.getCollectionNames().forEach(function(collname) { print(collname);db[collname].drop();} );
 quit()


# Restore the prod DB, which has been copied to the mongod-o container
mongorestore -d agshift -u agshift -p yrhy4Nqfuc9XEcdBqKyNCpeL /tmp/mongoDB_DUMP_agshift_12_21_2018_03_00_01/agshift
```

## MongoDB setting up replicate set on vanilla VM's

Ref: <https://computingforgeeks.com/how-to-setup-mongodb-replication-on-ubuntu-18-04-lts/>

## Scala structure

```
src/main/scala/<project>
                       -> <project>Process.scala (templatizes the structure of DB collections for siphoning out the data)
                       -> <project>Model.scala   (inherits Process; this is the main bullwark of DB read/write happens here)
                       -> <project>Service.scala (inherits model, this defines /api endpoint) 
        /scala/utils
                       -> Database.scala (configures the DB, including userid, password, connectionOptions and Credentials, AuthenticationMechanism etc...)

        /scala/ServiceController.scala -> Entrypoint. This extends/inherits the "App", defines the routes and binds to HTTP
        /scala/CorsHandler.scala       -> All CORS related stuff goes here

Example:
Fit the above structure to the following example from GitHub
tooling/src/main/scala/databaseaccess
```

## Scala Setup

```
Look at the Dockerfile. That will be a good start

Steps:
1. ssh -i ~/.ssh/ ubuntu1403_keypair.pem ubuntu@<domain-name>

2. mkdir -p GIT_STUFF
3. cd GIT_STUFF/
4. sudo apt update

5. sudo apt install -y openjdk-8-jdk
6. sudo apt install -y wget
7. mkdir $HOME/Downloads
8. cd $HOME/Downloads
9. wget www.scala-lang.org/files/archive/scala-2.11.8.deb
10. wget dl.bintray.com/sbt/debian/sbt-0.13.11.deb

11. sudo dpkg -i scala-2.11.8.deb
12. sudo dpkg -i sbt-0.13.11.deb
13. sbt (Revisit)
14. cd $HOME/GIT_STUFF
15. git clone https://github.com/agshift/cootbackend.git
16. git clone https://github.com/agshift/cootfendweb.git
17. sudo apt install -y mongodb
18. sudo apt install -y
       ○ build-essential
       ○ software-properties-common
       ○ apt-utils
       ○ gnupg 

○ curl
       ○ unzip
       ○ python
       ○ python-setuptools
       ○ vim
       ○ cron

       ○ openssh-client

○ xauth

○ git
       ○ xvfb
       ○ python-pip

○ s3cmd
       ○ python-arrow

○ iputils-ping
       ○ procps

○ dnsutils
 19. sudo apt install -y bsd-mailx mailutils
 20. curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
 21. sudo apt-get install -y nodejs
 22. sudo apt install -y npm //Gave Error -->Aman (don’t run, not is shell script)
 23. sudo npm install -g firebase-tools

24. sudo apt -y install nginx
 25. pip install --upgrade pip
 26. sudo apt install -y awscli
 27. wget 
https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.4/wkhtmltox-0.12.4_
linux-generic-amd64.tar.xz
 28. tar vxf wkhtmltox-0.12.4_linux-generic-amd64.tar.xz
 29. sudo cp wkhtmltox/bin/wk* /usr/local/bin/
 30. sudo apt -y install ssmtp

 # Configurations
 # MongoDB Backup
The location of the backup may vary
1. Fetch backup to local machine
     scp -i ~/.ssh/ubuntu1403_keypair.pem
ubuntu@ 52.38.137.129:/volxvdf/HydraMogoDBbackup/ mongoDB_DUMP_agshift_12_
31_2018_21_00_01.tar.gz ./
2. Extract directory from the tar file
     tar zxvf mongoDB_DUMP_agshift_12_31_2018_21_00_01.tar.gz
3. Restore data from that backup
     mongorestore -d agshift tmp/*/agshift



 #Deploy CootBackEnd (Without Kubernetes)
 1. cd
 2. cd $HOME/GIT_STUFF/cootbackend/agshift-rest-services/
3. git checkout develop (Branch is As Per Desire-->The most stable code is in Development
Branch)
4. Make Necessary Changes Required In application.conf file (Database Location, SMTP email
and password, cors domain name and domain url)
5. src/main/resources/application.conf (Make Necessary Changes and Save it.)
6. Change Database.scala file (Package Location --> com/agshift/utils/Database.scala)
(Changes required in password. If db instance requires password, we have to uncomment
password section)
7. Go to GIT_STUFF/cootbackend/agshift-rest-services/

8. sbt clean
9. export SBT_OPTS="-XX:+UseNUMA -XX:-UseBiasedLocking -Xms256M -Xmx1G
-Xss1500M -XX:MaxPermSize=128m -XX:+UseParallelGC -XX:MaxGCPauseMillis=100
-XX:GCTimeRatio=19"
10. sbt dist
11. cd target/universal/
12. rm -rf agshift-rest-services-1.0/ (agshift-rest-services-1.0 folder exists)
13. unzip agshift-rest-services-1.0.zip
14. cd agshift-rest-services-1.0/bin

15. sudo nohup ./agshift-rest-services &
16. ps -eaf | grep java (It should show java process running)
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://bhabs.gitbook.io/allsetup/java-scala-mongo-installation.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.