Development

Cloudbreak application

Local Development Setup

To use this development environment on OSX, you need to have Docker and Boot2docker installed.

Simplest way to prepare the working environment is to start the Cloudbreak on your local machine is to use the Cloudbreak Deployer.

First you need to create a sandbox directory which will store the necessary configuration files and dependencies of Cloudbreak Deployer. This directory must be created outside of the cloned Cloudbreak git repository:

mkdir cbd-local

To start the complete Cloudbreak ecosystem on your machine just execute the following sequence of commands:

cd cbd-local
curl https://raw.githubusercontent.com/hortonworks/cloudbreak-deployer/master/install | sh && cbd --version
cbd update master
cbd init
cbd start

If everything went well then the Cloudbreak will be available on http://192.168.59.103:3000. For more details and config parameters please check the documentation of Cloudbreak Deployer.

The deployer has generated a certs directory under cbd-local directory which will be needed later on to set up the IDEA properly.

The next step is to edit the cbd-local/Profile file with any editor and add the CB_SCHEMA_SCRIPTS_LOCATION environment variable which configures the location of SQL scripts that are in the 'core/src/main/resources/schema' directory in the cloned Cloudbreak git repository. Please note that the full path needs to be configured and env variables like $USER cannot be used.

export CB_SCHEMA_SCRIPTS_LOCATION=/Users/myusername/prj/cloudbreak/core/src/main/resources/schema

In order to kill Cloudbreak container running inside the boot2docker and redirect the Cloudbreak related traffic to the Cloudbreak running in IDEA use the followig command:

cbd util local-dev

IDEA

Cloudbreak can be imported into IDEA as gradle project. Once it is done, you need to import the proper code formatter by using the File -> Import Settings... menu and selecting the idea_settings.jar located in the config directory in Cloudbreak git repository.

To launch the Cloudbreak application execute the com.sequenceiq.cloudbreak.CloudbreakApplication class with VM options:

-XX:MaxPermSize=1024m
-Dcb.cert.dir=FULL_PATH_OF_THE_CERTS_DIR_GENERATED_BY_CBD
-Dcb.client.id=cloudbreak
-Dcb.client.secret=cbsecret2015
-Dcb.db.port.5432.tcp.addr=192.168.59.103
-Dcb.db.port.5432.tcp.port=5432
-Dspring.cloud.consul.host=192.168.59.103
-Dcb.identity.server.url=http://192.168.59.103:8089
-Dserver.port=9091

The -Dcb.cert.dir=FULL_PATH_OF_THE_CERTS_DIR_GENERATED_BY_CBD value above needs to be replaced with the full path of certs directory generated by Cloudbreak Deployer e.g. -Dcb.cert.dir=/Users/myusername/prj/cbd-local/certs

Command line

To run Cloudbreak from command line you have to create a property file, for example application.properties, with the content below, and execute ./gradlew bootRun -Dspring.config.location=file:/path/of/property/application.properties command at project root.

cb.cert.dir=FULL_PATH_OF_THE_CERTS_DIR_GENERATED_BY_CBD
cb.client.id=cloudbreak
cb.client.secret=cbsecret2015
cb.db.port.5432.tcp.addr=192.168.59.103
cb.db.port.5432.tcp.port=5432
spring.cloud.consul.host=192.168.59.103
cb.identity.server.url=http://192.168.59.103:8089
server.port=9091

Database development

If schema change is required in Cloudbreak database (cbdb), then the developer needs to write SQL scripts to migrate the database accordingly. In Cloudbreak the schema migration is managed by MYBATIS Migrations and the cbd tool provides an easy-to-use wrapper for it. The syntax for using the migration commands is cbd migrate <database name> <command> [parameters] e.g. cbd migrate migrate status.

Create a SQL template for schema changes:

cbd migrate cbdb new "CLOUD-123 schema change for new feature"

As as result of the above command an SQL file template is generated under the path specified in CB_SCHEMA_SCRIPTS_LOCATION environment variable, which is defined in Profile. The structure of the generated SQL template looks like the following:

-- // CLOUD-123 schema change for new feature
-- Migration SQL that makes the change goes here.



-- //@UNDO
-- SQL to undo the change goes here.

Once you have implemented your SQLs then you can execute them with:

cbd migrate cbdb up

If you would like to rollback the last SQL file, then just use the down command:

cbd migrate cbdb down

On order to check the status of database

cbd migrate cbdb status

#Every script that has not been executed will be marked as ...pending... in the output of status command:

------------------------------------------------------------------------
-- MyBatis Migrations - status
------------------------------------------------------------------------
ID             Applied At          Description
================================================================================
20150421140021 2015-07-08 10:04:28 create changelog
20150421150000 2015-07-08 10:04:28 CLOUD-607 create baseline schema
20150507121756 2015-07-08 10:04:28 CLOUD-576 change instancegrouptype hostgroup to core
20151008090632    ...pending...    CLOUD-123 schema change for new feature

------------------------------------------------------------------------

Building

Gradle is used for build and dependency management. Gradle wrapper is added to Cloudbreak git repository, therefore building can be done with:

./gradlew clean build

Cloudbreak deployer

Contribution

Development process should happen on separate branches. Then a pull-request should be opened as usual.

To build the project

# make deps needed only once
make deps

make install

To run the unit tests:

make tests

If you want to test the binary CircleCI build from your branch named fix-something, to validate the PR binary cbd tool will be tested. It is built by CircleCI for each branch.

cbd update fix-something

Snapshots

We recommend to always use the latest release, but you might want to check new features or bugfixes. All successful builds from the master branch are uploaded to the public S3 bucket. You can download it:

curl -L s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/cloudbreak/cloudbreak-deployer_snapshot_$(uname)_x86_64.tgz | tar -xz

Instead of overwriting the released version, download it to a local directory and useit by refering as ./cbd

./cbd --version

Testing

Shell scripts shouldn’t raise exceptions when it comes to unit testing. basht is used for testing. See the reasoning: why not bats or shunit2.

Please cover your bash functions with unit tests and run test with:

make tests

Release Process of the Clodbreak Deployer tool

The master branch is always built on CircleCI. When you wan’t a new release, all you have to do:

make release-next-ver

make release-next-ver performs the following steps:

Acceptance

Now you should test this release. You can get it by cbd update release-x.y.z. Comment with LGTM (Looking Good To Me).

Once the PR is merged, CircleCI will:

Edit on GitHub